Name: Data Intelligence 5 of 21 - Data Quality Rules
Uploaded: 2023-9-6
Duration: 374 s

Share feedback

<p>Planned maintenance of Practice Systems</p>

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Subscribe now

Learn more

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div><div>In this video series, we'll introduce some of the applications when using Data Intelligence. In this video, we'll go through using the Metadata Explorer to create rules, which help to determine whether data complies with business constraints and requirements. We'll also look at the concept of Rulebooks.<br><br>0:00 - Introduction<br>0:50 - About rules<br>2:00 - Create new rule and add conditions<br>4:00 - Test rule<br>4:30 - About rulebooks<br><br>Please note:<br>                     This video was originally published on the SAP HANA Academy YouTube Channel                      and therefore may contain references to SAP HANA Academy and YouTube.                      The tech world is always evolving and the SAP HANA Academy YouTube channel                      is set to close on December 31, 2023.                      At that time, references will no longer be valid                      and will gradually be removed when the videos are updated.<br>Also, this video includes automatically generated captions                      and therefore might misrepresent the spoken content or display incorrect product names.</div></div></div>

Data Intelligence 5 of 21 - Data Quality Rules

Welcome to the SAP HANA Academy. My name is Bob and in this series of videos by Digital Partner Engineering and the SAP HANA Academy we'll be looking at Data Intelligence 3.1. In this video, we'll be looking at data quality rules and the concept of rule books when using the Metadata Explorer. In one of the previous videos, we looked at uploading a parquet file using data intelligence into our semantic data lake. Now we know one of those files, the customer's file, has an issue with one of the columns. That column is the country column, and as you can see here, the country isn't standardized. So to demonstrate what we can do within the Metadata Explorer, we're going to look at, in this video, the concept of rules. Rules help us determine whether the data complies with business constraints and requirements. Now when you've built a rule, what you'll then use is a rule book to bind your rules to your data sets. And you can do this by assigning a column to a specific parameter. Then what you can do is run all your rules in a single rulebook. Then after you've run your rules, you can view the results and create passing and failing thresholds. Now you'll also see that we have these rule dashboards. Once you've got your rules, your categories and your rule books defined, you can create a dashboard which will reflect whether the dataset has passed the rules. You can then configure the passing threshold and apply weighting to certain rules or categories. And these rules can be applied to multiple datasets. So to start we'll go to view rules and here we can see within our rules overview all our rule categories. Now what we want to do is create a rule to check that the country code contains only the value US in capitals. So for us this is an accuracy rule. So on the right of accuracy what we'll want to do is create a new rule here by selecting this icon. We don't need a category, we can use an existing one. For the ID of the rule we'll put this as country us-sha and we'll put the name as the same. For the description we'll put the US customer file must not be null and must have a standardized country value of US. Then all we need to do is select save. The next step is to add a parameter. One parameter must exist in each rule. And also one condition must exist in a rule. So for us, our parameter is going to be country. Once we've entered that, we can click on save. Now to add the condition. So again, we're going to, on the right hand side, select that condition option. And, of course, the condition we want is that the country is not null. So I'm going to change the condition name to country, not null. We'll select the parameter that we've previously created. And of course, the operator is not null. So we can add another condition as well. So I'll add another condition and we'll change this condition name to Country US. And for this one we're going to select the parameter from the drop-down. Again we're going to select Country. And again, what we're going to select here is for the operator, we're going to say equals us. And then we'll perform a save. What we'll see is an indicator once we've saved that the rule is valid or invalid. Now that we know it's valid, as you can see in the top of the screen, what we'll want to do is test the rule. So on the top right hand side we'll select test rule. And then what we'll do is we'll add a value to test this rule. So of course it's expecting a country. So we'll add the value. Let's add in some values. Now let's select run tests. Here we can see the results. As you can imagine the only one that passes the test is US. All the others fail those rules. Imagine in the real world, you are not going to just have one rule, but you are going to have multiple rules. And you might have multiple rules which actually you want to reuse because you want to apply them to multiple data sets. The way we can do that is with rule books. Rule books contain a group of rules that can be run on one or more data sets and of course we're going to run that on that parquet file, the customer's US parquet file. So to build it what we need to do is on the top right hand side we select the plus button and then we'll need to give our rule book a name. So I'm going to call mine SHA and then there's nothing really here we have to modify. All we need to do is click on save. And then what we need to do for the rule book is choose any of the rules that we want to include in this rule book. We've only got one that we've built. So, if I import the rule, if we go to that accuracy category then we're going to simply add that countryUSHA rule. Again, the rule book can contain many different rules from many different categories. So all we need to do is now select save and what we'll do in the next video is look at rule bindings where we associate the rule book with a data set and then we'll look at the dashboards where we can look at scorecards which will show the results. You can find more tutorials on our YouTube channel, and if you'd like to be informed as soon as new video tutorials are published, then please subscribe. Thanks for watching.