Name: Data Intelligence 8 of 21 - Data Lineage
Uploaded: 2023-9-6
Duration: 459 s

Share feedback

<p>Planned maintenance of Practice Systems</p>

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Subscribe now

Learn more

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div>In this video series, we'll introduce some of the applications when using Data Intelligence. In this video, we'll go through using the Metadata Explorer to perform data lineage, which helps review data transformation history and metadata to quickly understand how, where, and why data has been altered.<br><br>0:00 - Introduction<br>1:00 - Start profiling<br>2:00 - View lineage<br>3:00 - Bind to rulebook<br>5:00 - View results<br>5:45 - Add to dashboard<br><br><br>Please note:<br>                     This video was originally published on the SAP HANA Academy YouTube Channel                      and therefore may contain references to SAP HANA Academy and YouTube.                      The tech world is always evolving and the SAP HANA Academy YouTube channel                      is set to close on December 31, 2023.                      At that time, references will no longer be valid                      and will gradually be removed when the videos are updated.<br>Also, this video includes automatically generated captions                      and therefore might misrepresent the spoken content or display incorrect product names.</div></div>

Data Intelligence 8 of 21 - Data Lineage

Welcome to the SAP HANA Academy. My name is Bob, and in this series of videos by Digital Partner Engineering and the SAP HANA Academy, we'll be looking at Data Intelligence 3.1. In this video, we'll be looking at data lineage when using the Metadata Explorer. In the previous video we looked at doing self-service data preparation within Data Intelligence. What we did is we had two data sources, a CSV file and a Parquet file. Both of them were imported into our semantic data lake. And the data in those files, one was a dimensional data structure and one was a fact table structure. So what we did is we cleaned the data with some manual cleansing. And then we joined the data and then ultimately aggregated the data to produce one data set. Now if you imagine this on a large scale, you might have multiple data sources which lead to cleansed data files. But you might want to be able to have some accountability and traceability to see where those data files come from. So this is data lineage. Now you don't need to profile a data set to view its lineage. There's other options in order to extract the lineage. Today I'm going to be talking about how to get a profile image, for example when a data preparation is executed or even if a pipeline is executed. But we'll profile the data because we want to look at that profiling information later on. Of course, we should get an alert that the profile has started on the screen and also on the top right hand side that we've started that alerting. So, what I'll just do is refresh the screen and now we can see in the top left hand side that we've got access to view our lineage. You'll also see that we've got an additional lineage tab and of course, on the top right hand side, what we should get is a form of an alert to indicate that the lineage has finished. So, if I select it, here we can see that the profiling completed as well as the lineage. So I'll just clear all. Of course, to view it, I'll just click on the lineage tab. And here we can see the data lineage. Quite simply, we've got the source file, the parquet and the CSV. We've combined it into sales data and the output is that customer sales SHA.csv file. So we can see the complete flow. Of course, if you want, you can publish this data set. And that means, of course, your colleagues can search the metadata, add comments to the objects, and also tag data sets. However, we're going to focus on binding this new data set to our country rule, which will help establish whether or not our manual data changes have improved the data quality. So to do this, we're going to go back to our rule books. And we'll need to search for that rule book. So in the filtered rule book names, a quick way to get it is just for me to type SHA. This is the last one of that rule book so I'll select it. We know that a rule will be in a category. We've only got one category so we'll expand. Then we need to click on the rule bindings and we're going to go to those rule bindings. Again, we can see at the moment it's just connected to our parquet file, which is the source file. But we want to add this to a new file, the new file that we've generated. So I'll click on Create Rule Binding. And then all we need to do here is choose that new data set. So for me it's in the same connection, but it's not this file, this is just a recent file. So what I'll need to do is browse. We'll select the connection, which is that DI data lake. We'll go to S shared. We'll go to the SHA folder. And we're going to choose that customer sales SHA CSV. So all I need to do is select on OK. So now we've created that rule binding of that second file. So all I need to do is click on save. And now what's going to happen is this rule will be applied to both of the data sets on the right hand side when we click run all. As always we get the alert that the rulebook execution has started, so we just need to wait for it to finish. And we can go to our monitor tab to check the status. So if I go to monitor and we go to monitor tasks, here we can see that it's still running, that rule book is running. After a few seconds, we can see it's been completed. So if I go back to my rules and we go to our rule books, we know that the task is completed. Of course, after a few seconds we get that other alert to indicate that the rulebook execution completed. What we'll do again is search for our rulebook, and here we can see it. So what I'll do is I'll select it, and of course then we just need to view the results. Here I can just click yes to see the updated results. It's quite simple to explain. We have the two versions of that data set. The parquet file is the original data set and the data quality could be considered around 79% and then after our manual changes we increase the data quality to around 89%. Of course you've got those two data sets below and the values as well on the right hand side in the form of those scorecards as well. But we can see these in other places. So if we go back to our rules dashboard. So if I click on the drop down and we go to the rules dashboards, we can add those analytics to our SHA dashboard. So here we've got that original scorecard and we can add another one onto the screen. So I'll just edit the screen. We'll add another scorecard. And just like before, what we need to do is go through that stepwise approach. So we'll select the rule book. Well, we've only got one that we built. So it's the SHA1. Then if we go to step 2, we'll need to choose our data set. So I'm going to again choose a single data set. Then we'll go to step 4. We have to choose the specific data set. So we're going to choose that updated file, the new one. And then in Stepfire, we'll just need to give it a title and a subtitle. And there we've got that extra scorecard. I've only done it in this fashion. It looks a bit different. Just to demonstrate that you can have different types of analytics on this screen. So what we've done in the past few videos is we've looked at mainly the metadata explorer. We've looked at how to discover data, define your rules, profile your data, check the quality of data. We've looked at glossaries and reviews and we finished off at looking at both self-service data preparation and data lineage. Now that we've done that, we're going to move on from using the Metadata Explorer to focusing on another tool, which is the Modeler. The Modeler helps create data processing pipelines or graphs, and provides a runtime and design time environment for data driven scenarios. You can find more tutorials on our YouTube channel, and if you'd like to be informed as soon as new video tutorials are published, then please subscribe. Thanks for watching.