Name: SAP Data Intelligence ML Scenario Manager (04) Data Exploration in Notebooks
Uploaded: 2023-9-4
Duration: 422 s

Share feedback

<p>Planned maintenance of Practice Systems</p>

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Subscribe now

Learn more

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div><div>In this fourth of a nine part video series regarding the SAP Data Intelligence Machine Learning Scenario Manager, Tahir Hussain Babar (Bob) shows how to connect to a dataset within our semantic data-lake within a Jupyter Notebook, and then shows how to perform basic data exploration. <br><br>0:00 - Introduction<br>1:00 - Create Notebook<br>3:30 - Install Python libraries<br>4:00 - Data exploration using Pandas and matplotlib<br><br><br><a href="https://sap.to/6050Krmuq" rel="nofollow noopener noreferrer" target="_blank">https://sap.to/6050Krmuq</a><br><br>Please note:<br>                     This video was originally published on the SAP HANA Academy YouTube Channel                      and therefore may contain references to SAP HANA Academy and YouTube.                      The tech world is always evolving and the SAP HANA Academy YouTube channel                      is set to close on December 31, 2023.                      At that time, references will no longer be valid                      and will gradually be removed when the videos are updated.<br>Also, this video includes automatically generated captions                      and therefore might misrepresent the spoken content or display incorrect product names.</div></div></div>

SAP Data Intelligence ML Scenario Manager (04) Data Exploration in Notebooks

Welcome to the SAP HANA Academy. My name's Bob, and in this video series, we'll be looking at SAP Data Intelligence. In this video series, we'll be focusing on using the Machine Learning Scenario Manager. And what we'll cover is how to create predictions within the Machine Learning Scenario Manager and then expose them so that they can be accessed via an API. In this video we'll be using a Jupyter Notebook to perform two things. Firstly, we'll connect to a data source within our semantic data lake within that Jupyter Notebook and then we'll perform some data exploration of that data. In the previous video we looked at the basics of using the machine learning scenario manager and we looked at what happens when you create a scenario and the components used within those scenarios. We then went through a basic example of creating a Jupyter notebook which is one of those components. Now in this video we're going to create ourselves a new notebook but we're going to to do some actual data science. So I'll create the notebook and we'll call it 10 Data Exploration and Model Training. When we click create, again we're going to have to specify that kernel, so again we'll specify Python 3, and then we'll be ready to create our code. For our machine learning scenario, we're going to be using the Intel extension for scikit-learn. As you can probably tell, I'm not a data scientist, so the aim of these videos is to get some code and execute it and give you basic explanations of what the code does in order to achieve our goals. We'll look at psychic learn in the next video, which is where we look at training our regression model. However, in this video, we're going to focus on connecting to our data source and doing some basic data exploration within our Jupyter Notebook. Now, if we think back to our source data, the aim of our project is to estimate a person's marathon time. We've got around 117 rows of data, but we'll want to put in any half marathon time and then retrieve what we think will be the full marathon time based on some machine learning. Often, the different steps of a data science or machine learning project will follow the cross-industry standard process for data mining, which is called CRISP-DM. For these videos, we'll follow a simplified workflow. In the first phase, the creation phase, we'll explore the data to understand the structure and the content. This is also where you train and interpret your machine learning models. Again we'll cover connectivity to our data source and basic data exploration in this video and then in the next video we'll look at training our regression model. The second phase which is where we deal with deployment is about putting things into production. So once we know the machine learning logic and it's been identified, we'll use then graphical pipelines to train the machine learning model and apply it for inference, which is where those predictions are made. Essentially, we're going to build two pipelines. In the business process phase, which is our third phase, we'll then use that machine learning model within a business process. In our simple example, we're going to be simulating this step using Mailman. Initially we'll need to access the data, so what we'll have to do is install the library that's required for Python to access our CSV in the internal data lake, which is our DI data lake connection. So to do this we're going to run a pip install HDFS. I've already executed this, so for me I'm just re-executing the commands to show you the commands needed. Now that we've loaded those libraries, what we'll need to do is connect to our CSV file. This is the CSV file which, of course, contains our Marathon data. And it's in this folder path within our DI data lake connection. So if I go back to my machine learning scenario, what we'll do is we'll run the following command. So using that HDFS library, we can load that data into a Pandas data frame. The third line, of course, connects to our data lake based on this port. And then we're running a client.read command to connect to that specific file in that folder path. We're calling it reader and then we're just setting some parameters to do with delimiters within the CSV file because we're reading that CSV and the data frame is called df underscore data. Of course we want to see the data now so we can do this with a simple python command to explore the data. So let's say we want to look at the first five rows of that data. All we need to do is run a df underscore data dot head and the value of head will be five. Now this is quite an important step because it establishes that we've correctly created a connection to our data within that Python notebook. Now what we can do is plot the runner's half marathon time against their full marathon time. So we'll do this by defining the X and Y axes as half marathon minutes and marathon minutes using X and Y equals true from that pandas data frame. And then what we can do is use the %MapPlotLibInline which will turn on what's called Inline Plotting. And this means that plot graphics will appear within the notebook. After specifying we want to use this PyPlot function, all we need to do now is decide on what we want to plot on our screen. So for us, of course, it's the X and the Y, and we want to plot those on the screen in dark blue, as indicated by color. And if I execute this command, what we'll do is we'll get our graphic within our notebook. Of course, what would be nice would be to also have the X and Y label, and it's easy to add by changing the data or the code in cell 4. So all I'm going to do is append two lines to this code for the X label and the Y label. So again, if I execute, this is the output that we get. So looking at this chart, you can see there's clearly a linear relationship. This isn't really a surprise because if you can run a full marathon fast, you're also likely to be able to run a half marathon fast as well. But as you can imagine, we've done some important things in this video. In this video, we've connected to our data source, and we've done some basic investigation of that data. So in the next video, what we'll need to do is look at training the regression model, and we'll do this in a new notebook.