Name: SAP Data Intelligence ML Scenario Manager (05) Training Models in Notebooks
Uploaded: 2023-9-4
Duration: 594 s

Share feedback

<p>Planned maintenance of Practice Systems</p>

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Subscribe now

Learn more

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div>Scikit-learn is one of the most widely used Python packages for data science and machine learning (ML). In this fifth of a nine part video series regarding the SAP Data Intelligence Machine Learning Scenario Manager, Tahir Hussain Babar (Bob) shows how to perform linear regressions within a Jupyter Notebook using Scikit-learn. <br><br>0:00 - Introduction<br>1:00 - Scikit-learn<br>2:00 - Linear regression<br>4:00 - RMSE<br>5:00 - Serialise into binary format using pickle<br>7:30 - Create Notebooks<br>8:00 - Load dat file using pickle<br>8:15 - Make prediction<br><br><br><a href="https://sap.to/6052KrmR0" rel="nofollow noopener noreferrer" target="_blank">https://sap.to/6052KrmR0</a><br><br>Please note:<br>                     This video was originally published on the SAP HANA Academy YouTube Channel                      and therefore may contain references to SAP HANA Academy and YouTube.                      The tech world is always evolving and the SAP HANA Academy YouTube channel                      is set to close on December 31, 2023.                      At that time, references will no longer be valid                      and will gradually be removed when the videos are updated.<br>Also, this video includes automatically generated captions                      and therefore might misrepresent the spoken content or display incorrect product names.</div></div>

SAP Data Intelligence ML Scenario Manager  (05) Training Models in Notebooks

Welcome to the SAP HANA Academy. My name's Bob, and in this video series, we'll be looking at SAP Data Intelligence. In this video series, we'll be focusing on using the Machine Learning Scenario Manager. And what we'll cover is how to create predictions within the Machine Learning Scenario Manager and then expose them so that they can be accessed via an API. In this video we'll be looking at using Jupyter Notebooks to train linear regression models. In the previous video, we looked at performing some data exploration using a Jupiter notebook of data which is in an SAP Data Intelligence semantic data lake. It was very basic data exploration, but what we were able to produce was a linear relationship between two values. Having performed some data exploration in the previous video, we're now going to look at performing some freestyle data science by training a regression model. For our machine learning scenario, we're going to be using the Intel extension for Scikit-Learn. And scikit-learn is one of the most widely used Python packages for data science and machine learning. The Intel extension for scikit-learn provides optimized implementations of many scikit-learn algorithms. And this is done by dynamically patching scikit-learn estimators to use the Intel OneAPIData Analytics Library, or 1DAL, as the underlying solver, whilst getting the same solution faster. And Scikit-learn is very popular when doing machine learning within Python on tabular data, such as the one that we've got. So as for the connectivity, we had to install the HDFS library to connect to our semantic data lake. We're going to have to install the Scikit-learn library as well. So when I execute this command, we can see that the requirement has already been satisfied. Of course, I've been prepping this in this video series. So once we've done that, what we can do now is train our linear regression model with the newly installed psychiclearn library. And to be clear, linear regression performs the task to predict a dependent variable value based on a given independent variable. So in our example, what we want to do is take our data set, which we've imported into this Jupyter notebook, and we know we've got two columns. We've got a column for our half marathon minutes and another column, Y, for our full marathon minutes. We've only got 117 rows, but what we actually want to be able to do is input any value, which applies to the half marathon minutes and then what we want to retrieve is a predicted value for the full marathon minutes. So we'll train the Linear Regression model with this newly installed scikit-learn library. And when we've done that we'll need to plot the results. So firstly what we'll do is plot the data from our data exploration, which is our basic Linear Regression. So we're going to output it this in dark blue. Again, this is what we did kind of in the previous video. The difference is that in this video, we're also going to output a prediction, which is indicated in red. And again, this prediction comes from our Scikit-learn library. So like before, all we need to do now is specify the labels. So one is the actual minutes of the half marathon, and the other is the actual minutes of the full marathon as well. So we've got this on the X and Y labels. When we execute, what we'll see is a red line, which of course represents the model that was found by the linear regression. Now when you're performing data science or machine learning, you also need some form of quality indicator. So the way we're going to do this is by calculating the root mean squared error, which is RMSE, as our quality indicator of the model's performance on that training data. And this indicator will be shown later on in our machine learning scenario manager. Now because we don't really want to focus on data science in these videos, really the application of it when using SAP Data Intelligence, the RMSE will be calculated rather manually. This way it might be easier to understand its meaning in case you've not used it before or if you're not familiar with it. However, most data scientists would probably leverage a Python package to shorten this bit. So we'll be importing NumPy, which adds support for large multidimensional arrays and matrices, as well as a large collection of high-level mathematical functions to operate on those arrays. Then what we'll use is our predicted values combined with our source data for those marathon minutes to work out the mean squared error. Once we've done this, we can then work out the root of the mean squared error, we'll round it, and lastly we'll print both the RMSE on the screen, as well as the number of rows that we used within this model. So when we execute, what we'll see is the RMSE value, which is 16.96, and of course we know the number of rows that we had, n, is equal to 117, which are the number of runners within our source data. Now in the real world, a statistician might want to run further tests on this regression, for example on the distribution of the errors. However, again, this is not a data science video, it's a video on the application of data science within SAP Data Intelligence, so we'll skip these steps. So now what we could do is apply the model immediately on a new observation. So, for example, I can input a value and get an output, which is the predicted value. However, to deploy the model later on, we'll be using one graphical pipeline to train and save the model, and a second pipeline for inference or for the predictions. So what we'll do is we'll spread the Python code across two separate notebooks for clarity. Now when we create our production pipeline it'll save the model as a pickle object. So we'll do the same thing here within our notebook. Pickle is essentially a Python module that can save the current state of an object to a file. This process is actually called serializing, and by loading the file and deserializing it, the object can be recreated and reused at a later stage. We can do this with the following command. After importing the library, we just need to dump the contents of this model into a file called marathon underscore lm dot pickle dot dat. And what will happen is the file will appear here on the left hand side, which will be the output of the model, which we can use later on. So what I'll do is I'll execute this command. On the left hand side we should see that we've got a new file appear here, as you can see. And after the files are paired, what we can do is save this notebook. So now that we've saved it, we can go and load and apply our regression model. So I'll quickly create myself a new notebook. We'll call this 20 apply model, because that's what we're going to be doing here. We'll do the same as before. We'll select the default kernel of Python 3, and what we need to do now is import this model. And we'll be using this model to create our prediction. So again we're going to import the pickle library. And the command we're going to run is quite simply, we're going to load that same file using the following command. So I'll execute this command, and it's now time to apply our model. So what we'll want to do is predict a runner's marathon time if the person runs a half marathon in, for example, two hours. So like before, we covered this in the basic Jupyter Notebook overview. We'll need to create a variable. For me, we're assigning 120 minutes to a variable called x underscore new. Now we're going to use that variable with our loaded model and we're going to make a prediction. We're going to call this prediction, this variable predictions. And then we're just going to round off this prediction. So I'll execute the model. And what we can see, the model estimates a time of about 264 minutes, which is around 4 hours and 24 minutes to run a full marathon, if your half marathon is in 120 minutes. Of course, we can change this value for testing purposes. So if I make a change to our variable to 140 and rerun the model, you'll see that if you run a half marathon in 140 minutes, we'll predict that you will run a full marathon in about 309 minutes. Everything now is in place to start deploying the machine learning model, and we're going to do this within two graphical pipelines. We'll cover how to do this in the next few videos.