Name: SAP Data Intelligence ML Scenario Manager (06) Model Training Pipeline Key Operators
Uploaded: 2023-9-4
Duration: 487 s

Share feedback

<p>Planned maintenance of Practice Systems</p>

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Subscribe now

Learn more

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div><div>In this sixth of a nine part video series regarding the SAP Data Intelligence Machine Learning Scenario Manager, Tahir Hussain Babar (Bob) shows how to create graphical pipelines within the ML Scenario Manager. The initial pipeline will be used to train a model, and this video explains the key operators used in the pipeline.  <br><br>0:00 - Introduction<br>2:20 - Create pipeline<br>3:00 - Pipeline review<br>4:15 - Read File operator<br>5:25 - Python script<br><br><br><a href="https://sap.to/6052KrmTr" rel="nofollow noopener noreferrer" target="_blank">https://sap.to/6052KrmTr</a><br><br>Please note:<br>                     This video was originally published on the SAP HANA Academy YouTube Channel                      and therefore may contain references to SAP HANA Academy and YouTube.                      The tech world is always evolving and the SAP HANA Academy YouTube channel                      is set to close on December 31, 2023.                      At that time, references will no longer be valid                      and will gradually be removed when the videos are updated.<br>Also, this video includes automatically generated captions                      and therefore might misrepresent the spoken content or display incorrect product names.</div></div></div>

SAP Data Intelligence ML Scenario Manager (06) Model Training Pipeline Key Operators

Welcome to the SAP HANA Academy. My name's Bob, and in this video series, we'll be looking at SAP Data Intelligence. In this video series, we'll be focusing on using the Machine Learning Scenario Manager. What we'll cover is how to create predictions within the Machine Learning Scenario Manager and then expose them so that they can be accessed via an API. In this video we're going to look at using the Machine Learning Scenario Manager to create pipelines. Now, this pipeline, the first pipeline we're going to create, will perform some model training, and in this first video out of the two, we'll look at the key operators within the pipeline, and we'll look at how the Python code that we've worked on in our Jupyter Notebook can be reused within these pipelines. In the previous few videos, we looked at using Jupyter Notebooks to load some data from our semantic data lake. We then used that data to do some data exploration, we looked at training regression models and then of course once we've trained and saved these regression models, we looked at loading and then applying these regression models to make predictions. In our example here, we have a variable which is X underscore nu, where the input value is 140 minutes. That's how long it took you to run a half marathon. Using this model, we're making a prediction that if you can run the half marathon in 140 minutes, you're then able to run a full marathon in 309 minutes. So in the previous videos, we looked at the creation phase, which is data exploration and freestyle data science. In the next few videos, we're going to be looking at pipelines for use in production. The pipeline we're going to look at in the next couple of videos is, will be used to train the model and save it into our machine learning scenario. And then lastly, we're going to look at another pipeline to surface the model as a REST API for inference. Lastly, we'll look at testing the pipeline using Mailman. So to create these pipelines, we'll go back to our machine learning scenario manager and we'll go to the pipelines tab. What we'll do is we'll create a pipeline and we'll call this 10 Train SHA. For the template we'll choose the Python producer template. And here you can see we've got a whole list of all the types of template that we can use. Now, after a few seconds we're going to see a pipeline which just needs to be adjusted. The aim of the videos coming up is not to explain how to build the exact pipeline, it's really to explain the fundamental aspects of the pipeline. We've actually covered a lot of these objects already before in previous videos within the Data Intelligence series. So an assumption here is that you've used pipelines or graphs before. So, for example, if you're not sure of how these pipelines work, whenever you select on an object, you have help on the right hand side and the configuration tab as well where you can configure each individual object. So what I'll do is I'll start by giving you an overview of the pipeline and then in the next two videos we'll look at the configuration aspects which needs to be changed. Now in principle the pipeline loads data with the read file operator. That data is then passed to a Python operator in which the machine learning model is trained. The same Python operator then stores the model in the machine learning scenario through what's called the artifacts producer. The Python operator's second output can then pass a quality metric of the model to the same machine learning scenario. Now once both the model and the metric are saved, the pipeline's execute is ended with the classical graph terminator. But of course we have to modify this to suit our specific needs. So I'll just zoom in again, and of course what we're going to firstly look at is modifying our data connection. And we do this within our read file operator. As the help says, the operator reads the contents of files from various storage services. Now all we need to do is go to our Configuration tab and then we need to change our connection. We'll click on this icon here and then within our configuration type we'll choose Connection Manager and then this will list our various connections. Now remember our CSV is in our semantic data lake called di-data lake, so we'll select that. And then we just need to click on save. Now there's only one other thing really here we need to change which is the path to our source data. So we can specify that here within path. So, again, we know it in our shared folder, within the Scenarios folder, there's a folder called Marathon, and all we need to do is select that file, running times.csv. Now, the next operator is just a data converter. So just to explain it so you can see how it works, if I select it and go to our configuration, we can see it's converting to a string. So the next step is to adjust the Python code that trains the regression model. So we'll select that Python 3 operator, and we'll go to the script. The template code will open up, and it shows how to pass the model and metrics into the machine learning scenario. Now we're going to use a specific set of code, and I'm going to go through the highlights of what the code is doing. The code has already been pre-prepared, and it obviously matches the data source that we're using just for ease of use. So of course, the first step is to receive the data from the read file operator. So that's what these lines are doing, and we're just doing some formatting of the data. Again, like before, we'll get those two columns, half marathon minutes and minutes, and we'll assign them to variables. We'll import the model, that linear regression model, and we know we need that scikit-learn library. So we're using similar code to which we used in the Jupyter Notebook to train the model, and then we'll pass the trained model as well as its quality indicator, of course the RMSE, to the machine learning scenario. Now this part of the code will send any of the metrics to the SubmitMetrics operator and then those metrics will be persisted with this command. Now as well as the metrics being persisted and saved, what we'll also have to do is the same with the model. So in this, this command will create and send the model blob to the output port and that will be the input to the artifact producer. Lastly, we're just defining our input port in the operator. Again, this script is specific for our data source, but you can see how easy it is to manipulate the script once you understand how the script works. And it's very similar to what we did in that Jupyter Notebook. All I'm going to do now is save this graph not to lose any of our work. So before we can actually run this pipeline, what we have to do is create a Dockerfile for the Python operator. This docker file is needed so that the Python operator has access to the Python libraries that the code is using. We'll cover how to do this in the next video.