Name: SAP Data Intelligence ML Scenario Manager (07) - Model Training Pipeline Docker Files
Uploaded: 2023-9-4
Duration: 643 s

Share feedback

Planned maintenance of Practice Systems

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Choose your subscription

Manage your subscription

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div>In this seventh of a nine part video series regarding the SAP Data Intelligence Machine Learning Scenario Manager, Tahir Hussain Babar (Bob) shows how to configure graphical pipelines that use Python Libraries with docker files within the ML Scenario Manager. We'll then look at executing the pipeline. 0:00 - Introduction 1:30 - Create Docker File 2:50 - Tags 3:30 - Build 4:00 - Configure Python operator 5:00 - Pipeline review (continued) 7:40 - Execute pipeline 8:55 - Result <a href="https://sap.to/6057Krmph" rel="nofollow noopener noreferrer" target="_blank">https://sap.to/6057Krmph</a> Please note: This video was originally published on the SAP HANA Academy YouTube Channel and therefore may contain references to SAP HANA Academy and YouTube. The tech world is always evolving and the SAP HANA Academy YouTube channel is set to close on December 31, 2023. At that time, references will no longer be valid and will gradually be removed when the videos are updated. Also, this video includes automatically generated captions and therefore might misrepresent the spoken content or display incorrect product names</div></div>

SAP Data Intelligence ML Scenario Manager (07) - Model Training Pipeline Docker Files

Welcome to the SAP HANA Academy. My name's Bob, and in this video series, we'll be looking at SAP Data Intelligence. In this video series, we'll be focusing on using the Machine Learning Scenario Manager. And what we'll cover is how to create predictions within the Machine Learning Scenario Manager and then expose them so that they can be accessed via an API. In this video, we're going to be looking at modifying our model training pipeline that we created with the Machine Learning Scenario Manager and we're going to look at two main concepts, using Docker files and also executing those pipelines. In the previous video, we looked at configuring a pipeline to train a model and save it into our machine learning scenario. Within this pipeline, we have a Python operator that needs access to the Python libraries that our code is using. We configured this Python code in the previous video. So before we can run the pipeline, we need to create what's called a Dockerfile for this Python operator. The usage of these Dockerfiles along with your Python operator gives flexibility to leverage virtually any Python library. And again, in order to do this we just need to provide the Dockerfile which will install all the necessary libraries. So what we need to do is go to our repository tab on the left hand side and we'll see that there's a folder called Dockerfiles. All we need to do is right click and select create Dockerfile. Now we'll have to give this file a name, so I'm going to call it like so. And when we've done this, all we need to do is click on OK. What you'll then see after a few seconds is this code window is created. So in our code we used various libraries within the Jupyter Notebook and we were able to import those libraries within that Jupyter Notebook. Here we're going to have to do it with these following commands. So we're running a pip 3.6 install command on the following libraries. Again, we use these same ones within our Jupyter Notebooks. The code leverages a base image that comes with SAP Data Intelligence and we'll install all the necessary libraries on it. In our example, we're specifying the versions of the libraries to ensure that new versions of the libraries don't impact our environment. Now, when we use SAP Data Intelligence for this task, we'll use tags to indicate the content of the Dockerfile. These tags are used in a graphical pipeline to specify in which Docker image an operator should run. So the way we get this to work is we open the configuration file for the Dockerfile with the icon on the top right hand corner, and we'll add a custom tag to be able to specify that this Dockerfile is for the Marathon example that we're using. In this example, which I'm showing here, we won't need any further tags, as the base image already contains the tags that we need for other libraries used in this pipeline. So all we need to do now is save this Dockerfile and then build it. So I'll select save and we'll see a confirmation below that we've saved the file and the tags. And then all we need to do is execute this to start the build. What we'll then see is within the build status that the Dockerfile is building. And it will take a few seconds for it to build. So after a few seconds we should see build status completed. And afterwards we just need to configure our Python operator. This is the one that trains the model to use this docket image. So I'll close this page. So, we'll go back to our graphical pipeline and all we need to do is right-click on our Python 3 script and go to Group. So, these groups can contain one or more graphical operators and on this group level we can specify the specific Docker image that should be used. So for us all we need to do is go to the groups configuration and select our tag. So within our tags we should be able to find our MarathonShare tag that we created a few minutes ago. So I'll click on plus, I'll click on the drop down and if I scroll down we should find that tag that we created. I've got a few, of course, because I've been prepping this for this video series. When we've done that, all we need to do is close the window or close the configuration tab. We've changed the group and now the pipeline is complete. So all I'll need to do is save the pipeline, and we can run it. So before we run it, we'll just go through some of the other operators within this pipeline. In this operator, next to the Python 3 operator, all we're doing is doing message conversion. So if I select it and we go to our configuration tab, we can see it says we're running a two message converter. This is because we need this for our Artifact Producer. So if I go to the Artifact Producer and we go to our help, we can see that the Artifact Producer registers the artifacts given as an input and returns the artifacts generated ID. And we need this for our model. When we've got this ID, what we'll want to do is write the ID. So we're using a write file operator to write that value. We're then performing another conversion, this time to a string, which will be then fed into our operators complete, which again is some Python. Now, before we look at that, let's go back to our submit metrics operator. So this is the output from our grouped Python 3 operator. So if I go to the help file, we can see that we're using this operator to submit the models, KPI or metrics to the machine learning tracking API. Again, we looked at the root mean squared error in our Jupyter notebook, which will be a value which deals with tolerances of errors, as well as the number of rows of data that we're dealing with. And again, all of this was configured in our Python 3 operator, which again outputs that metrics dictionary to our submit metrics operator. Now there's one last Python operator called operators complete before we terminate our pipeline. And as it says, when both input port signals arrive, the artifact producer and the submit metrics have completed successfully. So that means it's safe to terminate the graph and there'll be a message which confirms this. Lastly we're just setting our port callback. So that's the pipeline explained. All we'll need to do now is terminate the graph based on the operators complete. So the last thing I'll need to do here is save this pipeline, and again we can run it. So we're going to run it back in our Machine Learning Scenario Manager. So what I'll do is I'll go back. We can see that we've got this one pipeline here. So all I need to do is select it, and on the right hand side we have execute. So I'll select execute. Here we can add a configuration description if we want, but it's an optional step, so I'll just go to the next step. You can reuse previous configurations if you want, but we don't need to do that here. And here we need to set our new artifact name to be our LM underscore model. This means that the trained regression model will be saved under this name. So when we've done that all we need to do is click somewhere else on the screen and then click on save. Now, after a few seconds the pipeline executes and it will take a few seconds to complete. Of course, depending on the data size, you might need to refresh the screen a few times because for us, we've only got 120 rows or so, but it could take longer, of course, depending on the size and number of columns within your data source. But what we should eventually see is that the screen changes, and we'll see this within our Execution tab. So just to explain this screen, you can see that here we've got Execution equals Completed. We've got the models and data sets that it's produced, and also our list of metrics, as we can see here, our RMSE. So as you can imagine, you create the pipeline, you add any configuration if you want, you then execute the pipeline which will produce both metrics and your models and data sets. Now we can actually go and look at these in a more granular fashion because you can see that we've got various tabs, progress flow, configuration, metrics and models and datasets. Now if I click on my metrics you can see that we've got our RMSE and our number of rows which was 117. And of course right at the bottom we've got our model and dataset. This was the model that we created called LM underscore model. And on the right you can see you've got various data on that model including the technical identifier and when it was created. So now that we've got our model, at the moment we run it and it executes and it executes once and completes. But if we want we can now use this for real-time inference, which will mean we can supply a value and repeatedly supply different values and it will repeatedly give us predictions back. We'll go through how to do that in the next video.