Name: Data Intelligence 16 of 21 - Python Operators
Uploaded: 2023-9-6
Duration: 389 s

Share feedback

<p><u><strong>Planned maintenance of Practice Systems</strong></u></p>

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Choose your subscription

Manage your subscription

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div><div>In this video series, we'll introduce some of the applications when using Data Intelligence. In this video, we'll go through using the Modeler to run some python using the Python 3 Operator.<br><br>0:00 - Introduction<br>0:50 - Python3 Operator with script on failed rows<br>2:20 - Script  walkthrough<br>4:30 - Add Wiretap operator, save and execute<br><br>Please note:<br>                     This video was originally published on the SAP HANA Academy YouTube Channel                      and therefore may contain references to SAP HANA Academy and YouTube.                      The tech world is always evolving and the SAP HANA Academy YouTube channel                      is set to close on December 31, 2023.                      At that time, references will no longer be valid                      and will gradually be removed when the videos are updated.<br>Also, this video includes automatically generated captions                      and therefore might misrepresent the spoken content or display incorrect product names.</div></div></div>

Data Intelligence 16 of 21 - Python Operators

Welcome to the SAP HANA Academy. My name is Bob and in this series of videos by Digital Partner Engineering and the SAP HANA Academy we'll be looking at Data Intelligence 3.1. In this video we'll be looking at using the Python 3 operator when using the modeler. In the previous few videos we looked at using the Data Intelligence Modeler to build a graph which outputs data in CSV format. This is what some of that data looks like and what we want to do in this video is apply some code to it and we're going to apply some Python to it. This will just serve as a demonstration into some ways in which you can modify operators within data intelligence. Now the Python that we're going to add is simply going to rename some of the columns and also add some extra values in some columns. It's a very simple Python script. Again, it's just to show the capabilities of what you can do with these operators in the modeler. So I'll go and search for that Python operator. So if I do a search for Python, we're going to use this Python 3 operator. So I'll select it on the canvas. And we're only going to deal with those failed roles which come to us through the second wiretap. So what we'll need to do first is configure this Python operator. So as you can see the Python operator at the moment doesn't have any output ports. But we can add them by right-clicking and selecting add port. So the first port we'll add, we'll call it failed. And this will be an input port and the data type is of string. The next port we'll add will be an output port. So we'll call this ticket, and then for the data type we'll change this to data type message, and we'll make it an output port. So when I click OK now we can see that new output port and input port. So now we need to add the script. So the way we'll do that is by right clicking on the Python 3 operator and there's an open script tab. You can get this script from github by going to our SAP HANA Academy github repository and looking for data intelligence. And what I'll do is I'll paste that code here over this existing code which is just an example. We can also maximize this screen by minimizing the operators tab. So to briefly explain the code, the first three lines are importing some libraries, as we're using Panda's data frame. And then what we do is we read the data from the CSV file into a data frame. Line 9 is creating the column headers, and what you'll see is we've got three extra columns. Error Action, Error Columns and Row ID, which were given to us by the validation rule. We'll then simply rename those columns to get rid of the spaces. So we'll put an underscore instead of a space between any column that has two words. Next we'll add some ticketing information. In this simple example we're adding three columns, Timestamp, Status and Comment. We're adding the timestamp, which is now, and then we'll add a status with the value open, and of course a comment, which I'm just putting SHA to indicate that I've added that extra value to that column. We'll also define the row ID as a type integer. Then what we'll do is we'll explicitly state the order of the columns, and lastly we'll send this to the output port which we've named ticket, and we'll put the whole body of the text, and in our example we're formatting the date and including the header. We're also including some attributes which is data colon failed as a header to the CSV output. Lastly we're setting the callback port and if you remember we created an import port called failed. So what I'll do is I'll close this tab. Here's that input port, and as again you can see the connection from the wiretap to our Python operator. But what I can do is I can remove it by selecting it, and then rejoining to that failed input port of that Python 3 operator. Of course we'll want to see the results, so I'll go and grab myself a wiretap. I'll drag that onto the screen, and I'll connect the output ports from the Python 3 operator to the wiretap. We'll save the graph, and then we'll execute the graph. So after a few seconds we should be able to go and look at that UI. So if I go back to my wiretap, here we can see that fourth wiretap that we've just added. And now we've got those three extra columns. So the first one is the timestamp as of now. Then we've got the two comments that I added. And of course the rest of the columns came to us from that validation rule including those last three columns as you can see here. Now as an aside you can also select the additional wiretaps that you've got in this graph so you can view them here. So of course if I would have renamed the wiretap, it would be easier to distinguish which type wiretap applies to which part of my graph. So a very short video showing how we can use a little bit of Python code to create some headers and to output three extra columns of data. Of course the Python can be a lot more elaborate if you so choose. If I go back to the graph, of course I can stop this graph from running. So now that we've looked at how to add some Python to our graph, what we're going to do in the next video is look at outputting this again to a HANA database. But this time we'll use a different operator. Today we're going to be using the SAP HANA Client Operator, and we'll cover that in the next video. You can find more tutorials on our YouTube channel, and if you'd like to be informed as soon as new video tutorials are published, then please subscribe. Thanks for watching.