Name: Data Intelligence 10 of 21 - Write Files & Message Filters
Uploaded: 2023-9-6
Duration: 373 s

Share feedback

<p>Planned maintenance of Practice Systems</p>

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Subscribe now

Learn more

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div><div><div>In this video series, we'll introduce some of the applications when using Data Intelligence. In this video, we'll go through using the Modeler to use the Writefiles operators and Message Filters operators. <br><br>0:00 - Introduction<br>0:30 - Write File operator<br>3:00 - Message Filter and configure conversion filter<br>5:00 - Add Graph Terminator<br><br>Please note:<br>                     This video was originally published on the SAP HANA Academy YouTube Channel                      and therefore may contain references to SAP HANA Academy and YouTube.                      The tech world is always evolving and the SAP HANA Academy YouTube channel                      is set to close on December 31, 2023.                      At that time, references will no longer be valid                      and will gradually be removed when the videos are updated.<br>Also, this video includes automatically generated captions                      and therefore might misrepresent the spoken content or display incorrect product names.</div></div></div></div>

Data Intelligence 10 of 21 - Write Files & Message Filters

Welcome to the SAP HANA Academy, my name is Bob and in this series of videos by Digital Partner Engineering and the SAP HANA Academy we'll be looking at Data Intelligence 3.1. In this video, we'll be using the Modeler to demonstrate the Write Files and Message Filters operators. In the previous video, we looked at an introduction into the Data Intelligence Modeler, and we used two operators to read the contents from an S3 bucket. What we want to do now is continue building up this graph and we're going to start by including a write file operator. Now the write file operator will write any content that is provided to its input node as a file to various object store services. So the first thing we need to do is search for that operator. So I'll do a search for write. We've got that write file operator within the files folder and I'll drag it onto the canvas. And again all we need to do is connect the output from the read file operator into the input of the write file operator. Now you'll notice there are two outputs from the list files and the read files. One is the actual message and the other output node is to deal with error messages. So we'll connect the file output node to the input of the write file operator. We'll also configure this so if I select the operator here we can do some modifications. So firstly what I'll do is I'll set the path mode as static from configuration. And then even though we can get this from the previous operators what we can do is have a look at that connection. So if I go into connection rather than write it to the S3 bucket we're going to write it to our semantic data lake in data intelligence. So within Connection Manager, I'm going to drill down and choose the DI data lake, and I'll click on Save. Now that we've done that, we need to choose a path where we're going to put the output of those files. So for me it's the folder doesn't exist, so I'm going to put it in the shared folder, in the SHA folder, and I'm going to put it in a file called performance.csv. Additionally, if we want we can rename the label. So rather than write file, I'll change this to append file. Which means I'm going to ultimately change the mode. At the moment it will create the file only, but what I'll want to do is change this mode format to append. We can also choose overwrite as well if we wish. Now the way the write file operator works is that when it's successfully written the last batch of data, it'll mark its last output with an attribute called last batch. That means we can use this as a trigger to safely terminate the pipeline. So the way we set that up is by using what's called a message filter. So what I'll do again is I'll search for message filter and then I'll drag on message filter onto the screen. And what we'll need to do is join the output of a penned file to the input of my message filter. Again, we're going to choose file and connect it to the input. Now when we do this, because the output node is of a different data type than the input node, we'll be prompted to select a conversion method. So for us, all we need to do is choose the first from file. Now if the screen gets a bit messy, you can always move the objects around. So if I can move this object in between and you can even move all the objects together and also remove the joins between each of the operators. So now to configure that message filter operator. So the way we do that is we'll select the message filter and then we've got here condition. So the text I'm going to put here is simply in JSON format that the message.lastBatch is equal to true. And that means we can safely terminate the pipeline when the write file operator writes its last batch of data because it will mark its last output with an attribute which will equal last batch. So the last step is we need to indicate that we want to stop the graph, and we can do this with what's called a graph terminator. So the operator terminates the pipeline execution when receiving any input to its input node. So, if I go ahead and do a search for graph, we can go and select that graph terminator. And just like before, we're going to put the output of the message filter into the input of that graph terminator. Again, since the message filter operator is filtering out all messages that don't contain the last batch attribute, the graph won't be terminated until we're sure that the last batch was written by the right file operator. So now all we need to do is perform a save. It's telling us that we haven't completed the layout, that's no problem. It's giving us an old warning, because we had a single operator. But if we go, we clear this and we go to the log, we can see that the file collection performance SHA graph saved OK. So in the next video, what we're going to do is look at running and testing that this graph successfully executes. And of course we'll be looking at the results as well. You can find more tutorials on our YouTube channel, and if you'd like to be informed as soon as new video tutorials are published, then please subscribe. Thanks for watching.