Name: Data Intelligence 14 of 21 - Wiretaps
Uploaded: 2023-9-6
Duration: 393 s

Share feedback

<p>Planned maintenance of Practice Systems</p>

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Subscribe now

Learn more

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div><div>In this video series, we'll introduce some of the applications when using Data Intelligence. In this video, we'll go through using the Modeler to utilise Wiretaps. The Wiretap operator can wiretap a connection between two operators in a Modeler graph and display the traffic to the browser window or to an external websocket client that connects to this operator.<br><br>0:00 - Introduction<br>1:00 - Wiretaps<br>1:30 - Create new graph with Workflow Trigger operator<br>2:00 - HANA Table Consumer operator<br>2:50 - Flowagent CSV Producer operator<br>3:20 - Wiretap operator<br>4:30 - Save and run graph<br>5:00 - View Wiretap output<br><br>Please note:<br>                     This video was originally published on the SAP HANA Academy YouTube Channel                      and therefore may contain references to SAP HANA Academy and YouTube.                      The tech world is always evolving and the SAP HANA Academy YouTube channel                      is set to close on December 31, 2023.                      At that time, references will no longer be valid                      and will gradually be removed when the videos are updated.<br>Also, this video includes automatically generated captions                      and therefore might misrepresent the spoken content or display incorrect product names.</div></div></div>

Data Intelligence 14 of 21 - Wiretaps

Welcome to the SAP HANA Academy. My name is Bob and in this series of videos by Digital Partner Engineering and the SAP HANA Academy, we'll be looking at Data Intelligence 3.1. In this video, we'll be looking at the concept of using wiretaps when using the modeler. In the previous video, we used the modeler within Data Intelligence to connect to a series of files within an SRE bucket and consolidate them all into a singular table running in a schema in the HANA cloud. Now what you'll see is we have a date, a cell ID and various keys. We have four keys in total. Now in the next video, we're going to set up some validational rules to check that the values of these keys fall into certain thresholds. And before we set up those validational rules, we'll have to do some prep, which is what we're going to cover in this video by looking at the concept of wiretaps. Now wiretaps give us the ability to look at the data in the modeler when you're using data intelligence. You don't have to then go back to Metadata Explorer to evaluate the data. The first thing we need to do is build a new graph. And the first object I'm going to add is a workflow trigger. So I'll do a search for triggers and we'll add that workflow trigger. The reason we need this workflow trigger is because some of our operators within data intelligence require an input in order to start pipeline execution. We covered this in the last video, but if I open up the documentation again we can see that it sends a start signal when it's activated. So now what I'll do is I'll do a search for HANA table. And we're going to take that HANA table consumer. As you can imagine this is going to read from our data set which is in HANA. So I'll join the two together and then we'll configure that HANA table consumer. We just need to connect to it. We've got a connection in the Configuration Manager that we set up in the Connection Manager, and that table exists in a HANA Cloud system. The source table, we need to specify it. We've got a schema called SAP IT, which I can search for. And if I select that schema, the table we're connecting to is SHA underscore cell status. And again we configured that and built it in the previous video. Now because we want to see this data within the modeler, one way to do it is to use a flow agent CSV producer. So this will output the data in CSV format. We'll need this also for some operators later on in this video series. So I'll put a flow agent CSV producer and I'll join the output of the HANA consumer into the input of that CSV producer. Again later on in one of our operators we're going to be able to need to read CSV data. So now that we've got it in CSV format we can search for a wiretap operator. And this just gives us the ability to view the data during execution. So you'll see from the CSV producer there are three outputs. The top one out content outputs the data. The middle one outputs a message object which provides more information. And the last one outputs any errors. And we just need the data and nothing else. So I'll join the out content port to the input port of the wiretap. Now that second port provides a lot more information. You've got the header of the CSV and you've got indicators to let you know if the current batch is the last batch. And again it's very easy to get documentation on each of these ports as you can see here. Now going back to the wiretap there's nothing for us to configure. Here we can see in the documentation it simply enables us to view the data in a separate tab during runtime. So I'll save this graph and I'll give it a name. So I'm going to call it SHA.validateDataQuality. And again all it's going to do is read the data from that HANA table and output it on the screen in the form of CSV format. So of course now what we need to do is execute that graph because the data will only be in the wiretap user interface when the graph is running. So we'll wait for status to say running and now what it means is that if we right click on our wiretap we've got access to a link which is open UI and here we can see that CSV data. Again, this is used to verify that we're connecting to that HANA table correctly, but we'll also need the data in CSV format for the next operator. Now, the data will remain in this wiretap while the graph is running. If I go back to the graph and if I stop the graph, or if the graph completes, or if you have, for example, a workflow terminator, then as soon as the graph is finished, the data in the wiretap will be removed to our stop. And when the graph has been completed we'll no longer have access to that open UI link when we right click on the wiretap. So in this video we've looked at a couple more operators, the HANA table consumer and wiretaps. Now that we've got the data in the format that we need, which is that CSV format, what we're going to do in the next video is look at validational rules. You can find more tutorials on our YouTube channel, and if you'd like to be informed as soon as new video tutorials are published, then please subscribe. Thanks for watching.