Name: Creating Dataflows in SAP Customer Data Cloud
Uploaded: 2024-2-26
Duration: 494 s
Description: In this video on Dataflows, previously known as IdentitySync, you will learn what Dataflows are and the components of a Dataflow. This is followed by a demo where we’ll use Dataflows to export customer accounts into a cloud storage bucket. If you would like more information, see the Dataflows page on the SAP Help Portal.

Share feedback

<p>Planned maintenance of Practice Systems</p>

Planned maintenance of Practice Systems

Why create an SAP account?

Enhance your learning experience

SAP Learning

Subscribe now

Learn more

Live Sessions

Hands-on Practice

Certification

My Learning

Ready to start learning?

<div><div><div>In this video on Dataflows, previously known as IdentitySync, you will learn what Dataflows are and the components of a Dataflow. This is followed by a demo where we’ll use Dataflows to export customer accounts into a cloud storage bucket.  <br><br>If you would like more information, see the <a href="https://help.sap.com/docs/SAP_CUSTOMER_DATA_CLOUD/8b8d6fffe113457094a17701f63e3d6a/2b717202e42a4165aa7f75f7eecab64f.html?q=dataflows" rel="nofollow noopener noreferrer" target="_blank">Dataflows</a> page on the SAP Help Portal.<br><br><br><br></div></div></div>

Creating Dataflows in SAP Customer Data Cloud

Hello and welcome to this video on Dataflows,  previously known as IdentitySync  in SAP Customer Data Cloud. In this video, you will learn: What are Dataflows The components of a Dataflow Followed by a demo where we’ll use Dataflows for exporting customer accounts  into a cloud storage bucket.  Dataflow is an Extract, Transform,  and Load solution to transfer data  in bulk between platforms. With Dataflows, you can export and import data to Customer Data Cloud or a third party platform Additionally, you can transfer users from one SAP Customer Data Cloud site to another. The building blocks of a dataflow  are dedicated components. A component is a pre-configured unit that is used to perform a specific data integration operation.  The components include readers, writers,  transformers, deleters, and lookups.  Each component is responsible  for performing a single task,  such as creating a CSV file  or uploading a file to S3 storage. For information on working with each component please refer to the Component Repository page  on the SAP Help Portal. In this demo, we're going to demonstrate  exporting full customer accounts  into an S3 storage bucket  on the SAP BTP platform.  On the left side menu,  choose Profiles under Identity Access.  You’ll see all the existing customer accounts displayed.  Choose Dataflows under Orchestrate.  Choose Create Data Flow.  For integration, select "Gigya with SFTP"  and for template select “Export from Gigya to SFTP”.  We're going to export full accounts  to an S3 bucket.  Choose Continue to design the dataflow steps. A full dataflow diagram is created automatically  in the graphics editor. The first step is to read the existing customer accounts  from Customer Data Cloud.  Double click on the step to enter  the necessary parameters. In this case, we can accept the default values.  Next, the import job renames  the customer profile fields  to the target fields in the CSV file. Here, we remove the profile field.  Next, we write the customer account data into a CSV file. Each customer account will be written  on a separate line.  To easily find the file in the cloud storage,  we insert the current date in the target file name. After the CSV file is created,  we compress the text file into a gzip format.  For large text files with hundreds or thousands of customer accounts,  this saves the network bandwidth  and improves the data transfer performance. Now let’s replace the last step – sftp  with a cloud storage step.   Click on the trash can icon on the sftp step  to delete it from the dataflow.  Expand the Writers  from the component repository on the right side.  Drag the Amazon.S3 component  and drop it onto the dataflow canvas.  Next, drag a success path connector  from the gzip step  and drop it onto the amazon.S3 step.  The last step is sending the gzip file  to the S3 bucket. Double click on the amazon.s3 step  to enter the bucket name,  access key, secret key, and to select the storage region.  We’ll add an additional step that allows us to handle a possible data transmission  or storage error.  Expand Writers from the component repository  on the right side,  drag the sftp component and drop it onto the dataflow canvas. Although we can connect the amazon.S3 step  to the sftp step with a red error path connector,  we can also implement it in the dataflow source codes.  Choose the Source tab located  next to the Graph tab.  Scroll down to find the amazon.S3 step  and append an “error” attribute  after the “params” attribute.  On the “error” attribute,  put the “sftp” step in an array.  Don’t forget to add a comma  after the “params” attribute. Now go back to the Graph tab.  You’ll see a red dotted arrow  from the amazon.S3 step  to the sftp step, indicating that this is an error path.  In case the job fails to send the file  to the S3 bucket,  we upload that file to our SFTP site  as a fallback.   You'll need to enter the SFTP host name and credentials. Let’s rename the dataflow.  Once finished editing the dataflow,  you can choose Save Draft to edit it later  or Publish to activate the dataflow.  Back on the dataflows page,  open the three-dot Actions menu. We can run a test to see if our dataflow  is defined correctly,  but instead, let's open the scheduler.  Here, we can create a one-time  or recurring schedule for the dataflow.  Choose Create Schedule  and enter a name for it.  Choose the run time,  and whether it runs once or at scheduled intervals.  You can choose the log level:  Choose Error and only error logs  will be displayed in the job trace. Choose Info and info and error logs will be displayed in the trace. Or choose Debug where besides info  and error logs, each record will be logged  between every two steps.  This should be used only if the dataflow  is not working as expected  and is limited to a batch of 3 records.  In addition, you can limit the job  to a specified number of records, and email addresses to which feedback  of job success or failure will be sent. Choose Create. Back on the scheduling dashboard,  you can refresh the dashboard  and see that since we chose to run once,  the schedule disappears  when the job is completed. When you select "Back to dataflows",  you can see the last successful run of this job. Navigate to the job status page via the actions  button to see the details of the job’s execution.  There are 4 statuses that may be shown for a job: succeeded, completed with errors,  failed, and stopped,  which means someone stopped the dataflow mid-run.  Choose the information icon  to view further details. So, our job completed successfully,  and we should now be able to see the gzip file in the S3 bucket.  You can download the gzip file,  extract the CSV text file,  and open it to see the exported customer  accounts from Customer Data Cloud.  We can also open our email  and see that a success notification  was sent for this job. This concludes our demo of using Dataflows  to export customer accounts  into a cloud storage bucket.  Thank you for watching.