Loading Data with ODP_SAP

Objectives

After completing this lesson, you will be able to:

  • Load Data with ODP_SAP

Check Extraction of DataSources

Extraction Check

Now that you've learned about ODP_SAP DataSources, let's see how to load the data to SAP BW/4HANA. Before starting a DTP to load data, it's useful to first test the extraction to check if it provides the correct data.

Extraction Architecture. Extractor-based DataSources are loaded to SAP BW/4HANA using ODP.

The extractor-based DataSources are loaded to SAP BW/4HANA using ODP, as the S-API interface is not used anymore. However, the Extractor Checker can still be used.

Extractor Checker S-API. The transaction code RSA3 is highlighted. You can review the data supplied by a traditional SAP DataSource.

The Extractor Checker is launched with the transaction code RSA3. This is a good way to look at the data supplied by a traditional SAP DataSource.

The DataSource runs in SAP ECC. Then, you can look at the data records that are selected for extraction. Using this technique, you can prove that the DataSource runs properly in SAP ECC. Before you execute the DataSource, you can also control the number of data packets (calls) and the number of data records per data packet.

Load SAP Data Using ODP_SAP

Watch this video to see how text data is loaded using an ODP_SAP DataSource.

Operational Delta Queue (ODQ)

Operational Delta Queue Monitor (ODQMON)

When you use Operational Data Provisioning, the transaction ODQMON allows you to monitor the operational delta queue.

Monitor Delta Queue Requests screen and the Monitor Delta Queue Data Units.

The role of a provider is to provide one or more delta queues of a specific type. The BW DataSource is an example of a Provider. The target application of the delta queue is referred to as the Subscriber.

A subscription can be defined as follows: a specific subscriber orders data changes from one or more queues and continues processing the transferred data. Subscribers are categorized by their subscriber type. A subscription occurs when the subscriber requests data. Every subscription has a unique transaction number (for example, 2022-09-02 08:53:26 000000 CET). A subscriber can have more than one subscription. A queue can also be in multiple subscriptions for the same subscriber.

The data, which is highly compressed, is retained in the delta queue for a specified time period, in case the subscriber wants to retrieve the data records again. More than one subscriber can request the data changes to a queue.

Data cannot be changed in the ODQ (a feature that previously did exist with the PSA). Instead, the inbound table data of a DataStore Object (advanced) can be edited before activating the request. If raw data must be persisted in SAP BW/4HANA, a field-based DataStore Object (advanced) can be used to replace this role of the former PSA.

Views in the ODQ

Monitor Delta Queue Data Units screen.

The following calculations are performed in the monitor view:

  • Queues:

    For each queue, the system counts the number of units, totals the rows and data volume (before and after compression), and finds the compression rate. It also finds the smallest and largest sequential transaction number (TSN).

  • Subscriptions:

    For each subscription, the system counts the number of changed units per queue that are saved (in a retrievable state) in the delta queue. The system totals the changed rows (data changes) and the volume of saved, nonretrieved data (before and after compression), and also finds the compression rate.

  • Requests:

    For each request, the system counts the number of units per queue, totals the rows and data volume (before and after compression) and finds the compression rate.

  • Units:

    The Unit view displays the data that was extracted when you double-click on the unique time stamp. The preselection made in the monitor or the navigation route to this view (and the associated time stamps and queues) define which units are displayed here. For a unit, the system totals the rows and the volume of saved, nonretrieved data (before and after compression), and also finds the compression rate. Note that the system always performs the calculation at the unit level.

When you calculate the data volume (extended view) in the upper screen area of the monitor, the data volume and related key figures are calculated and displayed.

Note
This calculation can take a long time if a lot of units are saved in the delta queues. Therefore, the calculation is not performed automatically when the monitor is called. To perform this calculation, select the relevant checkbox. When you change to a different monitor view, the calculation setting applies to the new view.

Monitoring Requests in the Delta Queue Monitor

A request can be defined as a data request from a subscriber. There are two different types of requests, as follows:

  • A composite request transfers data from one or more queues that have been grouped together into a subscription.

  • An extraction request transfers queue data from the provider to the queue storage.

A composite request can contain several extraction requests. If only data changes that were written to the queue by the source application are requested, the composite request does not contain any extraction requests.

You monitor the requests in the Requests overview in the delta queue monitor view. You can perform various actions for the unconfirmed requests: in other words, requests that the subscriber hasn't transferred.

Use to check all the displayed requests to see if any of the extraction processes were canceled or terminated. The status in the database and in the display is updated (corrected).

You may want to investigate failed extraction requests. To analyze the cause of the error, follow these steps:

  1. Use Job Overview to open the job overview. Here, you can display the log of the relevant background job (if it has not been deleted).
  2. In the application log (transaction SLG1), you can analyze the data extraction logs, using object ODQ and sub-object Extraction.
  3. Use to reschedule failed extraction requests. The extraction process is delayed by 60 seconds. This gives you the chance to activate debugging for the background job - in the process overview (transaction SM50) or in the global work process overview (transaction SM66) - and find the cause of the error.

You may want to close unconfirmed requests.

Use to conclude requests that the subscriber hasn't confirmed or canceled. For the following reasons, you may need to close a request in the monitor:

  • The request without subscription has not been retrieved (for example, the connection to target system was deleted).

    In this case, don't assume that the subscriber will close the request themselves.

  • The system could not extract the data changes (delta) .

    In this case, none of the subscribers to this queue can continue transferring the new data, because data changes must be transferred seamlessly in the correct sequence. Usually, closing the failed request is not enough to continue transferring the data changes for this queue. You must solve the cause of the problem, otherwise the next extraction attempt will also fail.

If a reorganization job is scheduled, it deletes all requests that have been successfully retrieved by all the downstream targets based on the Recovery period (the default is 24 hours). The following figure shows the normal SAP job log, and also highlights the cleanup job that is normally run automatically.

Job overview screen. The Duration column is highlighted.
Note
The retention period setting (low versus high) is not currently programmed to behave differently. For more information on the clean-up process, use this link:https://wiki.scn.sap.com/wiki/pages/viewpage.action?pageId=449284375#OperationalDeltaQueueCleanup(ODQ_CLEANUP)-Classificationofrelevanceofdata

Review the ODQ

Watch this video to see how transaction ODQMON is used to analyze the data in the ODQ. It also covers the cleanup job to delete records when they are no longer needed.

Check ODP Extraction

Troubleshoot Extract from source system to SAP BW/4HANA.

You have already learned about the Extractor Checker. If you only need to check if data has been successfully extracted from a specific DataSource, you can refer to the standard table RSDDSTATEXTRACT. The table is used to store extractor statistics, such as the time of last delta load. But how can you test if ODP behaves as you expect, without extracting data physically to SAP BW/4HANA?

Program RODPS_REPL_TEST - Debug ODP Issues: Extract Data screen.

Problems with the provider part (extraction and fetch of data) can be reproduced with the program RODPS_REPL_TEST.

RODPS_REPL_TEST acts as a separate extra subscriber, so it is not a simulation. The queue entries in ODQMON, with corresponding table entries for RODPS_REPL_TEST, are created and adjusted. The application data is physically written to the queue. The created requests are subject to the regular ODQ clean-up job and will disappear after the retention time.

Be careful when using RODPS_REPL_TEST and Init/Delta replication mode and reset the subscription that your tests created. The reason is that an ODP source uses only one delta queue, which is shared between the various subscribers. The delta data is only cleaned up if all subscribers picked the delta. This does not happen for subscriber RODPS_REPL_TEST; therefore the delta table ODQDATA can grow big over time and have an impact on general ODQ loading performance.

The report selection fields are as follows:

  • ODP Context: Choose the provider context here, such as Server Application Programming Interface (SAPI) or SAP Landscape Transformation (SLT).
  • ODP Name: Name of the ODP. In SAPI-context the DataSource name.
  • Replication Mode (Full, Delta, or Recovery.)
  • Selections: Optional parameter where you can enter selection criteria (current restriction: if the ODP has more than 75 selection fields, no selection is offered).
  • Projections: Optional. Here, you can restrict the field list.
  • Read Open Pointer: Here, you can enter a pointer to repeat a delta. This pointer relates to subscriber RODPS_REPL_TEST. Currently, it's not possible to enter a pointer of another subscriber, for example, SAP_BW or BOBJ_DS.
  • Settings for execution: Choose the option that you want to test.
  • Output: The application data is stored in the queue in a highly compressed format. With this option, you can simulate the output for different wrapping and unwrapping options.

It is possible to protocol how the RODPS_REPL_ODP_OPEN and RODPS_REPL_ODP_FETCH function modules are called. This is especially helpful if it is unclear whether problems are on the subscriber or provider side. Problems can include the following:

  • Wrong data selection. Which selection was passed from the subscriber, such as SAP Data Services?

  • Processing mode. A repeat was expected, but a delta was done. What was requested from the subscriber?

  • How to enable the trace is described in SAP Note 1580242 Point 4.

  • It requires the setting of RSADMIN-parameter RODPS_REPL_TRACE = X and a new extraction afterwards. Refer to SAP Note 2453327.

How to debug the ODP-extraction is described in SAP Note 1580242:

  • Debugging the ODP-extraction works by forcing the program into an endless loop, then catching the process through SM50. Therefore, debugging an ODQ-extraction requires authorizations for debug and also the authorizations to catch a process in SM50 for debugging.
  • The wait loop is activated for a specific user via a check-point group (transaction code SAAB).
  • There are different checkpoints for the OPEN and FETCH part of the extraction in transaction code SAAB.
  • Function Module RSDS_DATA_PULL is a good entry point for debugging ODP-SAPI extraction FROM InfoPackage or DTP.
  • Class CL_RSDS_ACCESS_ODP (Method EXTRACT) is a good entry point for debugging DTP (simulation).

Log in to track your progress & complete quizzes