Introducing Filter and Rule Options

There are several reasons why you might want to filter and transform your data, such as the following:

• To retrieve only the data of the last fiscal year

• To save memory and transfer time by filtering

• To avoid duplicates during N:1 replication

• To recalculate fields on the fly

• To scramble HR data

There are different tools to solve challenges such as those listed. Some of them are easier to use, while others require more technical knowledge and are thus for experts only. The different tools are described later.

The figure outlines the concept of transformation rules, which can be explained as follows:

• The data for transformation is split into portions.

  The default is as follows:

  • Load: 10,000 lines
  • Replication: 5,000 lines

• The portions are processed successively, mapped, and transferred to the sender.

• The extension of functionality, for example, data conversion, is possible at several points of the process.

• The implementation of filter or data transformation occurs as ABAP coding.

A rule can filter or transform data during the load and replication process.

In transaction LTRS, choose Rule Assignment and add the relevant table. A new user interface appears, and you can create a new rule for the table.

You can choose between two different rule types:

1. Event-Related Rule

   • More flexible
   • Knowledge of data processing within the SAP Landscape Transformation Replication Server
   • Has access to all fields of a record

2. Field-Related Rule

   • Less flexible
   • Easy to create by using parameters

As briefly discussed, there are different use cases for rules. However, you can group them into two different types:

1. Data Transformation Rule

   For example, the SID field is newly created and is not available in the source table. No value is available.

   • You want to fill the SID field with the unique value T40, which you see on SAP HANA, where the record is coming from.
   • This is especially important in an N:1 scenario so that you avoid duplicates and see the origin.

2. Filter Rule

   For example, within the table, you have several carriers, such as Lufthansa (LH), American Airlines (AA), and many more.

   You want to move the data from the carrier Lufthansa ( CARRID = "LH") to SAP HANA.

Field-based rules are used to calculate a target field from up to three source fields. Field-based rules are easier to use, but offer less flexibility than event-based rules.

In the latest SLT versions, for specifying custom coding, an include program must be defined locally in the SAP Landscape Transformation Replication Server system. However, in previous SLT versions, it was also possible to store a line of code directly under the Rule Assignment folder.

Note that each line of code is restricted to 72 characters. This means that only simple mapping rules can be realized. (When saving the settings, a syntax check is performed.) In contrast, include programs do not have any size limitations and can be used to implement complex mapping rules with the support of the ABAP editor. You specify the name of the include program in the Include Name field.

When you create a new rule, it has the status New. To view the import parameter fields, a rule must have the status Released. You can also set the status of a rule to Obsolete.

Import parameters are used within ABAP code to apply values. These can be the values of fields in the sender record or literals.

In the case of source fields, the three import parameters have the following format:

• i_＜fieldname_1＞_1
• i_＜fieldname_2＞_2
• i_＜fieldname_3＞_3

The sections ＜fieldname_1＞, ＜fieldname_2＞, and ＜fieldname_3＞ are the names of the fields that are entered as import parameters, for example, i_mandt_1.

If you have defined a literal, the technical name is as follows:

• i_p1
• i_p2
• i_p3

The export parameter specifies the field name of the receiver structure, which has to be filled within the parameter rule. The parameter of the code has the format 'e_<fieldname>', where <fieldname> is the name of the field entered as the assignment target field, for example, e_mandt.

For event-related rules, custom coding can be inserted at eight predefined points of processing (the events). When you create a new event-related rule, the system displays a dialog box where you can choose the event, for example, Begin of Processing (BOP). All fields of source and target tables can be accessed.

The data of source and target work areas can be accessed through field symbols in the Begin of Record (BOR) and End of Record (EOR) events.

The field symbols follow a generic naming convention, where tablename denotes the source table name as follows:

• <wa_s_tablename> for the source system

  For example, <wa_s_zsflight>-sid

• <wa_r_tablename> for target work areas

  For example, <wa_r_zsflight>-sid

In the case of event-related rules, even though the UI allows assigning parameters, this is not supported yet.

Note that for migration objects referring to INDX-like tables that are to be declustered, not all events are available. For such migration objects, only events BOP, BOT, EOT, and EOP are available.

The figure shows the process flow of a data replication and the points in the code (events) where you can apply rules. The events are defined as follows:

BOP (Begin of Processing)

• Processed only once, before the data transfer really starts

• Can be used to initialize certain global fields that might be used in subsequent events, for example, fill internal mapping tables

EOP (End of Processing)

Processed only once, after the data transfer is completed

BOT (Begin of Block)

Access to all data records of a portion read from the sender system

EOT (End of Block):

Access to all data records immediately, before they are passed to the receiver system

BOL (Begin of Loop):

As with BOT, if only one table is included in the conversion object

In the case of objects with multiple tables, it can be applied to each specific table.

EOL (End of Loop):

As with EOT, if only one table is included in the conversion object

In the case of objects with multiple tables, it can be applied to each specific table.

BOR (Begin of Record):

This event is processed before the field mapping of the individual fields is started.

EOR (End of Record):

This event is processed after field mapping of the individual fields of a certain data record has finished.

Note that the Begin of Block (BOT) and Begin of Loop (BOL) events, as well as the End of Loop (EOL) and End of Block (EOT) events, are executed directly after each other for simple structures, such as transparent tables.

For more complex structures, such as cluster tables, the Begin of Block (BOT) and End of Block (EOT) events are triggered before looping over the clusters. However, the loop events, Begin of Loop (BOL) and End of Loop (EOL), are executed before looping over the entries of the logical tables in a cluster. Similarly, the Begin of Record (BOR) event and the End of Record (EOR) event do the same if no parameter rules are defined.

The naming convention to access tables for events (BOT, BOL, BOL, EOT, EOL, EOL) is as follows:

• ＜it_s_tablename＞ for the source table

• ＜it_r_tablename＞ for the target table

You only want to transfer a subset of the records in the source table. For instance, you only want to transfer specific records belonging to a specific company code or plant. For this purpose, you can use the SLT macro SKIP_RECORD in a transformation rule.

For example, for the filter rule, you work with the export parameter for the carrier (E_CARRID) and compare it with the value of Lufthansa ('LH'). If the value of CARRID is not equal (NE) to 'LH' , then the system will execute SKIP_RECORD, and the record will not be transferred to the target.

123456

IF E_CARRID NE 'LH’. 
  SKIP_RECORD. 
ENDIF.

You can also use the SKIP_RECORD macro in the ABAP includes of event-related rules.

You specify the filter code as follows:

1234
IF <wa_s_zsflight_00>-carrid NE 'LH'.
	SKIP_RECORD.

ENDIF.

If you need only a subset of the data from your source system, you can improve the performance of the initial load and the replication process by transferring only the data that you need.

To do this, specify the filter conditions for the table in transaction LTRS under Performance Options. You can specify filter conditions for key fields and for non-key fields. You can filter the initial load, the replication process, or both. You can also save the filters without using them.

For example, table ZSFLIGHT_00 contains 6 billion records. There are six different clients (the key field in position 1 in table), and the records are distributed evenly across all clients. According to your business requirements, you only require the data from two clients in the target system. You specify that only two clients should be included in the initial load. SLT then only transfers 2 billion records to the target system. This also significantly improves the performance of the initial load, as follows:

• Filter 1: MANDT = '000'

• Filter 2: MANDT = '650'

Only records from clients 000 and 650 are transferred to the target system.

You can define several filters and, for each filter, define multiple filter conditions. For each filter condition, you can define a value range for every key field, such that only specific values are replicated to the target.

For any additional settings to take effect, you must save your changes in the table before you start the initial load.

Note the following general aspects when you specify filters:

• The filter options are not available for objects with a name that is longer than 20 characters. The Filter Options tab is hidden for such objects.
• When specifying filters for a field, you should consider creating an index for the field to improve performance.
• Using filters on a non-key field for the initial load can negatively impact performance. You can either create an index for the field or use reading type 4 or 5 to improve performance.
• If you have manually specified filters that are not compliant with the dialog logic or you have specified different filters for the initial load and for the replication process, you can view these filters on the Filter Options tab but cannot edit them.
• For INDX-like tables, filter conditions can be specified for the replication process only. It is not possible to use filters for just the initial load or for the initial load and replication process. In addition, it is only possible to specify filter conditions that use the key fields of the INDX-like table.
• For cluster tables, the filter conditions are used for both the initial load and the replication process. It is not possible to use filters for just the initial load or for just the replication process.

It is not possible to filter the initial load if you use reading type 3.

• When a filter is specified on a non-key field, the filter does not work on data which is deleted or archived.
• When a filter is specified for a table being part of a cluster table, for example BSEG of RFBLG cluster, the filter does not work on data which is deleted or archived. If you specify the filter on the cluster table (RFBLG), the filter will only apply to delete and archive operations.
  Caution

  Due to the way SLT deals with cluster tables - first deleting all data for the key of a respective record of the cluster table (RFBLG) and then inserting the data of the table (BSEG) – putting a filter on the cluster table may lead to inconsistent data/errors in the replication process.
• Filters for the replication process are not supported:
  • when replicating data to SAP BW by Using the ODP Framework
  • when replicating data to Central Finance

If you still want to filter the data of the table, although one of the limitations mentioned here apply, you can still use your own rules to filter the data which can be specified in the Rule Assignment area of transaction LTRS (except when replicating data to Central Finance).

If you need multiple value ranges for the same key field, you must create a separate filter for each range because each key field can only be used once in the filter condition of one filter. All filters are used when data is selected from the source system (using an AND operator), so only those records that fit defined filter conditions are transferred.

You can also create more complex filters by using the available syntax for a WHERE clause of a SELECT statement (for example AND, OR, LIKE).

SLT captures all updates, inserts, and deletes for a table that is in replication. If your configuration does not use the new Subscription-based CDC recording mechanism and you expect a high change rate on the table, but your use case requires only a subset of the records, you can minimize the replication volume by a trigger filter. Compared to replication filters (explained previously), where data is filtered on the SLT server, in this case, data is already filtered in the trigger. Therefore, you might gain performance benefits if filtering when already in the source system.

Any filters that you define here are only applicable for the replication process; they do not affect the initial load. To filter the initial load and replication, additional filter rules have to be defined.

Within the trigger options in LTRS, you can specify a trigger code that will be added when SLT creates the database trigger on the source system. Ensure that the trigger condition is correct. The trigger option is an expert function and the code is database-specific. Remember that with the wrong settings you can harm your source system.

There are two options for changing the settings for triggers:

• You can specify general settings that apply to all triggers in the source system.
• You can also specify table-specific settings. Table-specific settings take priority over any general settings.

Creating a projection view, with a subset of table fields, means filtering on the source system. Only data in fields that have been selected for the view are transferred by SLT into the target system. The table in the target system has the same name as in the source system, but only contains the subset of fields.

View for Initial Load

You can use a database view to select data from the source table during the initial load. For example, you can use a view to filter columns so that not all columns are loaded to the target system, or you can use a view to join fields from other tables.

The view that you specify here must contain at least all of the key fields from the source table.

View for Replication

You can use a database view to select data from the source table during the replication process. For example, you can use a view to filter columns so that not all columns are loaded to the target system. Or you can use a view to join fields from other tables.

The view that you specify here must contain at least all of the key fields from the source table.

Note

As per the product functionality, when the source system is SAP S/4HANA 2020 or higher (capable of replicating CDS views), it is not possible to specify an alternate database view for Initial Load and view for Replication. Refer to note: 3222382 - View for Initial Load/Replication is not available in LTRS Table Settings when the source system is S4/HANA 2020 or higher - SLT.

Note

If the source system is connected to the SAP Landscape Transformation Replication Server system by means of a database connection, you must enter the name of a database table or view in the Structure Template field. This database table or view must contain all of the key fields from the source system table.

If you enter a database view in the Structure Template field, ensure that it exists in the SAP Landscape Transformation Replication Server system. The value of the Template Location field must be SLT System.

As of DMIS 2011 SP08, the replication of views is possible for non-ABAP sources.

As well as projection views, you can also load data from views that are joining two tables. If you join fields of different tables into one view, one table (A) is the leading table, and fields from other tables can be added.

In the SLT system, the table deviation has to be built as a transformation of table A or by using the Structure Template option. The table keeps the name, as in the source system, but is enlarged with the additional fields.