Troubleshooting Common ZDO Issues

This section show how to find and read log files from Software Update Manager (SUM) to troubleshoot issues and track the progress.

When an issue occurs, we recommend that you view the error using the SUM UI as your very first step. In many cases, the SUM UI displays an excerpt of a log file that is a good starting point for analyzing the errors.

In this example, the phase RUN_IMPACT_ANALYSIS_ZDO failed. The SUM UI already displays the file name where you can find the detailed information for this phase. In this example, the detailed error is written in file IMPANAUPG.PPM where the last three letters represent for the system ID. This information can also be retrieved from the file, PHASE_RUN.ELG.

If you have no information on which log file was written by a specific phase, check the file SAPup.log that shall point you to the correct log file. Taking again the example of phase RUN_IMPACT_ANALYSIS_ZDO, SAPup.log shows the following information:

1234567
CURRENTPHASE MAIN_SHDIMP/SUBMOD_SHD2_RUN/SUBMOD_LC4BRI_PREP/RUN_IMPACT_ANALYSIS_ZDO
...started at 20210422120446
# Using phase log file 'PHASE_RUN.LOG'.
...begin processing at 20210422120446
# Resource usage self:     0.0/    0.0/0MB usr/sys/mem, children     0.0/    0.0/0MB usr/sys/maxmem.
...finished at 20210422120449 with status FAILED.
# Error message set: 'Single errors (code <= 8) found in logfile 'PHASE_RUN.ELG''

If the failed phase is already passed but you still want to check for the relevant log file, the content of the ELG file might has been appended already to the SAV file. In this case, check PHASE_RUN.ELG or PHASE_RUN_ELG.SAV to find the right output. The file PHASE_RUN_ELG.SAV contains the following information:

12345678910111213
...
################################################################################################
Phase: RUN_IMPACT_ANALYSIS_ZDO # Description: Analysis of the impact of table classification to productive usage statistics ERRORS
################################################################################################
#==============================================================================================#
# Find the detailed information in log 'IMPANAUPG.PPM':
#==============================================================================================#
A4 ESUPG 665 Report name ...: "RSUPG_RUN_IMPACT_ANALYSIS" Component: "BC-UPG-TLS-TLA" "(Upgrade Tools for ABAP)"
A4 ESUPG 002 " "
A4 ESUPG 001 -------------------------------------------------------------------------
A4 ESUPG_IMPANA 007 Report "RSUPG_RUN_IMPACT_ANALYSIS" started
A4 ESUPG_IMPANA 001 -------------------------------------------------------------------------
...

As seen in the SUM UI already, the relevant log file for phase RUN_IMPACT_ANALYSIS_ZDO is named IMPANAUPG.PPM.

To show all log files, you can either stay in the SUM UI or you navigate to the directory SUM/abap/log, using transaction AL11 in your ABAP system or use OS command tools.

In general, check the most recent files first since the SUM updates the log files frequently. In some cases, it might be helpful to check the most recent log files in the directory SUM/abap/tmp.

This section introduces enhanced consistency checks as a prerequisite for a successful execution of a Zero Downtime Option (ZDO) maintenance event.

When running a zero-downtime upgrade, it is essential that the ABAP Dictionary objects and CDS views are in a consistent state prior the start of the ZDO procedure. In order to prepare the database schema used by the bridge subsystem, SUM will create views for every database table in the system. Additionally, the existing CDS views will be generated for the bridge subsystem. In case objects are inconsistent, the generation of those objects would fail. Hence, it's important to check the consistency in an early state of the upgrade procedure.

For ABAP Dictionary objects, SUM is equipped with an additional consistency check running during the ZDO procedure as part of the Checks roadmap step. The ZDO consistency check is split into three phases:

The initialization phase RUN_ZDO_CONSISTENCY_CHECK_INIT collects all ABAP Dictionary tables and writes the information into an internal control table. Exemplary log file output (ZDO_CONSISTENCY_CHECK_INIT.<SID>):

12345678910
4 ETG057 Job name ......: "CL_ZDM_CONSISTENCY_CHECK" Component: "BC-UPG-TLS-TLA" "(Upgrade Tools for ABAP)"
4 ETG040 Start time ....: "16.11.2021" "15:28:58"
4 ETG011 " "
4 ETG039 -------------------------------------------------------------------------
A4 ESUPG 654 "149.617" objects added to SZDM_DDIC_CHECK
4 ETG039 -------------------------------------------------------------------------
4 ETG011 " "
4 ETG057 Job name ......: "CL_ZDM_CONSISTENCY_CHECK" Component: "BC-UPG-TLS-TLA" "(Upgrade Tools for ABAP)"
4 ETG040 Start time ....: "16.11.2021" "15:28:58"
4 ETG041 End time ......: "16.11.2021" "15:28:58"

In case an inconsistency is found for any of the database tables, the phase PARRUN_ZDO_CONSISTENCY_CHECK stops with an error. In this example, two tables are found by the check which are in an inconsistent state.

Exemplary log file output (ZDO_CONSISTENCY_CHECK_*.<SID>)

12345678910111213141516171819202122
4 ETG039 -------------------------------------------------------------------------
4 ETG011 " "
4 ETG057 Job name ......: "CL_ZDM_CONSISTENCY_CHECK" Component: "BC-UPG-TLS-TLA" "(Upgrade Tools for ABAP)"
4 ETG040 Start time ....: "16.11.2021" "15:29:00"
4 ETG011 " "
4 ETG039 -------------------------------------------------------------------------
A4 ESUPG 652 Start "parallel" processing of consistency check
...
A4 ESUPG 651 "ZTABLEMODIFDB" is consistent
A4 ESUPG 650 Check consistency of "ZTABLENODB"
A2EESUPG 006 Unable to retrieve the DB fields of table "ZTABLENODB" due to rc "4"
A2EESUPG 007 Unable to determine the table delta between "ZTABLENODB" and "ZTABLENODB"
A4 ESUPG 650 Check consistency of "ZTABLENOINDEX"
A2EESUPG 234 Table "ZTABLENOINDEX" is inconsistent in key information NT <-> DB
A2EESUPG 007 Unable to determine the table delta between "ZTABLENOINDEX" and "ZTABLENOINDEX"
A4 ESUPG 650 Check consistency of "ZTIMESTAMP"
A4 ESUPG 651 "ZTIMESTAMP" is consistent
4 ETG039 -------------------------------------------------------------------------
4 ETG011 " "
4 ETG057 Job name ......: "CL_ZDM_CONSISTENCY_CHECK" Component: "BC-UPG-TLS-TLA" "(Upgrade Tools for ABAP)"
4 ETG040 Start time ....: "16.11.2021" "15:29:00"
4 ETG041 End time ......: "16.11.2021" "15:30:37"

When SUM is resumed without correcting the inconsistencies, the third phase RUN_ZDO_CONSISTENCY_CHECK_POST collects and aggregates the error messages. The consolidated error messages are written into the file CHECKS.LOG that is used at the end of the sub-module. Furthermore, this phase writes its own consolidated log file ZDO_CONSISTENCY_CHECK_POST.<SID> like in this example:

12345678910111213141516
4 ETG057 Job name ......: "CL_ZDM_CONSISTENCY_CHECK" Component: "BC-UPG-TLS-TLA" "(Upgrade Tools for ABAP)"
4 ETG040 Start time ....: "16.11.2021" "15:31:29"
4 ETG011 " "
4 ETG039 -------------------------------------------------------------------------
A4 ESUPG 652 Start "serial" processing of consistency check
A4 ESUPG 650 Check consistency of "ZTABLENODB"
A2EESUPG 006 Unable to retrieve the DB fields of table "ZTABLENODB" due to rc "4"
A2EESUPG 007 Unable to determine the table delta between "ZTABLENODB" and "ZTABLENODB"
A4 ESUPG 650 Check consistency of "ZTABLENOINDEX"
A2EESUPG 234 Table "ZTABLENOINDEX" is inconsistent in key information NT <-> DB
A2EESUPG 007 Unable to determine the table delta between "ZTABLENOINDEX" and "ZTABLENOINDEX"
4 ETG039 -------------------------------------------------------------------------
4 ETG011 " "
4 ETG057 Job name ......: "CL_ZDM_CONSISTENCY_CHECK" Component: "BC-UPG-TLS-TLA" "(Upgrade Tools for ABAP)"
4 ETG040 Start time ....: "16.11.2021" "15:31:29"
4 ETG041 End time ......: "16.11.2021" "15:31:29"

The following section describes how the inconsistency of the two examples seen above can be resolved.

Example 1 shows an error for table ZTABLENODB. The error log shows the following messages:

12
A2EESUPG 006 Unable to retrieve the DB fields of table "ZTABLENODB" due to rc "4"
A2EESUPG 007 Unable to determine the table delta between "ZTABLENODB" and "ZTABLENODB"

The first step to check the root cause is to run a database object check. This can be done through the SAP GUI, either using transaction SE11 or SE14.

The database object check for table ZTABLENODB returns Table is not created in the database. When you create the database table and re-running the same check, the issue is resolved.

Moving on to example 2 for table ZTABLENOINDEX. The error log shows the following messages:

12
A2EESUPG 234 Table "ZTABLENOINDEX" is inconsistent in key information NT <-> DB
A2EESUPG 007 Unable to determine the table delta between "ZTABLENOINDEX" and "ZTABLENOINDEX"

Similar to example 1, also in this case, the database object check shows the root cause. For table ZTABLENOINDEX, the primary index is missing on the database. After creating the database index and re-running the same check, the issue is resolved.

When you have resolved all inconsistencies, SUM can be resumed. The consolidated log file ZDO_CONSISTENCY_CHECK_POST.<SID> is updated and shows the success message No objects found:

1234567891011
4 ETG057 Job name ......: "CL_ZDM_CONSISTENCY_CHECK" Component: "BC-UPG-TLS-TLA" "(Upgrade Tools for ABAP)"
4 ETG040 Start time ....: "16.11.2021" "15:48:38"
4 ETG011 " "
4 ETG039 -------------------------------------------------------------------------
A4 ESUPG 652 Start "serial" processing of consistency check
A4 ESUPG 653 No objects found
4 ETG039 -------------------------------------------------------------------------
4 ETG011 " "
4 ETG057 Job name ......: "CL_ZDM_CONSISTENCY_CHECK" Component: "BC-UPG-TLS-TLA" "(Upgrade Tools for ABAP)"
4 ETG040 Start time ....: "16.11.2021" "15:48:38"
4 ETG041 End time ......: "16.11.2021" "15:48:38

This section introduces you to possible errors that may occur during table classification of Software Update Manager (SUM).

1
A2EESZDM 197 AIM "TEST_AFTER_IMPORT" for object "L" "TSTS" is not enabled for ZDM

1234
A2EESZDM 287 "Transparent" table "CDHDR" has no BRITYPE
...
A2EESZDM 663 XCLA table "CDHDR" has invalid change of BRITYPE from '"S"' to '"C"' 
A2EESCRR_GENERAL_INF 102 Operation "CL_CLASSIFY_OP_ACTION" for table "CDHDR" failed

Specific phases of Zero Downtime Option (ZDO) require exclusive access to database tables by Software Update Manager (SUM). This may lead to errors in SUM tool, if the lock cannot be acquired due to parallel productive operations.

When performing zero-downtime upgrades, all structural and functional changes to database objects will be done in uptime. Here is an excerpt of such changes:

• Table field changes
• Index-related changes
• Table content changes
• And so on

As described in the lesson 'Concept of Cloned and Read-only Tables' of unit 2, all changes performed during the upgrade have to be hidden for the end users working productively on the bridge subsystem based on the source release. Dependent on the type of change, a table can be kept as shared or the table has to be cloned. For both, shared and cloned tables, the ZDO procedure has to execute certain database operations on tables that are accessed by the end users on the bridge subsystem as well as in parallel by Software Update Manager.

With SAP HANA 2.0, database operations executed on shared tables are preferably performed online, which does not require an exclusive lock. These activities include:

• Creation of database triggers required by ZDO that prevent the upgrade subsystem changing content in shared database tables
• Adding new non-primary key fields without default values
• Adding new indexes on non-primary key fields
• And so on

For all tables that will be cloned by the ZDO procedure, a rename table statement has to be executed. This so-called smart-switch happens in phase EU_SWITCH_ZDM directly after the rollover to the bridge subsystem (phase REQ_USER_ROLLOVER_FINAL). This example shows the technical upgrade phases (excerpt of log file SAPupConsole.log):

12345678910111213141516171819
...
MAIN_BRISETUP/REQ_USER_ROLLOVER_PREP (2021/09/15 04:15:17)
...
MAIN_BRITRANS/BRIDGE_RECONNECT (2021/09/15 04:23:31)
...
MAIN_BRITRANS/REQ_USER_ROLLOVER_FINAL (2021/09/15 04:26:31)
...
>> START OF PHASE MAIN_SWITCH/EU_SWITCH_ZDM (2021/09/15 04:33:50)
  Determine list of tables to be switched. ...
  0:14: Checking which tables exist on the database...
  0:14: Checking which indexes exist on the database...
  0:15: Renaming 29054 indexes to old names...
  6:01: Switching 1097 indexes...
  7:12: Performing smart switch for 27954 tables...
  1:10:09: Switching 946 tables to new ones...
  ...
  1:10:27: Finished!
<< END OF PHASE MAIN_SWITCH/EU_SWITCH_ZDM (2021/09/15 05:44:17)
...

This pseudo code describes the steps performed by phase EU_SWITCH_ZDM:

• Acquire an exclusive table lock for table tab1
• Rename table tab1to tab1temp
• Release the exclusive table lock for table tab1

If business processes on the bridge subsystem are holding database lock for a long time, SUM will not be able to acquire an exclusive table lock in phase EU_SWITCH_ZDM. The ultimate goal is to not impact business users. Hence, SUM will fail with an error in phase EU_SWITCH_ZDM if the exclusive lock cannot be acquired. The business process will not be interrupted.

Exemplary error message in log file SAPupConsole.log.

12345678910111213141516171819202122232425
<< 2021/09/14 19:57:01 END OF PHASE MAIN_SWITCH/EU_SWITCH_ZDM
=========== Repeat Phase ===========
Severe error(s) occurred in phase MAIN_SWITCH/EU_SWITCH_ZDM!
Last error code set:
1 errors during parallel execution of processes, check 'SQLSTMTSTD.*' for details
*ERROR* :
The following errors were detected in the log files:
# /usr/sap/S4H/SUM/abap/log/SMART_SWITCH03.LOG:
4 ETQ393 START Statement with ID 'ZDM3185' already executed - skipped.
4 ETQ393 START Statement with ID 'ZDM3185' already executed - skipped.
4 ETQ393 START Statement with ID 'ZDM3185' already executed - skipped.
1 ETQ000 ==================================================
4 ETQ010 Date & Time: 20210914195447 1EETQ008 Error message: DBSL error 99 (db code 131):
1EETQ009Xtransaction rolled back by lock wait timeout: Lock-wait time out
1EETQ009Xexceeded [OWNER=32392647202, TYPE=OBJECT_LOCK, CURRENT_MODE=EXCLUSIVE,
1EETQ009XREQUESTED_MODE=EXCLUSIVE]

A trouble ticket and an archive with all relevant log files have been generated.
Trouble ticket: "/usr/sap/S4H/SUM/abap/log/SAPup_troubleticket.log"
Log archive: "/usr/sap/S4H/SUM/abap/log/SAPup_troubleticket.sar"

01) * Repeat phase MAIN_SWITCH/EU_SWITCH_ZDM to continue at the point it stopped
02) - Exit program : Repeat phase MAIN_SWITCH/EU_SWITCH_ZDM to continue at the point it stopped
================================================================================

In most cases, the error in phase EU_SWITCH_ZDM can be overcome by repeating the phase once or several times. As soon as SUM is able to acquire the exclusive table lock, the rename statement will be executed and the phase finishes.

If SUM still fails even after several retries to repeat phase EU_SWITCH_ZDM, the exclusive lock holder (work process ID) has to be identified. This can be done for instance using SAP GUI transactions DB01 or DBACOCKPIT. Then, decide together with the process owner whether to wait until the process finishes or if the process has to be canceled in order to proceed with the SUM procedure.

DB Table Lock Analyzer of SUM Toolbox

As part of the Software Update Manager (SUM) Toolbox a tool called DB Table Lock Analyzer is delivered which helps to find a suitable time to perform database lock intense phases like EU_SWITCH_ZDM.

Hint

The full documentation of the tool can be found in the application help of the transaction itself. Just choose the information icon next to the OK code field.

After launching the transaction SUMTOOLBOX, select the tool DB Table Lock Analyzer in the navigation tree. The tool is split into three areas:

1. Job schedule: The data has to be collected over a greater period of time in the productive system. The collection is done using an ABAP report that has to be scheduled as per the description in the embedded application help.
2. DB table lock: Every job run creates a new line item which is rated either in green, yellow, or red. If a line turns into yellow or red, it points to a critical situation like tables, which are very frequently locked in the system. More details can be seen by double-clicking on a line item.
3. Table lock details: By double-clicking on a row, the details show the table name and the likeliness of acquiring an exclusive table lock in the monitored interval.

The result of the DB table lock section should be checked carefully. The rollover to bridge, which is followed by the smart-switch phase, should be planned around the timings where the system is less busy.

However, not all yellow and red marked line items are potentially causing an issue. To further analyze this, the SUM table classification data from the sandbox system can be utilized.

Note

The file PUTTB_SHD.ZIP that contains the SUM table classification data can be downloaded in the sandbox system after the phase RUN_RSPTBFIL_ZDM_CLASSIFY is completed. The download is performed using the tool Export of SUM classification data in SUM Toolbox.

Choose the Upload button and select the file exported (default file name would be PUTTB_SHD.ZIP) from the sandbox system. The result table is automatically refreshed. In this particular example, after merging the information of the DB table lock analyzer and the SUM table classification result, all time frames are marked as green.

The reason for this is simple: only tables classified as clone will be handled by the smart-switch phase EU_SWITCH_ZDM. All other tables which remain shared, are not affected by the smart-switch procedure. If the tables that are listed in the table lock details section are classified as shared, the will be considered as uncritical and the line turn from a yellow or red state into green.

This section offers steps to troubleshoot potential challenges during transition to bridge subsystem.

After starting the rollover to the bridge subsystem in phase REQ_USER_ROLLOVER_PREP, all users have to be transferred to the bridge database schema. In case of distributed installations having multiple application servers, the rollover failed in phase BRIDGE_RECONNECT. An example of this is as follows:

The following errors were detected in the log files:

Code Snippet

Copy code

Copy code

Switch to dark mode

Switch to dark mode

12345678910111213
# /usr/sap/PPM/SUM/abap/log/RLFW_RFC_RECONNECT.PPM:
A4 ESUPG 302 Log name: "/usr/sap/PPM/SUM/abap/log/RLFW_RFC_RECONNECT.PPM"
A4 ESUPG 304 Start time.....: "11.11.2021" "08:03:05"
A4 ESUPG 002 " "
A4 ESUPG 001 -------------------------------------------------------------------------
A4 ESZDM 232 Starting RLFW_GET_RECONNECT_STATE to get the reconnect state.
A2EESZDM 233 Reconnect state could not be determined. Error: "An exception was raised.".

A4 ESZDM 513 Schema check action "BRIDGE_CHECK": call direct from SUM. Logfile "RLFW_RFC_RECONNECT.PPM"
A4 ESZDM 521 Schema check Start, action:"BRIDGE_CHECK" schema:"SAPHANADBSHD" version:"2"
A4 ESZDM 503 Bridge Read V1 schema: "SAPHANADB" of "20211111" "080220" on " host_primary_PPM_00"
A4 ESZDM 504 Bridge schema is: "SAPHANADBSHD" at "20211111" "080315" on " host_primary_PPM_00"
A2EESZDM 509 Bridge schema check, RFC couldnt get schema of instance "host_dialoginstance_PPM_00" on "host_primary_PPM_00"

The error A2EESZDM 509 describes that the primary application server cannot connect using RFC to at least one of the additional application servers.

This section provides recommendations for potential errors during execution of Software Update Manager (SUM).

12345678910111213141516171819202122
4 EPU201XBEGIN OF SECTION BEING ANALYZED LASTOFFSET               103212 NOW BEGIN OF RUN_CHECK_ILLACCESS ======
A4 ESUPG 001 -------------------------------------------------------------------------
A4 ESUPG 002 " "
A4 ESUPG 301 Report name ...: "RSZDM_CHECK_ILLACCESS"
A4 ESUPG 302 Log name: "SUM/abap/tmp/CHECK_ILLACCESS.PPM"
A4 ESUPG 304 Start time.....: "15.01.2021" "16:17:41"
A4 ESUPG 002 " "
A4 ESUPG 001 -------------------------------------------------------------------------
A4 ESZDM 320 Report "RSZDM_CHECK_ILLACCESS" started
...
A4 ESZDM 330 Phase "XPRAS_AIMMRG" containing illegal table access started at "210113 17:38:08"
A2EESZDM 324 Illegal access on table "TESTTABLE1" at "210113 19:05:50"
A4 ESZDM 325 Phase "XPRAS_AIMMRG" containing illegal table access finished at "210113 20:28:00"
A4 ESZDM 321 Report "RSZDM_CHECK_ILLACCESS" successfully finished
A4 ESUPG 001 -------------------------------------------------------------------------
A4 ESUPG 002 " "
A4 ESUPG 301 Report name ...: "RSZDM_CHECK_ILLACCESS"
A4 ESUPG 304 Start time.....: "15.01.2021" "16:17:41"
A4 ESUPG 305 End time ......: "15.01.2021" "16:17:42"
A4 ESUPG 002 " "
A4 ESUPG 001 -------------------------------------------------------------------------
4 EPU202XEND OF SECTION BEING ANALYZED END OF RUN_CHECK_ILLACCESS ==========================================

This section provides an overview of tooling provided by SAP to stop or start the bridge subsystem.

After the rollover to the bridge subsystem, all application servers are connected to the bridge database schema (like SAPHANADBSHD). In general, the standard commands such as Start, Stop, StartSystem, or StopSystem of the sapcontrol framework can be used to handle the bridge subsystem - especially, if only a subset of application servers should be restarted, this way would be preferred.

Alternatively, the upgrade executable SAPup is also equipped to restart the application servers.