Tell us what you think

We want to create the best possible experience for our learners. Your feedback helps us to improve the experience for everyone.

Log and Trace Information for System Start and Stop

Objectives

After completing this lesson, you will be able to:

List log and trace information for troubleshooting
Use SAP MC and SAPControl to access log and trace information
Appendix: further analysis options

Statuses of Application Server ABAP

Before drilling down to log and trace files, it makes sense to have a closer look at the different statuses of an ABAP Application Server (AS) instance. With this information in mind, it should be easier to identify a potential problem and isolate the root cause. This knowledge is essential for administrative personnel and should support a target-oriented troubleshooting procedure. Treat it as background information for your practical work. Within this class there is no further detailed discussion about this topic, but the different statuses might be seen in multiple log and trace files when reviewing a start or stop procedure.

In case some instances of the SAP system are in status Active and others not, and you are able to log on to the SAP system, use transaction SM51 or SMMS to check the status of the Application Server instances.

Without being able to log on to the ABAP stack, log and trace files can be used to search for the specific status of an SAP Application Server instance (see section below for details). The typical statuses in the context of a start action of an AS ABAP are shutdown, stop, initial, started, and active and can be seen in the log files. A special status called "Passive" can be established to take out an instance temporarily from load balancing for maintenance reasons. When this status is pushed, existing requests are further processed until finished, but newly incoming requests are only distributed to the remaining AS ABAP instance(s). In this way the instance will once be isolated and can be maintained when all existing requests were finished. At the end of the maintenance, the instance is brought again to status active, for example using transaction SM51 or SMMS.

Log and Trace Files for the Start Process

Preliminary Note

In this lesson you should get an idea about:

Where to find log and trace information relevant for start and stop procedures of SAP systems
Which files are most relevant for the start procedure
Which tools are used to access this information

This is our main goal, and not troubleshooting. As a team member of the administrative department, responsible for the start and stop procedures of SAP systems, you should get a mental map on where to have a look at which information, and using which tools, so that in case of a problem you are able to start the analysis of what happened and (hopefully) how to solve the issue.

Troubleshooting is much more. It requires knowledge from different affected areas. Troubleshooting is a systematic approach to solve a problem with the goal to determine why something does not work as expected and explain how to resolve the problem. But the starting point of troubleshooting is always describing the problem. It helps to find where to start your search for the cause with questions like:

What are the symptoms?
When and where does it occur?
What are the conditions under which it occurs?
Can it be reproduced?

Where to find Start-relevant Information

The start process is an especially important phase that is logged by the operating system, the SAP system, and the underlying database. If the SAP system does not start-up successfully, you can find the relevant error messages in log and trace files written on operating system level.

Note

Not all mentioned log file types can be found in every work directory of instances (ASCS is different from PAS/AAS).

In the figure above, the most important location to find start-relevant information is propagated, the home directory DIR_HOME, in slang words also called "work directory". Every SAP instance holds an own "work directory" and stores there the appropriate log and trace files. If problems occur during the starting procedure, these logs and traces provide useful information, such as warnings, error messages, error codes, or even problem descriptions in text form.

In addition, a kind of "Best Practice" can already be seen in the figure above (looking at the clock on the left):

When it comes to a problem in the context of a starting activity, access the "work directory" of the affected Instance and sort the files according to date and time stamp in descending order to get the most up-to-date files on top. This can already be an indicator of the problem origin.

STDERR<number> Files

The log files STDERR<number> are created by the instance-specific sapstartsrvdaemon (on UNIX/Linux platforms) or service (on Windows platform) during the start of the instance. The instance processes started by the sapstartsrv write to the individual STDERR<number> files, depending on the sequence in which they are listed in the Instance profile. The content of these log files, therefore depends on the individual system setup.

Note

The sequential number as part of the STDERR<number> files refers to the parameter Start_Program_<2-digit_number> in the Instance profile. When the file is created, the leading zero is hidden.

Here are some typical examples for STDERR<number> files:

STDERR<number> - Information about the start process of the database system
STDERR<number> -Information about the start process of the message server
STDERR<number> - Information about the start process of the dispatcher

Hint

A STDERR<number> file does not necessarily include errors. From a severity perspective It might include also warnings or even "only" information.

Log Files of sapstart and sapstartsrv

The sapstartsrv.log is the log file of the sapstartsrv process itself. It contains information about the sapstartsrv activities.

There are some differences on the operating system platforms (as already discussed in another lesson).

Only as reminder: On UNIX/Linux platforms, the sapstartsrv triggers the sapstart process to start the instance. The sapstart process writes consequently to a own sapstart.log file.

Trace Files of further important Processes and Services

All instance processes (and services) write to their own trace files, such as the Message Server to the file dev_ms or the Internet Communication Manager to the filedev_icm.

In case there are general problems in the initialization phase of the work processes, trace files with the naming convention dev_w<number> will include details. These files are also highly relevant for troubleshooting live, so not only for the start procedure. The work processes are used by the logged-on users in ABAP. If a work process fails or creates errors, you can identify the work process by its number and evaluate the trace file to figure-out the reason for the issue.

Configuration of the Log and Trace Level

You can set the granularity of the logged and traced information using the profile parameter rdisp/TRACE. The possible values for this parameter are:

0: No trace
1: Error messages
2: Full trace, whereas the trace entries depend on the SAP program that is being traced
3: As with 2, but data blocks are also traced

The trace level can be set separately for individual work processes in the process overview (transaction SM50). Alternatively the trace level can be increased or decreased using for example the SAP MC. This should be typically done for troubleshooting purposes and only on a temporary basis (see note below).

Note

The higher the trace level, the larger the amount of logged information, and therefore the larger the size of the files. Specifically the increase of the trace level can cause further problems in the areas capacity and performance. So be careful and change the default trace level only explicitly and on a temporary basis for troubleshooting.

As a consequence: Use trace levels >1 only for short periods of time, for analyzing error situations.

Instance Availability

The available.log registers times during which the instance was not operational, for example when starting or stopping.

Problem Analysis Areas

The SAP system can fail to start correctly for a number of reasons. If, for example, there may have been problems starting the underlying database, the SAP system - more precisely the PAS and/or AAS instances - could not subsequently be started. As shown in the figure above, the problem analysis can be done as follows:

Check the error messages and warnings of the relevant operating system with the corresponding operating system tools.
Check the status of the relevant database system using the error log files.
Check the start logs in the SAP MC. To do so, select the affected instance, open the context menu, and choose List Developer Traces.
Check the log files, STDERR<number> created by the sapstartsrv.
Check the trace files of the individual instance processes and work processes:
- dev_ms: Developer trace for the Message Server
- dev_rd: Developer trace for the RFC Gateway Service
- dev_disp: Developer trace for the ABAP Dispatcher
- dev_w<number> (<number> denotes the number of the work process): Developer trace for the work processes

If you can still log on to the SAP system, check the system log of the SAP system using transaction SM21. The system log can be also displayed and analyzed with the SAP MC or SAPControl.

SAP MC: Access Log and Trace Files

The SAP MC provides access to important log and trace files via the context menu. This is a core functionality besides the visualization of the status information in the Navigation Pane.

Functions provided in the Navigation Pane at the left - based on <SAPSID> and SAP Instances

The following functions are provided for an SAP system and for any SAP instance in the context menu of the Navigation Pane:

Show Version Information: Lists important binaries, their version and time stamp.
Show Environment Information: Lists variables with the corresponding values.
Analyze Log Files: Merge of important log and trace files and provides an output with filter options like period to scan, severity of messages, instance name, file name, details and the text of the messages.

Additional Functions based on an explicit SAP instance: There are additional functions, which are only provided on the level of an SAP instance.
Show Profile Parameters: Show Profile Parameters: Displays a list of all profile parameters with their values like RSPFPAR in the ABAP SAP system
Show Start Profile: Displays the instance profile.
Show Trace File: Displays the sapstartsrv.log file.
List Developer Traces: Lists all log and trace files of the work directory with file name, size, and modification time. You can select one file at a time to display the content.

Functions provided in the Result Pane on the right - based on processes

The following functions are provided for processes of an SAP instance in the Process List. In the list of processes in the Result Pane use the context menu.

Process List: Show Developer Trace: Based on the process you flagged, the Show Developer Trace function opens the respective trace file (for example, of the ABAP dispatcher or the ICM process).

AS ABAP WP Table: Show Developer Trace: Based on the work process, which you flagged, the Show Developer Trace function opens the trace file of the work process.
Log Files: Select an Instance in the Navigation Pane, expand the node and select Log Files. The complete content of the work directory is shown in the Result Pane. Select a log or trace file and choose in the context menu the function Show Log File or double-click on the corresponding row to open the Developer Trace.

SAPControl: List Webmethods to Access Log and Trace Information

With SAPControl it is also possible to access log and trace information using specific Webmethods.

Every function provided by the SAP MC can also be executed with SAPControl on command line level. That is the reason, why the call of sapcontrol --help leads to such a comprehensive output.

Here are some examples for functions used to analyze start and stop issues:

Note

This list is not exhaustive. For further details please check the already referenced How to document for the SAPControl web service.

GetVersionInfo: Returns a list of version information for the most important files of the instance.
ABAPGetWPTable: List output of the ABAP work processes like in SM50 or in the SAP MC.
GetTraceFile: The content of the instance-specific Web service trace file sapstartsrv.log will be the output of this command.
ListDeveloperTraces: Output is a list of all instance trace files in DIR_HOME. Read a specific trace file with the next function ReadDeveloperTrace.
ReadDeveloperTrace: Returns the content of a given trace file with <filename> <filesize> (for example, with size=0 to read the entire file.
ListLogFiles: This command returns a list of all instance log files.
AnalyseLogFiles: This option scans all log files for a given time period and returns a merged list of all matching log file entries (see How to use the SAPControl Web Service Interface for details).

Using Snapshots

The Snapshot technology discussed here is based on the Web Service of sapstartsrv using SAPControl.

Hint

Snapshots are always instance-specific, because every instance uses an own sapstartsrv. If you want to create Snapshots as a historical reference or a kind of "auditing" for parts of or complete SAP systems, scripts can be used to coordinate the time or event-based creation of the Snapshots. This has to done by the customer administrative department.

Snapshots can be used in SAP environments to collect important information about the current situation of a server as an automatism or manually. Automatically initialized snapshots dump all essential information into the developer trace (for example, dev_disp of the ABAP dispatcher) and create an archive including further related trace files, in order to have a kind of "package" to analyze the situation after it occurred.

Note

Do not confuse this topic with Storage Snapshot Technology, which is often used in the context of backup technology.

Snapshots: Automatic Creation

The trigger to create snapshots automatically is given in the following scenarios (list is not complete):

A hard shutdown occurs
A deadlock situation occurs
A resource bottleneck was detected
A stop of request processing for more than 30 seconds occurs
A crash occurs in a work process, ICM, or the RFC Gateway Service

In case a snapshots is triggered in an ABAP based SAP system, the processes write a special snapshot section inside the developer traces, which is needed to identify important meta data within the traces, like "begin", "end", "<id>", and "<reason>". In between of the "begin" and "end" the respective process dumps what happened. in terms of

The ABAP dispatcher dumps the request, the session, and the process which executed the request.
The ABAP work process dumps the C-STACK, the ABAP stack, and details about critical sections.

Note

Further details and background information for snapshots in the SAP system can be retrieved from the following space in the Support Wiki: https://wiki.scn.sap.com/wiki/display/SI/Snapshots+in+the+SAP+System

But it is also possible to create snapshots manually in different ways:

In the SAP MC via the context menu of individual instances
Using SAPControlWebmethodCreateSnapshot
In the SAP system using transaction SNAPSHOTS

Hint

The feature to use snapshots is available since kernel release 7.40. The transaction is provided since SAP_BASIS release 7.40 SP7. In addition there are parameters to limit the creation of snapshots, both, from a perspective of content triggering the creation, and from a point of view of the amount of snapshots which can be created.

For details refer to the documentation of the following parameters:

service/max_snapshots
rdisp/snapshot

Note

For sure, the scan of snapshots via a transaction code, like SNAPSHOTS, does not help to troubleshoot in a non-starting scenario of an SAP system, if any PAS and AAS instance is not starting successful and the administrator is not able to log-on to the ABAP stack.

Nevertheless, the snapshots are stored at a default location on operating system level, for example on UNIX/Linux the file path is: /usr/sap/<SAPSID>/SYS/global/sapcontrol/snapshots

The SAP MC can also be used to handle snapshots.

Note

Within the SAP MC it is possible to load already created snapshots and make them graphically visible, including the associated log and trace files. Use File → Load Snapshot... and browse to the location where the archive file is available. The snapshot will be display in the Navigation Pane under an explicit node Snapshot.

Finally snapshots can also be handled using SAPControl on command line level. There are some Webmethods for this purpose.

CreateSnapshot: This command creates an instance snapshot and stores it in the system DIR_GLOBAL directory as a ZIP archive.
ReadSnapshot: To make use of this Webmethod, the local file name has to be specified. It then returns the binary ZIP archive content.
ListSnapshots: Checks for the available snapshots.
DeleteSnapshots: Deletes a given list of snapshots in the DIR_GLOBAL file system.

(Optional) Analyze and Download a Snapshot

Prerequisites

Business Example

As an SAP system administrator, you want to locate and analyze snapshots created manually or by the SAP system. In addition you want to download a snapshot to access the content on OS level.

Note

In this exercise, when an object name or value contains ##, replace ## by the number your trainer assigned to you.

Steps

Check which snapshots exist in your assigned SAP system using transaction SNAPSHOTS.
1. You are working as user TRAIN-## in your SAP system.
2. Start transaction SNAPSHOTS.
3. Have a look at column File Name and verify the host name assigned to you.
4. There might be snapshots created for the PAS and/or AAS instance as part of your SAP system. The naming convention of the snapshots is <hostname>_<instance_number>_<date>_<time>.
5. Scroll down and recognize the column Description, which displays the reason for the snapshot (for example, Reason: Hard Shutdown or Reason: Workprocess <no.> died.
Analyze a snapshot of your choice.
1. Choose one snapshot you are interested in and select Display (Ctrl+Shift+F6) or double-click this entry.
2. At the left, in the navigation panel, an expanded hierarchy is visible, showing some traces provided by the current snapshot, like Dispatcher-Trace, Gateway Trace, ICM Trace, and other traces.
3. Expand the Dispatcher-Trace to see how this trace file is structured. In this way you can easily access the individual topic area covered by the trace (for example, Last Error Information, Scheduler Info or Memory Infos.
4. Double-click on Scheduler Info in the navigation panel to see some details on the right.
5. Some configuration details about the work processes can be seen, for example, the amount of different work process types, which are started by the ABAP dispatcher.
6. Alternatively scroll through a complete trace in the result panel at the right.
7. When done, choose Back (F3) using the arrow pointing to the left in the application toolbar.
Download a snapshot of your choice to the WTS.
1. Still in transaction SNAPSHOTS make a selection in the first column to choose one specific snapshot of your choice.
2. Choose Download (Ctrl+Shift+F7).
3. In the dialog Browse for Files or Folders select the following file path Application (N:)Documents and select OK.
4. Return to the SAP Easy Access menu by choosing Back (F3). SAP Easy Access is the standard initial screen in SAP systems. The left side of the screen displays the available menu in a tree structure. You can display your company’s logo on the right side.
Locate and open the downloaded snapshot by using an archive tool.
1. On the WTS open a File Explorer by selecting the Windows start button and then typing File. In the resulting list select File Explorer.
2. In the navigation panel of the File Explorer select Documents.
3. Double-click the zip archive named <hostname>_<instance_number>_<date>_<time>.
4. In the column Name of the archive tool double-click first <SID> and then <hostname> <instance_number> to see some details.
5. All content of a snapshot is stored in an xml format.
6. In the archive tool double-click the following path ReadConfigFileusrsap<SID>SYSprofile.
7. Select file <SID_<instance_name><instance_number>_<hostname>.xml and from the context menu use View file (Alt+V).
8. The instance profile is shown in an xml format.
  Result
  Note
  When downloading snapshots, files are converted into xml as most analysis tools are using this format.
9. Close the file and the archive tool by using the X in the upper right corner.
  Result
  You are able to locate and analyze snapshots in the SAP system in order to review errors and shutdown actions.

Using the Support Log Assistant

The Support Log Assistant is a Self Service Tool provided as application by the SAP Business Technology Platform that allows users to scan their own support related files for known issues and recommendations.

In the background there is a kind of Q&A database against which the customer-specific log and trace files are checked.

Since autumn 2019, the tool has been generally released by SAP for customers and supports since June 2021 six different languages with machine translation technology.

Currently there is already a wide range of supported file formats. From a security perspective it is important to mention that the files are not transferred to SAP.

In case you are interested in this topic, feel free to use the corresponding wiki page under the following link. Here you will find also an FAQ area:

To get an idea about the look and feel of the tool, check the following figures.

Support Log Assistant Wiki: https://wiki.scn.sap.com/wiki/display/ATopics/Support+Log+Assistant
Support Log Assistant - Link: https://supportportal-pslogassistant-app.cfapps.eu10.hana.ondemand.com/
Using the Support Log Assistant to automate support-related file analysis: https://me.sap.com/notes/2838708
Release Notes: https://support.sap.com/content/dam/support/en_us/library/ssp/my-support/releasenotes/2019/release-notes-wave6-2019.pdf
File Types supported: https://supportportal-pslogassistant-app.cfapps.eu10.hana.ondemand.com/utilities/FileTypeList/index.html
Expert Level File Analysis using the Support Log Assistant: https://blogs.sap.com/2020/11/16/expert-level-file-analysis-using-the-support-log-assistant/
Support Log Assistant 2.0 – Self Service Tool for Analyzing issues: https://blogs.sap.com/2020/11/09/support-log-assistant-2.0-self-service-tool-for-analyzing-issues/

Continue to quiz