Tell us what you think

We want to create the best possible experience for our learners. Your feedback helps us to improve the experience for everyone.

Monitoring the System Start and Stop Procedure

Objectives

After completing this lesson, you will be able to:

Find the storage locations of trace and log files
Name important Trace and Log Files for Troubleshooting

Log and Trace Files

In the case of an error or unexpected behavior of the Java Startup and Control Framework, it is important to check the work directory of the AS Java instances (PAS/AAS) under the following path: /usr/sap/<SID>/J<instance no.>/work.

As a kind of Best Practice, sort the files descending by date and time stamp and check the content of the most up-to-date log and trace files for errors and warnings. The same strategy should be taken into account also for problems regarding the Central Services Instance (SCS). Always remember, there are lots of potential problems and error situations. So, there is a wide range of root causes. Log and trace files will help the administrator analyzing the situation by pointing to a hardware (lack or defective) or software (missing or incorrect version) based problem, as well as identifying erroneous parametrization. Some of the most important log and trace files are listed below.

Important Log and Trace Files:

dev_jstart
dev_<node name> (e.g. dev_server0, dev_icm)
jvm_<node name>.out (e.g. jvm_bootstrap.out)
std_server<no.>.out (e.g. std_server1.out)
log_bootstrap<no.>.log
stderr<no.> (e.g. stderr1)
sapstart.log, sapstart<no.>.trc, sapstartsrv.log
available.log

Note

Also take care about the SAP Central Services (SCS) instance. Use the following path to access the work directory: /usr/sap/<SID>/SCS<instance_number/work

The trace and log files are stored in the work directory of an instance. This directory is called /usr/sap/<SID>/J<instance_number>/work (UNIX) and analogously in the Microsoft Windows environment.

dev_jstart is the trace file for the Instance Controller. It is the most important trace file for problem messages when starting AS Java. Current messages are written at the end of the file.

dev_<node name> is the trace file for the Server Processes, the ICM and further processes. The trace file dev_<node name> is written for each process started and thus for each ICM and server process.

jvm_<node name>.out is the output file for the Java Virtual Machine (JVM). Each node (except ICM) represents a Java node such as a bootstrap or a server process and therefore a JVM. The output of a JVM is forwarded to the file jvm_<node name>.out in the work directory of a Java instance.

std_server<X>.out is the default output file for the started managers and services of the corresponding nodes, one of the most common used files for troubleshooting the framework startup of a Server Process.

log_bootstrap<number>.log is the log file for the synchronization of binary data in the Java database with the local file system, reading the Java instance description from the Java database and updating the instance.properties file.

stderr<number> are created by the sapstartsrv process during the start of the instance. The instance processes that are started by the sapstartsrv write to the individual STDERR<n> files, depending on the sequence in which they are listed in the instance profile.

available.log displays the instance availability.

For most of the log files listed above, you will also find log files in the work directory with the ending .<no.>, with older versions of the files, which can also often be used for troubleshooting. In the start process of an instance, the files mentioned above are rewritten and the file names are changed from x.<no.> to x.<no.+1>. The number of the old versions complies with the parameter jstartup/keep_old_logfiles (default = 2).

Beside the node names for ICM and server processes there are also developer traces for data collections (datcol). In case of start problems, the data collector provides information about the problem and writes to the developer traces.

This is only available in english

The developer traces from the work directory can also be evaluated easily with the SAP MC (or SAP MMC). See the above figure. The analysis displays the error messages from the dev_<...> files (as well as other messages from the log and trace files that you will get to know in the Monitoring unit). Warnings or other messages from the dev_<...> files may also be displayed.

Optional: Logs of the Start and Stop Processes of an AS Java-based SAP System

Business Example

The start process of an SAP system is a critical action. If problems occur, the administrator must be familiar with the most important log and trace files written during the start up procedure. The error analysis is highly dependent on the expert knowledge of the administrative personnel. To identify the root cause of the issue, one of the first steps is to narrow down the area responsible for the problem (OS, database, JVM, Parameter etc.). If the problem could not be solved until this point in time, a further investigation step is necessary and you might filter error codes found in log and trace files to get details for the failure situation. Those text extracts, messages, return codes (etc.) can be used as search terms, making use of different sources provided by SAP (e.g. Notes search via the Support Launchpad or Wiki pages).

Valid for this Exercise

Parameter	Value
SAP Classroom	WTS
SAP System ID	SMJ
Host name (FQDN)	smhost.wdf.sap.corp
Operating System	Windows
Operating system user / password for Q-Group	smaadm / <password>
Operating system user / password for P-Group	smjadm / <password>
Central Services Instance	SCS90
Primary Application Server (PAS)	J91
SAP Management Console	SAP MC

Note

In this Exercise we are not expecting lots of errors. The goal of this practical work is, to be familiar with the potential locations and file names on OS level and know how the analysis should typically be done (e.g. sorting by time stamp).

Note

It is not mandatory to log on at OS level for troubleshooting purposes. Not every administrator has the authorization and gets the credentials of an OS admin user. The SAP MC is a very good starting point for the analysis, if the minimum prerequisites are given (e.g. sapstartsrv services are running).

Task 1: Optional: Log and Trace files of the Start and Stop Procedure

Check the most recent log and trace files which are written when starting a Java-based SAP System. Get an overview of the content of the work directory and check some important log and trace files.

Note

There are some differences on UNIX/Linux in comparison to Windows OS and some classes might run on another OS platform. In this class the Java-based SAP System is running on Windows platform.

Steps

Windows only: If not already done (see former Exercises) log on to the server smhost.wdf.sap.corp of your training system with Remote Desktop Connection using the smjadm operating system user with password <password>.
Hint: If you are still connected via RDP to the OS level (see former exercise) continue with the next relevant step.
1. On your WTS choose the Windows Start button in the lower left corner.
2. In the tiles area scroll down and select tile Remote Desktop...
3. In dialog Remote Desktop Connection enter the following value into the field Computer: smhost.wdf.sap.corp.
4. In the next dialog Windows Security - Enter your credentials provide the following user and password for the authentication.
  - Q- Group: smaadm / <password>
  - P- Group: smjadm / <password>
5. Confirm the next security related dialog Remote Desktop Connection with the text The identity of the remote computer cannot be verified. Do you want to connect anyway? with Yes.
Windows only: List the most recently written log and trace files of the Primary Application Server (PAS) Instance.
1. Prerequisite: you are connected to Windows OS level using Remote Desktop Connection(see former Step). If it is still open, you can find the application in the Windows taskbar of the WTS. If you select it, RDP shows a deep blue bar on top with the name of the Windows server, like smhost or any other Windows host name used in the training class.
2. Double-click the yellow folder icon in the Windows taskbar of the Remote Desktop Connection.
3. On the left in the navigation panel of the File Explorer choose drive (D:) and navigate in the content area on the right hand side to the following directory by double-clicking the folders:usr → sap → <SID> → (e.g. SMJ) → J<instance name><instance no.> → (e.g. J91) → work (e.g. D:\usr\sap\SMJ\J91\work).
4. In the content area on the right hand side of the Window click on the columnDate Modified → to sort the elements descending.
  Hint
  To set the File Explorer to full screen, use the Maximize icon on the top right corner.
  Note
  In a troubleshooting scenario typically the most actual information written to files is relevant for the analysis. Therefore the work directory is sorted descending according to the date/time and the list gives a first idea about which services and processes have most recently written to this directory.
5. Open some files in the File Explorer which is showing the work directory in descending order in the following way: double-click some files (for examples see list below). By opening files with this action a pop-up window How do you want to open this file? will ask to select a tool. Choose Notepad or WordPad and select OK.
  Hint
  To see the most up-to-date information within the files opened with Notepad or WordPad, choose the key combination CTRL+END on your keyboard or scroll down to the bottom of the file content.
  - dev_<node name> (e.g. dev_server0)
  - std_<node name><no.>.out (e.g. std_server1.out)
  - jvm_<node name>.out (e.g. jvm_server0.out)
  - sapstart.log or dev_sapstart
  Hint: The term node corresponds to a process in AS Java
Windows only:
Have a look at very important log and trace files of the Central Services Instance.
- Prerequisites for this step:
- 1. Connection to Windows OS level (WTS) using Remote Desktop Connection (see former Step).
- 2. Open File Explorer in the work directory of an Application Server Instance (PAS or AAS), using the path D:\usr\sap\<SID>\<instance name><instance no.>\work(see former Step).
- dev_ms (Message Server log file)
- dev_enqlisten (Enqueue listener log file)
- enquelog (Enqueue operations and conflict detection)
- dev_rd (Gateway Service log file )
- stderr<no.> (Start of individual Services - in case of written status like info/warning/error)
Hint: This list is not a exhaustive!
1. In the Remote Desktop Connection use the File Explorer to display the content of a PAS/AAS work directory (D:\usr\sap\<SID>\<instance name><instance no.>\work.
2. Open the relevant files (see list above) for example with Notepad (see former Step on how to proceed) and scroll to the end. Check the entries since the last start of the system.

Continue to quiz