Challenge
PowerCenter Integration Service process running on a UNIX machine can be terminated for various reasons such as, exceeding CPU time or memory limits. Informatica recommends tracking the process core files and then analyzing them with the help of system administrators to troubleshoot the cause.
Description
Fatal run-time errors in UNIX programs usually result in the termination of the UNIX process by the operating system. Usually, when the operating system terminates a process, a "core dump" file is also created, which can be used to analyze the reason for the abnormal termination.
What is a Core File and What Causes it to be Created?
UNIX operating systems may terminate a process before its normal, expected exit for several reasons. These reasons are typically for bad behavior by the program, and include attempts to execute illegal or incorrect machine instructions, attempts to allocate memory outside the memory space allocated to the program, attempts to write to memory marked read-only by the operating system, and other similar incorrect low-level operations. Most of these bad behaviors are caused by errors in programming logic in the program.
UNIX may also terminate a process for some reasons that are not caused by programming errors. The main examples of this type of termination are when a process exceeds its CPU time limit, and when a process exceeds its memory limit.
When UNIX terminates a process in this way, it normally writes an image of the processes memory to disk in a single file. These files are called "core files", and are intended to be used by a programmer to help determine the cause of the failure. Depending on the UNIX version, the name of the file may be "core", or in more recent UNIX versions, "core.nnnn" where nnnn is the UNIX process ID of the process that was terminated.
Core files are not created for "normal" runtime errors such as incorrect file permissions, lack of disk space, inability to open a file or network connection, and other errors that a program is expected to detect and handle. However, under certain error conditions a program may not handle the error conditions correctly and may follow a path of execution that causes the OS to terminate it and cause a core dump.
Mixing incompatible versions of UNIX, vendor, and database libraries can often trigger behavior that causes unexpected core dumps. For example, using an ODBC driver library from one vendor and an ODBC driver manager from another vendor may result in a core dump if the libraries are not compatible. A similar situation can occur if a process is using libraries from different versions of a database client, such as a mixed installation of Oracle 8i and 9i. An installation like this should not exist, but if it does, core dumps are often the result.
Core File Locations and Size Limits
A core file is written to the current working directory of the process that was terminated. For PowerCenter, this is always the directory the services were started from. For other applications, this may not be true.
UNIX also implements a per user resource limit on the maximum size of core files. This is controlled by the ulimit command. If the limit is 0, then core files will not be created. If the limit is less than the total memory size of the process, a partial core file will be written.
Analyzing Core Files
Core files provide valuable insight into the state and condition the process was in just before it was terminated. It also contains the history or log of routines that the process went through before that fateful function call; this log is known as the stack trace. There is little information in a core file that is relevant to an end user; most of the contents of a core file are only relevant to a developer, or someone who understands the internals of the program that generated the core file. However, there are a few things that an end user can do with a core file in the way of initial analysis. The most important aspect of analyzing a core file is the task of extracting the stack trace out of the core dump. Debuggers are the tools that help retrieve this stack trace and other vital information out of the core. Informatica recommends using the pmstack utility.
The first step is to save the core file under a new name so that it is not overwritten by a later crash of the same application. Usually appending a timestamp to the core is best, but it can be renamed to anything:
mv core core.ddmmyyhhmi
The second step is to log in with the same UNIX user id that started up the process that crashed. This sets the debugger's environment to be same as that of the process at startup time.
The third step is to go to the directory where the program is installed. Run the "file" command on the core file. This returns the name of the process that created the core file.
file <fullpathtocorefile>/core.ddmmyyhhmi
Core files can be generated by the PowerCenter executables (i.e., pmserver, infaservices, and pmdtm) as well as from other UNIX commands executed by the Integration Service, typically from command tasks and pre- or post-session commands. If a PowerCenter process is terminated by the OS and a core is generated, the session or server log typically indicates Process terminating on Signal/Exception as its last entry. Informatica recommends working with the UNIX System Administrator to find out the location of the core files.
Using the pmstack Utility
Informatica provides a pmstack utility that can automatically analyze a core file. If the core file is from PowerCenter, it will generate a complete stack trace from the core file, that can be sent to Informatica Customer Support for further analysis. The track contains everything necessary to further diagnose the problem. Core files themselves are normally not useful on a system other than the one where they were generated.
The pmstack utility can be downloaded from the Informatica Support knowledge base (Article 13652). Once downloaded, run pmstack with the c option, followed by the name of the core file:
$ pmstack -c core.21896
================================= SSG pmstack ver 2.0 073004
================================= Core info :
-rw------- 1 pr_pc_d pr_pc_d 58806272 Mar 29 16:28 core.21896
core.21896: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from
''''''''pmdtm''''''''
Process name used for analyzing the core : pmdtm
Generating stack trace, please wait..
Pmstack completed successfully
Please send file core.21896.trace to Informatica Technical Support
Then examine the generated trace file or send it to Informatica Customer support.
Pmstack also supports a p option, which can be used to extract a stack trace from a running process. This is sometimes useful if the process appears to be hung to determine what the process is doing.