crash(1M)

crash -- examine system images

Synopsis

/usr/sbin/crash [-a analysisfile] [-b batchfile] [-d dumpfile] [-m moduledir] [-n namelist] [-w outputfile]

Description

The crash command is used to examine the system memory image of a running or a crashed system by formatting and printing control structures, tables, and other information.

The amount and type of information stored in a system memory dump can be altered using the kernel parameter SYSDUMP_SELECTIVE. Set this parameter to

0: to dump both kernel and user mapped memory.
1: to dump only kernel mapped memory. This is the default.
2: to dump only a portion of kernel mapped memory.

If you change the value of SYSDUMP_SELECTIVE you must run idbuild(1M) to rebuild the kernel, and reboot the system.

When the dump is saved, you are prompted whether to save it in compressed format -- compression reduces transmission time if you are forwarding the dump to a remote site for analysis.

If the Encryption Utilities package is installed on your system, you are also presented with a prompt asking if you want to encrypt the dump. If you choose to encrypt the dump, you must supply an encryption key [see crypt(1) ]. To decrypt the dump:

cat dumpfile | crypt key > newdumpfile

where dumpfile is the encrypted dump file, crypt key is the encryption key that was specified when the dump was encrypted, and newdumpfile is the resulting, unencrypted dump file.

If the PANICBOOT parameter in /etc/default/init is set to ``YES'', the system automatically saves the memory dump and reboots if a system panic occurs. In this case, you are not prompted to compress or encrypt the dump.

Options

crash takes the following options:

-a analysisfile

Use the -a option to have the crash utility perform an analysis of the dump, and save the analysis in plain text format to the directory specified by analysisfile. The default analysisfile is /tmp/analysis.

-b batchfile

Use the text file batchfile, which contains a list of crash commands to be executed on the dump.

-d dumpfile

Use dumpfile for the file containing the system memory image. The default dumpfile is /dev/swap. If you enter a value other than /dev/swap, it is assumed that the system has crashed.

If a system dump is directed to a dump device other than the default swap device, the dump will not be detected when the system boots up. To enable the system to detect a dump that is not in the swap area when the system boots, add the following entry to the file /etc/dumptab.

   /dev/dump   0       -

-n namelist

Use the text file namelist, which contains the symbol table information needed for symbolic access to the system memory image to be examined. The default namelist is /stand/unix. If a system image from another machine is to be examined, the corresponding text file must be copied from that machine. For active systems, crash uses system calls to the running kernel to get symbol table information. For that reason, only global symbols are available.

-m moduledir

Use the specified directory, moduledir to look for modules. When working on a dump, crash looks for the modules that were loaded at the time of the dump, and adds those symbols to its symbol table information. By default, crash tries to find the modules in the directories from which they were loaded. The -m option specifies a directory where crash should look for the modules instead. If you use the -m option, crash only looks in the specified directory. You can only use a single instance of the -m option on the command line, and if you attempt to use the -m option more than once, only the last one is valid; the others are ignored.

-w outputfile

The output from a crash session is directed to outputfile. The default outputfile is the standard output.

Input

When you execute the crash command, a session is initiated. If you enter the ``?'' character at the command prompt, crash provides a help menu of the available commands.

Input during a crash session is of the form:

function [argument...]

where function is one of the crash functions described in ``Usage'', and argument is a list of qualifying data that indicate which items of the system image are to be printed.

The default for process-related items is the current process for a running system, or the process that was running at the time of the crash for a crashed system. If the contents of a table are being dumped, the default is all active table entries.

The following function options are available to crash functions wherever they are semantically valid.

-e: Display every entry in a table.
-f: Display the full structure.
-n: Display privilege names symbolically. Default is to display in hexadecimal.
-p: Interpret all address arguments in the command line as physical addresses. If they are not physical addresses, results are inconsistent.
-s process: Specify a process slot other than the default.
-w file: Redirect the output of a function to file.

The functions mode, defproc, and redirect correspond to the function options -p, -s, and -w. The mode function may be used to set the address translation mode to physical or virtual for all subsequently entered functions; defproc sets the value of the process slot argument for subsequent functions; and redirect redirects all subsequent output.

Output

Output from crash functions may be piped to another program as follows:

function [argument . . .] ! shell_command

For example, mount ! grep rw writes all mount table entries with an rw flag to the standard output. The redirection option (-w) cannot be used with this feature.

An argument can be either a symbol or a numeric argument. Numeric arguments are assumed to be decimal unless preceded by 0x or 0b prefixes for hexadecimal or binary numbers. Any argument that is not either a symbol or a number will be reported as an error. Each function, when executed, determines whether the argument specified a valid slot or address.

Default bases on all arguments may be overridden. The C conventions for designating the bases of numbers are recognized.

Aliases for functions may be any uniquely identifiable initial substring of the function name. Traditional aliases of one letter, such as p for proc, remain valid.

Many functions accept different forms of entry for the same argument. Requests for table information will accept a table entry number or a range. A range of slot numbers may be specified in the form a-b where a and b are decimal numbers. An expression consists of two operands and an operator. An operand may be an address, a symbol, or a number; the operator may be +, -, *, /, &, or |. An operand that is a number should be preceded by a radix prefix if it is not a decimal number (0 for octal, 0x for hexadecimal, 0b for binary). The expression must be enclosed in parentheses. Other functions accept any of these argument forms that are meaningful.

Two abbreviated arguments to crash functions are used throughout. Both accept data entered in several forms. They may be expanded into the following:

table_entry = address | slot | range

start_addr = address | symbol | expression

Usage

Following are the available functions in crash:

? [-w file]: List available functions.
!command: Escape to the shell and execute command.
abuf [-w file] [-mode]: Print audit buffer data in mode format. mode is one of long (-l), short (-t), or byte (-b). The default mode for character and ASCII formats is byte; the default mode for decimal, hexadecimal, and octal formats is long. When mode is omitted, the previous value is used. At the start of a crash session, the mode is long.
addstruct [-w file] structure_name: adds a structure to the list shown by size or size -f, to the structures and their field names (its particularly useful for od). For example, addstruct stat sys/stat.h parses the declaration of struct stat in /usr/include/sys/stat.h, and creates a /usr/lib/crash/statoffs.so used thereafter (even when you reinvoke crash). If the header file turns out to need other header files included before it can be parsed, then you can add those dependent header files. If it does not parse correctly, you can edit /usr/lib/crash/addstruct to correct its parsing.
as [-e] [-f] [-w file] [proc...]: Print address space information on process segments.
base [-w file] number...: Print number in binary, octal, decimal, and hexadecimal. A number in a radix other than decimal should be preceded by a prefix that indicates its radix as follows: 0x, hexadecimal; 0, octal; and 0b, binary.
buffer [-w file] [-format] [-p] addr: Alias: b
Print the contents of a buffer in the designated format. The following format designations are recognized: -b, byte: -c, character; -d, decimal; -x, hexadecimal; -o, octal; and, -i, inode. If no format is given, the previous format is used. The default format at the beginning of a crash session is hexadecimal.
bufhdr [-f] [-w file] [[-p] addr...]: Alias: buf
Print system buffer headers. The -f option produces different output depending on whether the buffer is local or remote.
callout [-w file] CG: Print the callout table for a CPU group.
cg [-w file] [cgnum]: Set the current CPU group to cgnum. If cgnum is not specified then the current CPU group is displayed.
cglocal [-w file] [-e | eng]: Print the local information for the current CPU group, or for the CPU specified by eng. If -e is specified, information is printed for all CPU groups.
class [-w file] [table_entry...]: Print information about process scheduler classes.
deflwp [-w file] [-c]
deflwp [-w file] [slot]: Set the value of the lwp slot for the current process. If slot is not specified, the current slot is displayed. The default lwp slot for each process is 0. If -c is specified, a reset is performed which gets context for the current slot number.
defproc [-w file] [-c]
defproc [-w file] [slot]: Set the value of the default process slot argument. The default process slot argument may be set to the current slot number (-c) or the slot number may be specified. If no argument is entered, the value of the previously set slot number is printed. At the start of a crash session, the process slot is set to the current process.
dis [-w file] [-a] start_addr [count]
dis [-w file] [-a] [-c | [-p | -sproc] address] [count]: Disassemble count instructions starting at start_addr. The default count is 1. The absolute option (-a) specifies a non-symbolic disassembly. The -c option can be used in place of start_addr to continue disassembly at the address at which a previous disassembly ended.
dispq [-l|-g] [-w file] [table_entry...]: Print the dispatcher (scheduler) queues. The -l option prints only the local dispatcher queues. The -g option prints only the global dispatcher queue.
ds [-w file] virtual_address...: Print the data symbol whose address is closest to, but not greater than, the address entered.
eng [-w file] [eng_num] [-b]: Set the current engine to eng_num. If no eng_num is specified, then the current engine number is displayed. The -b option acts as a toggle to bind or unbind an engine.
file [-e] [-w file] [[-p] table_entry...]: Alias: f
Print the file table.
filepriv [-n] [-w file] [-p table_entry...]: Alias: fprv
Print the kernel privilege table.
fs [-w file] [[-p] table_entry...]: Print the filesystem information table.
gdt [-e] [-w file] [slot [count]] table_entry...]: Print the global descriptor table.
help [-w file] [function... | -e]: Print a description of function, including syntax and aliases. Use -e to print a help list for all functions.
hexmode [-w file] [ON | OFF]: If hexmode is ON, commands expect hexadecimal input and display hexadecimal output. If hexmode is OFF, input and output are decimal.
idt [-e] [-w file ] [slot [count]]: Print the interrupt descriptor table.
inode [-e] [-f] [-w file] [[-p] table_entry...] [-r] [-l]: Alias: i
Print the inode table, including filesystem switch information. Use -r to print free inodes.
kmahist [-w file] [-a] [-c] [-2] [-f] [-l] [-n] [-s]: Prints kernel memory allocator history. Options include: -a for address, -c -2 -f for file, -l for lwp, -n and -s for size.
lck [-w file] [filockaddr...]: Print record locking information. If file lock address arguments are given, the record lock list is printed. If no argument is entered, information on locks relative to inodes is printed.
ldt [-e] [-w file] [process [slot [count]]]: Print the local descriptor table for the specified process. The table is printed for the current process if process is not specified.
lidcache [-w file]: Print out the level identifier (LID) translation cache. The LID cache is supported only if the Enhanced Security Utilities are installed and running.
linkblk [-e] [-w file] [[-p] table_entry...]: Print the linkblk table.
map [-w file] mapname...: Print the map structure of the given mapname.
mount [-e] [-w file] [[-p] table_entry...]: Alias: m, vfs
Print information about mounted filesystems.
nm [-w file] symbol...: Alias: symval, ts
Print value and type for the given symbol.
od [-p] [-w file] [-format] [-mode] [-s process] start_addr [count]: Alias: rd
Print count values starting at start_addr in one of the following formats: character (-c), decimal (-d), hexadecimal (-x), octal (-o), ASCII (-a), or hexadecimal/character (-h), and one of the following modes: long (-l), short (-t), or byte (-b). The default mode for character and ASCII formats is byte; the default mode for decimal, hexadecimal, and octal formats is long. The format -h prints both hexadecimal and character representations of the addresses dumped; no mode needs to be specified. When format or mode is omitted, the previous value is used. At the start of a crash session, the format is hexadecimal and the mode is long. If no count is entered, 1 is assumed.
page [-e] [-w file] [[-p] table_entry...]: Print information about pages.
panic: Print the latest system notices, warnings, and panic messages from the limited circular buffer kept in memory.
pcb [-e] [-w file] [process] [[-k | [[-u] [proc | #pio | idle] [lwp | eng]] [-i type=1]: Print the process control block (TSS). If no arguments are given, the active TSS for the current process is printed. If arguments are given, only one of -k, -u, or -i can be specified.
plocal [-w file] [-e | eng]: Print the processor local data for the current engine. Use the -e option to print data for all engines, or use eng to specify the engine number.
proc [-e] [-f [-n]] [-w file] [[-p] table_entry...#procid...]
proc [-f [-n]] [-w file]: Alias: p
Print the process table. Process table information may be specified in two ways. First, any mixture of table entries and process IDs may be entered. Each process ID must be preceded by a ``#''. The full option (-f) details most of the information in the process table as well as the region for that process.
ptbl [-e] [-w file] [-sprocess] [[-p] addr [count]]: Print information on page descriptor tables.
pty [-f] [-e] [-w file] [-s] [-h] [-l] [-t type]: Print the pseudo ttys presently configured. The -l, -h and -h options give information about the STREAMS modules ldterm, ptem and pckt, respectively.
qrun [-w file]: Print the list of scheduled STREAMS queues.
queue [-e] [-s style] [-w file] [[-p] table_entry...]
queue [-f] [-s style] [-w file] [[-p] table_entry...]: Print STREAMS queues. Up to three -s arguments can be specified: the first for the write side, the second for the read side, and a third for read/write. However, only one -s can be specified if other arguments are to be used. The -s option displays the symbolic STREAMS configuration.
quit: Alias: q
Terminate the crash session.
redirect [-w file] [-c]
redirect [-w file] [newfile]: Used with a file name, redirects output of a crash session to newfile. If no argument is given, the file name to which output is being redirected is printed. Alternatively, the close option (-c) closes the previously set file and redirects output to the standard output.
resmgr [-f [-w file]: Print the resource manager database.
rtdptbl [-w file] [table_entry...]: Print the real-time scheduler parameter table. See fp_dptbl(4).
rtproc [-w file]: Print information about processes in the real-time scheduler class.
search [-p] [-w file] [-m mask] [-s process] pattern start_addr length: Print the long words in memory that match pattern, beginning at the start_addr for length long words. The mask is ANDed (&) with each memory word and the result compared against the pattern. The mask defaults to 0xffffffff.
sinode [-e] [-f] [-w file] [-r] [-l] [[-p] table_entry...]: Alias: si
Print the inode table for ufs or sfs filesystems. Since the ufs/sfs incore inode contains the icommon inode and the alternate inode, this function displays, in addition to the icommon inode information, all security data stored in the alternate inode. This includes the level identifier, the Access Control List (ACL) count, the extended ACL disk block pointer, and any inode resident ACL entries.
size [-f] [-w file] [structure_name...]: Print the size of the designated structure. The (-x) option prints the size in hexadecimal. If no argument is given, a list of the structure names for which sizes are available is printed.
snode [-f] [-w file] [[-p] table_entry...]: Print information about open special files. Along with other information, it prints the security attributes of a device: mode, stat, high level range, low level range, release flag, and other security flags; these attributes are supported only if the Enhanced Security Utilities are installed and running.
stack [-e]
stack [-t stackptr] [-w file] [process]: Alias: s
Dump the stack. If no arguments are entered, the kernel stack for the current process is printed. If -t is used, it expects a hexadecimal value for the stack pointer. The interrupt stack and the stack for the current process are not available on a running system.
stat [-w file]: Print system statistics.
stream [-f] [-w file] [[-p] table_entry...]: Print the STREAMS table.
strstat [-w file]: Print STREAMS statistics.
test [-w file] [debuglevel]: Displays or changes the debug level.
trace [-w file] [-e] [-t] [process]: Alias: t
Print stack trace.
tsdptbl [-w file] [table_entry...]: Print the time-sharing scheduler parameter table. See ts_dptbl(4).
tslwp [-w file]: Alias: tsproc
Print information about light-weight processes (LWPs) in the time-sharing scheduler class.
tty [-e] [-f] [-l] [-w file] [-t type [[-p] table_entry...]|[-p] start addr]: Valid types: console, sr, sx, sc.
Print the tty table. If no arguments are given, the tty table for both tty types is printed. If the -t option is used, the table for the single tty type specified is printed. If no argument follows the type option, all entries in the table are printed. A single tty entry may be specified using start_addr. The -l option prints the line discipline information.
user [-e] [-w file] [process]: Alias: u
Print the ublock for the designated process.
var [-w file]: Alias: v
Print the tunable system parameters.
vfs [-e] [-w file] [[-p] table_entry...]: Alias: mount, m
Print information about mounted filesystems.
vfssw [-w file] [[-p] table_entry...]: Alias: fs
Print information about configured filesystem types.
vnode [-w file] [[-p] vnode_addr...]: Print information about vnodes.
vtop [-w file] [-s process] start_addr...: Print the physical address translation of the virtual address start_addr.
vxinode [-w file] [-e] [-f] [-l] [-r] [vx_inodeaddr...]: Alias: vi, vinode
Print the VxFS filesystem (vxfs) inode table; valid only for vxfs type filesystems. -l prints the contents of all the icache lists (these hold the vx_inodes) together with information for each inode indicating if it is a free or attribute (or both) inode. -r prints all the free vx_inodes. vx_inodeaddr prints the vx_inode located at address vx_inodeaddr.

Extensible functions

It is possible to add functions to crash without having access to the source code. This is useful for writing platform specific functions or to obtain more information than is available in the default command set.

Adding new functions is accomplished with shared objects. The shared object is created from a file consisting of the functions that are to be added to the existing command set. When crash is started, these shared objects are read and the additional functions are incorporated in the command set.

To create a shared object:

Create a file that contains the new commands. This file should define the variable functab, which should be an array type struct func. This array points to the new commands that need to be added.
Compile the file with the same options as the kernel build. For example, if crash is going to be run on top of a uniprocessor kernel, then the file should be compiled with the appropriate definitions for a uniprocessor kernel.
Use the -G option to the compiler to create the shared object.

crash reads the shared objects from directory /usr/lib/crash and also from files defined by environment variable CRASH_LIBS.

Extension example

An example of a file containing a new command (lbolt) is given below:

   #include <stdio.h>
   #include <sys/types.h>
   #include <syms.h>
   
   extern int get_lbolt();
   

   /* function definition */
   struct func {
           char *name;
           char *syntax;
           int (*call)();
           char *description;
   };
   

   struct func functab[] = {
           "lbolt", " ", get_lbolt, "lbolt",
           0,0,0,0
   };
   

   static struct syment *lbolt_sym = NULL;
   extern struct syment *symsrch(char *);
   extern FILE *fp;
   

   get_lbolt(void)
   {
           time_t lbolt;
   

           if (lbolt_sym == NULL && (lbolt_sym = symsrch("lbolt")) == NULL)
                   return 0;
           readmem((void *)(lbolt_sym->n_value), 1, -1, &lbolt, sizeof(lbolt),
                    "lbolt");
           fprintf(fp, "%lx\n",lbolt);
   }

Notices

This command has been updated to handle Intel Pentium III Streaming SIMD instructions; see ``Pentium III extended floating point support'' for more information.