Project

General

Profile

Wiki » Documentation » Installation Configuration and Administration Guide »

Gathering Diagnostic Data

Logs

Introduction

TBD: Describe the logging that is available, differentiate between the legacy logging (LOG-MANAGER) and the logging from FWD itself (server or client), link to the various other pages related to logging

LOG-MANAGER

TBD: summary: the logfile location and name will be determined by the 4GL code (setting the LOG-MANAGER:LOGFILE-NAME attribute) but FWD has a feature in which the logfile can be on the server or on the client (by default because 4GL filesystem references are on the client by default); link to docs about the server vs client usage

Server

TBD: how to find the logs, link to the docs for how to configure the logfile location and name

Client

Started Explicitly by User or Script

This includes interactive clients using ChUI (native console or Swing) or GUI (Swing). It also includes direct launching of batch processes.

TBD: how to find the logs, link to the docs for how to configure the logfile location and name

Started Implicitly by the Server or Spawner

This includes interactive clients using ChUI (native console or Swing) or GUI (Swing). It also includes direct launching of batch processes.

TBD: how to find the logs, link to the docs for how to configure the logfile location and name

Database Tools

pgBadger (PostgreSQL 9.x)

This is a free, open source, PostgreSQL log analyzer which parses the log file and provides some interesting analytics. This can help you track down database performance bottlenecks typically caused either by slow queries or by executing the same query many times (i.e., lots of database round trips). It will also highlight errors and warnings that you might otherwise miss in voluminous log files.

The pgBadger home page is at http://dalibo.github.io/pgbadger/. Technology prerequisites are a modern Perl installation and a modern browser (the presentation of results is rendered using JavaScript).

To use pgBadger to postprocess your database log files, you will need to (1) enable logging; and (2) modify your logging options in postgresql.conf, so that PostgreSQL logs information in a format pgBadger can understand. The specific options to change depend upon the way your particular PostgreSQL cluster is installed. The details are documented in the Documentation section of the pgBadger website. One thing to keep in mind is that you generally want to use very verbose logging to generate log files for troubleshooting, but you probably will not want the same level of verbosity for production use, since the logging itself can impact production performance and disk use. Thus, you may want to maintain two sets of postgresql.conf files; one for troubleshooting and one for production.

The high level troubleshooting process with pgBadger is to:
  1. enable pgBadger-friendly database logging and restart the database cluster;
  2. start up the FWD application server;
  3. perform the test/routine which you want to analyze;
  4. use the pgBadger command line script to process the PostgreSQL log file(s) (you can use optional parameters to scope the analysis to a certain time window);
  5. review the results in a browser.

Don't forget to reset the logging options and bounce your cluster again after your troubleshooting session, if you are using different logging options normally.

Thread Dumps

jstack

Run ps -aef and look for the java processes. In this example, 2 app servers startup with the server (actAS1 and mobAS1). These will be the 3 at the start of a server, with no users connected. Something like:

rfb       932639    3737  0 03:16 pts/6    00:00:00 /bin/bash ./server.sh -c server.xml
rfb       932918  932639 99 03:16 pts/6    00:04:41 java -XX:GCTimeRatio=19 -XX:MaxTenuringThreshold=7 -XX:StringTableSize=1000003 -Xmx8192m -server -Djava.locale.providers=SPI,JRE -Djava.ext.dirs=/usr/lib/jvm/java-8-openjdk-amd
rfb       933308    3737  0 03:17 pts/9    00:00:00 /opt/spawner/spawn 0 3333 localhost standard 0be20d98-b59f-4e48-ba2c-e3f0d7bf4547 net:socket:nio=true net:ssl:trackSeqNo=false -O server:clientConfig:outputToFile=/dev/null
rfb       933309  933308 13 03:17 pts/9    00:00:10 java -Xmx64m -Djava.awt.headless=true -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5000 -Djava.library.path=/home/rfb/projects/fwd/trunk_dev/build/lib/ -cla
rfb       933325    3737  0 03:17 pts/10   00:00:00 /opt/spawner/spawn 0 3333 localhost standard f2be5676-9860-4a93-9409-4851874ed51e net:socket:nio=true net:ssl:trackSeqNo=false -O server:clientConfig:outputToFile=/dev/null
rfb       933327  933325 11 03:17 pts/10   00:00:09 java -Xmx64m -Djava.awt.headless=true -Djava.library.path=/home/rfb/projects/fwd/trunk_dev/build/lib/ -classpath /home/rfb/projects/fwd/trunk_dev/build/lib/p2j.jar com.goldenco

Notice that the first java process I show above is 932918, which is a child of 932639. That tells us it is the server process (/bin/bash ./server.sh). The second and third java processes are a little more difficult to determine which is which. They are the app server clients, children of the /opt/spawner/spawn process.

So a jstack of the server process would be performed with jstack 932918:

If a user connects via a web interface, you will see a new process come in as a child of a new /opt/spawner/spawn process:

rfb       936040    3737  0 03:34 pts/11   00:00:00 /opt/spawner/spawn 1 rfb /usr/c-act java -Xmx1024m -Djava.awt.headless=true -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=2999 -Djava.library.path=/home/rfb/
rfb       936041  936040 99 03:34 pts/11   00:00:28 java -Xmx1024m -Djava.awt.headless=true -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=2999 -Djava.library.path=/home/rfb/projects/fwd/trunk_dev/build/lib/ -c

Because we are running on the same system, it shows rfb, which is me. You will see the user ID of the user that signs in. That's one way to tie the process to a user. To get jstack on the user's client, it would be jstack 936041

VisualVM

TBD

Heap Dumps

When a review of code is not sufficient to determine the root cause of a memory leak, a heap dump analysis often is necessary. A good tool for this is the eclipse Memory Analysis Tool (MAT - https://www.eclipse.org/mat/). It can be installed either standalone or as an eclipse IDE plug-in. Installation is left as an exercise to the reader (see https://www.eclipse.org/mat/downloads.php).

This is written from the perspective of using the MAT as a plug-in from within the eclipse IDE. A standalone version is also available but not described here.

Note: to use the MAT with larger heap dumps, eclipse itself will need to be given a larger maximum heap setting, generally enough to accommodate the size of the heap dump captured. This can be done by adjusting the -Xmx<n>g setting within the eclipse.ini file and restarting eclipse. See your eclipse documentation for the location of eclipse.ini on your particular system.

Capture a Heap Dump

The simplest way to acquire a heap dump is to do so from within the eclipse IDE. When installed as a plug-in, a "Memory Analysis" perspective is added to eclipse. This perspective includes two useful buttons in eclipse's tool bar:

The button on the left is used to acquire a heap dump by attaching to a running JVM. You will be prompted to select the JVM to which to attach, and to provide a file name/location within which to store the acquired dump. When you acquire a heap dump in this way, you can choose to open the heap dump in the MAT as soon as it is acquired.

The button on the right is used to open a previously acquired heap dump from the file system.

It is advisable to use a disk with the fastest possible I/O (e.g., an SSD) if the heap dumps you acquire aree large. Larger dumps take a while to acquire and to write to disk, and take even longer to open in the tool for the first time, as multiple indices are created and cached on disk. Be prepared to consume a lot of disk quickly, and to either archive or discard old heaps as soon as possible to avoid running out of space.

Eclipse can acquire a heap dump from a locally running JVM, but if your target JVM is not running on the same system as eclipse, you will need to use a different tool to acquire the heap dump and then transfer the resulting file to a system where the MAT is available. Once such tool that comes with the JDK is jmap (https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr014.html). jmap has several modes of operation; use the mode which produces a binary heap dump [TODO: document usage].

Analyzing Heap Dumps

TBD

Statistics

TBD: make sure our Monitoring chapter has the details and link there.

© 2020-2025 Golden Code Development Corporation. ALL RIGHTS RESERVED.

mat_toolbar_buttons.png (912 Bytes) Greg Shah, 02/18/2025 02:03 PM