FWD » Core Development » Runtime Infrastructure

We don't have generic documentation for production runtime monitoring, nor do we have a good recommendation for customers. Every customer will need this to some degree and we have open requests from some customers right now.

We should certainly have good documentation about how to enable and use the recommended tools, including some guidance about when to use them and whether the tools have any impact on runtime performance or capacity.

Our current docs are more developer focused:

JMX MBean Instrumentation (this should all be safe for production usage and VisualVM is a reasonable tool to use)
JDK Mission Control (this probably should not be used in production, but perhaps I am being too cautious)

The chapter to add would be in the Installation Configuration and Administration Guide which is our authoritative documentation for administration and deployment of the runtime system.

My guess is that VisualVM is the easiest tool to access and use, but that might only be because it was included in Java 8 (removed in Java 9). If that is not suitable, then another open source tool like JConsole would be a good choice. Commercial tools are not to be considered for this usage.

Please ensure you answer these questions:

• JVM/Application Monitoring
  • Basic
    • CPU usage
    • threading (including ad-hoc inspection of stack traces)
    • heap usage
    • garbage collection
  • JMX MBean Enable/Disable/Monitoring
  • Integration with Cloud Monitoring tools (if possible, we would only focus on AWS for now)
  • command line vs GUI usage
  • setup (installation, JVM command line options needed, requirements/cfg to enable local and remote access)
• PostgreSQL Monitoring
  • ECF/OM/AL2... should detail the requirements here (e.g. SQL query logging, stats and so forth)

I've included a "cast of thousands" as watchers so that we can get different perspectives and ensure that we don't miss anything critical. The focus here is on production systems so we should only specify approaches that would be safe for a customer to enable on a live system.

Alexandru: Is our documentation for Available JMX Instrumentation Points accurate and complete?

Added:

FWD Server - Dynamic Query Processing¶

• SpreadsheetCacheHitCount
• SpreadsheetCmdNanos
• ThreadsCpuUtilization

It is important to note that many (I think most) tracepoints are wrapped in a conditional activated by fwd_jmx_debug system property. Thus, the tracepoints are not in-use for production environments, but only in those using fwd_jmx_debug system property, which I expect to be development environments. I think this was a performance decision took by Stefanel in #9019 out of performance reasons.

It is important to note that many (I think most) tracepoints are wrapped in a conditional activated by fwd_jmx_debug system property. Thus, the tracepoints are not in-use for production environments, but only in those using fwd_jmx_debug system property, which I expect to be development environments. I think this was a performance decision took by Stefanel in #9019 out of performance reasons.

Hmmm, that is a very good point. This suggests a few changes are needed to our documentation:

1. We should explain what fwd_jmx_debug is, how and when to enable it (and when not to do so). We absolutely don't want to have any impact on performance by adding instrumentation.
2. The Available JMX Instrumentation Points section should be in its own page. It should describe how to enable JMX in general as well as any special enabling support (like fwd_jmx_debug). The Profiling page can point to it, as can any page in Installation Configuration and Administration Guide.
3. The detailed tables of the tracepoints should have an additional column that marks them as requiring fwd_jmx_debug or not. From our existing docs you can't tell if fwd_jmx_debug is needed for a given tracepoint.

Unknown to me and should be documented:

• SpreadsheetCacheHitCount
• SpreadsheetCmdNanos
• ThreadsCpuUtilization

Hynek: Please document these.

The Profiling page can point to it, as can any page in Installation Configuration and Administration Guide.

Should we have a dedicated part (i.e. 6) named Monitoring to which a brand new chapter (i.e. 43) be referred to as "Available Instrumentation Points"? I am thinking that other chapters can be deferred to that part as well - as a result of #9514.

Alexandru Lungu wrote:

The Profiling page can point to it, as can any page in Installation Configuration and Administration Guide.

Should we have a dedicated part (i.e. 6) named Monitoring to which a brand new chapter (i.e. 43) be referred to as "Available Instrumentation Points"? I am thinking that other chapters can be deferred to that part as well - as a result of #9514.

I think we can just insert a chapter 36 named Monitoring into the Operation section (and renumber the following chapters). The "Available Instrumentation Points" can be a section of that chapter.

Moved "Available Instrumentation Points", but the wiki is still WIP.

For #9514-1, I think logging is also a thing that should be added. With the CentralLogger in place, I suppose having a good configuration of the loggers in place for production environments is relevant in terms of observability. In terms of development, me and Artur did some very slim changes to improve the log outputs, but certainly more work can be done in that area. OFC, different logging levels provide different performance levels.

For #9514-1, I think logging is also a thing that should be added. With the CentralLogger in place, I suppose having a good configuration of the loggers in place for production environments is relevant in terms of observability.

Agreed.

In terms of development, me and Artur did some very slim changes to improve the log outputs, but certainly more work can be done in that area. OFC, different logging levels provide different performance levels.

Please create a new task for this part. We really need to clean things up so that by default there is very little logging. We have many things that are warnings that don't need to be warnings, some severe that aren't really severe and lots of info that probably should be fine. Bottom line: our logs are clogged with useless entries that hide real problems. This was fine when we were in the heat of adding a new feature and didn't discuss the log levels. Now it needs to be fixed. One way is to go through the logs for real customer test systems and clean up the stuff that should not be there.

Greg Shah wrote:

Unknown to me and should be documented:

• SpreadsheetCacheHitCount

This is an unused metric and is going away.

• SpreadsheetCmdNanos

Added to the Other Miscellaneous Metrics section.

• ThreadsCpuUtilization

This doesn't seem to be used anywhere.

• Adding -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints to the application startup
• Running the application under Java 17
• Including the --gc option to server.sh sets up the jmxremoteport, but I wasn't sure about how VisualVM would leverage that.

I would imaging monitoring CPU is important, so if there's more expertise on that aspect of VisualVM, it would be appreciated. It looks like a very good tool. I could take thread dumps easily.

I tried to do a quick assessment of VisualVM by downloading from https://visualvm.github.io/. It was very easy to get it up-and-running. With minimal reading of the docs, and watching their video, I could get it going. However... I was not successful monitoring the CPU. I am getting "Not supported for this JVM." in the CPU Monitor area. There were various solutions posed via Mr. Google, but none I tried worked:

• I can do heap thread dumps and heap dumps
• Track live statistics
• Keep track of MBeans using MBeans plugin
• Keep track of GC using Visual GC plugin
• Force GC
• Sometimes doing sampling

I just run the server with JVM OpenJDK 64-Bit Server VM (17.0.13+11-Ubuntu-2ubuntu122.04, mixed mode, sharing) and was able to see the live CPU (Java version 17.0.13, vendor Ubuntu). No arguments out of the ordinary:

-ea
-XX:StringTableSize=1000003
-Xmx8192m
-Dh2.collatorCacheSize=1000000
-Dh2.objectCacheSize=1000000
-XX:GCTimeRatio=19
-XX:MaxTenuringThreshold=7
-Xdebug
-Xnoagent
-Djava.compiler=NONE
-Xrunjdwp:transport=dt_socket,address=2080,server=y,suspend=n
-Djava.locale.providers=SPI,CLDR,COMPAT
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.naming/javax.naming=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
-Djava.library.path=../lib/
-Djava.system.class.loader=com.goldencode.p2j.classloader.MultiClassLoader

Roger, mind that I have the VisualVM-Extensions plugin set-up. Maybe this is the thing that makes Java 17 work. But mind that Java 21 is not yet compatible with Visual VM (or at least on my environment)

Thanks for that, Alexandru. I'm trying to run with Java 8. I'll try the extension.

I also want to add that there was a huge performance hit when I was trying to sign into the web client with the customer application. I took a very, very long time. I hesitate to try it with the customer's environment until I understand if that is my configuration, or not.

I also want to add that there was a huge performance hit when I was trying to sign into the web client with the customer application. I took a very, very long time. I hesitate to try it with the customer's environment until I understand if that is my configuration, or not.

TBH, I didn't see any performance downgrade when attaching with Visual VM to any server/client process - unless you activate the sampling, profiling or some MBeans. I am curious to understand where it comes from.

I may have been "playing" with sampling created the problem.

I was trying to connect to a customers Docker container, which is running their application in order to monitor it. It presented a few layers of problems. To avoid trying to connect over the internet through their VPN, I connected to a Windows desktop on their network, to at least eliminate that part of the network latency. I installed VisualVM, and it ran just fine on Windows. But the next step was to add a remote connection. I added the remote connection to the host of the container, knowing that I may still be able to monitor some of the main things I am concerned with... CPU, and thread dumps on the running VM. I might be naive about being able to access the processes that are part of the container, but I thought I'd start with something basic.

So adding the host was easy. I read that to remotely monitor, I'd need to utilize jstatd by running it on the computer. But I received an error:

fwd@custtest:~$ jstatd
Could not create remote object
access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
java.security.AccessControlException: access denied ("java.util.PropertyPermission" "java.rmi.server.ignoreSubClasses" "write")
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
    at java.security.AccessController.checkPermission(AccessController.java:886)
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
    at java.lang.System.setProperty(System.java:808)
    at sun.tools.jstatd.Jstatd.main(Jstatd.java:139)

So there may be more to do. Some Googling started with creating a custom security policy file, and adding -J-Djava.security.policy=<path_to_policy_file> to the command.

grant codebase "file:${java.home}/../lib/tools.jar" 
{
    permission java.security.AllPermission;
};

Some recent observations for using jstack in a container. Some settings have to be passed to docker upon building the container. In a docker compose yaml:

    cap_add:
          - SYS_PTRACE
    security_opt:
          - seccomp=unconfined
    pid: host

At the command line: docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined ...

Those can pose serious security holes (see #9785-33).