FWD » Core Development » Runtime Infrastructure

1. appserver configuration, which includes:
   • min and initial agents need to be set to a value which can handle the load of normal operations, without having to start (and then trim) agents
   • max agents needs to include the number of possible calls into the appserver, both internally and from external applications
   • tempClient/poolSize needs to be configured to a value which can handle the number of initial agents at server startup
   • clientConfig/spawnerLaunchTimeout needs to be set to a value which is reasonable for starting an agent 'on demand', while the server is running and under load
   • a possible problem for misconfiguration is that a very high number of 'on demand' agents can be started at once, which we had improved in #9534, but that was for initial agents launched at server startup; doing this while the FWD server is running is still problematic
   • number of max agents must account for maximum load (i.e. number of UI users, batch processes, or other appservers internal or external) which can call into this appserver
2. number of database connections (for each database)
   • this configuration is required both in directory.xml for maxPoolSize and max_connections in i.e. postgresql database server
   • the value of max_connections needs to be greater than maxPoolSize, for a database
   • maxPoolSize must be greater than the sum of: max agents, max load of expected UI users, batch processes and any external connections to the database (i.e. from external FWD servers)
3. dispatcher threads
   • these are used for calls from outside of the FWD server, like appserver calls
   • the value must be set to a value which can handle the average-to-maximum number of calls
   • if this value is small, requests will be queued and may time out on the requester side
   • this is configured at the FWD server bootstrap config via net:dispatcher:threads
4. router threads
   • this is configured via net:router:threads at the FWD server bootstrap config
   • used for server-to-server connections/communication? (Greg - please expand, I don't recall exactly this usage)
5. heap size of the FWD server must handle the max load
6. the available heap on the machine must handle the FWD server heap size, the max number of FWD UI clients/batch processes, and max number of appserver agents.
7. other issues which need attention are the various caches configured via the directory (TBD the list)
8. postgresql tuning (shared memory, etc)

Please help in completing this list. Once we have this, I think we should create a 'punchlist' in the wiki with various values depending on the number of users/agents/etc.

We also need to consider which values can increase (safely) automatically while the FWD server is running (tempClient/poolSize and net:dispatcher:threads look to be a good candidates).

max_connection is per-cluster, so all databases in that cluster (including multi-tenant) will use this value. So, to rephrase: max_connections needs to be greater than the sum of maxPoolSize from all databases being accessed in that cluster, from all FWD servers (a database can be accessed by more than one FWD server).

Roger: where do you configure max_connections in the docker postgresql image? What is the default value?

Ovidiu: for SQLServer, how do we set the max connections?

Constantin Asofiei wrote:

Roger: where do you configure max_connections in the docker postgresql image? What is the default value?

In the container, it's in /opt/<app>/db/fwdcluster/postgresql.conf, default is 100

The container project's newer scripts that incorporate PG versions 14 to 17 strictly speaking do not allow you to control max_connections. But a change to fwd_create_pg_cluster.sh would be where this would be done.

Roger Borrello wrote:

The container project's newer scripts that incorporate PG versions 14 to 17 strictly speaking do not allow you to control max_connections. But a change to fwd_create_pg_cluster.sh would be where this would be done.

If we want to use this for load testing or even production usage, then this needs to be easily configurable. And I think this needs to be allowed when the customer's docker image is being created (somehow).

Otherwise, if you increase maxPoolSize in directory.xml to 50 per each database, if there are 4 databases, then the postgresql cluster needs to have max_connections set to 210 (to allow external/non-FWD connections, also).

For MS SQL Server, the number of maximum connections can be obtained using:

SELECT @@MAX_CONNECTIONS AS 'Max Connections';

It can be set (in this example to 9876 connections) using the following statements executed on master database:

EXEC sys.sp_configure N'show advanced options', N'1'  RECONFIGURE WITH OVERRIDE
GO
EXEC sys.sp_configure N'user connections', N'9876'
GO
RECONFIGURE WITH OVERRIDE
GO
EXEC sys.sp_configure N'show advanced options', N'0'  RECONFIGURE WITH OVERRIDE
GO

I am adding a --max-connections VALUE option (defaults to 100) to the fwd_create_pg_cluster.sh script. This script needs to be integrated into the Hotel samples. It is intended to be the new standard by which PostgreSQL clusters are created (#10491).

Updated #10722-8... default is 100 connections

As Constantin noted, we also need to be able to configure this as part of the project json file, since it affects cfg inside FWD itself.

Greg Shah wrote:

As Constantin noted, we also need to be able to configure this as part of the project json file, since it affects cfg inside FWD itself.

This is the configuration of the OS/PostgreSQL environment, not FWD. The JSON (in its present form) configures the application's setup itself. The environment that the application setup program is running within is not setup by that JSON. Start thinking about the bootstrapped environment. You had indicated we want control of that setup, but design needs to be done. Should the cluster be created off an exit that determines there isn't any? Or should it be entirely manual? Will this be the same JSON setup for the application?

At the present time, there are some tools to create the Docker images, but the integration needs design.

Roger Borrello wrote:

Greg Shah wrote:

As Constantin noted, we also need to be able to configure this as part of the project json file, since it affects cfg inside FWD itself.

This is the configuration of the OS/PostgreSQL environment, not FWD.

We can't change just the PostgreSQL max-connections on its own. We have related configuration settings in our directory that must match up.

The JSON (in its present form) configures the application's setup itself.

Elements in the json change how we generate the directory, right? That is all I'm talking about here. Let's not over-complicate this.

Greg Shah wrote:

Roger Borrello wrote:

Greg Shah wrote:

As Constantin noted, we also need to be able to configure this as part of the project json file, since it affects cfg inside FWD itself.

This is the configuration of the OS/PostgreSQL environment, not FWD.

We can't change just the PostgreSQL max-connections on its own. We have related configuration settings in our directory that must match up.

The JSON (in its present form) configures the application's setup itself.

Elements in the json change how we generate the directory, right? That is all I'm talking about here. Let's not over-complicate this.

At the current time fwd_create_pg_cluster.sh is our first foray into the world of tools to standardize the setup of the environment, so forgive me for having that pop to top-of-mind.

I assume any tools we create would set a standard. Either that or we just document: "Make max_connections in your cluster's postgresql.conf equal to..." And the formula would need to be vetted, such as "the sum of maxPoolSize values in a given server's directory, plus some extra in case out of band connections are expected."

Some background on why we made the design decision to not use pg_createcluster... that tool spreads the cluster configuration outside the cluster's directory (where postgresql.conf resides) to the /etc/postgresql directory (where pg_hba.conf resides). This allows the configuration to be more self contained.

I don't have an issue with changing fwd_create_pg_cluster.sh. I'm saying we need other changes too, to deal with the rest of the cfg.