On 6/23/26 10:45, Hynek Cihlar wrote:

Greg,

regarding Lyra and Artillery. Lyra's core is designed such that the recorded tests can be executed with tools like Artillery. The recorded tests are self-contained for example. However during the implementation phase some technical depth accumulated, which would have to be cleaned up first.

Currently Lyra only supports single-node test orchestration. There is one process per VU (i.e. one browser process group and Node process per VU). OTOH Artillery runs multiple VUs in a single process.

I see two product paths. 1) Lyra supporting execution in Artillery (or similar) and 2) Lyra itself be able to drive tests on multiple nodes.

For both the paths there is the same prerequisite: Allowing multiple VUs to be executed in a single process. The following are the gaps that would need to be addressed:

• The helpers ignore the Playwright page object they're handed.
• Module-global state vs in-process VU concurrency. Lyra's helpers keep all per-VU state as module-level singletons, these must be moved into a runtime context.

• Named-timers and metrics, implement an integration layer for the multi-node player (like Artillery).

All the above is about 2 MDs of work.

Additionally for Artillery a thin integration layer would need to be implemented to allow storing metrics and the named timers in its context.

For our own multi-node orchestration some additional work would be needed, like multi-node clustering and remote test spawning, globally unique VUs, metrics transport and merging, clock alignment, etc. This would be substantially more effort, roughly 10 MDs.

Do you want me to create a Redmine issue and provide more details there?

Thanks,
Hynek

As a clarification of the 1 VU -> 1 node process architecture. <customer app> still uses OsSsoAuthenticator, which requires 1 OS user per client. Because the spawner with this auth mode spawns the ClientDriver process from the OS's user shell it means that the client process is owned by that OS user.

In Lyra I did use 1 node process for all VUs, but since they were owned by the master OS user, I got several problems when it came to collecting statistics. jstack and jmap wouldn't work, I would get permission denied. Another component that relies heavily on this architecture is the PID tracking, Lyra can easily trace the Java processes because their unique UID, this allows for lifecycle management: cleanup, proper artifact generation, correct dump target (several bugs where since all processes were owned by sb, the wrong ones were used for dumping). Any tool that has strict permissions will not work unless this architecture is used.

We can totally remove this and move to whatever architecture Artillery requires, but unless <customer app> switches the SSO definitively the entire diagnostics component will be missing. I think the best solution is to add the option to switch between modes, such that once the load test machine is ready nothing is effectively changed and we can make quick progress, generate reports for client vs memory usage record performance properly etc. instead of adjusting now, and then having to re-build again the diagnostics component.

Gemini chat output:

Prompt 1¶

I'm considering a toolset for implementing load testing for my web application. I use Playwright to write my specific tests. I'm considering Artillery (https://www.artillery.io) as the harness or overall orchestration tool for running the load tests. How does it compare to other options?

Response 1¶

Artillery is the standout choice for teams looking to leverage Playwright for browser-based load testing. Its native engine seamlessly reuses existing Playwright code, tracks Web Vitals, and scales to thousands of headless browsers via serverless AWS setups.

How Artillery Compares to Other Options¶

• k6 (Grafana Labs): The current industry favorite for developer-first teams. k6 is written in Go, making it lighter, faster, and more resource-efficient than Node-based Artillery. While k6 supports browser testing via k6-browser, integrating raw Playwright scripts into Artillery is more straightforward.
• Locust: The premier choice for Python developers. It is code-driven, highly scalable, and excellent for complex, state-heavy scenarios. However, if your Playwright code is in JavaScript/TypeScript, Locust won't help you reuse it.
• Gatling: Built for Scala and Java developers. Gatling yields high concurrency per testing agent, but it lacks the direct, no-hassle Playwright integration that Artillery provides.
• Apache JMeter: The enterprise heavyweight. It's excellent for testing legacy systems and has a massive plugin ecosystem, but its GUI-heavy nature makes it difficult to pair with modern JavaScript-based Playwright suites.

Why Artillery Wins for Your Use Case¶

• Zero Rewrite for Browsers: Artillery lets you import functions from your existing Playwright Page Object Models (POMs) directly into your performance tests with minimal tweaking.
• Flexible Execution: It scales out of the box using serverless infrastructure (AWS Fargate, AWS Lambda) without requiring dedicated orchestration infrastructure.
• Hybrid Testing: You can easily mix pure HTTP API tests (very fast) with Playwright-driven browser paths (resource-heavy) within the same scenario.

Prompt 2¶

The test suite is actually in a derivative of Playwright called Lyra (https://proj.goldencode.com/projects/p2j/wiki/Lyra_Automated_Testing_Evolved). We wrote Lyra and can change or extend it as needed. It is written in Javascript, as are the tests. We are self-hosting. Our expectation is that we will have to implement an integration layer to make Lyra work with Artillery, but we also expect that this layer should be relatively easy since Lyra is based on Playwright.

Response 2¶

Because Lyra is built on top of Playwright and written in JavaScript, integrating it into Artillery is highly achievable. However, you will need to bypass Artillery's standard test-runner architecture and instead treat Lyra as a custom Processor module.

Understanding how Artillery interacts with JavaScript code reveals exactly how to build this integration layer for a self-hosted environment.

How Artillery's Playwright Integration Works Under the Hood¶

To map out your integration layer, you must first understand a major constraint in Artillery: Artillery does not use playwright test (the test runner). It bypasses the standard testing hooks, fixtures, and assertions (test(), beforeEach(), expect()).

Instead, Artillery uses the core Playwright Library API (playwright.chromium.launch()). When an Artillery virtual user (VU) spawns, Artillery automatically spins up a headless browser instance and hands a raw Playwright page object over to a standard JavaScript asynchronous function defined in a processor file.

# artillery-config.yml
config:
  target: "https://myapp.local" 
  engines:
    playwright: {} # Loads Artillery's Playwright engine
  processor: "./artillery-lyra-bridge.js" # Your integration layer
scenarios:
  - engine: playwright
    testFunction: "runLyraScenario" # The bridge function Artillery calls

Use code with caution.

Designing the Lyra-Artillery Integration Layer¶

Since you own the Lyra codebase, your integration layer needs to act as an adapter that accepts Artillery's raw page context and feeds it directly into Lyra's internal execution engine.

1. Decoupling the Runner from the Actions: If Lyra currently forces scripts to run through a custom command-line interface (CLI) or test runner that handles browser creation internally, you will need to expose a lower-level API. Modify Lyra so its automated testing actions can be executed against an existing, externally provided Playwright page instance.
2. The Bridge Script (artillery-lyra-bridge.js): You will write a bridge script that exports a standard function for Artillery to call. This function will receive the Artillery-managed browser context and inject it into your Lyra framework:

// artillery-lyra-bridge.js
const { LyraExecutor } = require('./path-to-your-lyra-core');

async function runLyraScenario(page, vuContext, events) {
  // 1. Lyra intercepts Artillery's browser page object
  const lyraTest = new LyraExecutor({ externalPage: page });

  try {
    // 2. Execute your Lyra-specific compiled test JSON or script
    await lyraTest.run('./specs/login-and-checkout.lyra.js');

  } catch (error) {
    // 3. Inform Artillery's aggregator that a virtual user failed
    events.emit('counter', 'lyra.errors', 1);
    throw error;
  }
}

module.exports = { runLyraScenario };

Use code with caution.

Architectural Gotchas for Self-Hosting¶

Since you are entirely self-hosting the infrastructure, keep these two critical constraints in mind when scaling up your Lyra tests:

• Memory Bloat per Virtual User: Because Lyra is an abstraction layer built on top of Playwright, it inherently adds processing overhead to every Virtual User. Headless browsers typically allow only 10 to 20 VUs per CPU core before collapsing from memory starvation. Ensure your self-hosted nodes are provisioned primarily for high RAM capacity.
• Metrics Tracking & Custom Events: Artillery automatically tracks HTTP metrics and core web vitals through its own Playwright engine. However, it will not natively understand custom functional checkpoints inside Lyra. You will want to utilize the events emitter argument provided by Artillery to inject custom metrics or transaction timers straight from Lyra into Artillery's final terminal report using events.emit('histogram', 'lyra.transaction_time_ms', duration).

If we indeed can effectively integrate and use Artillery, I'd prefer to do that instead of "recreating the wheel". If Artillery is up to the task, let's take advantage of it. If there is some reason no to do that, let's discuss it here.

Serban, you should think more outside the box, literally :-). Lyra is executed on its own server node (or nodes) completely independent on FWD. This is the typical and required deployment model. Running everything on one system is typically needed only during development.

Șerban Bursuc wrote:

As a clarification of the 1 VU -> 1 node process architecture. <customer app> still uses OsSsoAuthenticator, which requires 1 OS user per client. Because the spawner with this auth mode spawns the ClientDriver process from the OS's user shell it means that the client process is owned by that OS user.

Could we expose the diagnostics tooling as admin REST APIs in the FWD server? That would allow the support to be distributed.

Hynek Cihlar wrote:

Serban, you should think more outside the box, literally :-). Lyra is executed on its own server node (or nodes) completely independent on FWD. This is the typical and required deployment model. Running everything on one system is typically needed only during development.

Șerban Bursuc wrote:

As a clarification of the 1 VU -> 1 node process architecture. <customer app> still uses OsSsoAuthenticator, which requires 1 OS user per client. Because the spawner with this auth mode spawns the ClientDriver process from the OS's user shell it means that the client process is owned by that OS user.

And my point is that the development is not ready :-). The current state should be kept at least until we can test 10860c on our load test machine, because there might be some changes that need to be done before planning for Artillery. This would also allow the so-wanted data collection from load testing I've mentioned. If Lyra gets changed BEFORE this happens, then this prospect becomes increasingly difficult. There's still skepticism as to why tests are failing with multiple users, and while we do have evidence, I would rather consider waiting for a proper Lyra load test. The load test harness still hasn't been tested with many users, and when the distributed obfuscation comes in, then we will get these answers even later.

So in short, I have no problem with creating a complex distributed Lyra system, but for this:

1. Need a way to actually test changes as we go, not making speculative changes that will need to be fixed later anyway.
2. Need to clear some current questions, as the load test harness has been tested on a poor environment.

The order of development is sort of bothering me ATM with Lyra. Of course we can use a special branch for the targeted Artillery changes and add them when needed, but if there's a clash with the changes done after doing a load test on our machine, then it becomes complicated.

Could we expose the diagnostics tooling as admin REST APIs in the FWD server? That would allow the support to be distributed.

Some kind of harness still needs to collect those statistics. I think this is moving the job from one place to another, instead of a Lyra component doing the diagnostics on the server machine and sending it to the node machine, you rely on other means that essentially have to do the same thing. An REST API will have to be implemented for sure, but what collects this data should still be Lyra, even if in a distributed system.

This all goes out the window if jstack and jmap can somehow be used remotely without strict permissions, if there's a way to do that ignore my point.

Șerban Bursuc wrote:

Hynek Cihlar wrote:

Serban, you should think more outside the box, literally :-). Lyra is executed on its own server node (or nodes) completely independent on FWD. This is the typical and required deployment model. Running everything on one system is typically needed only during development.

Șerban Bursuc wrote:

As a clarification of the 1 VU -> 1 node process architecture. <customer app> still uses OsSsoAuthenticator, which requires 1 OS user per client. Because the spawner with this auth mode spawns the ClientDriver process from the OS's user shell it means that the client process is owned by that OS user.

And my point is that the development is not ready :-). The current state should be kept at least until we can test 10860c on our load test machine, because there might be some changes that need to be done before planning for Artillery. This would also allow the so-wanted data collection from load testing I've mentioned. If Lyra gets changed BEFORE this happens, then this prospect becomes increasingly difficult. There's still skepticism as to why tests are failing with multiple users, and while we do have evidence, I would rather consider waiting for a proper Lyra load test. The load test harness still hasn't been tested with many users, and when the distributed obfuscation comes in, then we will get these answers even later.

I don't disagree. Adding another layer may make things more complex and harder to diagnose. But to be able to run many many test VUs it will require multi-node playback, there is no way around this.

Also replicating the target deployment setup closely will help to reduce noise and hopefully simplify diagnosing found issues.

The order of development is sort of bothering me ATM with Lyra. Of course we can use a special branch for the targeted Artillery changes and add them when needed, but if there's a clash with the changes done after doing a load test on our machine, then it becomes complicated.

The direction towards a distributed load testing (whether Aretillery or an in-house orchestrator) was on the path since the beginning. No way you could achieve reasonable load from a single node.

Greg Shah wrote:

Could we expose the diagnostics tooling as admin REST APIs in the FWD server? That would allow the support to be distributed.

We already have JMX, that I think would be more lighweight and standard mechanism.

Hynek Cihlar wrote:

Șerban Bursuc wrote:

Hynek Cihlar wrote:

Serban, you should think more outside the box, literally :-). Lyra is executed on its own server node (or nodes) completely independent on FWD. This is the typical and required deployment model. Running everything on one system is typically needed only during development.

Șerban Bursuc wrote:

As a clarification of the 1 VU -> 1 node process architecture. <customer app> still uses OsSsoAuthenticator, which requires 1 OS user per client. Because the spawner with this auth mode spawns the ClientDriver process from the OS's user shell it means that the client process is owned by that OS user.

And my point is that the development is not ready :-). The current state should be kept at least until we can test 10860c on our load test machine, because there might be some changes that need to be done before planning for Artillery. This would also allow the so-wanted data collection from load testing I've mentioned. If Lyra gets changed BEFORE this happens, then this prospect becomes increasingly difficult. There's still skepticism as to why tests are failing with multiple users, and while we do have evidence, I would rather consider waiting for a proper Lyra load test. The load test harness still hasn't been tested with many users, and when the distributed obfuscation comes in, then we will get these answers even later.

I don't disagree. Adding another layer may make things more complex and harder to diagnose. But to be able to run many many test VUs it will require multi-node playback, there is no way around this.

Also replicating the target deployment setup closely will help to reduce noise and hopefully simplify diagnosing found issues.

The order of development is sort of bothering me ATM with Lyra. Of course we can use a special branch for the targeted Artillery changes and add them when needed, but if there's a clash with the changes done after doing a load test on our machine, then it becomes complicated.

The direction towards a distributed load testing (whether Aretillery or an in-house orchestrator) was on the path since the beginning. No way you could achieve reasonable load from a single node.

Of course, we proceed as planned. But for this step we really need that load testing machine. We need it to both test the current state of the harness/CACT and to be able to test development for distributed systems. I am trying to say to not glance over this and directly jump to the next major Lyra step, I think the current state needs to be consolidated first and these massive changes need to have a test environment ready.

Șerban Bursuc wrote:

Of course, we proceed as planned. But for this step we really need that load testing machine. We need it to both test the current state of the harness/CACT and to be able to test development for distributed systems. I am trying to say to not glance over this and directly jump to the next major Lyra step, I think the current state needs to be consolidated first and these massive changes need to have a test environment ready.

Can you use VirtualBox for the devel part? It should suffice on the small scale.

Hynek Cihlar wrote:

Șerban Bursuc wrote:

Of course, we proceed as planned. But for this step we really need that load testing machine. We need it to both test the current state of the harness/CACT and to be able to test development for distributed systems. I am trying to say to not glance over this and directly jump to the next major Lyra step, I think the current state needs to be consolidated first and these massive changes need to have a test environment ready.

Can you use VirtualBox for the devel part? It should suffice on the small scale.

No, it consumes too much RAM, even without load testing I seldom can have a FWD server open + VirtualBox because I get frequent OS crashes. I have it at 8GB and if I lower I have instability (using Windows). This would be even worse with the load testing overhead. I could use a very low overhead distro, but if the server is on my machine I still will struggle with memory if the node count increases.

With a machine at GCD we could test 1 orchestrator (probably at the GCD machine) and 2 test runners, which seems like a fair development setup for this next phase. Maybe could even turn my own laptop into a compute node.

Serban, I will create new task branch 11562a for this work. My plan is to to perform the refactorings first and merge them before implementing the Artillery specific changes. This should minimize the impact on your current work.

Hynek Cihlar wrote:

Serban, I will create new task branch 11562a for this work. My plan is to to perform the refactorings first and merge them before implementing the Artillery specific changes. This should minimize the impact on your current work.

Yes, unfortunately no load test machine means we need changes in the dark. I'm fine with this as long as the single node current approach is not regressed. I can test your changes in a distributed environment (but just barely) once they're ready.

Șerban Bursuc wrote:

Hynek Cihlar wrote:

Serban, I will create new task branch 11562a for this work. My plan is to to perform the refactorings first and merge them before implementing the Artillery specific changes. This should minimize the impact on your current work.

Yes, unfortunately no load test machine means we need changes in the dark. I'm fine with this as long as the single node current approach is not regressed. I can test your changes in a distributed environment (but just barely) once they're ready.

Sounds good.

I have a working Artillery support implemented, not yet checked in. A simple (but distributed) smoke test passed.

Hynek Cihlar wrote:

11562a contains the required refactorings. I did some elementary testing, more testing is WIP. Serban, please review.

Sorry for the delay, I mistook a bug from trunk Lyra with changes from this branch. The changes on the branch are good.

Șerban Bursuc wrote:

Hynek Cihlar wrote:

11562a contains the required refactorings. I did some elementary testing, more testing is WIP. Serban, please review.

Sorry for the delay, I mistook a bug from trunk Lyra with changes from this branch. The changes on the branch are good.

Serban, plrease run your tests with the branch.

Serban, plrease run your tests with the branch.

I initially did but it's good that you pressed on this because it made me realize they were done with the old helpers file :). The new helper file is fine but requires a small patch to work, I attached it. I'm not sure it's correct though but it allowed me to run all tests successfully, otherwise there are these two errors:

Error: Dataset 'credentials' not found. Call loadDataset first.                                                                                                                                                                                                                         
      at getData (/home/sb/gcd/trunk/tools/lyra/lyra-test-helpers-1.2.0.js:147:11)                                                                                                                                                                                                        
      at runTest (/home/sb/gcd/trunk/tools/lyra/kbp1_account_customer_order.js:41:16)                                                                                                                                                                                                     
      at async wrapTestExecution.viewport.width (/home/sb/gcd/trunk/tools/lyra/kbp1_account_customer_order.js:9649:5)                                                                                                                                                                     
      at async _wrapTestExecutionWorker (/home/sb/gcd/trunk/tools/lyra/lyra-test-helpers-1.2.0.js:2796:5)  

 Error: [lyra] Counter 'REPRINT_DOC' not configured — call configureCounter('REPRINT_DOC', ...) first                                                                                                                                                                                    
      at nextUnique (/home/sb/gcd/trunk/tools/lyra/lyra-test-helpers-1.2.0.js:3065:19)                                                                                                                                                                                                    
      at typeUnique (/home/sb/gcd/trunk/tools/lyra/lyra-test-helpers-1.2.0.js:3114:22)                                                                                                                                                                                                    
      at runTest (/home/sb/gcd/trunk/tools/lyra/kbp2_cash_collect_sale_reprint_document.js:4651:9)                                                                                                                                                                                        
      at async /home/sb/gcd/trunk/tools/lyra/kbp2_cash_collect_sale_reprint_document.js:4935:5                                                                                                                                                                                            
      at async _wrapTestExecutionWorker (/home/sb/gcd/trunk/tools/lyra/lyra-test-helpers-1.2.0.js:2805:5) 

Serban, please attach the patch.

I send it to you via email when Redmine was down a few days ago. Here it is again.

--- lyra-test-helpers-1.2.0.js.orig
+++ lyra-test-helpers-1.2.0.js
@@ -142,7 +142,9 @@
  * @returns {*} The data item for this user.
  */
 function getData(name) {
-  const dataset = _lyraCtx().datasets.get(name);
+  // Datasets are typically loaded at module scope, before wrapTestExecution() enters
+  // the per-VU context — those land in the default context, so fall back to it.
+  const dataset = _lyraCtx().datasets.get(name) || __lyra_default_ctx.datasets.get(name);
   if (!dataset) {
     throw new Error(`Dataset '${name}' not found. Call loadDataset first.`);
   }
@@ -3052,7 +3054,9 @@
  * @returns {Promise<string>} Formatted unique code, e.g. 'INV01'
  */
 async function nextUnique(name) {
-  const cfg = _lyraCtx().counterPatterns.get(name);
+  // Counters are typically configured at module scope, before wrapTestExecution() enters
+  // the per-VU context — those land in the default context, so fall back to it.
+  const cfg = _lyraCtx().counterPatterns.get(name) || __lyra_default_ctx.counterPatterns.get(name);
   if (!cfg) throw new Error(`[lyra] Counter '${name}' not configured — call configureCounter('${name}', ...) first`);

   const counterFile = _lyra_counter_file();

Hynek, I see nothing wrong with the changes but they need to be tested using the current CACT suite, at least 2 clients.