FWD » Testing

I opened this task to discuss about various ways of testing the performance of FWD.

Prerequisites¶

There are several layers that can be tested in FWD: conversion (preprocessing, lexing, parsing, conversion), run-time (UI, databases, base language, etc.), infrastructure (appservers, REST, SOAP, Admin Console, multi-tenancy, etc.). While most of the current dicussion targets functional testing, some of the areas mentioned are also critical from the POV of performance (e.g. database querying and results caching, conversion pipeline, UI drawing, server-client communication, context look-ups etc.). For that matter, I propose seeking a way to design performance tests and use them for regression testing.

Non-functional requirements¶

The core aspect that should be discussed is how to ensure we run comprehensive, independent and relevant performance tests:

• comprehension requires a specific scenario to hit specific edge cases in FWD to trigger different mechanisms that may be slow. A test that is too straight-forward may work fast, but under very particular circumstances it may start to work slow.
• independency is about checking performance of a certin feature in a quiet and consistent environment. Running the same tests may yield different times due to: different hardware, different resuorces overload (caused by other processes on the same OS), different OS settings (like setting CPU cores on performance), etc. The testing protocol should be decided once so that tests will run consistently on the same hardware under the same circumstances.
• relevance comes from designing realistic test-cases. It is easy to spot the pitfall of writing a test that does 1M inserts and expecting it to work fast, but realize that 99% of the time is the database communication, that can't be quite improved from FWD. This might a test that is unoptimizable and out of our control. By relevance we need to design a guide of writing tests so that they are actually capturing some realistic FWD use-cases that are more high-level and relevant to the customers.

Functional requirements¶

1. A manager should be implemented that:

• captures the times of a performance testing run.
• the times should be compared with a baseline. If the times are different with >10%, then the test should fail form the performance POV.
• If the times are in <10%, I guess we can be notified in order to update the baseline.

The manager can be within Jenkins? The artifacts generated by the tests run can be saved as files and passed to the manager by Jenkins. The manager should do the baseline work and report back which tests failed from the POV of performance or not + update baselines.

2. AFAIK, functional testing is mostly relying on converting ABLUnit, using Harness or running automatic UI testing. IMHO, all such techniques can capture time and share them to a system managing the performance testing. Thus, we can reuse the functional tests to serve as performance tests as well. The downfall is that:

• some of these tests run concurrently (Harness?) and this may break the independency requirement. TODO
• some of these tests may not be relevant as they will run very fast. TODO
• some of the tests may check a very specific feature of FWD under very specific circumstances. I guess that is OK as overall we are interested in ensuring the customer code is working fast, not necessary very esoteric test-cases to start with. I can agree that some testing are specific in their nature and customer applications may determine very specific circumstances that can cause slow-down

I propose having these existing tests adapted for performance testing would be a requirement. Thus, such testing should yield granular times that can be aggregated and used by the manager.

3. I also suggest writing specific performance tests. I suppose most can be implemented from within the ABLUnit, but let me know if you have other opinions on that. Anyway, these performance tests can:

• either represent simple unit tests that run among the functional tests and at most be tagged as performance tests. This has the benefit that we can use the same project structure and reside the construct X performance testcases near the construct X functional testcases.
• or store the performance tests in another project. If you feel this is a good idea, please let me know.

Overall, I agree with most of the above. Some thoughts:

• I like the idea of implementing common code in your manager concept as a performance results analyzer.
  • Yes, it most often will be used from Jenkins but it might not only be used from there.
  • It will depend upon performance metrics which will need to be generated by the Harness and ABLUnit.
  • These metrics should be reported in a standard way. If we can include them in the junit results, that is good, but it is not required. I don't want to "dumb down" our solution just to live within the existing output for junit.
• Range
  • The detection range should be configurable, not hard coded to 10%.
  • We should report + or - of the range. Faster results can be good news but they can also be an indication that something failed but wasn't otherwise detected (e.g. a test that executed 10x faster than normal might just have silently failed).

I propose having these existing tests adapted for performance testing would be a requirement.

I'm not sure about this. I do want to measure existing test suites. I agree we want to ensure our approach can deliver a consistent mechanism for that measurement. But I don't want this to take the form of only using a subset of the tests for performance measurement. Likewise I don't want to require alternate version of tests that are used for performance while then not measuring the larger set of tests we run.

The problem with only measuring a subset is that we have to duplicate our test coverage for performance, which is not practical to do. The performance subset would always lag behind functional testing, probably by a significant percentage. We'll never catchup or even get close.

That is why I think it is very important to find a way to get consistent measurements from the existing tests.

I also suggest writing specific performance tests.

Yes, we certainly will want to add tests specifically for performance. I just want to start from the base of all our existing tests.

either represent simple unit tests that run among the functional tests and at most be tagged as performance tests. This has the benefit that we can use the same project structure and reside the construct X performance testcases near the construct X functional testcases.

Using tagging is a good way to do this. The perf tests should certainly be stored in the same area as related functional testcases. I expect them to use the same data, the same techniques and so forth. Separating them would not be efficient.

I propose having these existing tests adapted for performance testing would be a requirement.

I think this was an under-statment. I was rather thinking of adapting the "mechanism" that runs tests rather the the tested code per-se. For instance, I was thinking of having Junit / converted ABLUnit, Harness, Sikuli, etc. adapted/exended/... to provide us with accurate times so that the manager can use. The tests (unit, harness, UI, etc.) should stay the same. Overall, I meant adjusting the functional testing to also yield "milliseconds/seconds" beside "OK/FAIL". Now that I mention this, you may be right ... a FAIL may automatically fail performance as well as it exists early. Or even an OK can be suspect if its time was 10x faster.

And just to be clear, I was not referring to using ETIME to measure the time.

I don't want to do any out of the blue performance tests just to see how hast construct "x" works unless there is a good reason to do so. Performance tests are way more time consuming than functional tests. So each functional tests should at least provide some times to have a good impression of an overall FWD performance. More specific scenarios may require more specific tests ... but that will be driven by customer tasks IMHO.

PS: I will update note 1 on the fly to record the ideas.