Project

General

Profile

Feature #9619

Forced test order

Added by Tomasz Domin over 1 year ago. Updated 5 months ago.

Status:
WIP
Priority:
Normal
Assignee:
Tomasz Domin
Start date:
Due date:
% Done:

30%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
reviewer:
production:
No
env_name:
topics:

Related issues

Related to Harness - Feature #6677: Performance test mode WIP

History

#1 Updated by Tomasz Domin over 1 year ago

  • Subject changed from Forced test orded execution to Forced test order
Harness includes features that allows for controlling test execution order like (source : Documentation and source code):
  • pre-condition (the test is executed first and will be executed on each session in the test-set)
  • none (no constraints)
  • critical path (execution of this test will stop any further tests from being dequeued/processed until this test has completed processing)
  • concurrent (the test will be started at the same time as other tests marked concurrent and assigned to the same group name)
  • sequential (the test will execute on the same session as others marked sequential and assigned to the same group name and the execution will be forced to be in the same order as encoded in the test-set)
  • specific (this test will not run until the specifically named test has successfully completed; if the test upon which this one is dependent fails, then this test will be marked as a failed dependency and will not run)
  • post-condition (the test is executed last and will be executed on each session in the test-set)
However there are two problems discovered during execution of complex test sets like regression tests:
  • only one dependency type per test is supported, e.g. its not possible to specify concurrent and specific nor concurrent in one group and sequential in another test group
  • in case there is no strict order of the tests there may be huge differences in execution order of tests depending on performance of a machine executing the tests, that is very visible when running tests in container environment where host resources are limited

The proposed solution is to provide an additional file that contains expected order of test execution.
There would be a kind of umbrella file allowing harness to keep order of test independent of underneath system platform and changes/optimizations to FWD runtime.
Would contain name of test together with its required state to proceed - either given test started and test finished would be needed to proceed with execution of further tests.

#3 Updated by Greg Shah over 1 year ago

Keeping dependency information in a separate file doesn't seem to be a good practice. Likewise, adding a completely separate mechanism for dependencies is confusing and error prone.

I would prefer to implement multiple dependencies per test. And/or add additional dependency types.

#4 Updated by Tomasz Domin over 1 year ago

Greg Shah wrote:

Keeping dependency information in a separate file doesn't seem to be a good practice. Likewise, adding a completely separate mechanism for dependencies is confusing and error prone.

My idea that came just next was to put additional sequential dependencies into index file.

I would prefer to implement multiple dependencies per test. And/or add additional dependency types.

That is much harder (requires more time) to implement as that requires playing with Queues/Barriers, and a positive outcome is not certain even then if tests are complex and have complex to track implicit dependencies.

Actually idea is about soft sequential dependencies. The ones which are implemented are hard - in case tests are to be executed in specific order they form a Queue executed on a single thread extending test execution time (usually). I was rather thinking about "frame" for dependencies. Knowing test execution order we can force them to keep in order with possibly minimal impact on multithreading (in worst case we can assign driver threads to tests based on scenario provided).

#5 Updated by Greg Shah over 1 year ago

I was rather thinking about "frame" for dependencies.

Like a phase of testing? We could add a dependency type for an ordinal phase number (1, 2, 3...). Given an ordinal of X for a specific test, it would be interpreted as "all tests in phases < X are complete".

#6 Updated by Tomasz Domin over 1 year ago

  • Assignee set to Tomasz Domin
  • Status changed from New to WIP
  • % Done changed from 0 to 10

Created task branch 9619a

#7 Updated by Tomasz Domin over 1 year ago

  • % Done changed from 10 to 30

Greg Shah wrote:

I was rather thinking about "frame" for dependencies.

Like a phase of testing? We could add a dependency type for an ordinal phase number (1, 2, 3...). Given an ordinal of X for a specific test, it would be interpreted as "all tests in phases < X are complete".

And how do we know how many phases are needed and what are dependencies for each phase ? We'd need an in-depth test analysis which is time consuming and still gives uncertain results ? :) (I know something about that).

Having generated tests' order from a single successful run is a border case for automatic creation of phases :)
Most importantly - I've implemented a dirty prototype of ordered test execution - and it seems it works :) (for 2 cpu). The problem I have is CPU scaling - how much should I multiply all timeouts when running on slower machine. I will run more tests over weekend if I have time.

#8 Updated by Greg Shah over 1 year ago

Tomasz Domin wrote:

Greg Shah wrote:

I was rather thinking about "frame" for dependencies.

Like a phase of testing? We could add a dependency type for an ordinal phase number (1, 2, 3...). Given an ordinal of X for a specific test, it would be interpreted as "all tests in phases < X are complete".

And how do we know how many phases are needed and what are dependencies for each phase ?

This is the responsibility of whomever is writing the tests. Normally, such things are not hidden or surprising.

We'd need an in-depth test analysis which is time consuming and still gives uncertain results ? :) (I know something about that).

We need it for this one case of ChUI regression tests because we didn't develop the tests nor know the code under test. Normally, test developers would not need this.

Having generated tests' order from a single successful run is a border case for automatic creation of phases :)

We can use these results to define the phases. I don't see the automatic creation of phases as being generally useful.

#9 Updated by Tomasz Domin over 1 year ago

Greg Shah wrote:

We'd need an in-depth test analysis which is time consuming and still gives uncertain results ? :) (I know something about that).

We need it for this one case of ChUI regression tests because we didn't develop the tests nor know the code under test. Normally, test developers would not need this.

I guess we need it in case something is not working, not in generic case. That worked this way until now, as in case any of test was failing it was on FWD change developer to analyze the problem, which resulted in get deeper into the failing test, but not into other tests.

Having generated tests' order from a single successful run is a border case for automatic creation of phases :)

We can use these results to define the phases. I don't see the automatic creation of phases as being generally useful.

Greg, but even knowing some number of phases that wont make tests 100% reliable. Note that a lot of tests use timer based waiting for some background task to do some work. The time background work is done depends on a load of a host machine running tests. In case host running tests runs anything else it may interfere with tests execution leading to false negatives.
Having as many phases as possible (so ordered execution) would reduce impact of that kind of interference.

Isnt a goal running regression tests in automatic pipeline - possibly in many parallel sessions on the same server ? Considering a fact that single run takes hours.

#10 Updated by Tomasz Domin over 1 year ago

I am suspending work on Fixed Test execution feature.
In order to fix issues specified in #9619-9 additional functionality needs to be implemented (in separate tasks):
  • possibility to specify multiple specific referents for every single test, including concurrent and sequential
  • an option to autoscale Harness timeout multiplier based on selected tests' execution speed

#11 Updated by Greg Shah over 1 year ago

Isnt a goal running regression tests in automatic pipeline - possibly in many parallel sessions on the same server ? Considering a fact that single run takes hours.

Yes

#12 Updated by Tomasz Domin 5 months ago

Pushed up 9619b with following new, but not full tested features:
  • Fix for referencing a non-existant rectangle
  • Support for multiple specific dependencies
  • Support for Test Step timeout auto scaling
  • Added support for terminal encoding
  • Added terminal encoding charset check
  • Support for a correct decoding of multi-byte charsets
  • Faster processing (immediate) of incoming screen changes

Pushed up harness/9489b to revision 47

#13 Updated by Greg Shah 5 months ago

#14 Updated by Greg Shah 5 months ago

What do you suggest for the next steps here?

#15 Updated by Tomasz Domin 5 months ago

Greg Shah wrote:

What do you suggest for the next steps here?

Finalize using of those feature for stabilizing and optimizing regression testing.

Also available in: Atom PDF