Author: SVL

Issues.

  1. If an error occurred during data flush, a STOP condition should be raised rather than ERROR condition. This corresponds to OUTPUT THROUGH statement. When we use OUTPUT TO statement it behaves in the following way:

    1. If 4GL cannot open file for some reason it informs us about it and does not go further.

    2. If we have deleted output file during output, we will get no reaction both 4GL and P2J.

    3. It would be nice to emulate “insufficient disk space” error during output, but I do not know how to do that on devbox not disturbing anyone.

    Into P2J, when we handle a flush error, in most cases we deal with a Stream not specifying whether it is a ProcessStream or FileStream. But according to investigations described above, if we will have an exception, it will be exception because of stream opened by OUTPUT THROUGH statement, and therefore it will be STOP condition. So I assume it is safe to make all stream errors in P2J raise STOP condition.

    Testcases for this issue: output_raise_stop1,2,3.p

    Solution: change ErrorManager.recordOrThrowError(-1, ...) to ErrorManager.showErrorAndAbend(140, "** Pipe to subprocess has been broken"):

    1. private boolean TC.switchStreams(Frame frame)

    2. private boolean TC.flushRedirected(Stream stream, boolean bypass)

    3. public void TC.flushStream(Stream out)

    4. Stream.java contains many such places:

      // TODO: actual Progress compatible text and number are unknown
      ErrorManager.recordOrThrowError(-1, err);

      at least some of them (and may be all) should be carefully changed to ErrorManager.showErrorAndAbend.

  1. Stack overflow exception - see stack_overflow.p. Solution: if an error has occurred, mark the stream “broken” and silently drop all output to it. The solution is already implemented into ProcessStream.java.

  2. If a stream has been closed, then any output to it (DISPLAY/PUT) causes NPE. Testcase: npe_after_close.p. Solution not designed yet.

  3. The main problem. 4GL does not immediately raise STOP condition when a target process is broken or even if a target process does not exist. We believe this delay is driven by a variable-size buffer which stores data before sending it to a target process, and STOP condition is raised when the buffer is full and flush is performed. Out task it to closely reproduce the behavior of this buffer. I believe that you cannot reproduce it exactly, because results of all testcases I will be talking about further may heavily vary from one run to another.

Output buffer behavior.

Rules that describe the buffer flush behavior:

  1. Rules below apply to the case when an error occurred while performing output to a stream (named or unnamed) opened using OUTPUT THROUGH or INPUT-OUTPUT THROUGH statement. Possible errors:

    1. incorrect process name, e.g. output thru "?" ;

    2. process cannot perform writing operation, e.g. output thru "cat > file.txt", where file.txt is a write-protected file;

    3. output process was killed;

    4. any other process-related problems;

    5. not that if output process hangs or receives the STOP signal, 4GL hangs too.


  2. STOP condition *may* be raised when flush fails. Usually output fails when the flush is performed, and flush causes STOP condition to be raised. But if we have not much data we may output it having an invalid stream as target, close the stream and got no exception.

  3. Input blocking operations (implicit pauses (DISPLAY "TXT" with FRAME F0. HIDE FRAME F0.), readkey, update, CHOOSE, WAIT-FOR, MESSAGE SET/UPDATE, MESSAGE VIEW-AS ALERT, MESSAGE VIEW-AS ALERT SET/UPDATE, PROMPT-FOR, SET are input blocking operations, only MESSAGE isn't.) make flush happen earlier.

  4. Stream closing (implicit or explicit) itself does not cause STOP condition to be raised unless there is some input blocking operation or a certain ( > some limit) amount of data waiting for flush.

  5. Unnamed stream generally behave in the same way as named ones.

  6. In the UNBUFFERED streams *usually* flush happens earlier than in buffered ones.

So, generally, buffer size limit depends on:

  1. type of stream error;

  2. presence of blocking operations;

  3. presence of UNBUFFERED option;

  4. differs if we deal with the UNNAMED stream;

  5. presence of OUTPUT CLOSE operation;

  6. and after all, results vary from one testcase run to another.

Some assumptions about buffer size:

  1. Buffer is measured in bytes (as we do in P2J), not in variables or any other units.

  2. Buffer size does change its size in time, i.e. if we had first flush after 100'th outputted byte, we will have the second flush after 200'th outputted byte and so on (assuming that conditions like presence of blocking operations does not changed between 0-100'th and 100-200'th bytes interval).

  3. If we have different kinds of blocking operations, the buffer size will be smallest between buffer sizes caused by these kinds operations.

  4. After an output error, client enters a state where buffer size is about two times smaller than usual. I assume that it can be a trigger that drives client back to its original state, but so far I didn't find it and I assume that this state remains until the end of client execution.

  5. Increasing the number of input blocking operations does not make flush happen earlier than if a single input blocking operation of the same kind was executed.



Buffer_size* testcase generator.

The purposes of testcase generator are:

  1. Because buffer size depends on five factors at least, total we may have up to a thousand testcases, so it is more convenient go generate them automatically.

  2. Because testcase results differs from run to run, it is convenient to automatically count an “average” result among several runs for each testcase.

  3. To have a framework for the automatic testcase results analysis.

Directories:

  1. first_flush - used for buffer size determination without close operations;

  2. second_flush - used to make the assumption #2 above, normally you don't need to use it;

  3. close - used for buffer size determination with close operations;

  4. proj - Java project files;

  5. res - output directory for generated testcases.

Files:

first_flush, second_flush and close directories contain the similar set of files used during testcase generation:

  1. blocking_operations.p - contains the set of possible blocking operations;

  2. output_data.p - contains the set of possible output operations;

  3. output_destinations.p - contains the set of possible actions that will lead to a target process error;

  4. template.p - template used for buffer_size* files generation;

  5. buffer_run_template.sh - template used for generation of buffer_run.sh ;

  6. exceptions.txt - contains the data that describe the combination of conditions that shouldn't take place. E.g. if it is written

    parameter1: value1
    parameter2: value2

    then all testcases that have parameter1 = value1 and parameter2 = value2 and any other values of other parameters, will not be generated.

Markers:

Markers which are used into blocking_operations.p, output_data.p, output_destinations.p:

  1. /*-start-*/ - indicates the beginning of the next alternative;

  2. /*-EOF-*/ - stops the processing of this file on this line.

Markers which are used into blocking_operations.p, output_data.p, output_destinations.p and template.p:

  1. /*-inout-*/ - inserts “output” if we have the output stream or “input-output” if we have the input-output stream;

  2. /*-unnamed-*/ - inserts “stream s” if the stream in not unnamed;

  3. /*-buffered-*/ - inserts “unbuffered” if the stream in unbuffered.

Markers which are used into template.p:

  1. /*-destination-*/- inserts code that specifies the target process;

  2. /*-blocking-*/ - inserts code that specifies the blocking operation;

  3. /*-output-data-*/ - inserts code that specifies the output operation;

  4. /*-close-*/ - inserts code that specifies the close operation.

Markers which are used into buffer_run_template.p:

  1. # run - inserts code that runs all testcases in turn.

Note that you should specify:

  1. blocking_operation = "name of blocking operation" for each blocking operation into blocking_operations.p;

  2. output_data = "name of output operation" for each output operation into output_data.p;

  3. output_destination = "name of output desstination" for each blocking operation into output_destinations.p;

  4. close_operation = "name of close operation" for each close operation into close_operations.p;

in order to be used into further analysis.

How to create, run and analyze testcases:

  1. First copy killall.sh to you directory on devbox. I didn't find killall command in HP-UX, so I've created my own script.

  2. Build buffer_size project using ant jar from the buffer_size directory.

  3. Change files used for testcase generation if you need to.

  4. Execute first_flush.sh or close.sh depending on what you need.

  5. Copy all files from res directory to your directory on devbox.

  6. Remove old buffer_size_summary.txt (file which stores testcase results) if you don't need it anymore.

  7. Execute buffer_run.sh number_of_runs command on devbox. This will run the testcase harness given number of times.

  8. Press something heavy enough on the “Enter” key on keyboard in order to keep it in the pressed state.

  9. Wait until the end of testcases execution.

  10. Copy new buffer_size_summary.txt to buffer_size directory on your PC.

  11. If you ran the set of testcases several times, run analyze.sh in order to produce buffer_size_summary_analyzed.txt which will contain “average” testcase results and can be considered as a “reference” file which should be able to be generated by P2J.

  12. Testcase results format:

    output_data: put 1 variable
    destination: output thru ?
    close operation: close
    unnamed:
    unbuffered: unbuffered
    inout: output
    blocking: no blocking operations
    iteration 1 result: 260 - minimum number out output operation which lead to raising of STOP condition (no output errors has occurred before)
    iteration 2 result: 120 - minimum number out output operation which lead to raising of STOP condition (an output error has occurred before, see assumption #4)

Runtime changes.

I believe that we have enough testcases to determine the Progress behavior. So, now we need to focus on runtime.

The idea is to have the singleton com.goldencode.p2j.util.BufferSizeManager which will act in the following way:

  1. It will receive notifications about stream opening and closing (may be a good place for notification is into StreamDaemon.store()/cleanupMaps()?).

  2. It will receive notifications about blocking operations (from ThinClient and other places) and store a set of blocking operations has occurred while a stream was active for each stream (based on assumptions #3,5).

  3. It will receive notification if a stream has been broken.

  4. On stream error ProcessStream requests BufferSizeManager for the buffer size, then silently drops specified number of bytes of output data and then raises exception.

  5. The “reference” buffer sizes should be stored in a configuration file or in Java code. It's easy to change Multiarray.printArray() to produce output in suitable format.

  6. I think that we may constantly track buffer size in order to more precisely reproduce the behavior (note that in item #4 we do not consider how many bytes are already in the buffer). But from the other side, I think our main goal is to reproduce correct behavior for the first flush, because most part of process stream errors happen at the beginning of the output.

Currently ProcessStream and BufferSizeManager are only drafts, so treat to them accordingly.