FWD » Core Development » Database

This ticket is derived from #8912-216.

main.p

define variable external as handle no-undo.
run external.p persistent set external.
run proc1 in external.

external.p

define temp-table tt
   field f1 as character
   field f2 as integer
   index idx is unique f1 f2.

procedure proc1: 
   create tt.
   assign tt.f1 = "f1".
   create tt.
   assign tt.f1 = "f1".
end procedure.

When using composite indices in this context, getUnvalidatedIndices returns null instead of the bit set representing the unique indices, because of the null result, validateUniqueIndices is not called.

getUnvalidatedIndices returns null because the unique bit set is being checked against the Map of properties touched dirtyProps, in this case f2 is not in the dirtyProps, as a consequence the result is null.

// every component of the index must have been touched
            BitSet copy = (BitSet) nextUnique.clone();
            copy.and(dirtyProps);
            if (copy.equals(nextUnique))
            {
               if (result == null)
               {
                  result = new BitSet();
               }
               result.set(i);
            }

Artur, the first record that you created is flushed to the database because you re-use the buffer (causing an implicit release).
The second record you create is not flushed to the database (yet?).

TBD: In 4GL, VALIDATIONS may be delayed until the unique index is fully updated. In your case, the idx index is not complete yet, so 4GL (and FWD) may have delayed the validation. My guts tell me that FWD delayed it for too much, so it never actually occurred in the end.

The next steps: detect what is actually causing the flush: the end of proc1? the end of main.p? other?

Releasing tt after the second create does cause the error to be thrown.

Who is causing the release of tt the second time? The first time it is clear.

Alexandru Lungu wrote:

Who is causing the release of tt the second time? The first time it is clear.

I added a release as a test. In the test case from #9091-4, the release is not done.

run external.p is converted as an externalProcedure.

define variable external as handle no-undo.
run external.p persistent set external.
run proc1 in external.

I converted the example in #9091-11 and I get:

public class Start
{
   private static final InvokeConfig RUN_CALL_SITE_1 = new InvokeConfig().setTarget("external.p").persistent();

   private static final InvokeConfig RUN_CALL_SITE_2 = new InvokeConfig().setTarget("proc1");

   @LegacySignature(type = Type.VARIABLE, name = "external")
   handle external = TypeFactory.handle();

   /**
    * External procedure (converted to Java from the 4GL source code
    * in start.p).
    */
   @LegacySignature(type = Type.MAIN, name = "start.p")
   public void execute()
   {
      externalProcedure(Start.this, new Block((Body) () -> 
      {
         RUN_CALL_SITE_1.clone().setProcedureHandle(external).run();
         RUN_CALL_SITE_2.clone().setInHandle(external).run();
      }));
   }
}

which is an externalProcedure. Tested with 7156b/15117.

I meant that the proc1 is internalProcedure.

 public void proc1()
   {
      internalProcedure(Parser.this, "proc1", new Block((Body) () -> 
      {
         tt.create();
         tt.setF1(new character("f1"));
         tt.create();
         tt.setF1(new character("f1"));
      }));
   }

Artur Școlnic wrote:

run external.p is converted as an externalProcedure.
[...] is converted as an internalProcedure.

They are both converted similarly, sorry for the confusion.

In TransactionManager.processFinalazables(), the record buffer is missing from the blk.finalizables when using run proc1 in external..

When using run proc1 in external, the conversion for external.p is

   @LegacySignature(type = Type.MAIN, name = "parser.p")
   public void execute()
   {
      externalProcedure(Parser.this, new Block((Body) () -> 
      {
         RecordBuffer.openScope(tt);
      }));
   }

   @LegacySignature(type = Type.PROCEDURE, name = "proc1")
   public void proc1()
   {
      internalProcedure(Parser.this, "proc1", new Block((Body) () -> 
      {
         tt.create();
         tt.setF1(new character("f1"));
         tt.create();
         tt.setF1(new character("f1"));
      }));
   }

The opening of the scope in the external procedure is correct. To propagate it, FWD is checking if the procedure was run persistently. If so, the scope of the buffer will stay opened until the procedure will be deleted. If this doesn't happen, it means there is a bug. Look for BufferManager.scopeFinished. It checks if the scope closed was of a persistent procedure. If so, it saves the buffers in a PersistentProcScope.

The buffer is in PersistentProcScope, but the problem is still persisting, target.finished() is called to soon or is not called again for this buffer.

target.finished() for a BufferManager does

RecordBuffer.cleanupBatchMode(BufferManager.this);

On popScope of the buffer, finishedImpl is not executed, instead the worldScope attribute is set to true.

In scopeFinished for the proc1, the procedure gets deleted only if it is external and not persistent,
which in the case of proc1 is not true, instead of the delete, deleteResources is executed.

 if (extProc && !persistent)
    {
        deleted = delete(this, referent, true);

        // already deleted, nothing else to do
        delete = null;
    }
    else
    {
        // delete the static resources created in an internal proc/function/trigger 
        deleteResources(this, pdata.staticResources.pop());
    }

Considering that the proc1 is an internalProcedure, is persistent and the staticResources is empty, could someone suggest a safe way to delete it?

Artur Școlnic wrote:

Considering that the proc1 is an internalProcedure, is persistent and the staticResources is empty, could someone suggest a safe way to delete it?

internal procedures are not 'persistent' and can't be 'deleted'. What exactly do you want to delete? There are no static resources (like buffers) defined in proc1 in your example.

Do a compile test.p listing test.lst on your test program and show the listing from OpenEdge - this will show you the buffer scoping.

In ProcedureManager, for External.execute(), there are 3 finalizables objects added to the finalizables set, one of them is the buffer, I would expect that at some point the external procedure gets deleted and the finalizables are processed accordingly, this does not happen.

• main.p
  

  define variable external as handle no-undo.
  run external.p persistent set external.
  run proc1 in external.
  message "before delete".
  delete object external.
  message "after delete".
  

• external.p
  

  
  define temp-table tt
     field f1 as character
     field f2 as integer
     index idx is unique f1 f2.
  
  procedure proc1: 
     create tt.
     assign tt.f1 = "f1".
     create tt.
     assign tt.f1 = "f1".
     message "proc1 finished".
  end procedure.
  

and the validation error message is done when the external handle gets deleted. So the flush is not done via proc1, but when the external program deletes the temp-table.

Also, I've checked the compile listing of external.p and the scope of the temp-table is at the external program, and not proc1 - and this makes sense with the flush done at handle delete.

It seems like the external procedure is deleted at the end of the main program and all of it's resources are closed, this is not happening in FWD, however this is not the problem that this ticket was opened for, the fact that the test case from #9091-4 unveils an issue in FWD is a coincidence. For the problem found in #8912-216, I designed a more accurate and thorough test case, it shows the real issue.

start.p

define temp-table tt no-undo serialize-name "TempTable" 
   field f1 as character initial ?
   field f2 as character initial ?
   field f3 as integer
   index pk_idx is unique primary f3 f1.

define dataset dstt for tt.
define variable reader as handle no-undo.
define variable parser as handle no-undo.
define variable xml_content as longchar no-undo.
copy-lob from file "/home/as/gcd/dataset/abl/file.xml" to xml_content.
run parser.p persistent set parser.
create sax-reader reader.
reader:handler = parser.
reader:set-input-source("longchar", xml_content).
reader:suppress-namespace-processing = true.
reader:sax-parse().
dataset dstt:empty-dataset().
run proc2 in parser (output dataset dstt).
catch e as Progress.Lang.Error :
   message "Caught exception!".
end catch.
finally:
   delete object reader no-error.
   delete procedure parser no-error.
end finally.

parser.p

block-level on error undo, throw.

define temp-table tt no-undo serialize-name "TempTable" 
   field f1 as character initial ?
   field f2 as character initial ?
   field f3 as integer
   index pk_idx is unique primary f3 f1.

define dataset dstt for tt.

define variable delimeter_char as character no-undo initial "/".
define variable fullpath as character no-undo.
define variable current_f1 as character no-undo.
define variable current_f2 as character no-undo.

procedure StartElement:
   define input parameter par1 as character no-undo.
   define input parameter par2    as character no-undo.
   define input parameter qName        as character no-undo.
   define input parameter attributes   as handle    no-undo.
   fullpath = fullpath + delimeter_char + qName.

   case fullpath:
       when "/TempTables/TempTable/OldTempTableVal" then
          current_f1 = attributes:get-value-by-qname("val").

       when "/TempTables/TempTable/NewTempTableVal" then
          current_f2 = attributes:get-value-by-qname("val").
   end case.

end procedure.

procedure EndElement:
   define input parameter par1 as character no-undo.
   define input parameter par2 as character no-undo.
   define input parameter qName as character no-undo.

   fullpath = substring(fullpath, 1, length(fullpath) - length(qname) - length(delimeter_char)).
   case qName:
      when "TempTable" then do:
         create tt.
         assign tt.f1 = current_f1
                tt.f2 = current_f2
                current_f1 = "" 
                current_f2 = "".
      end.
   end case.

end procedure.

procedure proc2:

   define output parameter dataset for dstt.

end procedure.

file.xml

<?xml version="1.0"?>
<TempTables>
  <TempTable>
    <OldTempTableVal val="same_val"></OldTempTableVal>
    <NewTempTableVal val="new_val1"></NewTempTableVal>
  </TempTable>
  <TempTable>
    <OldTempTableVal val="same_val"></OldTempTableVal>
    <NewTempTableVal val="new_val2"></NewTempTableVal>
  </TempTable>
</TempTables>

• In 4GL, Progress.Lang.Error is caught, and Caught exception! is displayed, no other errors are thrown.
• In FWD, tt already exists with f1 "same_val" f3 0 is thrown, the catch block is not executed.

What is the error message being caught in OpenEdge? Who exactly throws this error - reader:sax-parse() or run proc2?

Constantin Asofiei wrote:

What is the error message being caught in OpenEdge?

Caught exception!

Constantin Asofiei wrote:

Who exactly throws this error - reader:sax-parse() or run proc2?

run proc2

Artur Școlnic wrote:

Constantin Asofiei wrote:

What is the error message being caught in OpenEdge?

Caught exception!

No, the output of ex:getMessage(1).

Artur Școlnic wrote:

Constantin Asofiei wrote:

Who exactly throws this error - reader:sax-parse() or run proc2?

run proc2

Remove the SAX code and keep only the proc2, plus proc1 from #9091-25 instead of the SAX code. My assumption is a flush happens when the define output parameter dataset for dstt. is processed.

Constantin Asofiei wrote:

No, the output of ex:getMessage(1).

tt already exists...

start.p

define temp-table tt
   field f1 as character
   field f2 as integer
   index idx is unique f1 f2.
define dataset dstt for tt.
define variable parser as handle no-undo.
run parser.p persistent set parser.
run proc1 in parser.
dataset dstt:empty-dataset().
run proc2 in parser (output dataset dstt).
catch e as Progress.Lang.Error :
   message e:getMessage(1).
end catch.
finally:
   delete procedure parser no-error.
end finally.

parser.p

define temp-table tt
   field f1 as character
   field f2 as integer
   index idx is unique f1 f2.
define dataset dstt for tt.
procedure proc1:
   create tt.
   assign tt.f1 = "f1".
   create tt.
   assign tt.f1 = "f1".
end procedure.
procedure proc2:
   define output parameter dataset for dstt.
end procedure.

In 4gl the error tt already exists... is caught, in FWD it is not.

Add a MESSAGE statement in proc2, so we know if that proc2 body is executed or not. Also, test this with and without the CATCH block.

procedure proc2:
   message "1".
   define output parameter dataset for dstt.
   message "2".
end procedure.

• In 4gl messages from proc2 are not displayed, in FWD, they are, both 1 and 2.
• In 4gl, without the catch block, the error is thrown, like in FWD.
• In FWD, the behavior is the same regardless of the catch block.

proc2 does not seem to be executed, however, if the output parameter is not used, the error is no longer thrown at all.
From what I can understand, run proc2 in parser (output dataset dstt) causes the error to be thrown.

OK, now test in OpenEdge with a TABLE parameter instead of DATASET, add a proc3 with a def output parameter table for tt1. and call that instead of proc2.

Another test: use a secondary buffer def buffer buftt1 for tt1. in proc1 instead of the implicit buffer, and test with both dataset and table parameters.

Constantin Asofiei wrote:

OK, now test in OpenEdge with a TABLE parameter instead of DATASET, add a proc3 with a def output parameter table for tt1. and call that instead of proc2.

The behavior is the same as using datasets.

Constantin Asofiei wrote:

Another test: use a secondary buffer def buffer buftt1 for tt1. in proc1 instead of the implicit buffer, and test with both dataset and table parameters.

The behavior is the same as using the implicit buffer for both 4gl and FWD.

Could running of a procedure with an output parameter cause a flush and expose the errors in the procedures ran previously?

I've added this in parser.p:

procedure checkNew.
   message "is record new" new(tt).
end.

and used run checkNew. in proc1 and in the finally block of start.p. In both cases it shows yes. This means that only a validation is performed, and not a flush/release. And this validation is happening when the parameter is being processed (the associate methods in FWD), accros all buffers associated with that dataset/temp-table.

Adding release tt in proc1 after the creates causes the error not to be caught in 4gl, this matches the behavior in FWD.

In FWD, tt is flushed 3 times, at the create, on the associate of the dataset (throws the error) and when exiting the scope of the start program (also throws the error).

As far as I know, errors like tt already exists are not supposed to be caught with catch e as Progress.Lang.Error and still it works, is there a form wrapping or reassignment of the error in this case?

Constantin Asofiei wrote:

... and used run checkNew. in proc1 and in the finally block of start.p. In both cases it shows yes. This means that only a validation is performed, and not a flush/release. And this validation is happening when the parameter is being processed (the associate methods in FWD), accros all buffers associated with that dataset/temp-table.

Coming back to this: if proc1 creates just a record, then checkNew will show 'false' for new(tt1) and 'true' for avail(tt1). This indicates that a flush is performed, and not a release (otherwise new would report true).

Regarding the tt already exists with exception; I think this is thrown as a structured exception (i.e. a CATCH block will catch it) only when the parameter is processed (and it tries to flush the buffers, thus validation is performed). The other ways to raise this error condition, like via RELEASE, will not force to morph this into a structured exception.

start.p

define variable parser as handle no-undo.
run parser.p persistent set parser.
run proc1 in parser.
delete procedure parser no-error.

define temp-table tt
   field f1 as character
   field f2 as integer
   index idx is unique f1 f2.
procedure proc1:
   create tt.
   assign tt.f1 = "f1".
   create tt.
   assign tt.f1 = "f1".
end procedure.

In ProcedureManager.delete()

if (!errors.isEmpty())
      {
         // force an ERROR condition to be raised and displayed - it will ignore NO-ERROR mode
         ErrorManager.displayAndRaiseError(errors);
      }

if (!errors.isEmpty() && !wa.tm.errHlp.isSilent())
      {
         ErrorManager.displayAndRaiseError(errors);
      }

Coming back to #9091-37, it looks like proc2 should not be executed at all, but it is executed in FWD, which causes the validation error to be thrown.
When it comes to the delete procedure parser no-error, again, the silent error is not honored, when reaching the if block discussed in #9091-50, isSilent() returns false, because the silentWorker was already executed for this work area.
These two issues seem to be unrelated.

Artur Școlnic wrote:

Coming back to #9091-37, it looks like proc2 should not be executed at all, but it is executed in FWD, which causes the validation error to be thrown.

It is important to mention that the error thrown by the validation, is morphed in a structured exception in 4GL, but in FWD, is is not.

start.p

define temp-table tt
   field f1 as character
   index idx is unique f1.
define variable parser as handle no-undo.
run parser.p persistent set parser.
run proc1 in parser (output table tt).
catch e as Progress.Lang.Error :
   message e:getMessage(1).
end catch.

define temp-table tt
   field f1 as character initial "val" 
   index idx is unique f1.
procedure proc1:
   define output parameter table for tt.
   create tt.
   create tt.
end procedure.

This test case isolates the problem with the error morphing in 4GL.
The validation is triggered by the output parameter in run proc1 in parser (output table tt), but gets morphed and caught in the catch block in start.p.
In FWD the validation error is caused by the output parameter, but it gets displayed and wrapped in an ErrorConditionException.

define temp-table tt
   field f1 as character
   index idx is unique f1.
create tt.
create tt.
catch e as Progress.Lang.SysError:
    message "Caught!".
end catch.

define temp-table tt
   field f1 as character.
find tt.
catch e as Progress.Lang.SysError:
    message "Caught!".
end catch.

For errors conditions being raised when database is accessed, we found quirks which we solved via calling ErrorManager.recordOrShowDatabaseError, see #8643, #8308-226. This may be another case related to this.

Replacing ErrorManager.recordOrThrowError with ErrorManager.recordOrShowDatabaseError did not solve the issue.

To summarize, there are 2 issues. The first one is that no-error is not honored in FWD when a handle is deleted and a validation is triggered, the test case from #9091-49 shows it.
To solve this, the exception should not be added in the error list in ProcedureManager.delete(), maybe a flag that indicates that the error is triggered by the delete could help.
The second issue is that the SysError tt already exists is not caught in a catch block, the test case for this problem is #9091-53 and #9091-54.
Other SysErrors are usually rethrown as a DeferredLegacyErrorException and are caught, this ValidationException is wrapped using ErrorManager.recordOrThrowError in TemporaryBuffer.copyAllRows().
I think this validation error needs to end up as a DeferredLegacyErrorException. Using the existing methods in ErrorManager or trying to throw directly a DeferredLegacyErrorException did not solve the issue.

Artur Școlnic wrote:

The second issue is that the SysError tt already exists is not caught in a catch block, the test case for this problem is #9091-53 and #9091-54.

ErrorManager.recordOrThrowError is actually being used and the exception is being rethrown as a DeferredLegacyErrorException, like any other SysError, the issue lies elsewhere.
Usually, a SysError is raised by the body of the procedure, in recordOrThrowError, recordForLegacyThrow is executed, ultimately, the error is caught in the BlockManager.processBody(), then processLegacyError is invoked.
In the case of this validation error, it is thrown after the body had been processed, calling recordForLegacyThrow will have no effect and the error will remain unprocessed.