FWD » Core Development » Database

TABLE Parameters:

• BY-REFERENCE (full/partial)
• BY-VALUE (full/partial)
• BIND (full/partial)

KW_BY_REF, KW_BIND, AND KW_BY_VALUE are all marked as partial runtime support. If they are fully supported, we need only fix the gap marking.

start.p

define temp-table ttTest 
    no-undo reference-only
    field testNum as integer.

define buffer bttTest for ttTest.

def var cls as TestClass.

cls = new TestClass().
cls:meth1(output table bttTest bind).

message "DEFAULT BUFFER: " buffer ttTest:NUM-REFERENCES.
message "TEMP-TABLE DEFAULT BUFFER: " temp-table ttTest:NUM-REFERENCES.
message "CUSTOM BUFFER: " buffer bttTest:NUM-REFERENCES.
message "TEMP-TABLE CUSTOM BUFFER: " temp-table bttTest:NUM-REFERENCES.

TestClass.cls

USING Progress.Lang.*.

BLOCK-LEVEL ON ERROR UNDO, THROW.

CLASS TestClass:
    define temp-table ttRefTest 
    no-undo 
    field testNum as integer.

    method public void meth1 ( output table for ttRefTest bind ):
    end.
END CLASS.

Output would be:

DEFAULT BUFFER: 0
TEMP-TABLE DEFAULT BUFFER: 2
CUSTOM BUFFER: 2
TEMP-TABLE CUSTOM BUFFER: 2

- if we try to interrogate the NUM-REFERENCES attribute of temp-table defined as reference-only before any initialization (or binding), an error will happen.
- after binding the reference-only temp-table will have the num-references increased, the actual temp-table (to which we bound the reference) don't change this number.
- not sure whether NUM-REFERENCES is a property of the buffer or temp-table handle. In the code snippet example, I made some experiments which shows that TEMP-TABLE handle always returns the NUM-REFERENCES. With BUFFER, NUM-REFERENCES will only be updated on buffer which was actually bound.

- if REFERENCE-ONLY temp-table have additional buffer defined, those also should be bounded to NON-REFERENCE temp-table.

I have been looking at DataSet implementation of num-reference, I think it is not quite right.

start.p

define temp-table ttTest 
    no-undo reference-only
    field testNum as integer.

define dataset dtt1 reference-only for ttTest.

define buffer bttTest for ttTest.

def var cls as TestClass. 
def var cls1 as TestClass1.

cls = new TestClass().
cls1 = new TestClass1().

cls:meth2(output dataset dtt1 bind).
cls1:meth2(cls).

message "Procedure" dataset dtt1:NUM-REFERENCES.

TestClass.cls

USING Progress.Lang.*.

BLOCK-LEVEL ON ERROR UNDO, THROW.

CLASS TestClass:
    define temp-table ttRefTest 
    no-undo 
    field testNum as integer.

    define dataset dttRefDataset for ttRefTest.

    method public void meth2 ( output dataset for dttRefDataset bind ):
       message "Main Class" dataset dttRefDataset:NUM-REFERENCES.
    end.
END CLASS.

TestClass1.cls

USING Progress.Lang.*.

BLOCK-LEVEL ON ERROR UNDO, THROW.

CLASS TestClass1:
    define temp-table ttRefTest 
    no-undo reference-only
    field testNum as integer.

    define dataset dttRefDataset reference-only for ttRefTest.

    method public void meth2 (input cls as class TestClass):
       cls:meth2(output dataset dttRefDataset bind).
    end.
END CLASS.

Output:

Main Class 0
Main Class 1
Procedure 2

From the above example shows that NUM-REFERENCES is updated at the end of the called procedure. I think the update of NUM-REFERENCES should postponed to the end of the procedure, maybe added as finalizable.

There is something in TemporarBuffer.associateImpl.

Set<Temporary> copy = new HashSet<>(dstBuf.explicitBuffers);
copy.add(dstDMO);

for (Temporary dmo : copy)
{
   TemporaryBuffer exBuf = (TemporaryBuffer) ((BufferReference) dmo).definition();

   if (exBuf != dstBufferTarget)
   {
      // create a new buffer
      Temporary newBuf;
      try
      {
         pm.disablePendingResources();
         newBuf = TemporaryBuffer.define(null,
                                         srcTable,
                                         exBuf.getDMOAlias(),
                                         exBuf.getLegacyName(),
                                         -1,
                                         false,
                                         false);
         exBuf.explicitBuffers.remove(newBuf);
      }
      finally
      {
         pm.enablePendingResources();
      }

      Temporary oldExBound = exBuf.mutableHandler.bind(newBuf, true);
   }
}

Let me show you what is bothering me on an example:

start.p

def temp-table tt reference-only
                  field f1 as int.

def buffer btt for tt.

TestClass.cls

def temp-table main-temp-table 
                field f1 as int.

Suppose I would like to BIND tt temp-table to main-temp-table.

In TB.associateImpl we will iterate over dstBuf.explicitBuffers (tt, btt is our case). I case of tt it will correctly bind it to the existing main-temp-table buffer. But for btt it will create a new buffer for main-temp-table and bind to it. I don't understand why the creation of a new buffer is necessary?

Eduard Soltan wrote:

From the above example shows that NUM-REFERENCES is updated at the end of the called procedure. I think the update of NUM-REFERENCES should postponed to the end of the procedure, maybe added as finalizable.

Never mind, I think I got why it is needed.

Some things that I noticed:

define temp-table ttTest 
    no-undo reference-only
    field testNum as char.

def var cls as TestClass. 

cls = new TestClass().
cls:meth1(output table ttTest bind).

USING Progress.Lang.*.

BLOCK-LEVEL ON ERROR UNDO, THROW.

CLASS TestClass:
    define temp-table ttRefTest 
    no-undo 
    field testNum as integer.

    method public void meth1 ( output table for ttRefTest bind ):
    end.
END CLASS.

ttTest from the calling procedure (start.p) and the ttRefTest temp-table from called procedure (TestClass.meth1) has different schema. testNum in one case is integer type, and in other is character type.

This raises an compilation error in OE, in FWD it will work without any error. This will eventually fail in FWD at some point, but from completely different reasons.

For example if a table has more fields that the other, an IndexOutOfBoundsException will be raised. Or ClassCastException could be raised in an attempt to access a field value.
A mismatch in index structure will result in a runtime error in OE, in FWD this case in not handled.

Is there any reliable why to compare the schema of 2 dmos? There is in RB.compare, but it seems to focus only on fields.

start.p

define temp-table ttTest 
    no-undo reference-only
    field f1 as int
    field f2 as char.

define variable h as handle.

run test23.p (output table ttTest bind).

create ttTest.

test23.p

define temp-table ttTest 
    no-undo
    field f1 as int
    field f2 as char.

define output parameter table for ttTest bind.

In this case I get an runtime error, Attempt to reference uninitialized temp-table.

In Parameter passing syntax I found the following statement:

Do not delete the object or routine to which a reference-only object is bound, or you might be left with references to an object that no longer exists. 

And my testcase above is exactly the case to which the documentation refers. ttTest temp-table (not defined as reference-only) is deleted once the test23.p procedure finishes its execution. And ttTest temp-table has a reference to a deleted object.

However in a slightly more complicated case, extracted from the testcases:

start.p

define temp-table ttTest 
    no-undo reference-only
    field f1 as int
    field f2 as char.

define variable h as handle.

run test22.p persistent set h.

run test21.p(h).
run proc2 in h. 

test21.p

define temp-table ttTest 
    no-undo
    field f1 as int
    field f2 as char.

define input parameter h as handle no-undo.

run proc1 in h (input-output table ttTest bind).

test22.p

define temp-table ttTest 
    no-undo reference-only
    field f1 as int
    field f2 as char.

procedure proc1:
    define input-output  parameter table for ttTest bind.
end.

procedure proc2:
    define query qh for ttTest.

    open query qh for each ttTest.
    get first qh.

    message "HELLO".
end.

Even though the test21.p procedure is deleted after the call in start.p, the ttTest reference in test22.p to the temp-table object in test21.p is still active and could actually be used afterwards.

In FWD once the test21.p procedure terminates, the scope of ttTest is finished and since it is not a persistent procedure it is not passed to the previous stack scope. Hence later use of the temp-table reference will be unable to find this particular buffer at the current scope.

I could not find this anywhere in the documentation. My best guess is that for calling procedures that passes an actual temp-table object (INPUT/INPUT-OUTPUT) to called routines with BIND option, it will actually perform an deep-copy (either right away or at the end of the procedure).

Eduard,

• 'forward' (direct, the temp table from caller is BINDed (sic!) to the one in called procedure):
  • the caller has a normal temp-table and uses it as parameter;
  • the callee declares the REFERENCE-ONLY temp-table and the parameter as INPUT ... BIND or INPUT-OUTPUT ... BIND;
• 'backward' (after return, the outer temp-table will 'point' to the table from procedure to which it was BINDed):
  • the caller defines the (empty) REFERENCE-ONLY temp-table to be used as parameter;
  • the callee defines the temp-table normally and the parameter as OUTPUT ... BIND.

Any other combination of these should raise some kind of error condition (like 13009, please look it up in OE, Help | Messages...). Official documentation is here.

Ovidiu Maxiniuc wrote:

Eduard,

You can only BIND a temp-table to a REFERENCE-ONLY temp-table. There are two cases here (these are my personal naming and probably you will not find them anywhere else):

I understand this, and already handled majority of this error cases in 9545a. What I am trying to handled now, is the cases when the callee procedure defines the temp-table normally is ended.

caller routine             ------------>  callee routine
REFERENCE-ONLY temp-table                 NORMAL temp-table   

After the end of the callee routine in the above schema, access to temp-table will raise uninitialization exception. This is expected behaviour described in documentation.

main program  ------------------> caller routine    ----------------> callee routine
                                  NORMAL temp-table                   REFERENCE-ONLY temp-table

Lets say in my schema callee routine is persisted and its methods can be called after the end of the caller routine. In this case a REFERENCE-ONLY temp-table in the callee routine which in theory should point to a deleted object, could actually be accessed without any problem. For an example please see #9545-12.

Eduard Soltan wrote:

I understand this, and already handled majority of this error cases in 9545a. What I am trying to handled now, is the cases when the callee procedure defines the temp-table normally is ended.

caller routine             ------------>  callee routine
REFERENCE-ONLY temp-table                 NORMAL temp-table

After the end of the callee routine in the above schema, access to temp-table will raise uninitialization exception. This is expected behaviour described in documentation.

This is the 'backward' case. The callee must define the parameter using OUTPUT ... BIND. In absence of these options, I think OE will issue some error or at least a warning.

main program  ------------------> caller routine    ----------------> callee routine
                                  NORMAL temp-table                   REFERENCE-ONLY temp-table

Lets say in my schema callee routine is persisted and its methods can be called after the end of the caller routine. In this case a REFERENCE-ONLY temp-table in the callee routine which in theory should point to a deleted object, could actually be accessed without any problem. For an example please see #9545-12.

This is the 'forward' case. The callee is expected to receive the parameters using INPUT ... BIND or INPUT-OUTPUT ... BIND options.
I do not understand why the REFERENCE-ONLY temp-table should point to a deleted object? It must be bound (BINDed) to the normal table the routine receives as parameter.

The problem in #9545-12 testcase is that start.p declares the temp-table ttTest as reference-only. Then you run test22.p persistent set h, but test22.p also declares ttTest as reference-only. This is the problem, you cannot have both tables reference-only. At least one must be a normal table, and in this case based on INPUT-OUTPUT parameter declaration, the definition from caller (start.p) should be a normal temp-table.

Ovidiu Maxiniuc wrote:

The problem in #9545-12 testcase is that start.p declares the temp-table ttTest as reference-only. Then you run test22.p persistent set h, but test22.p also declares ttTest as reference-only. This is the problem, you cannot have both tables reference-only. At least one must be a normal table, and in this case based on INPUT-OUTPUT parameter declaration, the definition from caller (start.p) should be a normal temp-table.

ttTest from start.p, has nothing to do with the actual test, in fact it is a leftover from my previous tests. Sorry for the confusion.

main program  ------------------> caller routine    ----------------> callee routine
                                  NORMAL temp-table                   REFERENCE-ONLY temp-table

If I apply above schema to #9545-12 example, this would be the actors: main program (start.p), caller routine (test21.p), callee routine (test22.p)

OK, sorry I was mislead by the persistent initialisation of test22.p.

Yet, I think the error is correct. Only the test22.p is persistent. test21.p is not. That means the ttTest temp-table from it (the only normal table) gets destroyed at the moment test21.p ends (the scope where it was declared ends).

The ttTest of test22.p was BINDed to ttTest of test21.p, but now it make reference to a deleted table. So the message

Do not delete the object or routine to which a reference-only object is bound, or you might be left with references to an object that no longer exists.

Ovidiu Maxiniuc wrote:

Yet, I think the error is correct. Only the test22.p is persistent. test21.p is not. That means the ttTest temp-table from it (the only normal table) gets destroyed at the moment test21.p ends (the scope where it was declared ends).

The ttTest of test22.p was BINDed to ttTest of test21.p, but now it make reference to a deleted table. So the message[...]from your note #9545-11 is legit.

If that was the case, that would be wonderful! But this is the case only for #9545-11 test case (exception is raised only for this case), it behaves according to documentation.

In #9545-12 test case it is a completely different story, the usage of test22.p ttTest after test21.p routine was ended (test22.p:proc2) does not raise any exception and the program continues execution normally.

Have these programs been tested on OE? Sometimes the OE documentation is incorrect, misleading or incomplete.

Greg Shah wrote:

Have these programs been tested on OE? Sometimes the OE documentation is incorrect, misleading or incomplete.

Yes, they have been tested in OE.

I can confirm, I have personally tested it.

And I must apologise again. In my note #9545-18 I have actually looked at note 12, not 11. Indeed, this is odd: it shows that OE will prevent deletion of the 'target' temp-table is there is at least one reference-only table bound to it.

FWD is more strict in this regard and will destroy the referred temp-table as soon the it gets out of the scope it was declared. Even for static temp-tables as in this case.

To fix, this we need to check at the moment the table gets out-of-scope if it was bound. In this case the resource is still in use and we must delay the destruction. Logically, when the last table to which it is bound is deleted / also gets out-of-scope the original temp-table will be deleted. This complicates a bit out temp-table management, but I expect it's not that difficult. Let's also investigate chained binding case (reference to reference to normal tables).

start.p

define temp-table ttTest 
     no-undo 
     field testNum as integer.

def var h as handle.
def var a as logical init yes.

h = temp-table ttTest:handle.
run test30.p (input-output table-handle h by-value) no-error.

def query qh for ttTest.

open query qh for each ttTest.
get first qh.

message available(ttTest).

test30.p

routine-level on error undo, throw.

define input-output parameter table-handle httTest.

doBlock1:
do transaction:
   httTest:default-buffer-handle:buffer-create().
   httTest::testNum = 5.

   undo, return error.
end.

Note how in my test case I pass my table-handle BY-VALUE, this means that in calling procedure (test30.p) a deep-copy of the temp-table will be created.

Then in test30.p I will create a record which is not committed to the database. undo, return error. will return control to start.p and the unflushed record will be lost.

In FWD, at the end topLevelBlock method there is a additional step which is OutputTableHandleCopier.finishedImpl. In this method the deep-copy of the parameter temp-table it is copied into the original tmep-table (for OUTPUT case). Here the unflushed record also gets its way into the temp-table parameter.
I think we need to make this method finalizable aware that a rollback happened previously.

Eduard Soltan wrote:

I think we need to make this method finalizable aware that a rollback happened previously.

Isn't this more of OutputTableHandleCopier being executed before the rollback happens?

Constantin Asofiei wrote:

Eduard Soltan wrote:

I think we need to make this method finalizable aware that a rollback happened previously.

Isn't this more of OutputTableHandleCopier being executed before the rollback happens?

No, it's executed after rollback. rollback don't clear the buffer content.

Is the buffer httTest:default-buffer still holding the record when OutputTableHandleCopier is copied?

Constantin Asofiei wrote:

Is the buffer httTest:default-buffer still holding the record when OutputTableHandleCopier is copied?

Yes.

After doBlock1 I would expect that in 4GL the buffer has no record (as the tx was rolledback). Please check this.

Constantin Asofiei wrote:

After doBlock1 I would expect that in 4GL the buffer has no record (as the tx was rolledback). Please check this.

It depends on the no-undo option of the temp-table definition. If we have no-undo then it is not rollbacked.

In FWD the same issue persist for both options.

routine-level on error undo, throw.

def temp-table tt field f1 as int.

define input-output parameter table-handle httTest.

do on error undo, throw:
   doBlock1:
   do transaction:
      httTest:default-buffer-handle:buffer-create().
      httTest::testNum = 5.

      find first tt. 
   end.

  message "Hello" httTest:default-buffer-handle:available.
end.

httTest buffer is still loaded after doBlock1 block, even if an error is thrown by find first tt.

Eduard, so please check why FWD doesn't clear the record from the buffer,or otherwise doesn't mark it as 'deleted'. When doing a copy for OUTPUT params, FWD flushes all buffers first, so that's why this still gets seen.

We can divide this in 2 cases temp-table with no-undo and without no-undo option.

routine-level on error undo, throw.

def temp-table tt field f1 as int.

define input-output parameter table-handle httTest.

do on error undo, throw:
   doBlock1:
   do transaction ON ERROR UNDO, LEAVE:
      httTest:default-buffer-handle:buffer-create().
      httTest::testNum = 5.

      find first tt. // ERROR IS THROWN HERE
   end.

   message "Hello" httTest:default-buffer-handle:available. // Execution continues from this point
end.

Undoable TEMP-TABLE¶

OE behaviour description:

httTest buffer holds an undoable temp-table. find first tt. raise on error, which rollback the doBlock1 and leaves it. After the doBlock1 execution, httTest handle does not hold the uncommitted record.

FWD behaviour description:

httTest buffer holds an undoable temp-table. find first tt. raise on error, which rollback the doBlock1 and leaves it. After the doBlock1 execution, httTest handle still holds the uncommitted record.

Reason / Resolution:
TemporaryBuffer.createDynamicTable(_httTest, httTest, true, true); code is responsible for creation of a dynamic copy of temp-table passed as parameter. And this logic does not take into account undoable state of the source temp-table.

Not Undoable TEMP-TABLE¶

We will analyze the same example.

OE behaviour description:

httTest buffer holds an undoable temp-table. find first tt. raise on error, which rollback the doBlock1 and leaves it. After the doBlock1 execution, httTest handle still holds the uncommitted record.
I think that this proves that a rollback/UNDO should not release the temp-table buffer (with NO-UNDO).

FWD behaviour description:

FWD behaves in the same way.

Reason / Resolution:
The problem appears when we rollback/undo all blocks up to topLevelBlock. In this case in OE, the unreleased buffer does get copied in the OUTPUT parameter. In FWD it does. I think we need to make OutputTableHandleCopier aware of the rollback state of the topLevelBlock

Eduard, the root cause is that the dynamic temp-table created for an argument does not inherit the undoable from the argument:

def temp-table tt1 field f1 as int.

procedure proc0.
def input-output parameter table-handle tth.
   message "undoable" tth:undo. // yes
   do transaction:
      tth:default-buffer-handle:buffer-create().
      tth::f1 = 1.
      undo, leave.
   end.

   message tth:default-buffer-handle:available.
end.

def var h as handle.
h = temp-table tt1:handle.
run proc0(input-output table-handle h).

find first tt1 no-error.
message avail tt1.

In FWD, the undo state is no.

Constantin Asofiei wrote:

Eduard, the root cause is that the dynamic temp-table created for an argument does not inherit the undoable from the argument:

Yes, indeed for undoable temp-table this is the problem.

However for NO-UNDO the root cause it something else. Rollback should not delete/release the transient record in 4GL, this also happens in FWD.

routine-level on error undo, error.
def temp-table tt1 NO-UNDO field f1 as int.

def temp-table tt field f2 as int.

procedure proc0.
def input-output parameter table-handle tth.
   message "undoable" tth:undo. // yes
   do transaction on error undo, throw:
      tth:default-buffer-handle:buffer-create().
      tth::f1 = 1.

      find first tt. // ERROR RAISED HERE
      FINALLY:
         message "First FINALLY " tth:default-buffer-handle:available.         
      END FINALLY.
   end.

   message tth:default-buffer-handle:available.
   FINALLY:
         message "Second FINALLY " tth:default-buffer-handle:available.      
   END FINALLY.
end.

def var h as handle.
h = temp-table tt1:handle.
run proc0(input-output table-handle h) no-error.

find first tt1 no-error.
message avail tt1.

Output:

undoable no
message "First FINALLY " yes.
message "Second FINALLY " yes.
no

I modified a bit your example, error raised by find first tt. statement will be propagated up to proc0 procedure (transient record still not release/deleted). However the transient record will not be copied into the original temp-table.

I have been investigated the error handling cases of passing parameters between calling and called procedure, when one temp-table has BEFORE-TABLE option and the other one doesn't.

BY-VALUE

BY-REFERENCE

BIND

MyClass1.cls

CLASS MyClass1:
    define temp-table ttNewTest
           field testNum as integer.

    method public void meth1(input-output table-handle htt):
       delete object htt no-error.

       create temp-table htt.
       htt:create-like(temp-table ttNewTest:handle).
       htt:temp-table-prepare ('ttNewTest').

       delete object htt no-error.
    end.
END CLASS.

start.p

define temp-table ttTest 
       no-undo 
       field testNum    as integer.

def var obj as MyClass1.

obj = new MyClass1().
obj:meth1(input-output table ttTest).

I think is a problem in FWD. I expose a small break down of what happens in OE and FWD in this example.

When you pass a static temp-table BY-VALUE, OE deep-copies the entire structure and data into a newly created dynamic temp-table for the duration of the call. This is exactly what happens in my example, when passing ttTest handle to MyClass1:meth1 method.

First delete object htt statement should deleted from heap memory, this exact dynamic temp-table created automatically by OE. But because this handle is an OUTPUT parameter this will delete operation will be postponed until the scope of the method will end.

Second delete object htt no-error statement is meant to delete the dynamic temp-table created by user. Which also should be postponed to the end of the procedure.

FWD¶

In FWD we postponed the deletion of the OUTPUT parameter by setting the postponedDelete parameter in TempTableBuilder. And we get to the end of the procedure, we will delete only the dynamic temp-table which is present in the handle at that moment. And the dynamic temp-table is now sitting in the memory not being reference by anything.

Buffer / Temp-Table explicit deletion¶

define temp-table ttTest 
    no-undo
    field f1 as int
    field f2 as char.

define variable h1 as handle.
h1 = temp-table ttTest:handle.

delete object h1.

In my above example, I am trying to perform a explicit delete of static temp-table. However this can't actually be done for static temp-tables, only for dynamically created ones. The static temp-table will be deleted then the procedure will go out of scope. The delete of static temp-table is prevented in FWD by !implicitDeletion check.

Procedure explicit deletion¶

define temp-table ttTest 
    no-undo reference-only
    field f1 as int
    field f2 as char.

define variable h as handle.
define variable h1 as handle.

run test41.p persistent set h.
run meth1 in h (output table ttTest bind).

delete object h.
create ttTest.
ttTest.f1 = 4.

When I perform an explicit deletion of a persistent procedure, its resources are implicitly deleted (StaticTempTables , Buffers, etc). In my example after the explicit delete of the test41.p procedure handle. I can still access the bounded temp-table from test41.p.

Changes in current form passed initial conversion and regression tests on ETF.

Committed on 9545a, rev. 16473.

Fixed different error handling logic.

Committed on 9545a, rev. 16473.

- improved shouldRaiseBeforeError method logic, to handle FWD actor and parameter switch.
- added check for same table in OUTPUT dynamic table logic.

With this changes I don't get any more errors on tests/table/parameter.

Test run finished after 8855 ms
[        61 containers found      ]
[         0 containers skipped    ]
[        61 containers started    ]
[         0 containers aborted    ]
[        61 containers successful ]
[         0 containers failed     ]
[      1024 tests found           ]
[         0 tests skipped         ]
[      1024 tests started         ]
[         0 tests aborted         ]
[      1024 tests successful      ]
[         0 tests failed          ]

Rebased to trunk rev. 16588.

Eduard, please rebase again and I'll move to full review and testing.

Rebased to trunk rev. 16656.