Project

General

Profile

Commit Callback (FWD Extension)

This is part of FWD extension. It was developed in Redmine task #11383

Overview

The COMMIT callback is an FWD-only extension of the 4GL callback mechanism. It notifies application code after the records affected by a transaction have been physically committed to the database, so that an external consumer (a message broker, a search index, a replication feed) can read the committed state instead of an in-flight, possibly-to-be-undone image.

This addresses the problem that a REPLICATION-WRITE trigger will publish a record to an external consumer while the FWD transaction is still open. From the point of view of the consumer, a SQL query at that specific time against the same row returns the pre-transaction data (or an empty result set, if the record was created during the transaction).

The impact on conversion was kept to a minimum: all changes are runtime-only. The existing SET-CALLBACK-PROCEDURE and SET-CALLBACK 4GL methods were extended to accept an additional value for their first parameter (the event name) and an additional parameter that selects batch mode.

The callback differs from the existing buffer and DATASET callbacks (BEFORE-FILL, ROW-CREATE, AFTER-ROW-FILL, ...) in three ways:

  • it is registered on a buffer of a permanent table, as opposed to a temp-table or dataset buffer;
  • it is registered per user context, not per buffer instance — any buffer of the table may be used to register it, and the registration outlives that buffer;
  • by design, it cannot be raised by the application; it is raised only by the persistence layer, at commit time.

Registration

The callback is configured with the existing SET-CALLBACK-PROCEDURE and SET-CALLBACK buffer-handle methods, using the new event name "COMMIT". No conversion changes are required: these handle methods are mapped to the CallbackProcedure interface in rules/convert/methods_attributes.rules, and the event name is an ordinary string argument. New overloaded methods were added to the interface to accommodate the additional parameter for batch mode; the batch size is declared as NumberType, so any numeric 4GL expression (INTEGER, INT64, DECIMAL) is accepted and converted to an integer.

Two delivery modes exist. The mode is selected by the presence and the value of the batch-size argument.

Batch size k Delivery
absent, unknown, or negative per-record: one invocation per record, buffer handle signature
0 batch: one single invocation per table, carrying every affected record
1 or more batch: ceil(N / k) invocations, each carrying at most k records

Note that k equal to 1 is still batch mode: the callback is invoked with the JSON signature and a one-element extent, not with a buffer handle. Per-record delivery is requested by omitting the argument, or by passing a negative or unknown value.

Per-record mode (buffer handle)

Registering without a batch size — or with a negative or unknown one — selects per-record delivery. The callback is invoked once per committed record and receives a handle to a read-only buffer with that record loaded.

DEFINE VARIABLE hBook AS HANDLE NO-UNDO.

CREATE BUFFER hBook FOR TABLE "book".

/* context defaults to THIS-PROCEDURE */
hBook:SET-CALLBACK-PROCEDURE("COMMIT", "book-committed").

/* explicit context: the internal procedure is looked up in hCtx */
hBook:SET-CALLBACK-PROCEDURE("COMMIT", "book-committed", hCtx).

Batch mode (longchar in JSON format)

Registering with a batch size of 0 or more selects batch delivery. The affected records of the table are JSON-serialized individually and passed as a LONGCHAR EXTENT parameter. A positive value causes the callback to receive at most that many records per invocation, so the callback is invoked ceil(N / k) times per table, where N is the number of affected records of that table and k the batch size. A batch size of 0 requests a single invocation carrying all tracked records of the table.

DEFINE VARIABLE hCust AS HANDLE NO-UNDO.

CREATE BUFFER hCust FOR TABLE "customer".

/* at most 500 customer records per invocation */
hCust:SET-CALLBACK-PROCEDURE("COMMIT", "customer-committed", 500).

/* all affected customer records in one single invocation */
hCust:SET-CALLBACK-PROCEDURE("COMMIT", "customer-committed", 0).

Batching bounds the number of records materialized in memory at once, and bounds the size of the LONGCHAR array handed to the 4GL. A batch size of 0 gives up both bounds in exchange for a single notification per table.

Class methods as targets

SET-CALLBACK accepts an object reference, so a method of a class instance can serve as the callback target. Both delivery modes are available.

/* per-record mode, method of the current class instance */
hBook:SET-CALLBACK("COMMIT", "BookCommitted", THIS-OBJECT).

/* batch mode, method of the current class instance */
hBook:SET-CALLBACK("COMMIT", "BookCommitted", THIS-OBJECT, 500).

/* batch mode, internal procedure in an explicit procedure context */
hBook:SET-CALLBACK("COMMIT", "book-committed", hCtx, 500).

When the context argument is unknown, the target defaults to THIS-OBJECT if the registration happens inside a class, and to THIS-PROCEDURE otherwise.

Argument summary

Form Optional arguments Mode
hBuf:SET-CALLBACK-PROCEDURE("COMMIT", routine) absent per-record
hBuf:SET-CALLBACK-PROCEDURE("COMMIT", routine, hCtx) HANDLE per-record
hBuf:SET-CALLBACK-PROCEDURE("COMMIT", routine, k) numeric per k, see above
hBuf:SET-CALLBACK("COMMIT", method, THIS-OBJECT) Progress.Lang.Object per-record
hBuf:SET-CALLBACK("COMMIT", method, THIS-OBJECT, k) Progress.Lang.Object + numeric per k, see above
hBuf:SET-CALLBACK("COMMIT", routine, hCtx, k) HANDLE + numeric per k, see above

The third argument is overloaded on type: a HANDLE is the execution context, an object reference is the owning class instance, and a numeric value is the batch size.

Conversion notes

The two methods are emitted as instance calls on the CallbackProcedure interface, so the number and the types of the arguments are resolved when the generated Java is compiled; no rule had to be taught about the new event name or the new argument. One detail is worth knowing when writing the 4GL: com/goldencode/p2j/convert/SignatureHelper.java describes SET-CALLBACK-PROCEDURE as two mandatory parameters out of three, the third typed as HANDLE, and SET-CALLBACK likewise with the third typed as BaseDataType. Those descriptors are consulted only when an argument's type must be inferred from its context, so a literal or a typed numeric variable in the batch-size position converts correctly, as does the fourth argument of the SET-CALLBACK forms. Prefer a literal or a typed numeric variable there, rather than an expression whose type conversion has to infer.

Removing a callback

Passing the empty string as the routine name removes the registration for that table.

hBook:SET-CALLBACK-PROCEDURE("COMMIT", "").

Passing the unknown value (?) as the routine name is a no-op and returns TRUE. Registering a second callback for the same table replaces the first: there is exactly one COMMIT callback per table per user context.

Callback signatures

The signature of the target routine is determined by the delivery mode. A mismatch is not detected at registration time — only when the event is raised, and even then it does not disturb the committing code: the failure is recorded as error -109 and the remaining records of that table are skipped. See Restrictions and error conditions below.

Per-record mode

PROCEDURE book-committed:
   DEFINE INPUT PARAMETER hBuf AS HANDLE NO-UNDO.

   /* hBuf is a read-only buffer of 'book' with one committed record loaded */
   MESSAGE "committed:" hBuf:BUFFER-FIELD("title"):BUFFER-VALUE.
END PROCEDURE.

The buffer passed in is created by the runtime, is marked read-only, and is released after the last record of the table has been reported. The temporary record is dropped from the buffer between invocations.

Batch mode

PROCEDURE customer-committed:
   DEFINE INPUT PARAMETER table-name AS CHARACTER       NO-UNDO.
   DEFINE INPUT PARAMETER jsons      AS LONGCHAR EXTENT NO-UNDO.

   DEFINE VARIABLE i AS INTEGER NO-UNDO.

   DO i = 1 TO EXTENT(jsons):
      RUN publish-to-broker (INPUT table-name, INPUT jsons[i]).
   END.
END PROCEDURE.

The payload parameter must be declared as an indeterminate EXTENT; the runtime sizes it to the number of records in the current batch, which is at most the configured batch size, or the full set when the batch size is 0. LONGCHAR is required — a serialized record can exceed the 32K limit of CHARACTER (see #11383-34).

The table name is passed as the first parameter so that one generic callback can serve several tables. The same routine may be registered for book and for customer and dispatch on table-name.

JSON encoding

Each record is serialized to a single-line JSON object. The _rowid pseudo-field is emitted first, followed by every field of the table in DMO property order, keyed by its legacy (4GL) name.

{"_rowid":"0x0000000000000406", "book-id":1001, "title":"Dune", "author":"Frank Herbert", "price":24.5, "published":"1965-06-01", "in-stock":true, "tags":["scifi","classic",null]}

This format was chosen for its flexibility:

  • one record is one self-contained JSON document, shorter and cheaper to parse than a single document wrapping the whole set, and the PK — which uniquely identifies a record in any table — is intentionally the first node encountered;
  • if the set of records has to be forwarded to a public API and one call per record is expensive, the individual documents can simply be concatenated into a JSON array and sent as a single request, where the consumer supports such a syntax.

Value mapping:

4GL type JSON
INTEGER, INT64 number
DECIMAL number, always with a fractional part (24.0, not 24)
CHARACTER, LONGCHAR string
LOGICAL true / false
DATE, DATETIME, DATETIME-TZ ISO-8601 string
RAW, BLOB base64 string
unknown value null
EXTENT field JSON array, one element per extent slot

Two notes on the encoding:

  • _rowid is emitted as a string in 0x%016x form (16 hex digits, 0x prefix). This deviates from the 4GL JSON encoding of the ROWID data type and is deliberate — it keeps the value usable as an opaque key on the consumer side;
  • String values are escaped per RFC 8259. The common control characters use their compact escapes (\t, \n, \r, \b); every other control character is emitted in \u00XX form.

Scalar values are produced by the same code path as the native 4GL JSON support, so the output is consistent with WRITE-JSON for the field types listed above.

When the event is raised

The event is raised by the persistence layer immediately after the JDBC COMMIT of the session's connection succeeds, and before the transaction bookkeeping is torn down. The callback therefore runs outside the transaction: the records it reports are durable, and a query issued from the callback (or by an external consumer it notifies) sees them.

Only records that are still present at the moment of the commit are reported:

  • records created or modified inside the transaction are collected as they are flushed;
  • records collected inside a sub-transaction that is rolled back are discarded, so an UNDO-ed block reports nothing;
  • records collected inside a sub-transaction that ends normally are rolled up into the enclosing block;
  • a full transaction rollback discards everything collected and the event is not raised.

Whether the transaction commits or rolls back, the collected data is released at the end of it, leaving the structure ready for the next transaction.

A record modified several times inside one transaction is reported once.

Collection is armed at the start of the transaction from the set of tables that have a callback registered at that moment. A callback cannot be registered while a transaction is active, even if no records were touched yet.

Introspection

Method Behaviour for "COMMIT"
hBuf:GET-CALLBACK-PROC-NAME("COMMIT") the registered routine name, or the unknown value if none
hBuf:GET-CALLBACK-PROC-CONTEXT("COMMIT") the context procedure handle, or an invalid handle if none
hBuf:APPLY-CALLBACK("COMMIT") always FALSE — see below

APPLY-CALLBACK is deliberately a no-op for this event. The event's contract is "these records are committed"; raising it by hand, either mid-transaction (where the records' fate is undecided) or outside a transaction (where there is nothing to commit), would violate that contract.

Restrictions and error conditions

  • Permanent tables only. Registering on a temp-table buffer or on a meta-schema buffer returns FALSE and does nothing. Temp-tables are not transactional in the sense this event describes;
  • Permanent, non-temporary databases only. Collection is armed only for a session whose database is a primary permanent database;
  • No registration inside a transaction. Attempting to register while a transaction is active raises an error and returns FALSE, because the set of monitored tables is fixed when the transaction starts. Register during application start-up, or at least outside any transaction block;
  • Per user context. The registry is context-local. A registration made by one session is not visible to another, and it is discarded when the context ends;
  • One callback per table. Re-registering replaces the previous entry. A single callback can serve several tables if it is written generically enough (see the example below), but a callback can equally be written for one specific table. The latter is usually cleaner than testing the table name at the moment the callback is fired;
  • Errors inside the callback are suppressed. The error handler is put into silent mode for the duration of the invocation. If the target routine cannot be resolved, if it raises ERROR, or if the record cannot be re-read, error -109 ("Failed to invoke COMMIT CALLBACK for table") is recorded and the remaining records of that table are skipped. The commit itself is already durable and is not affected, and the committing code does not see a condition;
  • Deletes are not reported. Only records that were created or updated are collected. A record deleted inside the transaction produces no notification.

Examples

Publishing two tables through one generic callback

DEFINE VARIABLE hBook AS HANDLE NO-UNDO.
DEFINE VARIABLE hCust AS HANDLE NO-UNDO.

CREATE BUFFER hBook FOR TABLE "book".
CREATE BUFFER hCust FOR TABLE "customer".

hBook:SET-CALLBACK-PROCEDURE("COMMIT", "record-committed", 200).
hCust:SET-CALLBACK-PROCEDURE("COMMIT", "record-committed", 200).

/* the buffers are only needed for registration; the callbacks survive them */
DELETE OBJECT hBook.
DELETE OBJECT hCust.

PROCEDURE record-committed:
   DEFINE INPUT PARAMETER table-name AS CHARACTER       NO-UNDO.
   DEFINE INPUT PARAMETER jsons      AS LONGCHAR EXTENT NO-UNDO.

   DEFINE VARIABLE i AS INTEGER NO-UNDO.

   DO i = 1 TO EXTENT(jsons):
      RUN publish-to-broker (INPUT table-name, INPUT jsons[i]).
   END.
END PROCEDURE.

A transaction that inserts 3 books and updates 1 customer results in one call with table-name = "book" and EXTENT(jsons) = 3, and one call with table-name = "customer" and EXTENT(jsons) = 1. Registering with a batch size of 0 instead of 200 gives exactly the same two calls, whatever the number of affected records.

Per-record mode with a static buffer

The registration needs any buffer of the table, including a static one.

DEFINE BUFFER b-book FOR book.

BUFFER b-book:SET-CALLBACK-PROCEDURE("COMMIT", "book-committed").

PROCEDURE book-committed:
   DEFINE INPUT PARAMETER hBuf AS HANDLE NO-UNDO.

   MESSAGE "book committed, rowid:" STRING(hBuf:ROWID)
           "title:" hBuf:BUFFER-FIELD("title"):BUFFER-VALUE
      VIEW-AS ALERT-BOX.
END PROCEDURE.

Sub-transaction rollback

DO TRANSACTION:
   CREATE book.
   ASSIGN book.title = "Kept".

   DO TRANSACTION ON ERROR UNDO, LEAVE:
      CREATE book.
      ASSIGN book.title = "Discarded".
      UNDO, LEAVE.
   END.
END.

The callback is invoked for the "Kept" record only. The record created in the inner block never reaches the commit, so it is not reported.

Turning the notification off

DEFINE BUFFER b-cust FOR customer.

/* stop monitoring 'customer'; 'book' registration, if any, is unaffected */
BUFFER b-cust:SET-CALLBACK-PROCEDURE("COMMIT", "").

Alternative: a master (global) COMMIT callback

An implementation of a similar feature, one that would serve all tables from all databases, was attempted. It consists of a registration on the procedure handle rather than on a buffer handle. As for individual tables, it reuses the "COMMIT" event name and the same SET-CALLBACK-PROCEDURE / SET-CALLBACK methods, dispatching to BufferImpl.setCommitCallback(...) static overloads that store a single CallbackData in a context-local slot:

/* registers a single, application-wide COMMIT callback */
THIS-PROCEDURE:SET-CALLBACK-PROCEDURE("COMMIT", "tx-committed", 500).
THIS-PROCEDURE:SET-CALLBACK("COMMIT", "tx-committed", hCtx, 500).
THIS-PROCEDURE:SET-CALLBACK("COMMIT", "TxCommitted", THIS-OBJECT).

The motivation is stated in #11383-35 — a single global callback receiving one payload describing all changes of the full transaction — together with the scaling caveat raised in #11383-33: the per-table form does not scale to an application that monitors tens or hundreds of tables.

Current state: work in progress. This is a prototype, built to understand the full advantages of the approach before committing to it. Registration and introspection are implemented, but the stored callback is not yet consumed by the firing path, so a global-only registration does not raise any event. The registration problem is solved by anchoring on the procedure context, but that exposes a different one: a transaction spans all databases, so the affected records have to be collected from every active Session instance of the current user context and, at the moment of execution, serialised from all of those instances into the single parameter of the master callback.

Introspection is already routed to the master registration, through BufferImpl.getCommitCallbackProcName() and BufferImpl.getCommitCallbackProcContext(), and behaves exactly as the buffer form does for a per-table registration:

Method Behaviour for "COMMIT"
THIS-PROCEDURE:GET-CALLBACK-PROC-NAME("COMMIT") the registered routine name, or the unknown value if none
THIS-PROCEDURE:GET-CALLBACK-PROC-CONTEXT("COMMIT") the context procedure handle, or an invalid handle if none
THIS-PROCEDURE:APPLY-CALLBACK("COMMIT") always FALSE, as on the buffer form

The procedure anchor also ties the registration to the lifetime of the external program that owns it: when that program is deleted the callback is deregistered, and the program must therefore not be deleted while a full transaction is active. Anchoring on SESSION instead would avoid this, but the members of the SESSION system handle are enumerated one by one in rules/convert/methods_attributes.rules, so that variant would require conversion rule changes and would give up the runtime-only property of the current design.

Advantages of the global form

  • One registration for the whole application. No need to create a buffer per monitored table purely to register a callback, and no need to enumerate the monitored tables at all;
  • Registration is not tied to a buffer. It reads naturally from both procedures and class methods, and the anchor handle is the execution context that the callback needs anyway;
  • Fewer 4GL invocations. One call per transaction, instead of ceil(N / k) calls for each affected table. Once batches are activated for the global form too, this computation becomes more complicated;
  • It is the natural place for a transaction-scoped payload. A single event can carry a logical transaction id, which is what the original ticket asked for: the consumer needs to know that a set of records became durable together, not merely that individual rows changed;
  • It can be made correct for multi-database transactions. A single event must be raised after every database in the transaction has committed. That is impossible from Session.commit(), which runs once per database; the master-commit hook already provided by TransactionManager (registerTransactionCommit, raised at full-transaction level after all commitables) is the correct point.

Disadvantages of the global form

  • It gives up the cheap opt-out. The per-table form skips a record in Session.save() with a single map lookup that misses for unmonitored tables. A global callback must track every flushed record of every table, so the tracking cost is paid application-wide. Given the explicit performance requirement in the ticket (#11383-37, "my primary concern is to make this perform very fast"), this is the main risk. How much it matters depends on the customer's intent: if only one, two or a small set of tables are of interest, registering callbacks on those specific buffers is the optimal choice;
  • Unbounded payload. A transaction that touches 200,000 rows across 40 tables produces one payload. Bounding it by batching re-introduces multiple invocations and requires a sequence / final marker in the envelope, which weakens the "single event" contract;
  • The consumer must parse a nested structure. The per-table form hands the 4GL (table-name, jsons), which maps directly onto a broker API of the form send(topic, payload). A global payload has to carry the table discriminator inside the JSON, so the receiving code must re-parse the document to split it per table;
  • Lifetime bound to the anchoring program. The registration disappears when the external program that owns it is deleted, and that program must stay alive for the duration of every full transaction it is meant to observe;
  • Event-name overloading. Both forms use "COMMIT" and the same two methods; the scope of a registration is implied solely by the type of the receiving handle, so application code that passes the wrong handle registers the wrong scope silently;
  • A single slot per context. Only one global callback can exist; there is no composition if two independent subsystems both want transaction notifications.
  • Unwanted dependency. The ExternalProgramWrapper now calls static methods from persistence package (BufferImpl). This is a dependency I do not think we want.
  • Non-deterministic order. The order in which the records from tables are collected is given by iteration on registered committable objects.

Can the two coexist?

Basically, yes, and the current code is already structured for it: the registries are distinct (a DmoMeta-keyed map for the per-table form, a single-element array for the global form) and the registration anchors are distinct (buffer handle versus procedure handle). Neither can overwrite the other. Completing the global form does not require changing the per-table API or its semantics.

Introspection is already aligned between the two forms, so three things remain to be settled for them to coexist correctly:

  1. Arming. Collection must switch to "track every table" whenever a global callback is registered, while retaining the per-table filter when only per-table callbacks exist. This keeps the cheap path for applications that do not use the global form;
  2. A single firing point. Both forms should be raised from the same place, after all databases of the transaction have committed. Raising the per-table form from Session.commit() as it is done today means that in a two-database transaction, a notification for a table in the first database goes out while the second database is still uncommitted — the same class of problem the feature exists to solve;
  3. Order and duplication. A table that carries a per-table callback while a global callback is also registered is reported twice. The order should be defined (per-table first, then global) and the duplication either documented or made suppressible.

© 2026 Golden Code Development Corporation. ALL RIGHTS RESERVED.