FWD » Core Development » Database

In the 4GL, there is no concept of a DMO. From the beginning (in the 1980s), the 4GL has always directly mapped compile-time symbolic names into direct buffer references. Each of these buffers is specific to a table which has structure that defines the fields (their order, data type and extent). But the runtime access is very low level. I think they are directly accessing the in-memory representation of the disk layout of that table's record. From their perspective, there is no need to do anything different. It has compile-time safety, a simple optionally qualified symbolic reference to field data and is very fast. Presumeably, the runtime code that reads and writes fields implements the needed hooks for validation, flushing and triggers. Otherwise, it is just a fast access to the in-memory representation that itself may have been very quickly loaded via shared memory with the database. It is likely that the symbolic names don't exist in their post-compiled code. There is no need for such a layer. It is useful in the original source only and post-compile all field references can be directly mapped to the in-memory record representation.

In FWD, early on we mapped record access into a DMO concept. The initial ORM was Hibernate but Hibernate just doesn't match the 4GL behavior at all. To effectively use Hibernate, one needs a hand coded app that uses more modern approaches to things like transaction scoping and locking. With Hibernate, we got a runtime DMO implementation that was utilized directly from the converted code. Using proxies, we have hooks for our validation, flushing and triggers. At a minimum we have a runtime ORM layer between the converted code and the low level access to the BDT data from a field. I won't dispute that there is value to the refactoring and usage from the converted code.

However, I do think that we have over-used the DMOs in a way that adds a significant performance penalty in addition to adding a level of type-awareness which the 4GL does not have. I'm referring to our direct usage of DMOs in the area of inter-program linkage. By inter-program linkage, I mean buffer parameter passing (in all its varied forms and top-level code constructs) as well as other areas like datasets, buffer-copy, buffer-compare and so forth. The 4GL simply has no equivalent to the typing that we force into these cases.

In the 4GL, when code (4GL code like a method body or implicit code like buffer-copy or dataset fill) at runtime references a buffer it does so without any names, just directly using the structure itself. That is why they can pass buffers that are structurally the same but have complete different names (and even have differences in attributes/settings that are not structural). The names and any attributes/settings that are not structural are all things that are local to a program and have no association with the low level buffer that is passed around/referenced. The local code "sees" its own names and settings for the record while directly accessing the record itself that comes from another program with completely different names and settings, but the same structure.

In FWD, we naturally used the DMOs directly and decided to pass those around since they represent the record from the converted 4GL code perspective. But this forced a functional mismatch. Our DMOs don't have a concept of a lightweight structural record that can be passed around and accessed directly. Instead, we have to "tie ourselves in knots" to force DMOs into this parameter/remote reference mode. We add a layer of extra proxying to map the local DMO to the passed/referenced DMO. This proxy of a proxy adds a cost from a performance perspective. Not only do we take the hit on the extra invocation handler for every access to the proxy but we also have to take the hit to generate the proxy during the call (or presumeably during some level of setup in other implicit runtime code like buffer-copy or dataset processing).

With #9937, we take this even further adding a 3rd level of proxying to an already overcomplicated approach. This madness must stop. I'm not suggesting that we delay #9937. Nor am I suggesting that we eliminate DMOs entirely. But I am suggesting that we need to re-evaluate our approach to cross-program access to buffers. We should not use DMOs for parameters/buffer-copy/buffer-compare/datasets... instead, we should pass a structural representation of the record or a reference to a structural representation of the record.

The idea:

• DMOs provide a named representation of a record to a specific program.
• Any inter-program usage of a record should not be passed as a DMO, but instead we pass a lightweight "struct" (I'm loosely using the C/C++ concept of a simple data structure that has type fields but which does not have any heavyweight logic or settings. In C/C++, one directly accesses the memory for a field in the struct rather than going through any kind of more complicated bean-like getter/setter code).
• This struct (structural representation of the record or a reference to a structural representation of the record) is probably not the same thing as the BaseRecord that is used internally in a DMO. For one thing, the BaseRecord is implemented as an array rather than having a structural component. But we also would not want to expose the BaseRecord to the converted code.
• The struct would NOT have any customer-specific names associated with the fields. Instead, every struct with a unique signature would have well-known generated names that would not need any conversion and would not be referenced by converted code.
• The point here is not to directly reference the struct fields, it is to pass the minimal record state through which can then be shoved inside the local DMO so that the code can access it as if it was that DMO. Which is exactly what the 4GL is doing, as far as I can tell.
• With a struct we can get a level of compile-time type safety but we would do so at the same level as the 4GL is doing it. But we would not add all the other DMO "heaviness" to the type-safety. We should be able to eliminate both the 2-level and 3-level proxying of proxies.
• The local code gets all the goodness of a DMO but we just never pass DMOs around.
• The result is closer to the 4GL behavior by default, there is less complexity, it is less fragile and it must be faster.
• This also has big benefits at conversion time since temp-DMOs would never be referenced across programs unless they were exactly the same. Some of the app by app nonsense would be eliminated here. And I also presume that the overall inheritance hierarchy of DMOs can be a little simpler as well.

There are some questions to answer:

• To what level do we need to implement true compile-time type safety for these cases? The easiest thing to do here would be to just pass a reference to the BaseRecord/TempRecord or an opaque container that has a reference to the BaseRecord/TempRecord. But if the 4GL really has some level of compile-time checks, then we probably should also have those checks.
• Assuming we do need compile-time type safety, do we generate a fake struct class (one that has the fields of the right type in the right order and with the right extents) which will naturally provide a named class that can be passed but which would not actually have any data in the data members themselves? We don't want to spend time copying the BaseRecord/TempRecord into a struct and then back to the BaseRecord/TempRecord on the other side of the call. That would be a waste and is not needed. The struct could subclass from something that has a reference to the BaseRecord/TempRecord and the DMO could then just reference that BaseRecord/TempRecord directly. I don't love the idea of a struct whose structure is never used but it would be the a small price to pay to avoid all the problems of the current implementation.

Thoughts?

Greg,
I am not sure I got all detail, in spite of being focused on the 1st not for some time now. It's clear we will not be able to match the C/C++ performance from 4GL if the things are as you described (writing the records directly to a memory mapped 'fragment' of the table), and the current trend is not optimal.

What I understand is that each procedure has its version of a table, including the db schema and the additional 'decorations' which should not be taken into consideration when checking the compatibility of two table/buffers. Now we have a strict relation Buffer(Impl) <-> BaseRecord of one-to-one. The former is constructed around the DMO, the latter is just a container for data and except for the data type it holds it should have no idea what are the name of the table/fields and other attributes.

For parameters, the buffers should not construct their own BaseRecord, but use the 'pointer' to one already created by the caller, and access it using the local interface, even if it can differ (a bit even in the schema definition) from the caller. The BIND option should make the association 'persistent'.

The case of reference-only buffers is most interesting in that we are creating a BaseRecord for them, even if we should not.

There is another thing, related to this (proxy). In 4GL there are two syntaxes for accessing two different objects: <buf-name>.<field> to access the data in the field and buffer <buf-name>:<attribute> to access the attribute (like format). In FWD we merged into a single object so we have bufName.getField() for the former and bufName.attribute() for the latter, using the convention that the methods for getters and setters do not honour the bean naming schema. IMHO, we should also have two distinct entities in the converted code, with a clear separation between them: the record buffer which should expose the DMO implementing class and the buffer with the rest of attributes. I am thinking about this for some time, but I do not have a clear decision. Please ignore this paragraph if you consider it is not on topic of the task.

For parameters, the buffers should not construct their own BaseRecord, but use the 'pointer' to one already created by the caller, and access it using the local interface, even if it can differ (a bit even in the schema definition) from the caller. The BIND option should make the association 'persistent'.

Exactly.

The case of reference-only buffers is most interesting in that we are creating a BaseRecord for them, even if we should not.

Nice. Sounds like an additional optimization.

There is another thing, related to this (proxy). In 4GL there are two syntaxes for accessing two different objects: <buf-name>.<field> to access the data in the field and buffer <buf-name>:<attribute> to access the attribute (like format). In FWD we merged into a single object so we have bufName.getField() for the former and bufName.attribute() for the latter, using the convention that the methods for getters and setters do not honour the bean naming schema. IMHO, we should also have two distinct entities in the converted code, with a clear separation between them: the record buffer which should expose the DMO implementing class and the buffer with the rest of attributes. I am thinking about this for some time, but I do not have a clear decision. Please ignore this paragraph if you consider it is not on topic of the task.

Interesting. It does sound like an area that needs attention. It may not be directly related since it is all about the local usage. On the other hand, the attributes don't need a proxy, they could be accessed through a statically defined instance. We may want to create a separate task for that.