FWD » Core Development » Database

09/09/2020

09:37 AM Feature #4011: database/persistence layer performance improvements

We don't want to move to 4.1 before we release 4.0. On the other hand, this is a very major change. Until 4.0 is re... Greg Shah

09:17 AM Feature #4011: database/persistence layer performance improvements

Eric Faulhaber wrote:
> 4011e was merged to trunk and committed as revision 11348.
Was there a FWD revision numbe... Roger Borrello

04:42 AM Support #4701 (Review): try to improve H2 transaction commit performance

This task in finished. Waiting to rebase 4701b and eventually merge to trunk/3821c. However, I suggest to keep #4843 ... Adrian Lungu

09/08/2020

06:23 PM Feature #4011: database/persistence layer performance improvements

Trunk revisions 11347.1.316 and 11347.1.318 implemented a mechanism to keep track of the versions of persistent DMOs ... Eric Faulhaber

06:16 PM Feature #4011: database/persistence layer performance improvements

4011e was merged to trunk and committed as revision 11348. Eric Faulhaber

05:14 PM Feature #2156 (Closed): datetime-tz and hibernate

Please make the change in 3821c. I'm going to close this task.
> I say probably because the new ORM has not yet b... Greg Shah

05:11 PM Feature #2156: datetime-tz and hibernate

Greg Shah wrote:
> @datetime-tz@ field persistence is fully supported now?
Yes, that was one of the aspects we bene... Ovidiu Maxiniuc

04:56 PM Feature #2156: datetime-tz and hibernate

@datetime-tz@ field persistence is fully supported now? Do we need to change anything in our gap marking? Greg Shah

04:48 PM Feature #2156: datetime-tz and hibernate

Yes.
It is rather obsolete as we have now a much simpler access to DTZ fields using the new ORM. Ovidiu Maxiniuc

04:44 PM Feature #2156: datetime-tz and hibernate

4011e was merged to trunk as rev 11348.
Eric/Ovidiu: Can we close this task? Greg Shah

03:35 PM Support #4701: try to improve H2 transaction commit performance

Is there anything more to work on in this task or do we just need to get it rebased for merge to trunk (or merge into... Greg Shah

08/21/2020

03:07 AM Feature #3814: more schema metadata

Greg Shah wrote:
> Good job. I think we should pause the work there. Is it safe to include in the branch pending t... Igor Skornyakov

09:03 PM Feature #3814: more schema metadata

Good job. I think we should pause the work there. Is it safe to include in the branch pending the final changes? Greg Shah

08/20/2020

05:42 PM Feature #3814: more schema metadata

@UserTableStatUpdater@ core functionality implemented. The exposed public methods are:
* getTableStats(integer con... Igor Skornyakov

08/15/2020

08:06 AM Support #4701: try to improve H2 transaction commit performance

I've done a final profiling of FWD in regard with @h2_performance/perf-create.p@. I've added to the testcase 4 config... Adrian Lungu

08/14/2020

06:09 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Unless there is a way to write application logic to have some dependency on this interval, I... Igor Skornyakov

06:01 PM Feature #3814: more schema metadata

Unless there is a way to write application logic to have some dependency on this interval, I don't see any advantage ... Eric Faulhaber

02:13 PM Feature #3814: more schema metadata

The situation with @_UserTableStat@ VST looks like following:
# For every connection 4GL creates 50 slots with count... Igor Skornyakov

08:13 AM Feature #3814: more schema metadata

> you mean by multiple clients
You've been saying that it shows the stats for multuple users. Each user would be ... Greg Shah

07:27 AM Feature #3814: more schema metadata

Greg Shah wrote:
> Run multuple clients at different times and see if the data for earlier closed clients is seen by... Igor Skornyakov

06:08 AM Feature #3814: more schema metadata

Run multuple clients at different times and see if the data for earlier closed clients is seen by later clients. Greg Shah

03:29 AM Feature #3814: more schema metadata

Greg Shah wrote:
> > My understanding is that VSTs are created at the connect to the database and are not cleared un... Igor Skornyakov

08/13/2020

07:28 PM Feature #3814: more schema metadata

> My understanding is that VSTs are created at the connect to the database and are not cleared until the connection i... Greg Shah

06:45 PM Feature #3814: more schema metadata

My understanding is that VSTs are created at the connect to the database and are not cleared until the connection is ... Igor Skornyakov

06:41 PM Feature #3814: more schema metadata

> The data from the contexts that are closed before the start of the aggregation will be lost.
I would have though... Greg Shah

06:39 PM Feature #3814: more schema metadata

> Note that simple long types will do. There is no need to use AtomicLong, since these variables are counting operati... Greg Shah

06:27 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Igor Skornyakov wrote:
>
> > I'm not talking about database query. Me suggestion is have ... Igor Skornyakov

06:12 PM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> I'm not talking about database query. Me suggestion is have the map as a field in the @Da... Eric Faulhaber

05:28 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Igor Skornyakov wrote:
> > And keeping this map in the @Database@ instance will greatly sim... Igor Skornyakov

05:15 PM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> And keeping this map in the @Database@ instance will greatly simplify the aggregation.
P... Eric Faulhaber

04:51 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
.> Yes, a map lookup at @RecordBuffer@ creation (or initialization) seems an acceptable level ... Igor Skornyakov

04:44 PM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> To ensure that the @TableStats@ should be one per table I think that it is better to hold t... Eric Faulhaber

04:36 PM Feature #3814: more schema metadata

BTW: I do not think that the overhead will be substantially increased if the map will be not context-local but a fiel... Igor Skornyakov

04:31 PM Feature #3814: more schema metadata

Greg Shah wrote:
> [...]
>
> There would be one of these for each table ever accessed. At the first time a table... Igor Skornyakov

04:19 PM Feature #3814: more schema metadata

... Greg Shah

04:12 PM Feature #3814: more schema metadata

Greg Shah wrote:
> This still seems expensive. We don't want to take a map lookup hit in order to increment a count... Igor Skornyakov

04:06 PM Feature #3814: more schema metadata

> I understand that it should be a map of arrays of counters. If it will be ConcurrentMap with Atomic values no addit... Greg Shah

03:17 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> I assume the entries for a particular session should be removed when that session ends. Othe... Igor Skornyakov

03:12 PM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> I understand that it should be a map of arrays of counters. If it will be @ConcurrentMap@... Eric Faulhaber

02:59 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Greg Shah wrote:
> > That doc page suggests @_usertablestat-num@ is the table number. Perh... Igor Skornyakov

02:52 PM Feature #3814: more schema metadata

Greg Shah wrote:
> That doc page suggests @_usertablestat-num@ is the table number. Perhaps this is some kind of un... Eric Faulhaber

02:43 PM Feature #3814: more schema metadata

Greg Shah wrote:
> That doc page suggests @_usertablestat-num@ is the table number. Perhaps this is some kind of un... Igor Skornyakov

02:33 PM Feature #3814: more schema metadata

That doc page suggests @_usertablestat-num@ is the table number. Perhaps this is some kind of unique number associat... Greg Shah

02:32 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Eric Faulhaber wrote:
> > What is the @_usertablestat-num@ field about?
>
> Is this a "... Igor Skornyakov

02:31 PM Feature #3814: more schema metadata

Eric,
Yes, the table effectively holds counters, 4 for 4 different CRUD operations per user/table combination. At le... Igor Skornyakov

02:27 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> What is the @_usertablestat-num@ field about?
Is this a "foreign key" of sorts to the @_... Eric Faulhaber

02:21 PM Feature #3814: more schema metadata

Igor, to best answer you, please help me understand the purpose of these fields:... Eric Faulhaber

01:50 PM Feature #3814: more schema metadata

It is also possible to have the updater, but it will not try to persist statistics constantly but keep it in the in-m... Igor Skornyakov

01:34 PM Feature #3814: more schema metadata

Greg Shah wrote:
> Even if the overhead is minimized, the current approach wastes CPU cycles for a feature that is a... Igor Skornyakov

01:16 PM Feature #3814: more schema metadata

Even if the overhead is minimized, the current approach wastes CPU cycles for a feature that is almost never used.
... Greg Shah

12:07 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> I don't know enough about the information you need to collect to answer this. This method ge... Igor Skornyakov

11:48 AM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> I'm working now on the @_usertablestat@ VST support. This VST holds information about the C... Eric Faulhaber

06:44 AM Feature #3814: more schema metadata

Greg Shah wrote:
> Is the @_usertablestat@ contents specific to the current user's session? If so, then it is impor... Igor Skornyakov

06:29 AM Feature #3814: more schema metadata

Is the @_usertablestat@ contents specific to the current user's session? If so, then it is important to capture and ... Greg Shah

03:19 PM Bug #4071: FWD server stops allowing new web client logins after abend

Greg Shah wrote:
> > for interactive sessions, it should interrupt and wait for the Conversation thread to finish an... Constantin Asofiei

08:07 AM Bug #4071: FWD server stops allowing new web client logins after abend

> for interactive sessions, it should interrupt and wait for the Conversation thread to finish and cleanup the contex... Greg Shah

05:05 AM Bug #4071: FWD server stops allowing new web client logins after abend

And there's one more case to consider for the 'unplanned' socket termination, for virtual sessions: if there is an ac... Constantin Asofiei

01:05 PM Feature #3813: misc DB features part deux

OK, then lets wait until 4011b is in trunk (and 3821c rebased) before we finish this last item. Greg Shah

01:01 PM Feature #3813: misc DB features part deux

Greg Shah wrote:
> So there are no pending properties changes in 3821c?
It looks like that. Please note that it w... Igor Skornyakov

12:55 PM Feature #3813: misc DB features part deux

So there are no pending properties changes in 3821c? Greg Shah

12:09 PM Feature #3813: misc DB features part deux

Greg Shah wrote:
> I understand there will be some edits. Am I correct in my understanding that there are already o... Igor Skornyakov

12:04 PM Feature #3813: misc DB features part deux

I understand there will be some edits. Am I correct in my understanding that there are already other such edits that... Greg Shah

11:56 AM Feature #3813: misc DB features part deux

Greg Shah wrote:
> We have other such property changes in 3821c, right? If you finish @BUFFER-FIELD:DECIMALS@ in 38... Igor Skornyakov

10:59 AM Feature #3813: misc DB features part deux

> According to #3813-162, #3813-163, the only remaining issue is Built-In Function Usage in WHERE Clauses but there i... Greg Shah

08:34 AM Feature #3813: misc DB features part deux

Greg Shah wrote:
> What is left open in this task?
According to #3813-(162,163), the only remaining issue is @Bui... Igor Skornyakov

08:12 AM Feature #3813: misc DB features part deux

What is left open in this task? Greg Shah

10:52 AM Feature #4613: add missing support for database user defined functions (UDF) if they are supporte...

Some items mentioned in #3813 should be handled as part of this task:
> +Built-In Function Usage in WHERE Clauses+... Greg Shah

07:59 AM Support #4701: try to improve H2 transaction commit performance

Committed 4701a rev.11625 - added switch between shared and private temporary database. Adrian Lungu

08/12/2020

05:59 PM Bug #4071: FWD server stops allowing new web client logins after abend

I agree that all context cleanup for interactive sessions must be on the main Conversation thread.
For the virtual... Constantin Asofiei

09:06 AM Bug #4071: FWD server stops allowing new web client logins after abend

I think we also need to exclude context-cleanup from happening on @RelatedThread@ instances. Such threads share the ... Greg Shah

09:01 AM Bug #4071: FWD server stops allowing new web client logins after abend

I think we need to consider if there is ever a scenario in which context cleanup is needed for reader/writer threads.... Greg Shah

03:27 PM Feature #3814: more schema metadata

I'm working now on the @_usertablestat@ VST support. This VST holds information about the CRUD operations. The corres... Igor Skornyakov

06:58 AM Feature #3814: more schema metadata

Greg Shah wrote:
> When will you get to a good stopping point?
Most likely today. Tomorrow at the latest.
Igor Skornyakov

06:35 AM Feature #3814: more schema metadata

> Maybe it makes send to do it only for the new ORM.
Yes, probably so. Get it to the point where everything is re... Greg Shah

06:23 AM Feature #3814: more schema metadata

Greg Shah wrote:
> How quickly do you estimate that can you complete this work?
I plan to finish with the updater... Igor Skornyakov

06:01 AM Feature #3814: more schema metadata

How quickly do you estimate that can you complete this work? Greg Shah

08:46 AM Support #4701: try to improve H2 transaction commit performance

Created branch 4701a and committed the changes regarding per-user temporary databases rev. 11624. Please review.
... Adrian Lungu

08/11/2020

05:01 PM Feature #3814: more schema metadata

Yes, if that is how it works in the 4GL for a non-multi-tenant case. Greg Shah

04:47 PM Feature #3814: more schema metadata

Greg Shah wrote:
> > I do not know if we already have any support for the notion of tenant.
>
> No, we do not. W... Igor Skornyakov

04:39 PM Feature #3814: more schema metadata

> I do not know if we already have any support for the notion of tenant.
No, we do not. We are not currently plan... Greg Shah

02:30 PM Feature #3814: more schema metadata

As I can see the following fields are already supported, at least the "Minimal*" interfaces in the @ConnectTableUpdat... Igor Skornyakov