FWD » Core Development » Database

08/21/2020

03:07 AM Feature #3814: more schema metadata

Greg Shah wrote:
> Good job. I think we should pause the work there. Is it safe to include in the branch pending t... Igor Skornyakov

09:03 PM Feature #3814: more schema metadata

Good job. I think we should pause the work there. Is it safe to include in the branch pending the final changes? Greg Shah

08/20/2020

05:42 PM Feature #3814: more schema metadata

@UserTableStatUpdater@ core functionality implemented. The exposed public methods are:
* getTableStats(integer con... Igor Skornyakov

08/15/2020

08:06 AM Support #4701: try to improve H2 transaction commit performance

I've done a final profiling of FWD in regard with @h2_performance/perf-create.p@. I've added to the testcase 4 config... Adrian Lungu

08/14/2020

06:09 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Unless there is a way to write application logic to have some dependency on this interval, I... Igor Skornyakov

06:01 PM Feature #3814: more schema metadata

Unless there is a way to write application logic to have some dependency on this interval, I don't see any advantage ... Eric Faulhaber

02:13 PM Feature #3814: more schema metadata

The situation with @_UserTableStat@ VST looks like following:
# For every connection 4GL creates 50 slots with count... Igor Skornyakov

08:13 AM Feature #3814: more schema metadata

> you mean by multiple clients
You've been saying that it shows the stats for multuple users. Each user would be ... Greg Shah

07:27 AM Feature #3814: more schema metadata

Greg Shah wrote:
> Run multuple clients at different times and see if the data for earlier closed clients is seen by... Igor Skornyakov

06:08 AM Feature #3814: more schema metadata

Run multuple clients at different times and see if the data for earlier closed clients is seen by later clients. Greg Shah

03:29 AM Feature #3814: more schema metadata

Greg Shah wrote:
> > My understanding is that VSTs are created at the connect to the database and are not cleared un... Igor Skornyakov

08/13/2020

07:28 PM Feature #3814: more schema metadata

> My understanding is that VSTs are created at the connect to the database and are not cleared until the connection i... Greg Shah

06:45 PM Feature #3814: more schema metadata

My understanding is that VSTs are created at the connect to the database and are not cleared until the connection is ... Igor Skornyakov

06:41 PM Feature #3814: more schema metadata

> The data from the contexts that are closed before the start of the aggregation will be lost.
I would have though... Greg Shah

06:39 PM Feature #3814: more schema metadata

> Note that simple long types will do. There is no need to use AtomicLong, since these variables are counting operati... Greg Shah

06:27 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Igor Skornyakov wrote:
>
> > I'm not talking about database query. Me suggestion is have ... Igor Skornyakov

06:12 PM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> I'm not talking about database query. Me suggestion is have the map as a field in the @Da... Eric Faulhaber

05:28 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Igor Skornyakov wrote:
> > And keeping this map in the @Database@ instance will greatly sim... Igor Skornyakov

05:15 PM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> And keeping this map in the @Database@ instance will greatly simplify the aggregation.
P... Eric Faulhaber

04:51 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
.> Yes, a map lookup at @RecordBuffer@ creation (or initialization) seems an acceptable level ... Igor Skornyakov

04:44 PM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> To ensure that the @TableStats@ should be one per table I think that it is better to hold t... Eric Faulhaber

04:36 PM Feature #3814: more schema metadata

BTW: I do not think that the overhead will be substantially increased if the map will be not context-local but a fiel... Igor Skornyakov

04:31 PM Feature #3814: more schema metadata

Greg Shah wrote:
> [...]
>
> There would be one of these for each table ever accessed. At the first time a table... Igor Skornyakov

04:19 PM Feature #3814: more schema metadata

... Greg Shah

04:12 PM Feature #3814: more schema metadata

Greg Shah wrote:
> This still seems expensive. We don't want to take a map lookup hit in order to increment a count... Igor Skornyakov

04:06 PM Feature #3814: more schema metadata

> I understand that it should be a map of arrays of counters. If it will be ConcurrentMap with Atomic values no addit... Greg Shah

03:17 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> I assume the entries for a particular session should be removed when that session ends. Othe... Igor Skornyakov

03:12 PM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> I understand that it should be a map of arrays of counters. If it will be @ConcurrentMap@... Eric Faulhaber

02:59 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Greg Shah wrote:
> > That doc page suggests @_usertablestat-num@ is the table number. Perh... Igor Skornyakov

02:52 PM Feature #3814: more schema metadata

Greg Shah wrote:
> That doc page suggests @_usertablestat-num@ is the table number. Perhaps this is some kind of un... Eric Faulhaber

02:43 PM Feature #3814: more schema metadata

Greg Shah wrote:
> That doc page suggests @_usertablestat-num@ is the table number. Perhaps this is some kind of un... Igor Skornyakov

02:33 PM Feature #3814: more schema metadata

That doc page suggests @_usertablestat-num@ is the table number. Perhaps this is some kind of unique number associat... Greg Shah

02:32 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> Eric Faulhaber wrote:
> > What is the @_usertablestat-num@ field about?
>
> Is this a "... Igor Skornyakov

02:31 PM Feature #3814: more schema metadata

Eric,
Yes, the table effectively holds counters, 4 for 4 different CRUD operations per user/table combination. At le... Igor Skornyakov

02:27 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> What is the @_usertablestat-num@ field about?
Is this a "foreign key" of sorts to the @_... Eric Faulhaber

02:21 PM Feature #3814: more schema metadata

Igor, to best answer you, please help me understand the purpose of these fields:... Eric Faulhaber

01:50 PM Feature #3814: more schema metadata

It is also possible to have the updater, but it will not try to persist statistics constantly but keep it in the in-m... Igor Skornyakov

01:34 PM Feature #3814: more schema metadata

Greg Shah wrote:
> Even if the overhead is minimized, the current approach wastes CPU cycles for a feature that is a... Igor Skornyakov

01:16 PM Feature #3814: more schema metadata

Even if the overhead is minimized, the current approach wastes CPU cycles for a feature that is almost never used.
... Greg Shah

12:07 PM Feature #3814: more schema metadata

Eric Faulhaber wrote:
> I don't know enough about the information you need to collect to answer this. This method ge... Igor Skornyakov

11:48 AM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> I'm working now on the @_usertablestat@ VST support. This VST holds information about the C... Eric Faulhaber

06:44 AM Feature #3814: more schema metadata

Greg Shah wrote:
> Is the @_usertablestat@ contents specific to the current user's session? If so, then it is impor... Igor Skornyakov

06:29 AM Feature #3814: more schema metadata

Is the @_usertablestat@ contents specific to the current user's session? If so, then it is important to capture and ... Greg Shah

03:19 PM Bug #4071: FWD server stops allowing new web client logins after abend

Greg Shah wrote:
> > for interactive sessions, it should interrupt and wait for the Conversation thread to finish an... Constantin Asofiei

08:07 AM Bug #4071: FWD server stops allowing new web client logins after abend

> for interactive sessions, it should interrupt and wait for the Conversation thread to finish and cleanup the contex... Greg Shah

05:05 AM Bug #4071: FWD server stops allowing new web client logins after abend

And there's one more case to consider for the 'unplanned' socket termination, for virtual sessions: if there is an ac... Constantin Asofiei

01:05 PM Feature #3813: misc DB features part deux

OK, then lets wait until 4011b is in trunk (and 3821c rebased) before we finish this last item. Greg Shah

01:01 PM Feature #3813: misc DB features part deux

Greg Shah wrote:
> So there are no pending properties changes in 3821c?
It looks like that. Please note that it w... Igor Skornyakov

12:55 PM Feature #3813: misc DB features part deux

So there are no pending properties changes in 3821c? Greg Shah

12:09 PM Feature #3813: misc DB features part deux

Greg Shah wrote:
> I understand there will be some edits. Am I correct in my understanding that there are already o... Igor Skornyakov

12:04 PM Feature #3813: misc DB features part deux

I understand there will be some edits. Am I correct in my understanding that there are already other such edits that... Greg Shah

11:56 AM Feature #3813: misc DB features part deux

Greg Shah wrote:
> We have other such property changes in 3821c, right? If you finish @BUFFER-FIELD:DECIMALS@ in 38... Igor Skornyakov

10:59 AM Feature #3813: misc DB features part deux

> According to #3813-162, #3813-163, the only remaining issue is Built-In Function Usage in WHERE Clauses but there i... Greg Shah

08:34 AM Feature #3813: misc DB features part deux

Greg Shah wrote:
> What is left open in this task?
According to #3813-(162,163), the only remaining issue is @Bui... Igor Skornyakov

08:12 AM Feature #3813: misc DB features part deux

What is left open in this task? Greg Shah

10:52 AM Feature #4613: add missing support for database user defined functions (UDF) if they are supporte...

Some items mentioned in #3813 should be handled as part of this task:
> +Built-In Function Usage in WHERE Clauses+... Greg Shah

07:59 AM Support #4701: try to improve H2 transaction commit performance

Committed 4701a rev.11625 - added switch between shared and private temporary database. Adrian Lungu

08/12/2020

05:59 PM Bug #4071: FWD server stops allowing new web client logins after abend

I agree that all context cleanup for interactive sessions must be on the main Conversation thread.
For the virtual... Constantin Asofiei

09:06 AM Bug #4071: FWD server stops allowing new web client logins after abend

I think we also need to exclude context-cleanup from happening on @RelatedThread@ instances. Such threads share the ... Greg Shah

09:01 AM Bug #4071: FWD server stops allowing new web client logins after abend

I think we need to consider if there is ever a scenario in which context cleanup is needed for reader/writer threads.... Greg Shah

03:27 PM Feature #3814: more schema metadata

I'm working now on the @_usertablestat@ VST support. This VST holds information about the CRUD operations. The corres... Igor Skornyakov

06:58 AM Feature #3814: more schema metadata

Greg Shah wrote:
> When will you get to a good stopping point?
Most likely today. Tomorrow at the latest.
Igor Skornyakov

06:35 AM Feature #3814: more schema metadata

> Maybe it makes send to do it only for the new ORM.
Yes, probably so. Get it to the point where everything is re... Greg Shah

06:23 AM Feature #3814: more schema metadata

Greg Shah wrote:
> How quickly do you estimate that can you complete this work?
I plan to finish with the updater... Igor Skornyakov

06:01 AM Feature #3814: more schema metadata

How quickly do you estimate that can you complete this work? Greg Shah

08:46 AM Support #4701: try to improve H2 transaction commit performance

Created branch 4701a and committed the changes regarding per-user temporary databases rev. 11624. Please review.
... Adrian Lungu

08/11/2020

05:01 PM Feature #3814: more schema metadata

Yes, if that is how it works in the 4GL for a non-multi-tenant case. Greg Shah

04:47 PM Feature #3814: more schema metadata

Greg Shah wrote:
> > I do not know if we already have any support for the notion of tenant.
>
> No, we do not. W... Igor Skornyakov

04:39 PM Feature #3814: more schema metadata

> I do not know if we already have any support for the notion of tenant.
No, we do not. We are not currently plan... Greg Shah

02:30 PM Feature #3814: more schema metadata

As I can see the following fields are already supported, at least the "Minimal*" interfaces in the @ConnectTableUpdat... Igor Skornyakov

08/10/2020

05:11 PM Feature #3814: more schema metadata

Igor Skornyakov wrote:
> 3821c rev. 11449 does not compile
> [...]
Fixed in rev 11450.
Igor Skornyakov

05:06 PM Feature #3814: more schema metadata

Yes Greg Shah

03:28 PM Feature #3814: more schema metadata

3821c rev. 11449 does not compile... Igor Skornyakov

11:33 AM Support #4701: try to improve H2 transaction commit performance

This approach makes me a bit nervous.
Note that the @Database@ object is used as a key in a number of maps across ... Eric Faulhaber

11:11 AM Support #4701: try to improve H2 transaction commit performance

@SecurityManager.getInstance().getSessionId()@ is what I needed.
Eric Faulhaber wrote:
> Can you help me underst... Adrian Lungu

10:48 AM Support #4701: try to improve H2 transaction commit performance

Constantin Asofiei wrote:
> Eric Faulhaber wrote:
> > You could use @SecurityManager.getInstance().getUserId()@ to ... Eric Faulhaber

10:00 AM Support #4701: try to improve H2 transaction commit performance

Eric Faulhaber wrote:
> You could use @SecurityManager.getInstance().getUserId()@ to get a unique ID per user sessio... Constantin Asofiei

09:56 AM Support #4701: try to improve H2 transaction commit performance

Can you help me understand the proposed "high" and "low" level implementations?
I guess we are really differentiat... Eric Faulhaber

09:30 AM Support #4701: try to improve H2 transaction commit performance

By now, I delayed the initialization of the @_temp@ database. Until now, FWD was initializing the @_temp@ table at se... Adrian Lungu

08/09/2020

01:49 PM Feature #3814: more schema metadata

Added population of the @_Field@ table fields, described in #3814-188.
Committed to 3821c revision 11447. Igor Skornyakov

09:22 AM Support #4701: try to improve H2 transaction commit performance

Recently I have finished building new FWD testcases based on #4701-42 (extracting SQL batches from @h2_performance@ t... Adrian Lungu

08/08/2020

06:59 AM Feature #3814: more schema metadata

It is interesting that for some system tables the @_Field@ table contains records for fields which are not shown in t... Igor Skornyakov

08/07/2020

05:52 PM Feature #3814: more schema metadata

Constantin Asofiei wrote:
> Igor, please see the @TYPE_@ constants in @LegacyJavaAppserverApi@ - I think @_dtype@ is... Igor Skornyakov

05:20 PM Feature #3814: more schema metadata

Igor, please see the @TYPE_@ constants in @LegacyJavaAppserverApi@ - I think @_dtype@ is the same. You can confirm t... Constantin Asofiei

05:10 PM Feature #3814: more schema metadata

Greg Shah wrote:
> Is @_field._dtype@ the same as the data-type list I've described in #3549-3 and #3549-4?
Greg,... Igor Skornyakov

04:30 PM Feature #3814: more schema metadata

Is @_field._dtype@ the same as the data-type list I've described in #3549-3 and #3549-4? Greg Shah

04:22 PM Feature #3814: more schema metadata

Based on the analisys of the content of the @_Field@ table in the large customer database (12000+ records in this tab... Igor Skornyakov

09:49 AM Support #4701: try to improve H2 transaction commit performance

Please build the necessary changes. It would be best to implement this such that we can set a flag and choose "per-u... Greg Shah

08/06/2020

11:46 AM Support #4701: try to improve H2 transaction commit performance

Greg Shah wrote:
> What is the development effort needed to make this work? Is the meta database a problem?
At t... Adrian Lungu

10:45 AM Support #4701: try to improve H2 transaction commit performance

Your results in #4843 (per-user "private" @_temp@ H2 instances vs the current "global" instance approach) are very en... Greg Shah

08/05/2020

12:50 PM Feature #3814: more schema metadata

Greg Shah wrote:
> Can you finish #3814-185 items 1-3 by Friday?
Yes, I think that it possible. If not, I will fini... Igor Skornyakov

12:47 PM Feature #3814: more schema metadata

Can you finish #3814-185 items 1-3 by Friday?
> It is difficult to estimate the time which is required for impleme... Greg Shah

12:42 PM Feature #3814: more schema metadata

Status summary update (see #3814-135)
# Adding of the mandatory meta-tables is enforced. Working now on handling som... Igor Skornyakov

10:02 AM Support #4701: try to improve H2 transaction commit performance

The preliminary results reported by Apache JMeter are not very promising. However, note that the used tests are pure ... Adrian Lungu

08/03/2020

08:00 AM Support #4701: try to improve H2 transaction commit performance

Eric Faulhaber wrote:
> Adrian, as I understand it, your JMeter test environment can simulate heavy multi-user use, ... Adrian Lungu

08/01/2020

05:54 PM Support #4701: try to improve H2 transaction commit performance

Adrian, as I understand it, your JMeter test environment can simulate heavy multi-user use, correct?
I would like ... Eric Faulhaber

07/31/2020

11:31 AM Feature #4011: database/persistence layer performance improvements

Ovidiu, OK, I see that these are emitted at the @Table@ annotation for the DMO interface. Before that , we had code ... Constantin Asofiei

11:08 AM Feature #4011: database/persistence layer performance improvements

I tried to keep as much compatibility as I was aware when I added the new annotations and even added new attributes. ... Ovidiu Maxiniuc

10:52 AM Feature #4011: database/persistence layer performance improvements

Ovidiu Maxiniuc wrote:
> Have you encountered this necessity in customer code?
Yes. The previous support needs to ... Constantin Asofiei

10:30 AM Feature #4011: database/persistence layer performance improvements

Constantin,
Have you encountered this necessity in customer code?
The problem might be more complicated than ad... Ovidiu Maxiniuc

06:21 AM Feature #4011: database/persistence layer performance improvements

Ovidiu Maxiniuc wrote:
> We are talking about two different types of 'attributes' of the temp-tables:
> * structur... Constantin Asofiei

07/30/2020

03:50 PM Feature #4011: database/persistence layer performance improvements

Constantin,
I replaced the last conditional from the code you posted in note-791 with... Ovidiu Maxiniuc

01:32 PM Feature #4011: database/persistence layer performance improvements

Ovidiu Maxiniuc wrote:
> The problem with @BufferImpl.fill@ is not only in the code from 791 but in previous lines, ... Eric Faulhaber

10:20 AM Feature #4011: database/persistence layer performance improvements

The problem with @BufferImpl.fill@ is not only in the code from 791 but in prevous lines, too.
# use of @dereference... Ovidiu Maxiniuc