Project

General

Profile

Feature #6830

find and fix all SQL SELECT statements with inlined literal arguments, and rewrite it using arguments and prepared statement

Added by Constantin Asofiei over 1 year ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
version:

6830.patch Magnifier (25.5 KB) Radu Apetrii, 10/24/2022 05:54 AM

6830-2.patch Magnifier (29 KB) Radu Apetrii, 10/26/2022 04:46 AM

6830-6129b.patch Magnifier (33.4 KB) Radu Apetrii, 01/09/2023 06:35 AM


Related issues

Related to Database - Bug #7165: Close multiplex scope using prepared statement Test

History

#1 Updated by Constantin Asofiei over 1 year ago

This is mostly I think about multiplex ID being inlined instead of using an argument. The cases I found are:

TemporaryBuffer.removeRecords
FastCopyHelper.copyTable
FastCopyHelper.restartSequence
TemporaryBuffer.count

The point here is that inlining an argument makes that SQL string 'unique' and using a prepared statement (and the psCache in TempTableDataSourceProvider.DataSourceImpl) useless. This also requires the re-parse of the SQL on each invocation.

#2 Updated by Alexandru Lungu over 1 year ago

  • Status changed from New to WIP
  • Assignee set to Radu Apetrii

#3 Updated by Constantin Asofiei over 1 year ago

The task title mentions SELECT, but all kind of queries need to be checked/fixed: INSERT, UPDATE, DELETE.

#4 Updated by Radu Apetrii over 1 year ago

  • % Done changed from 0 to 50

I have fixed the cases described previously by Constantin, apart from FastCopyHelper.copyTable, due to the fact that it was already modified.
Other spots I have found in which this fix could be applied are as follows:

Select:

ConversionData.clean (151)
MariaDbDialect.getSequenceSetValString (878)
P2jH2Dialect.getSequenceCurrValString (778)
P2JPostgreSQLDialect.getSequenceSetValString (1101)
P2JSQLServer2008Dialect.getDropIndexString (1162)
P2JSQLServer2012Dialect.getDropSequenceString (421)
P2JSQLServer2012Dialect.getSequenceCurrValString (455)
AbstractTempTable._hasRecords (2446)
ReportApi.applyFileFilter (779) (796) (811) (837)
ReportApi.persistSearchHistory (2517) (2526) (2528)
ReportWorker.saveReport (461) 

Insert: nothing

Update:

BufferImpl.acceptChanges (8733)
FastCopyHelper.updatePeerRecords (1308) (1327)
TemporaryBuffer.markNew (4545)
TemporaryBuffer.removeRecords (7300) (7340) (7358) 

Delete:

DefaultDirtyShareManager.delete (787)
TransactionTableUpdater.flush (208)
ReportApi.applyFileFilter (758) 

As a side note, the number between parenthesis represents the line of the possible change (for 3821c, rev. 14280). This might not be the same in your case. It was added as some sort of reference point, so that you don't have to navigate through the entire method to find one small spot. If there are multiple parenthesis it means that there is more than one spot where the fix could be applied.
I will start fixing all these methods, unless you suggest that I should skip some of them. Let me know if so.
Once finished, should I commit these changes to 3821c?

#5 Updated by Radu Apetrii over 1 year ago

Here's a report on how modifying went so far:
  • Out of 26 possible spots, 16 were modified with no further problems.
  • P2JSQLServer2008Dialect.getDropIndexString and P2JSQLServer2012Dialect.getDropSequenceString are used just for generating DDLs so there is no point in using prepared statements.
  • For ReportApi.persistSearchHistory (in all three places), I am not sure if H2 still doesn't allow parameter substitutions in Subselects. If it does allow it, I can proceed to modifying this method as well.
  • For FastCopyHelper.updatePeerRecords (in both places), I am not sure if parameter substitution is supported by the method executeSQLBatch.
  • In TemporaryBuffer.removeRecords (in all three places), the business logic is pretty complicated. It would be difficult to add prepared statements and the code readability would certainly decrease.

Next, I will test the corectness of these changes in order to make sure that it doesn't break something.

#6 Updated by Radu Apetrii over 1 year ago

I have tested these prepared statements (from the last post) and I got the following results:
  • Out of 16 modifications, 14 of them are working fine.
  • I could not test MariaDbDialect.getSequenceSetValString because I cannot access sequences in MariaDb (more in #6868).
  • Considering that sequences are only for persistent databases, I am not sure if P2JH2Dialect.getSequenceSetValString should be tested. (Also, I have a small problem with persistent H2 which prevents me from testing this at the moment.)

Should I commit these changes to 3821c?

#7 Updated by Ovidiu Maxiniuc over 1 year ago

Radu Apetrii wrote:

  • I could not test MariaDbDialect.getSequenceSetValString because I cannot access sequences in MariaDb (more in #6868).

I posted you there a temporary patch which solves the issue. Please do not commit the changes to 3821c. They will be merged naturally with that branch.

  • Considering that sequences are only for persistent databases, I am not sure if P2JH2Dialect.getSequenceSetValString should be tested. (Also, I have a small problem with persistent H2 which prevents me from testing this at the moment.)

The H2 is not limited to temporary databases. FWD supports persistent databases backed by H2. In fact this is a solution for a quick setups of project without the hassle of installing and configuring the a SQL server like PostgreSQL. To use this, just update the database section of your directory.xml. For an example, see directory_db_h2.xml.template in the hotel test application.

#8 Updated by Constantin Asofiei over 1 year ago

Radu, please post a patch here for review, and mention the branch and revision it was built on.

#9 Updated by Radu Apetrii over 1 year ago

  • % Done changed from 50 to 70
  • File 6830.patchMagnifier added

Here's the patch you requested. It was built on 3821c, rev. 14307.
I also modified ReportApi.persistSearchHistory since H2 now allows parameter substitution in subselects.
Ovidiu Maxiniuc wrote:

I posted you there a temporary patch which solves the issue. Please do not commit the changes to 3821c. They will be merged naturally with that branch.

Thank you for the patch, the error disappeared.

#10 Updated by Ovidiu Maxiniuc over 1 year ago

I reviewed the 6830.patch. Nice job!

A couple of notes related to affected code:
  • in FastCopyHelper.java I see we use an inlined alter sequence statement. Even if we know that the dialect will always be H2, I think we should use the buffer's dialect-specific getSequenceSetValString() method instead. This way we can reuse the code and manageability (benefit of any improvement we do in the dialect specific updates this place too.
  • you added some new APIs in Dialog class. I think the idea is good. I would like to go one step ahead and use the sequence name as parameter, too, but I guess this is not possible for all dialects since some of them (if not the majority) require the name directly. Also, the overloaded method is still necessary because it is used in DDL generation.

Greg, is this a good moment to add the patch to 3821c?

#11 Updated by Greg Shah over 1 year ago

Greg, is this a good moment to add the patch to 3821c?

How safe are the changes? Have we tested them in one of the large customer applications?

#12 Updated by Radu Apetrii over 1 year ago

Greg Shah wrote:

Greg, is this a good moment to add the patch to 3821c?

How safe are the changes? Have we tested them in one of the large customer applications?

I've made tests for every change and debugged them. The syntax is not a problem, but I could not test that each one of them gives the correct result (only for some of them I was able to check that).
In a large application almost half of the changes won't even be noticeable, because they appear in reports or in really specific cases. Still, I will test them as you said, in a large customer application, and I will provide the result obtained.

#13 Updated by Radu Apetrii over 1 year ago

I have applied the changes you suggested. Also, I have tested all of them against a large customer application and there were no errors. The branch I used is 3821c, rev.14322.
In addition to the last patch, I have also included the changes in P2JSQLServer2008Dialect.java and P2JSQLServer2012Dialect.java so that when you apply the patch no errors (hopefully) will appear.
Ovidiu Maxiniuc wrote:

in FastCopyHelper.java I see we use an inlined alter sequence statement. Even if we know that the dialect will always be H2, I think we should use the buffer's dialect-specific getSequenceSetValString() method instead. This way we can reuse the code and manageability (benefit of any improvement we do in the dialect specific updates this place too.

you added some new APIs in Dialog class. I think the idea is good. I would like to go one step ahead and use the sequence name as parameter, too, but I guess this is not possible for all dialects since some of them (if not the majority) require the name directly. Also, the overloaded method is still necessary because it is used in DDL generation.

I have successfully added the sequence name as a parameter in PostgreSQL and MariaDB and for the other dialects (e.g. H2) I have raised an exception.

#14 Updated by Constantin Asofiei over 1 year ago

Radu, what is the state of this task? Was anything committed to 3821c?

#15 Updated by Radu Apetrii over 1 year ago

Constantin Asofiei wrote:

Radu, what is the state of this task? Was anything committed to 3821c?

No, I did not commit these changes to 3821c. The only thing I did was posting the patch here.

#16 Updated by Constantin Asofiei over 1 year ago

Radu Apetrii wrote:

Constantin Asofiei wrote:

Radu, what is the state of this task? Was anything committed to 3821c?

No, I did not commit these changes to 3821c. The only thing I did was posting the patch here.

Please port it to 6129b and post the patch here. If you do this today, that would be great. Thank you.

#17 Updated by Radu Apetrii over 1 year ago

Here you go, the patch ported to 6129b, rev. 14344.
I tested some little examples to see that nothing breaks, but not all the changes were included in these examples. So, let me know if something unexpected happens and I will fix it.

#18 Updated by Constantin Asofiei over 1 year ago

Radu, the changes are OK with 6129b and a customer app (performance is improved). Please commit the 6830-6129b.patch to 6129b.

#19 Updated by Radu Apetrii over 1 year ago

Constantin Asofiei wrote:

Radu, the changes are OK with 6129b and a customer app (performance is improved). Please commit the 6830-6129b.patch to 6129b.

Done. Committed to 6129b, rev. 14346.

#20 Updated by Radu Apetrii over 1 year ago

  • % Done changed from 70 to 100
  • Status changed from WIP to Review

#21 Updated by Alexandru Lungu about 1 year ago

  • Status changed from Review to Test

This reached trunk with 6129b. It can be closed.
#7165 was opened in the meantime targeting a "sub-issue" from here.

#22 Updated by Alexandru Lungu about 1 year ago

  • Related to Bug #7165: Close multiplex scope using prepared statement added

#23 Updated by Eric Faulhaber 4 months ago

  • Status changed from Test to Closed

Also available in: Atom PDF