Feature #6830
find and fix all SQL SELECT statements with inlined literal arguments, and rewrite it using arguments and prepared statement
100%
Related issues
History
#1 Updated by Constantin Asofiei over 1 year ago
This is mostly I think about multiplex ID being inlined instead of using an argument. The cases I found are:
TemporaryBuffer.removeRecords FastCopyHelper.copyTable FastCopyHelper.restartSequence TemporaryBuffer.count
The point here is that inlining an argument makes that SQL string 'unique' and using a prepared statement (and the psCache
in TempTableDataSourceProvider.DataSourceImpl
) useless. This also requires the re-parse of the SQL on each invocation.
#2 Updated by Alexandru Lungu over 1 year ago
- Status changed from New to WIP
- Assignee set to Radu Apetrii
#3 Updated by Constantin Asofiei over 1 year ago
The task title mentions SELECT, but all kind of queries need to be checked/fixed: INSERT, UPDATE, DELETE.
#4 Updated by Radu Apetrii over 1 year ago
- % Done changed from 0 to 50
I have fixed the cases described previously by Constantin, apart from FastCopyHelper.copyTable
, due to the fact that it was already modified.
Other spots I have found in which this fix could be applied are as follows:
Select:
ConversionData.clean (151) MariaDbDialect.getSequenceSetValString (878) P2jH2Dialect.getSequenceCurrValString (778) P2JPostgreSQLDialect.getSequenceSetValString (1101) P2JSQLServer2008Dialect.getDropIndexString (1162) P2JSQLServer2012Dialect.getDropSequenceString (421) P2JSQLServer2012Dialect.getSequenceCurrValString (455) AbstractTempTable._hasRecords (2446) ReportApi.applyFileFilter (779) (796) (811) (837) ReportApi.persistSearchHistory (2517) (2526) (2528) ReportWorker.saveReport (461)
Insert: nothing
Update:
BufferImpl.acceptChanges (8733) FastCopyHelper.updatePeerRecords (1308) (1327) TemporaryBuffer.markNew (4545) TemporaryBuffer.removeRecords (7300) (7340) (7358)
Delete:
DefaultDirtyShareManager.delete (787) TransactionTableUpdater.flush (208) ReportApi.applyFileFilter (758)
As a side note, the number between parenthesis represents the line of the possible change (for 3821c, rev. 14280). This might not be the same in your case. It was added as some sort of reference point, so that you don't have to navigate through the entire method to find one small spot. If there are multiple parenthesis it means that there is more than one spot where the fix could be applied.
I will start fixing all these methods, unless you suggest that I should skip some of them. Let me know if so.
Once finished, should I commit these changes to 3821c?
#5 Updated by Radu Apetrii over 1 year ago
- Out of 26 possible spots, 16 were modified with no further problems.
P2JSQLServer2008Dialect.getDropIndexString
andP2JSQLServer2012Dialect.getDropSequenceString
are used just for generating DDLs so there is no point in using prepared statements.- For
ReportApi.persistSearchHistory
(in all three places), I am not sure if H2 still doesn't allow parameter substitutions in Subselects. If it does allow it, I can proceed to modifying this method as well. - For
FastCopyHelper.updatePeerRecords
(in both places), I am not sure if parameter substitution is supported by the methodexecuteSQLBatch
. - In
TemporaryBuffer.removeRecords
(in all three places), the business logic is pretty complicated. It would be difficult to add prepared statements and the code readability would certainly decrease.
Next, I will test the corectness of these changes in order to make sure that it doesn't break something.
#6 Updated by Radu Apetrii over 1 year ago
- Out of 16 modifications, 14 of them are working fine.
- I could not test
MariaDbDialect.getSequenceSetValString
because I cannot access sequences in MariaDb (more in #6868). - Considering that sequences are only for persistent databases, I am not sure if
P2JH2Dialect.getSequenceSetValString
should be tested. (Also, I have a small problem with persistent H2 which prevents me from testing this at the moment.)
Should I commit these changes to 3821c?
#7 Updated by Ovidiu Maxiniuc over 1 year ago
Radu Apetrii wrote:
- I could not test
MariaDbDialect.getSequenceSetValString
because I cannot access sequences in MariaDb (more in #6868).
I posted you there a temporary patch which solves the issue. Please do not commit the changes to 3821c. They will be merged naturally with that branch.
- Considering that sequences are only for persistent databases, I am not sure if
P2JH2Dialect.getSequenceSetValString
should be tested. (Also, I have a small problem with persistent H2 which prevents me from testing this at the moment.)
The H2 is not limited to temporary databases. FWD supports persistent databases backed by H2. In fact this is a solution for a quick setups of project without the hassle of installing and configuring the a SQL server like PostgreSQL. To use this, just update the database section of your directory.xml
. For an example, see directory_db_h2.xml.template
in the hotel test application.
#8 Updated by Constantin Asofiei over 1 year ago
Radu, please post a patch here for review, and mention the branch and revision it was built on.
#9 Updated by Radu Apetrii over 1 year ago
- % Done changed from 50 to 70
- File 6830.patch added
Here's the patch you requested. It was built on 3821c, rev. 14307.
I also modified ReportApi.persistSearchHistory
since H2 now allows parameter substitution in subselects.
Ovidiu Maxiniuc wrote:
I posted you there a temporary patch which solves the issue. Please do not commit the changes to 3821c. They will be merged naturally with that branch.
Thank you for the patch, the error disappeared.
#10 Updated by Ovidiu Maxiniuc over 1 year ago
I reviewed the 6830.patch
. Nice job!
- in
FastCopyHelper.java
I see we use an inlinedalter sequence
statement. Even if we know that the dialect will always be H2, I think we should use the buffer's dialect-specificgetSequenceSetValString()
method instead. This way we can reuse the code and manageability (benefit of any improvement we do in the dialect specific updates this place too. - you added some new APIs in
Dialog
class. I think the idea is good. I would like to go one step ahead and use the sequence name as parameter, too, but I guess this is not possible for all dialects since some of them (if not the majority) require the name directly. Also, the overloaded method is still necessary because it is used in DDL generation.
Greg, is this a good moment to add the patch to 3821c?
#11 Updated by Greg Shah over 1 year ago
Greg, is this a good moment to add the patch to 3821c?
How safe are the changes? Have we tested them in one of the large customer applications?
#12 Updated by Radu Apetrii over 1 year ago
Greg Shah wrote:
Greg, is this a good moment to add the patch to 3821c?
How safe are the changes? Have we tested them in one of the large customer applications?
I've made tests for every change and debugged them. The syntax is not a problem, but I could not test that each one of them gives the correct result (only for some of them I was able to check that).
In a large application almost half of the changes won't even be noticeable, because they appear in reports or in really specific cases. Still, I will test them as you said, in a large customer application, and I will provide the result obtained.
#13 Updated by Radu Apetrii over 1 year ago
- File 6830-2.patch added
I have applied the changes you suggested. Also, I have tested all of them against a large customer application and there were no errors. The branch I used is 3821c, rev.14322.
In addition to the last patch, I have also included the changes in P2JSQLServer2008Dialect.java
and P2JSQLServer2012Dialect.java
so that when you apply the patch no errors (hopefully) will appear.
Ovidiu Maxiniuc wrote:
in FastCopyHelper.java I see we use an inlined alter sequence statement. Even if we know that the dialect will always be H2, I think we should use the buffer's dialect-specific getSequenceSetValString() method instead. This way we can reuse the code and manageability (benefit of any improvement we do in the dialect specific updates this place too.
you added some new APIs in Dialog class. I think the idea is good. I would like to go one step ahead and use the sequence name as parameter, too, but I guess this is not possible for all dialects since some of them (if not the majority) require the name directly. Also, the overloaded method is still necessary because it is used in DDL generation.
I have successfully added the sequence name as a parameter in PostgreSQL and MariaDB and for the other dialects (e.g. H2) I have raised an exception.
#14 Updated by Constantin Asofiei over 1 year ago
Radu, what is the state of this task? Was anything committed to 3821c?
#15 Updated by Radu Apetrii over 1 year ago
Constantin Asofiei wrote:
Radu, what is the state of this task? Was anything committed to 3821c?
No, I did not commit these changes to 3821c. The only thing I did was posting the patch here.
#16 Updated by Constantin Asofiei over 1 year ago
Radu Apetrii wrote:
Constantin Asofiei wrote:
Radu, what is the state of this task? Was anything committed to 3821c?
No, I did not commit these changes to 3821c. The only thing I did was posting the patch here.
Please port it to 6129b and post the patch here. If you do this today, that would be great. Thank you.
#17 Updated by Radu Apetrii over 1 year ago
- File 6830-6129b.patch added
Here you go, the patch ported to 6129b, rev. 14344.
I tested some little examples to see that nothing breaks, but not all the changes were included in these examples. So, let me know if something unexpected happens and I will fix it.
#18 Updated by Constantin Asofiei over 1 year ago
Radu, the changes are OK with 6129b and a customer app (performance is improved). Please commit the 6830-6129b.patch to 6129b.
#19 Updated by Radu Apetrii over 1 year ago
Constantin Asofiei wrote:
Radu, the changes are OK with 6129b and a customer app (performance is improved). Please commit the 6830-6129b.patch to 6129b.
Done. Committed to 6129b, rev. 14346.
#20 Updated by Radu Apetrii over 1 year ago
- % Done changed from 70 to 100
- Status changed from WIP to Review
#21 Updated by Alexandru Lungu about 1 year ago
- Status changed from Review to Test
This reached trunk with 6129b. It can be closed.
#7165 was opened in the meantime targeting a "sub-issue" from here.
#22 Updated by Alexandru Lungu about 1 year ago
- Related to Bug #7165: Close multiplex scope using prepared statement added
#23 Updated by Eric Faulhaber 4 months ago
- Status changed from Test to Closed