Project

General

Profile

Feature #10270

« Previous | Next »

implement some temp-tables using the persistent database instance when they are joined to tables in that database

Added by Greg Shah about 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
Due date:
% Done:

0%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
reviewer:
production:
No
env_name:
topics:

Related issues

Related to Database - Feature #4018: possibly move temp-table support into persistent database New
Related to Database - Feature #10199: implement a temp-table size limit such that usage above that limit is backed by disk instead of memory New

History

#1 Updated by Greg Shah about 1 year ago

  • Related to Feature #4018: possibly move temp-table support into persistent database added

#4 Updated by Greg Shah about 1 year ago

The idea here is that we could analyze usage of each temp-table and implement heuristcs that mark specific temp-tables as needing to be implemented in the persistent database. They would still be temp-tables, but by including them in the same persistent database as the tables to which they are joined, would allow us to actually implement the join instead of the manual thing with do today (with CompoundQuery I think).

I am not suggesting the implementation of full temp-table support in the persistent database (#4018). I personally don't think #@4018 will prove to be a positive change.

But a selective change to allow joins might be useful.

#5 Updated by Alexandru Lungu about 1 year ago

The idea here is that we could analyze usage of each temp-table and implement heuristcs that mark specific temp-tables as needing to be implemented in the persistent database. They would still be temp-tables, but by including them in the same persistent database as the tables to which they are joined, would allow us to actually implement the join instead of the manual thing with do today (with CompoundQuery I think).

It would be nice to eventually leave the temporary tables reside mainly in-memory, but have the possibility to dump them somewhere: in-file or inside persistent database. Having an unified layer for this can also integrate smoothly with #10199. In fact, maybe that "backed by disk instead of memory" can even represent the persistent database.

My point is that having the effort to dump temporary tables to PG (or other persistent database) may suffice for #10199 and #10270. When?
  • if the table gets way too large to cover #10199
  • if the table is marked by heuristics as suitable for persistent database.

But this unification may require #4018.

#6 Updated by Alexandru Lungu about 1 year ago

  • Related to Feature #10199: implement a temp-table size limit such that usage above that limit is backed by disk instead of memory added

#7 Updated by Ovidiu Maxiniuc about 1 year ago

I think this might be useful, but the heuristic could be difficult to implement because of dynamic queries. What I mean is that we can miss an optimisable temp-table because we cannot see it is joined only with permanent tables id dynamic queries, but we must take care not to move those which should remain in _temp database because of opaque dynamic queries. Also we need to decide what happens with these tables in case they are joined with permanent tables from different databases.

Also available in: Atom PDF