DynamicTablesHelper.java
/*
** Module : DynamicTablesHelper.java
** Abstract : Helper class for dynamic temporary tables.
**
** Copyright (c) 2013-2025, Golden Code Development Corporation.
**
** -#- -I- --Date-- ---------------------------------------Description---------------------------------------
** 001 SVL 20130624 Created initial version.
** 002 SVL 20130822 Continued implementation.
** 003 SVL 20131115 Implemented tableExists, some refactoring for dynamic buffers registration
** and clean up.
** 004 CA 20131013 Added no-op deleted() method, required by the changes in Finalizable interface.
** 005 OM 20140117 Added support for runtime indexed access to fields from buffer.
** 006 SVL 20140123 Implemented get4GLTableName.
** 007 SVL 20140124 Refactored DMO class cleanup.
** 008 OM 20140123 Added legacy table name in createDynamicDMO. Also the fields ar normalized.
** 009 OM 20140128 Fixed historical annotation so the table/fields can be find by P2OLookup.
** Removed ConcurrentModificationException in cleanup.
** 010 ECF 20140226 Renamed generateDMOAst to generateSchemaAst. Added index ASTs to the schema
** AST produced by this method. Replaced Apache commons logging with J2SE
** logging.
** 011 ECF 20140404 Reimplemented dynamic conversion to use ConversionPool.
** 012 OM 20140414 Added case-sensitive field support for indexed character columns.
** 013 ECF 20140611 Fixed regression caused by 012; avoids remapping all character computed
** column information for every dynamic temp-table. DatabaseManager now handles
** this mapping within its registerDynamicTable method.
** 014 ECF 20140922 Reduced memory leak and increased performance by caching dynamic temp-table
** resources. This is an interim update, but the cleanup of expired cache
** resources currently is broken, in that DMO resources can be released under
** some circumstances while a Hibernate session is still using them. The
** temporary workaround is to define a high cache limit, so that the cache does
** not expire elements.
** 015 ECF 20150109 Preserve original case for legacy field names.
** 016 VMN 20150122 Fixes to use non-normalized legacy field and table names in annotations.
** 017 ECF 20150327 Added p2o_post processing step.
** 018 ECF 20150414 Replaced in-memory compiling of dynamically generated DMO source code with
** direct-to-bytecode assembly from Java ASTs.
** 019 ECF 20150430 Fixed memory leak of active temp-table builders in DMOResourceManager inner
** class. A proper clean-up mechanism for dynamic temp-tables which are no longer
** in use and have expired from the cache is still needed.
** 020 ECF 20150521 Fixed memory leak of temporary buffers.
** 021 ECF 20150819 Fixed NPE caused by race condition in createDynamicDMO. Increased cache size.
** 022 OM 20151028 Avoided NPE in context cleanup().
** 023 ECF 20160506 Slight optimization of get4GLTableName.
** 024 GES 20160504 Modified initial value processing for character field types, to make it safe
** for the full range of characters that might be present (including embedded
** double quotes).
** 025 ECF 20160521 Replaced ExpiryCache with simple HashMap (TODO: switch back when cleanup code
** works).
** 026 OM 20160905 Small optimizations.
** 027 SVL 20190614 Support for CLOB and BLOB fields.
** 028 CA 20190728 Added object type.
** ECF 20180802 Fixed findExistingTable to work with metadata tables.
** CA 20190826 Some fixes in generateSchemaAst related to initial field value.
** CA 20190902 Fixed problems when connecting to a FWD server running a database schema which
** not actively managed by the requester, too.
** 029 SVL 20190801 tableExists looks in shared tables too.
** 030 CA 20200622 Small performance improvement - use IdentityHashMap if the key is java.lang.Class.
** 031 CA 20200922 For temp-tables, the reserved properties need to be emitted, too. Fixed generateSchemaAst
** which had code as if it was generating a .p2o and not a .schema AST.
** OM 20200924 P2JIndexComponent carries multiple information to avoid map lookups for them.
** OM 20201001 Improved DMO manipulation performance by caching slow Property annotation access.
** OM 20201002 Use DmoMeta cached information instead of map lookups.
** CA 20201003 Use an identity HashSet where possible.
** OM 20201120 Added DECIMALS attribute support.
** SVL 20210330 Fixed handling of default values for dynamic LOGICAL fields.
** SVL 20210401 "?" text is treated as an unknown default value for any data type.
** ECF 20210511 Refactored string key normalization work.
** CA 20210823 Fixed extent field check - extent 1 is allowed.
** HC 20211201 Fixed runtime conversion of comhandle field for dynamic temp tables.
** ECF 20211202 Fixed initial value bug. Some naming cleanup.
** OM 20211213 Added missing COLUMN-LABEL attribute to fields.
** CA 20211222 Fixed memory leaks: remove the dynamic temp-table from the DMO users, when the dynamic
** temp-table gets deleted.
** OM 20220212 Use ReservedProperty static method for testing reserved properties names.
** OM 20220727 FieldId and PropertyId are different for denormalized extent fields.
** OM 20221003 When generateSchemaAst() set annotations only for present field attributes.
** BS 20221004 Changes for SAVE CACHE support.
** OM 20220706 New attributes supported in generateSchemaAst().
** OM 20220914 Use MAX-WIDTH to generate custom sized varchar columns.
** BS 20221004 Changes for SAVE CACHE support.
** IAS 20221006 Added RAW type to the switch in the 'generateSchemaAst'.
** IAS 20221013 Fixed setting of "serialize-name" and "serialize-hidden annotations".
** IAS 20221014 Fixed 'column-codepage' support.
** SVL 20230110 P2JIndex.components() provides direct access to the components array in order to improve
** performance.
** CA 20230116 Avoid using handle.unwrap, handle.getReference or other BDT usage from within FWD runtime.
** OM 20230214 Added NO-UNDO property for dynamically built temp-table.
** 032 CA 20230215 Improved other cases to reduce BDT usage from within FWD runtime.
** 033 IAS 20230427 Added 'legacyName' to the 'indexAst'.
** 034 GBB 20230512 Logging methods replaced by CentralLogger/ConversionStatus.
** 035 DDF 20230608 Made the size of the DMO resources cache configurable.
** DDF 20230613 Removed the logic that reads the configuration size of the cache from the static block.
** CacheManager will handle the finding of the configuration size and the cache creation.
** DDF 20230627 Made the cache final.
** 036 OM 20240402 Removed registration of dynamic tables with DatabaseManager.
** 037 CA 20240502 When a BEFORE-TABLE is being built too, for a dynamic temp-table, the associated DMOs for
** both the before and after table must have the after/beforeTable annotations set.
** 038 AD 20240403 Added support for caching to disk dynamically generated table DMOs.
** AD 20240430 Unified the 2 configuration options created for reading/writing to persistent cache.
** 039 ICP 20240703 Added null check in initializePersistentCacheData.
** ICP 20240711 Ensured that newly created temporary directories are cleaned up.
** ICP 20240805 When the jar is created, write the jar index as well.
** Ensured no duplicates in the jar.
** 040 OM 20240901 Improved Database API.
** 041 OM 20241128 Multi-tenant runtime support: selected the proper persistence context, eventually
** based on [sharedDb] parameter.
** 042 ICP 20250210 Added directory.xml and bootstrap option to disable the update of the persistent cache.
** When the update is disabled, the cache file can be now loaded from the classpath.
** Fixed edge case when DMO is found in persistentCacheMap, but classloading fails.
** 043 ICP 20250129 Used logical constants to leverage cached instances.
*/
/*
** This program is free software: you can redistribute it and/or modify
** it under the terms of the GNU Affero General Public License as
** published by the Free Software Foundation, either version 3 of the
** License, or (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
** GNU Affero General Public License for more details.
**
** You may find a copy of the GNU Affero GPL version 3 at the following
** location: https://www.gnu.org/licenses/agpl-3.0.en.html
**
** Additional terms under GNU Affero GPL version 3 section 7:
**
** Under Section 7 of the GNU Affero GPL version 3, the following additional
** terms apply to the works covered under the License. These additional terms
** are non-permissive additional terms allowed under Section 7 of the GNU
** Affero GPL version 3 and may not be removed by you.
**
** 0. Attribution Requirement.
**
** You must preserve all legal notices or author attributions in the covered
** work or Appropriate Legal Notices displayed by works containing the covered
** work. You may not remove from the covered work any author or developer
** credit already included within the covered work.
**
** 1. No License To Use Trademarks.
**
** This license does not grant any license or rights to use the trademarks
** Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
** of Golden Code Development Corporation. You are not authorized to use the
** name Golden Code, FWD, or the names of any author or contributor, for
** publicity purposes without written authorization.
**
** 2. No Misrepresentation of Affiliation.
**
** You may not represent yourself as Golden Code Development Corporation or FWD.
**
** You may not represent yourself for publicity purposes as associated with
** Golden Code Development Corporation, FWD, or any author or contributor to
** the covered work, without written authorization.
**
** 3. No Misrepresentation of Source or Origin.
**
** You may not represent the covered work as solely your work. All modified
** versions of the covered work must be marked in a reasonable way to make it
** clear that the modified work is not originating from Golden Code Development
** Corporation or FWD. All modified versions must contain the notices of
** attribution required in this license.
*/
package com.goldencode.p2j.persist;
import antlr.*;
import com.goldencode.asm.*;
import com.goldencode.ast.*;
import com.goldencode.cache.*;
import com.goldencode.p2j.cfg.*;
import com.goldencode.p2j.convert.*;
import com.goldencode.p2j.directory.DirectoryService;
import com.goldencode.p2j.pattern.*;
import com.goldencode.p2j.persist.orm.*;
import com.goldencode.p2j.schema.*;
import com.goldencode.p2j.security.*;
import com.goldencode.p2j.uast.*;
import com.goldencode.p2j.util.*;
import com.goldencode.p2j.util.ErrorManager;
import com.goldencode.p2j.util.Utils;
import com.goldencode.p2j.util.logging.*;
import java.io.*;
import java.util.*;
import java.util.logging.*;
import org.apache.commons.io.FileUtils;
import java.util.concurrent.atomic.*;
import java.util.function.Consumer;
import java.util.jar.*;
/**
* Helper class for dynamic temporary tables.
*/
public class DynamicTablesHelper
{
/** Base package for generated DMOs. */
static final String DMO_BASE_PACKAGE = "com.goldencode.p2j.persist.dynamic";
/** Category name for DMO interfaces, used in conversion rules. */
static final String DMO_IFACE_CATEGORY = "iface";
/** Param type to Progress type. */
static final Map<ParmType, Integer> propTypeToProgress = new HashMap<>();
/** Logger. */
private static final CentralLogger LOG = CentralLogger.get(DynamicTablesHelper.class.getName());
/** Singleton instance. */
private static volatile DynamicTablesHelper instance;
/** In-memory class loader */
private static AsmClassLoader systemInMemoryClassLoader;
/** Bootstrap configuration for {@link #updatePersistentCache} */
private static Boolean bootstrapUpdateDMOCache = null;
/** Cache for dynamically converted and compiled DMO resources */
private final Map<CacheKey, CacheData> cache;
/** Lock object on which to synchronize cache operations */
private final Object cacheLock = new Object();
/**
* Cache used to find out if a dynamic table DMO needs to be assembled or exists saved on disk.
* The key is a string value of a CacheKey object. The value is a list, starting with an id,
* and followed by the names of all the interface names used to generate the DMO class.
*/
private Map<String, List<String>> persistentCacheMap = new HashMap<>();
/** Marks if dynamic table DMO classes generated will be saved to/read from a persistent cache. */
private boolean usePersistentCache = false;
/** Marks if dynamic table DMO persistent cache will be updated during this server run. */
private boolean updatePersistentCache = false;
/** Name of a temporary directory where .class files of dynamic table DMO classes are saved. */
private String tempLibFilePath;
/** Path to the jar file where dynamic table DMO classes will be placed at server shutdown. */
private String persistentCacheJarPath;
/** Path to the file containing data used to identify cached dynamic table DMO classes. */
private String persistentCacheMapPath;
/** Path to the jar file that will temprarily exist at server shutdown. */
private String temporaryJarName;
/** File object corresponding to the jar file with dynamic table DMO classes. */
private File persistentCacheJarFile;
/** List of directories that are newly created. */
private List<File> createdDirectories = new ArrayList<>();
/** List of entries present in the jar. */
private static Set<String> existingEntries = new HashSet<>();
/** File object corresponding to the file with data used to identify cached dynamic table DMO classes. */
private File persistentCacheMapFile;
/** Containers for context-specific data. */
private final ContextLocal<Context> context = new ContextLocal<Context>()
{
public WeightFactor getWeight() { return WeightFactor.LEVEL_8; }
public Context initialValue() { return new Context(); }
protected void cleanup(Context ctx) { ctx.cleanup(); }
};
/** Counter used for generation of unique names for dynamic tables. */
private final AtomicLong tableCounter;
/** Cross-context manager which cleans up shared DMO resources once they go out of scope */
private final DMOResourceManager dmoResourceManager = new DMOResourceManager();
static
{
propTypeToProgress.put(ParmType.CHAR, ProgressParserTokenTypes.FIELD_CHAR);
propTypeToProgress.put(ParmType.INT, ProgressParserTokenTypes.FIELD_INT);
propTypeToProgress.put(ParmType.INT64, ProgressParserTokenTypes.FIELD_INT64);
propTypeToProgress.put(ParmType.DEC, ProgressParserTokenTypes.FIELD_DEC);
propTypeToProgress.put(ParmType.LOG, ProgressParserTokenTypes.FIELD_LOGICAL);
propTypeToProgress.put(ParmType.DATE, ProgressParserTokenTypes.FIELD_DATE);
propTypeToProgress.put(ParmType.DT, ProgressParserTokenTypes.FIELD_DATETIME);
propTypeToProgress.put(ParmType.DTTZ, ProgressParserTokenTypes.FIELD_DATETIME_TZ);
propTypeToProgress.put(ParmType.HANDLE, ProgressParserTokenTypes.FIELD_HANDLE);
propTypeToProgress.put(ParmType.OBJECT, ProgressParserTokenTypes.FIELD_CLASS);
propTypeToProgress.put(ParmType.RECID, ProgressParserTokenTypes.FIELD_RECID);
propTypeToProgress.put(ParmType.ROWID, ProgressParserTokenTypes.FIELD_ROWID);
propTypeToProgress.put(ParmType.RAW, ProgressParserTokenTypes.FIELD_RAW);
propTypeToProgress.put(ParmType.CLOB, ProgressParserTokenTypes.FIELD_CLOB);
propTypeToProgress.put(ParmType.BLOB, ProgressParserTokenTypes.FIELD_BLOB);
propTypeToProgress.put(ParmType.COMHANDLE, ProgressParserTokenTypes.FIELD_COM_HANDLE);
systemInMemoryClassLoader = AsmClassLoader.getInstance();
if (systemInMemoryClassLoader == null)
{
throw new IllegalStateException("It is expected to have InMemoryClassLoader " +
"in the chain of system class loaders.");
}
}
/**
* Private constructor to enforce singleton.
*/
private DynamicTablesHelper()
{
cache = CacheManager.createMapCache(DynamicTablesHelper.class, null, 8192);
initializePersistentCacheData();
if (usePersistentCache)
{
tableCounter = new AtomicLong(persistentCacheMap.size() + 1);
}
else
{
tableCounter = new AtomicLong(1);
}
}
/**
* Get singleton instance of the helper.
*
* @return singleton instance of the helper.
*/
public static DynamicTablesHelper getInstance()
{
if (instance == null)
{
synchronized (DynamicTablesHelper.class)
{
if (instance == null)
{
instance = new DynamicTablesHelper();
PatternEngine.setDebugLevel(ConversionDriver.MSG_NONE);
}
}
}
return instance;
}
/**
* Initializes the bootstrap configuration for dynamic DMO cache updates.
*
* @param config
* The {@link BootstrapConfig} instance used to retrieve the configuration value.
*/
public static void bootstrap(BootstrapConfig config)
{
bootstrapUpdateDMOCache = config.getBoolean("persistence", "dmo", "updateDynamicDMOCache", null);
}
/**
* Get DMO field name by 4GL field name for a dynamic DMO.
*
* @param dmoIface
* DMO interface that corresponds the target dynamic DMO class.
* @param name4gl
* 4GL field name.
*
* @return DMO field name for the given DMO class and 4GL field name or {@code null} if no
* corresponding name was found.
*/
static String getHibernateFieldName(Class<? extends DataModelObject> dmoIface, String name4gl)
{
DMOInfo dmoInfo = getInstance().context.get().getDMOInfo(dmoIface);
if (dmoInfo == null)
{
return null;
}
return dmoInfo.legacyToDmoNames.get(TextOps.rightTrimLower(name4gl));
}
/**
* Get DMO field name by field creation id for a dynamic DMO.
*
* @param dmoIface
* DMO interface that corresponds the target dynamic DMO class.
* @param fieldId
* 4GL field id.
*
* @return DMO field name for the given DMO class and 4GL field name or {@code null} if no
* corresponding name was found.
*/
static String getHibernateFieldName(Class<? extends DataModelObject> dmoIface, int fieldId)
{
// the algorithm here is based on the way the Hibernate field names are generated at
// "conversion" time. See (construction of clsFieldsXXX mappings in createDynamicDMO())
return "field" + fieldId;
}
/**
* Get 4GL field name by DMO field name for a dynamic DMO.
*
* @param dmoIface
* DMO interface that corresponds the target dynamic DMO class.
* @param dmoFieldName
* DMO field name.
*
* @return 4GL field name for the given DMO class and DMO field name or {@code null} if no
* corresponding name was found.
*/
static String get4GLFieldName(Class<? extends DataModelObject> dmoIface, String dmoFieldName)
{
DMOInfo dmoInfo = getInstance().context.get().getDMOInfo(dmoIface);
if (dmoInfo == null)
{
return null;
}
return dmoInfo.dmoToLegacyNames.get(dmoFieldName);
}
/**
* Get table name and database name from the string which represents full table name
*
* @param fullTableName
* The string which represents full table name. Can have two formats:
* "databaseName.tableName" or "tableName".
*
* @return String array with two elements. The first element is table name. The second element
* is database name or <code>null</code> if the database name wasn't specified in the
* full table name.
*/
static String[] parseTableName(String fullTableName)
{
String tableName;
String databaseName = null;
int pos = rightTrim(fullTableName).indexOf('.');
if (pos < 0)
{
tableName = rightTrim(fullTableName);
}
else
{
// TODO database name should be case-insensitive, but for now it is considered to be
// case-sensitive
databaseName = rightTrim(fullTableName.substring(0, pos));
tableName = rightTrim(fullTableName.substring(pos + 1));
}
return new String[] { tableName, databaseName };
}
/**
* Allows to determine if the buffer is for a dynamic table.
*
* @param buffer
* Buffer to check.
*
* @return <code>true</code> if buffer is for a dynamic table.
*/
static boolean isDynamicTableBuffer(RecordBuffer buffer)
{
TempTable tempTable = buffer.getParentTable();
return (tempTable != null && tempTable._dynamic());
}
/**
* Returns the {@link TempTableBuilder} object that contains information about the 4GL
* representation of the table or <code>null</code> if there is no corresponding dynamic table
* for the specified record buffer.
*
* @param buffer
* Record buffer that corresponds to the target dynamic table.
*
* @return See above.
*/
static TempTableBuilder getTableInfo(RecordBuffer buffer)
{
return (TempTableBuilder) buffer.getParentTable();
}
/**
* Determines if the specified table exists.
*
* @param persistence
* Persistence which corresponds the target database.
* @param tableName4GL
* 4GL table name.
*
* @return <code>true</code> if the specified table exists.
*/
static boolean tableExists(Persistence persistence, String tableName4GL)
{
String normalizedTableName = TextOps.rightTrimLower(tableName4GL);
if (persistence.isTemporary())
{
boolean res = BufferManager.get().getStaticTempTableDepth(normalizedTableName) != -1;
if (!res)
{
res = SharedVariableManager.lookupTempTable(normalizedTableName) != null;
}
return res;
}
else
{
String schema = DatabaseManager.getSchema(persistence.getDatabase(Persistence.PRIVATE_CTX));
return TableMapper.getDMOClass(schema, normalizedTableName) != null;
}
}
/**
* Search for an existing table with the specified name.
*
* @param fullTableName
* Table name in format "databaseName.tableName" or "tableName". In the latter case
* table search is performed in all permanent databases and then the temporary
* database. Dynamic tables are ignored.
*
* @return persistence which corresponds the target database or <code>null</code> if there is
* no such table.
*/
static Persistence findExistingTable(String fullTableName)
{
String[] arr = parseTableName(fullTableName);
String srcTableName = arr[0];
String srcDatabaseName = arr[1];
if (srcDatabaseName != null)
{
// database is explicitly specified
if (!ConnectionManager.connected(srcDatabaseName).booleanValue())
{
return null;
}
String physicalName = ConnectionManager.pdbName(srcDatabaseName).getValue();
Persistence persistence = PersistenceFactory.getInstance(physicalName);
if (tableExists(persistence, srcTableName))
{
return persistence;
}
else
{
// special handling for metadata table
if (srcTableName.startsWith("_"))
{
return findMetaPersistence(physicalName, srcTableName);
}
return null;
}
}
else
{
// TODO: OM: it looks to me like the _temp database is checked first!
// search through all databases, check permanent databases first
long numDbs = ConnectionManager.getConnectedDbCount();
for (int i = 1; i <= numDbs; i++)
{
String databaseName = ConnectionManager.ldbNameNative(i);
if (!ConnectionManager.connected(databaseName).booleanValue())
{
continue;
}
Database db = ConnectionManager.get().getDatabase(databaseName);
Persistence persistence = PersistenceFactory.getInstance(db);
if (!tableExists(persistence, srcTableName))
{
// TODO: if a buffer is dynamically created using CREATE BUFFER bh FOR TABLE "name", the
// table's DMO might have not been processed yet, leading to a negative result here.
// Try adding the table now
// special handling for metadata table
if (srcTableName.startsWith("_"))
{
persistence = findMetaPersistence(db.getName(), srcTableName);
if (persistence != null)
{
return persistence;
}
}
continue;
}
return persistence;
}
// check temporary database
Persistence persistence = PersistenceFactory.getInstance(DatabaseManager.TEMP_TABLE_DB);
if (tableExists(persistence, srcTableName))
{
return persistence;
}
else
{
return null;
}
}
}
/**
* Get original 4GL table name. Despite the fact that we specify 4GL table name as an input
* parameter, this parameter can have different letter case or may have trailing spaces (this
* form is allowed when specifying table name in 4GL code).
*
* @param persistence
* Persistence which corresponds the target database.
* @param tableName4GL
* 4GL table name (case-insensitive, may have trailing spaces).
*
* @return original 4GL table name.
*/
static String get4GLTableName(Persistence persistence, String tableName4GL)
{
// TODO implement when table names in the original letter case will be preserved
return TextOps.rightTrimLower(tableName4GL);
}
/**
* Traverse an AST and assign unique IDs to all of its nodes.
*
* @param root
* The root node of the AST.
* @param startId
* First available id for assignment.
*
* @return new first available id for assignment.
*/
static long assignIds(Aast root, long startId)
{
root.setId(startId++);
int childCount = root.getNumImmediateChildren();
if (childCount > 0)
{
for (int i = 0; i < childCount; i++)
{
Aast child = root.getChildAt(i);
startId = assignIds(child, startId);
}
}
return startId;
}
/**
* Find the {@code Persistence} object that is associated with the given database and table.
*
* @param pdb
* Physical database name of primary database.
* @param tableName
* Unqualified table name.
*
* @return {@code Persistence} object if table is found
*/
private static Persistence findMetaPersistence(String pdb, String tableName)
{
Database primaryDb = ConnectionManager.get().getDatabase(pdb);
Database metaDb = primaryDb.toType(Database.Type.META);
Persistence persistence = PersistenceFactory.getInstance(metaDb);
if (tableExists(persistence, tableName))
{
return persistence;
}
return null;
}
/**
* Convenience method to remove trailing whitespace (space, tab, line feed, carriage return) from the
* string.
*
* @param str
* Target string.
*
* @return the target string without trailing spaces.
*/
private static String rightTrim(String str)
{
return TextOps.trim(str, null, false, false);
}
/**
* Method used to save to disk a dynamic table class generated at runtime as a .class file.
*
* @param className
* The name of the newly generated class that will be saved to disk.
* @param code
* The contents of the class file as a byte array.
*/
private void saveDynamicDmoToDisk(String className, byte[] code)
{
String fileName = tempLibFilePath +
File.separator +
className.replace(".", File.separator) +
".class";
File file = new File(fileName);
file.getParentFile().mkdirs();
try (FileOutputStream fos = new FileOutputStream(file))
{
fos.write(code);
}
catch (IOException e)
{
LOG.severe("Failed to write dynamic DMO to jar file for [" + className + "]", e);
}
}
/**
* Method called at class initialization, meant to read from disk the data regarding
* the dynamic table classes cache, saved in the form of a HashMap object.
*/
private void initializePersistentCacheData()
{
DirectoryService ds = DirectoryService.getInstance();
try
{
if (ds != null)
{
if (!ds.bind())
{
throw new RuntimeException("Directory bind failed");
}
String directoryNodeParent = "dynamic-dmos/";
String customUsePersistentCache = directoryNodeParent + "use-persistent-cache";
String customUpdatePersistentCache = directoryNodeParent + "update-persistent-cache";
String customPersistentCacheMapPath = directoryNodeParent + "persistent-cache-map-path";
String customPersistentCacheJarPath = directoryNodeParent + "persistent-cache-jar-path";
String customTempJarFolderPath = directoryNodeParent + "temporary-cache-folder-path";
String customTempJarFilePath = directoryNodeParent + "temporary-cache-jar-path";
usePersistentCache = Utils.getDirectoryNodeBoolean(ds,
customUsePersistentCache,
usePersistentCache,
false);
if (usePersistentCache)
{
LOG.info("Dynamic table dmos are configured to be read from jar.");
persistentCacheJarPath = null;
temporaryJarName = "lib_cache" + File.separator + "temp_dynamic_dmos.jar";
persistentCacheMapPath = "dynamic_dmo_cache_data.txt";
tempLibFilePath = "temp_lib_cache";
updatePersistentCache = Utils.getDirectoryNodeBoolean(ds,
customUpdatePersistentCache,
updatePersistentCache,
false);
persistentCacheMapPath = Utils.getDirectoryNodeString(ds,
customPersistentCacheMapPath,
persistentCacheMapPath,
false);
persistentCacheJarPath = Utils.getDirectoryNodeString(ds,
customPersistentCacheJarPath,
persistentCacheJarPath,
false);
tempLibFilePath = Utils.getDirectoryNodeString(ds,
customTempJarFolderPath,
tempLibFilePath,
false);
temporaryJarName = Utils.getDirectoryNodeString(ds,
customTempJarFilePath,
temporaryJarName,
false);
// if a bootstrap configuration is passed, the value read from directory.xml is overridden
if (bootstrapUpdateDMOCache != null)
{
updatePersistentCache = bootstrapUpdateDMOCache;
}
if (persistentCacheJarPath == null)
{
usePersistentCache = false;
LOG.severe("Did not provide path for dynamic table dmos jar.");
}
}
}
}
catch (Exception e)
{
LOG.severe("Error retrieving custom data for DynamicTableHelper.", e);
}
finally
{
if (ds != null && ds.isBound())
{
ds.unbind();
}
}
if (usePersistentCache && updatePersistentCache)
{
createDirectoryHierarchy(new File(tempLibFilePath));
persistentCacheJarFile = new File(persistentCacheJarPath);
File parent = persistentCacheJarFile.getParentFile();
if (parent != null && !parent.exists())
{
parent.mkdirs();
}
persistentCacheMapFile = new File(persistentCacheMapPath);
if (!persistentCacheMapFile.exists())
{
cleanupDirectories();
return;
}
try (FileInputStream fis = new FileInputStream(persistentCacheMapFile);
ObjectInputStream ois = new ObjectInputStream(fis))
{
persistentCacheMap = (HashMap<String, List<String>>) ois.readObject();
}
catch (Exception e)
{
LOG.severe("Failed to read dynamic table cache data map from disk.", e);
}
}
if (usePersistentCache && !updatePersistentCache)
{
persistentCacheMapFile = new File(persistentCacheMapPath);
if (persistentCacheMapFile.exists())
{
try (FileInputStream fis = new FileInputStream(persistentCacheMapFile);
ObjectInputStream ois = new ObjectInputStream(fis))
{
persistentCacheMap = (HashMap<String, List<String>>) ois.readObject();
}
catch (Exception e)
{
LOG.severe("Failed to read dynamic table cache data map from disk.", e);
}
return;
}
cleanupDirectories();
// Try loading from classpath
try (InputStream is = getClass().getClassLoader().getResourceAsStream(persistentCacheMapPath))
{
if (is != null)
{
try (ObjectInputStream ois = new ObjectInputStream(is))
{
persistentCacheMap = (HashMap<String, List<String>>) ois.readObject();
LOG.info("Loaded dynamic DMO cache from classpath: " + persistentCacheMapPath);
return;
}
}
else
{
LOG.warning("Dynamic DMO cache file not found in classpath: " + persistentCacheMapPath);
}
}
catch (IOException | ClassNotFoundException e)
{
LOG.warning("Failed to load dynamic DMO cache from classpath: " + persistentCacheMapPath, e);
}
LOG.severe("Failed to load persistent cache. Dynamic DMOs will be generated at runtime.");
usePersistentCache = false;
}
}
/**
* Method used to update/create the file on disk that retains data about the
* dynamic table classes cache, in the form of a HashMap object.
*/
private void updatePersistentCacheMap()
{
try (FileOutputStream fos = new FileOutputStream(persistentCacheMapFile);
ObjectOutputStream oos = new ObjectOutputStream(fos))
{
oos.writeObject(persistentCacheMap);
}
catch (Exception e)
{
LOG.severe("Failed to save to file persistent cache map data.", e);
}
}
/**
* Method meant to create a Jar file containing dynamic table classes that were
* cached to disk, so that they are not generated every time the server is started.
* It creates a temporary jar file, copies all the classes from the old jar file,
* if it exists, then takes all the new classes generated that are saved in a
* folder that is then deleted. The old jar file is deleted, and the temporary
* jar file is renamed to match the old one.
*/
public void createCache()
{
if (!usePersistentCache)
{
cleanupDirectories();
return;
}
if (updatePersistentCache)
{
File file = new File(tempLibFilePath);
if (!file.exists())
{
LOG.severe("Trying to create dynamic dmos cache jar, but directory with classes is missing.");
cleanupDirectories();
return;
}
if (file.list() != null && file.list().length == 0)
{
LOG.info("Trying to create dynamic dmos cache jar, but directory with classes is empty.");
cleanupDirectories();
return;
}
File tempJar = new File(temporaryJarName);
createDirectoryHierarchy(tempJar.getParentFile());
if (tempJar.exists())
{
tempJar.delete();
}
try
{
tempJar.createNewFile();
}
catch (IOException e)
{
LOG.severe("Failed to create temporary JAR file for saving dynamic DMO classes jar to disk.", e);
cleanupDirectories();
return;
}
try (FileOutputStream fos = new FileOutputStream(tempJar);
JarOutputStream jos = new JarOutputStream(fos))
{
Set<String> packages = new HashSet<>();
File[] files = file.listFiles();
for (int i = 0; i < files.length; i++)
{
createCacheHelper(files[i], jos, packages);
}
writeJarIndex(jos, packages);
if (persistentCacheJarFile.exists())
{
try (FileInputStream fis = new FileInputStream(persistentCacheJarFile);
JarInputStream jis = new JarInputStream(fis))
{
JarEntry je;
while ((je = jis.getNextJarEntry()) != null)
{
if (!existingEntries.contains(je.getName()))
{
jos.putNextEntry(je);
byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = jis.read(buffer)) != -1)
{
jos.write(buffer, 0, bytesRead);
}
jos.closeEntry();
existingEntries.add(je.getName());
}
}
}
persistentCacheJarFile.delete();
}
tempJar.renameTo(persistentCacheJarFile);
updatePersistentCacheMap();
}
catch (Exception e)
{
LOG.severe("Failed to create JAR with dynamic DMO classes.", e);
}
finally
{
try
{
FileUtils.cleanDirectory(file);
if (tempJar.exists())
{
if (!tempJar.delete())
{
LOG.warning("Failed to delete temporary JAR file: " + tempJar.getAbsolutePath());
}
}
cleanupDirectories();
}
catch (IOException e)
{
LOG.severe("Failed to delete temporary dynamic DMO classes folder.", e);
}
}
}
}
/**
* Ensures the directory hierarchy exists by creating any missing directories.
* Newly created directories are tracked for cleanup later.
*
* @param dir
* The directory to create.
*/
private void createDirectoryHierarchy(File dir)
{
File parent = dir;
Stack<File> directoriesToCreate = new Stack<>();
while (parent != null && !parent.exists())
{
directoriesToCreate.push(parent);
parent = parent.getParentFile();
}
while (!directoriesToCreate.isEmpty())
{
File toCreate = directoriesToCreate.pop();
try
{
if (toCreate.mkdir())
{
createdDirectories.add(toCreate);
}
else if (!toCreate.exists())
{
LOG.warning("Failed to create directory: " + toCreate.getAbsolutePath());
}
}
catch (SecurityException e)
{
LOG.severe("Security exception while creating directory: " +
toCreate.getAbsolutePath() +
" - " +
e.getMessage());
}
}
}
/**
* Cleans up newly created directories.
*/
private void cleanupDirectories()
{
for (File dir : createdDirectories)
{
try
{
if (dir.exists() && dir.list().length == 0)
{
dir.delete();
}
}
catch (Exception e)
{
LOG.warning("Failed to delete directory: " + dir.getAbsolutePath());
}
}
}
/**
* Helper method for creating the dynamic table classes jar/cache. Receives the
* file/directory to be added to the jar, and the stream used to write to the jar.
* Calls itself recursively for every directory in the source File, if that is
* a directory. Otherwise, it adds the source File to the jar.
*
* @param source
* The class file or directory containing class files that will be added to the jar.
* @param target
* The stream used to add files to the jar.
* @param packages
* The set to collect package names.
*
* @throws IOException
* Exception that can be thrown by the JarOutputStream object, not handled here.
*/
private void createCacheHelper(File source, JarOutputStream target, Set<String> packages)
throws IOException
{
// +1, because of the following file separator
String name = source.getPath().replace("\\", File.separator).substring(tempLibFilePath.length() + 1);
if (source.isDirectory())
{
if (!name.endsWith(File.separator))
{
name += File.separator;
}
if (existingEntries.add(name))
{
JarEntry entry = new JarEntry(name);
entry.setTime(source.lastModified());
target.putNextEntry(entry);
target.closeEntry();
}
File[] files = source.listFiles();
for (int i = 0; i < files.length; i++)
{
createCacheHelper(files[i], target, packages);
}
}
else
{
if (existingEntries.add(name))
{
JarEntry entry = new JarEntry(name);
entry.setTime(source.lastModified());
target.putNextEntry(entry);
try (FileInputStream fis = new FileInputStream(source);
BufferedInputStream bis = new BufferedInputStream(fis))
{
byte[] buffer = new byte[1024];
while (true)
{
int count = bis.read(buffer);
if (count == -1)
{
break;
}
target.write(buffer, 0, count);
}
target.closeEntry();
}
}
int lastSlashIndex = name.lastIndexOf('/');
if (lastSlashIndex != -1)
{
String packageName = name.substring(0, lastSlashIndex).replace('/', '.');
packages.add(packageName);
}
}
}
/**
* Writes the jar index to the jar file's META-INF directory.
*
* @param jos
* The JarOutputStream to write the index to.
* @param packages
* The set of package names to include in the index.
*
* @throws IOException
* Exception that can be thrown by the JarOutputStream object, not handled here.
*/
private static void writeJarIndex(JarOutputStream jos, Set<String> packages)
throws IOException
{
String indexListPath = "META-INF/INDEX.LIST";
if (!existingEntries.contains(indexListPath))
{
JarEntry indexEntry = new JarEntry(indexListPath);
jos.putNextEntry(indexEntry);
jos.write("JarIndex-Version: 1.0\n\n".getBytes());
jos.write("./\n".getBytes());
for (String pkg : packages)
{
jos.write((pkg + "\n").getBytes());
}
jos.closeEntry();
existingEntries.add(indexListPath);
}
}
/**
* Generate DMO interface, implementation class and mapping document and register the new DMO type.
* If requested, also caches the new classes generated to disk.
*
* @param builder
* Temporary table builder containing definitions of the target table.
* @param legacyTableName
* The legacy name of the table.
* @param legacyPeerName
* The legacy name of the table's peer. May be {@code null}.
* @param isBeforeTable
* Flag indicating the DMO is being built for a before table.
*
* @return <code>true</code> on success.
*/
boolean createDynamicDMO(TempTableBuilder builder,
String legacyTableName,
String legacyPeerName,
boolean isBeforeTable)
{
Context local = context.get();
Collection<P2JField> allFields = builder.getFields();
Collection<P2JIndex> allIndexes = builder.getIndexes();
// check the cache first
CacheKey cacheKey = new CacheKey(allFields, allIndexes);
CacheData cacheData;
boolean classLoadingFailed = false;
synchronized (cacheLock)
{
cacheData = cache.get(cacheKey);
}
DmoMeta dmoMeta = null;
Class<? extends DataModelObject> dmoIface = null;
Class<? extends TempTableBuffer> dmoBufIface = null;
Class<? extends Record> dmoClass = null;
DMOInfo dmoInfo = null;
Map<String, Class<?>> classes;
Map<String, String> clsFields4GLtoORM;
Map<String, String> clsFieldsORMTo4GL;
ArrayList<P2JIndex> fwdIndexes = null;
try
{
if (cacheData == null)
{
fwdIndexes = new ArrayList<>();
Long tableId = null;
List<String> persistentData = null;
clsFields4GLtoORM = new HashMap<>();
clsFieldsORMTo4GL = new HashMap<>();
long fieldId = 1;
for (P2JField field : allFields)
{
String fieldName4GL = field.getName();
String fieldName4GLNorm = TextOps.rightTrimLower(fieldName4GL);
String fieldNameORM = ReservedProperty.isReservedProperty(fieldName4GL) ? fieldName4GL
: "field" + (fieldId++);
clsFields4GLtoORM.put(fieldName4GLNorm, fieldNameORM);
clsFieldsORMTo4GL.put(fieldNameORM, fieldName4GL);
}
if (usePersistentCache)
{
persistentData = persistentCacheMap.get(cacheKey.toString());
if (persistentData != null && persistentData.size() >= 1)
{
try
{
tableId = Long.decode(persistentData.get(0));
}
catch (Exception e)
{
tableId = null;
}
}
if (tableId != null)
{
String ifaceName = "DynamicRecord" + tableId;
String sqlTableName = "dtt" + tableId;
ClassLoader loader = ClassLoader.getSystemClassLoader();
String fullIfaceName = DMO_BASE_PACKAGE +
"." +
DatabaseManager.TEMP_TABLE_SCHEMA +
"." +
ifaceName;
String fullIfaceBufName = fullIfaceName + "$Buf";
Class<?> clazz;
classes = new HashMap<>();
try
{
for (int i = 1; i < persistentData.size(); i++)
{
clazz = loader.loadClass(persistentData.get(i));
classes.put(persistentData.get(i), clazz);
}
dmoIface = (Class<? extends DataModelObject>) classes.get(fullIfaceName);
dmoBufIface = (Class<? extends TempTableBuffer>) classes.get(fullIfaceBufName);
dmoMeta = DmoMetadataManager.registerDmo(dmoIface);
extractIndexes(dmoMeta, fwdIndexes);
dmoClass = dmoMeta.getImplementationClass();
String fullImplClassName = dmoClass.getName();
classes.put(fullImplClassName, dmoClass);
dmoInfo = new DMOInfo(classes.values(), clsFields4GLtoORM, clsFieldsORMTo4GL);
CacheData cd = new CacheData(dmoInfo,
fullIfaceName,
fullImplClassName,
sqlTableName,
fwdIndexes,
classes);
cache.put(cacheKey, cd);
}
catch (Exception e)
{
classLoadingFailed = true;
}
}
}
if (tableId == null || classLoadingFailed)
{
if (tableId == null)
{
tableId = tableCounter.getAndIncrement();
}
String sqlTableName = "dtt" + tableId;
String ifaceName = "DynamicRecord" + tableId;
Aast root = instance.generateSchemaAst(builder,
ifaceName,
sqlTableName,
legacyTableName,
clsFields4GLtoORM,
legacyPeerName,
isBeforeTable);
try
{
ConversionPool.Results results;
results = ConversionPool.runTask(ConversionProfile.P2O, root);
root = (Aast) results.getStoredObject("p2o.prog");
ConversionPool.runTask(ConversionProfile.P2O_POST, results, root);
root = (Aast) results.getStoredObject("p2o.peer");
root.putAnnotation("permanent", false);
root.putAnnotation("package", DMO_BASE_PACKAGE);
// generate interface Java ASTs
results = ConversionPool.runTask(ConversionProfile.JAVA_DMO, root);
Aast iface = (Aast) results.getStoredObject(DMO_IFACE_CATEGORY, ifaceName);
String fullIfaceName = DMO_BASE_PACKAGE +
"." +
DatabaseManager.TEMP_TABLE_SCHEMA +
"." +
ifaceName;
String fullIfaceBufName = fullIfaceName + "$Buf";
// generate bytecode from Java ASTs
results = ConversionPool.runTask(ConversionProfile.BREW_DMO_ASM, iface);
// even though we've already taken the hit of converting, give the cache one last
// try, to minimize the chance of duplicate classes
synchronized (cacheLock)
{
cacheData = cache.get(cacheKey);
if (cacheData == null)
{
// TODO: use a separate class loader for each table, so the classes can be
// garbage collected when no longer in use
AsmClassLoader loader = systemInMemoryClassLoader;
List<String> keys;
List<String> persistentCacheEntryValue = null;
if (usePersistentCache && updatePersistentCache && !classLoadingFailed)
{
persistentCacheEntryValue = new ArrayList<>();
persistentCacheEntryValue.add(tableId.toString());
}
classes = new HashMap<>();
byte[] code;
// load interfaces first
keys = (List<String>) results.getStoredObject("iface.keys");
String name;
for (int i = 0; i < keys.size(); i++)
{
name = keys.get(i);
code = (byte[]) results.getStoredObject("ifaces", name);
// enable the following line for debugging
// new FileOutputStream(name + ".class").write(code);
Class<?> clazz = loader.loadClass(name, code);
classes.put(name, clazz);
if (usePersistentCache && updatePersistentCache && !classLoadingFailed)
{
saveDynamicDmoToDisk(name, code);
persistentCacheEntryValue.add(name);
}
}
dmoIface = (Class<? extends DataModelObject>) classes.get(fullIfaceName);
dmoBufIface = (Class<? extends TempTableBuffer>) classes.get(fullIfaceBufName);
Consumer<DmoClass> callback = null;
if (usePersistentCache && updatePersistentCache && !classLoadingFailed)
{
callback = (x) ->
{
saveDynamicDmoToDisk(x.getInternalName(), x.getBytecode());
};
persistentCacheMap.put(cacheKey.toString(), persistentCacheEntryValue);
}
dmoMeta = DmoMetadataManager.registerDmo(dmoIface, callback);
extractIndexes(dmoMeta, fwdIndexes);
dmoClass = dmoMeta.getImplementationClass();
String fullImplClassName = dmoClass.getName();
classes.put(fullImplClassName, dmoClass);
dmoInfo = new DMOInfo(classes.values(), clsFields4GLtoORM, clsFieldsORMTo4GL);
CacheData cd = new CacheData(dmoInfo,
fullIfaceName,
fullImplClassName,
sqlTableName,
fwdIndexes,
classes);
cache.put(cacheKey, cd);
}
}
}
finally
{
// clean up after TRPL rules in the AST registry
String ifacePath = DMO_BASE_PACKAGE.replace('.', File.separatorChar) + File.separator +
DatabaseManager.TEMP_TABLE_SCHEMA + File.separator +
ifaceName + ".java";
AstManager.get().removeTree(ifacePath);
}
}
}
if (cacheData != null)
{
dmoInfo = cacheData.getDMOInfo();
String fullIfaceName = cacheData.getDMOIfaceName();
String fullIfaceBufName = fullIfaceName + "$Buf";
classes = cacheData.getClasses();
dmoIface = (Class<? extends DataModelObject>) classes.get(fullIfaceName);
dmoBufIface = (Class<? extends TempTableBuffer>) classes.get(fullIfaceBufName);
dmoMeta = DmoMetadataManager.getDmoInfo(dmoIface);
}
local.registerDMOInfo(dmoIface, dmoInfo);
builder.setDmoIface(dmoMeta, dmoIface, dmoBufIface);
local.registerBuilder(builder);
}
catch (Exception e)
{
LOG.log(Level.SEVERE, e.getMessage(), e);
ErrorManager.displayError("Error while generating dynamic table: " + e.getMessage());
return false;
}
return true;
}
/**
* Deregister the given temp-table builder from the DMO resource manager.
*
* @param builder
* Builder to be deregistered.
*/
void deregisterBuilder(TempTableBuilder builder)
{
dmoResourceManager.builderInactive(builder);
}
/**
* Remove this dynamic temp-table from the DMO users.
*
* @param builder
* Builder to be deregistered.
*/
void deregisterDmoUser(TempTableBuilder builder)
{
Class<? extends DataModelObject> dmoIface = builder.getDMOInterface();
context.get().dmoUsers.getOrDefault(dmoIface, Collections.emptySet()).remove(builder);
}
/**
* Extracts the list of indexes from an interface into an array.
*
* @param dmoMeta
* the metadata collection of the DMO interface.
* @param fwdIndexes
* The destination array where the index data will be stored.
*/
private void extractIndexes(DmoMeta dmoMeta, ArrayList<P2JIndex> fwdIndexes)
{
fwdIndexes.clear();
Iterator<P2JIndex> idxIter = dmoMeta.getDatabaseIndexes();
idxIter.forEachRemaining(fwdIndexes::add);
}
/**
* Generate AST that represents the target table and has the same structure as .p2o files from
* the conversion stage.
*
* @param builder
* Temporary table builder containing definitions of the target table.
* @param ifaceName
* The name of the target DMO interface.
* @param tableName
* The name of the target SQL table.
* @param legacyTableName
* The legacy (Progress) name of the table.
* @param fields4GLtoORM
* Map 4GL field names to ORM names.
* @param legacyPeerName
* The legacy name of the table's peer. May be {@code null}.
* @param isBeforeTable
* Flag indicating the DMO is being built for a before table.
*
* @return the root node of the generated AST.
*/
Aast generateSchemaAst(TempTableBuilder builder,
String ifaceName,
String tableName,
String legacyTableName,
Map<String, String> fields4GLtoORM,
String legacyPeerName,
boolean isBeforeTable)
{
Aast root = new ProgressAst(new CommonToken(ProgressParserTokenTypes.DATABASE,
DatabaseManager.TEMP_TABLE_SCHEMA));
ProgressAst tableSchema = new ProgressAst(new CommonToken(ProgressParserTokenTypes.TEMP_TABLE,
tableName));
tableSchema.putAnnotation("class-name", ifaceName);
tableSchema.putAnnotation("table-name", tableName);
if (legacyPeerName != null)
{
tableSchema.putAnnotation((isBeforeTable ? "after-table" : "before-table"), legacyPeerName);
}
tableSchema.putAnnotation(P2OLookup.HISTORICAL, legacyTableName);
root.addChild(tableSchema);
if (!builder.isUndoable())
{
// TODO: if other properties must be added, extract this node outside if
ProgressAst propsAst = new ProgressAst(new CommonToken(ProgressParserTokenTypes.PROPERTIES,
"properties"));
tableSchema.addChild(propsAst);
propsAst.addChild(new ProgressAst(new CommonToken(ProgressParserTokenTypes.KW_NO_UNDO, "no-undo")));
}
// add field nodes to table
Map<String, Aast> fieldAsts = new HashMap<>();
for (P2JField field : builder.getFields())
{
String fieldName4GL = TextOps.rightTrimLower(field.getName());
String fieldNameORM = fields4GLtoORM.get(fieldName4GL);
ParmType type = field.getType();
int progressType = propTypeToProgress.get(type);
ProgressAst tableField = new ProgressAst(new CommonToken(progressType, fieldNameORM));
fieldAsts.put(fieldName4GL, tableField);
ReservedProperty resPropName = ReservedProperty.getByName(fieldName4GL);
if (resPropName != null)
{
tableField.putAnnotation("__fieldid", (long) resPropName.propId);
}
if (field.getFormat() != null)
{
tableField.putAnnotation(P2OLookup.FORMAT, field.getFormat());
}
if (field.getLabel() != null)
{
tableField.putAnnotation(P2OLookup.LABEL, field.getLabel());
}
if (field.getColumnLabel() != null)
{
tableField.putAnnotation(P2OLookup.COLUMN_LABEL, field.getColumnLabel());
}
String cp = field.getCodePage();
if (cp != null && !cp.trim().equals(""))
{
tableField.putAnnotation("column-codepage", cp);
}
if (field.isSerializeHidden())
{
tableField.putAnnotation("serialize-hidden", field.isSerializeHidden());
}
if (field.getSerializeName() != null)
{
tableField.putAnnotation("serialize-name", field.getSerializeName());
}
tableField.putAnnotation("xml-data-type", field.getXmlDataType());
tableField.putAnnotation("xml-node-type", field.getXmlNodeType());
tableField.putAnnotation("xml-node-name", field.getXmlNodeName());
tableField.putAnnotation("valexp", field.getValidateExpression());
tableField.putAnnotation("valmsg", field.getValidateMessage());
tableField.putAnnotation("help", field.getHelp());
tableField.putAnnotation(P2OLookup.HISTORICAL, field.getName());
if (field.isCaseSensitive())
{
tableField.putAnnotation(P2OLookup.CASE_SENSITIVE, field.isCaseSensitive());
}
if (field.getScale() > 0)
{
tableField.putAnnotation(P2OLookup.DECIMALS, (long) field.getScale());
}
if (field.getMaxWidth() > 0)
{
tableField.putAnnotation("max_wid", (long) field.getMaxWidth());
}
long extent = field.getExtent();
if (extent > 0)
{
tableField.putAnnotation(P2OLookup.EXTENT, extent);
}
tableSchema.addChild(tableField);
// add initial value node to field
BaseDataType initial = field.getInitial();
if (initial != null && !((initial instanceof Text) && TextOps.lengthOf((Text) initial) == 0))
{
Integer valType;
if (initial.isUnknown() || (initial instanceof Text &&
ResourceIdHelper.UNKNOWN_STRING.equals(initial.toStringMessage())))
{
valType = ProgressParserTokenTypes.UNKNOWN_VAL;
}
else
{
switch (type)
{
case CHAR:
valType = ProgressParserTokenTypes.STRING;
break;
case INT64:
case INT:
case RECID:
valType = ProgressParserTokenTypes.NUM_LITERAL;
break;
case DEC:
valType = ProgressParserTokenTypes.DEC_LITERAL;
break;
case LOG:
valType = new logical(initial).equals(logical.TRUE)
? ProgressParserTokenTypes.BOOL_TRUE
: ProgressParserTokenTypes.BOOL_FALSE;
break;
case DATE:
valType = ProgressParserTokenTypes.DATE_LITERAL;
// TODO: what about builtins (today and now)? The above will not handle this correctly --
// we already have a date object at this point and the builtin info is lost
break;
case DT:
valType = ProgressParserTokenTypes.DATETIME_LITERAL;
break;
case DTTZ:
valType = ProgressParserTokenTypes.DATETIME_TZ_LITERAL;
break;
case RAW:
valType = null;
break;
// TODO: other data types?
default:
valType = null;
if (LOG.isLoggable(Level.WARNING))
{
LOG.warning("Unsupported initial value literal type in dynamic temp table field: " +
type + "; (" + legacyTableName + "." + field.getName() + ")");
}
}
}
String valStr;
switch (type)
{
case CHAR:
valStr = initial.toStringMessage();
valStr = character.javaRuntimeToProgressSourceString(valStr);
break;
default:
valStr = initial.toStringMessage();
}
if (valType != null)
{
ProgressAst initNode = new ProgressAst(new CommonToken(ProgressParserTokenTypes.KW_INIT, ""));
tableField.addChild(initNode);
ProgressAst initVal = new ProgressAst(new CommonToken(valType, valStr));
initNode.addChild(initVal);
}
}
}
// add index nodes to table
for (P2JIndex index : builder.getIndexes())
{
String indexName = TextOps.rightTrimLower(index.getName());
ProgressAst indexAst = new ProgressAst(new CommonToken(ProgressParserTokenTypes.INDEX, indexName));
tableSchema.addChild(indexAst);
String legacyName = TextOps.rightTrimNative(index.getLegacyName());
if (legacyName != null)
{
indexAst.putAnnotation(P2OLookup.HISTORICAL, legacyName);
}
if (index.isPrimary())
{
indexAst.putAnnotation(P2OLookup.PRIMARY, true);
}
if (index.isUnique())
{
indexAst.putAnnotation(P2OLookup.UNIQUE, true);
}
if (index.isWord())
{
indexAst.putAnnotation(P2OLookup.WORD, true);
}
// add index fields to index
ArrayList<P2JIndexComponent> comps = index.components();
for (int i = 0; i < comps.size(); i++)
{
P2JIndexComponent comp = comps.get(i);
ProgressAst compAst = new ProgressAst(
new CommonToken(ProgressParserTokenTypes.INDEX_FIELD, comp.getNormalizedName()));
indexAst.addChild(compAst);
if (comp.isDescending())
{
compAst.putAnnotation(P2OLookup.DESCEND, true);
}
// TODO: what about ABBREVIATED and CASE-SENSITIVE options (not handled as part of
// TRPL schema fixups today)?
}
}
assignIds(root, 1);
// now assign each index field's refid to the corresponding field's ID
Aast indexAst = tableSchema.getImmediateChild(ProgressParserTokenTypes.INDEX, null);
while (indexAst != null)
{
Aast compAst = indexAst.getImmediateChild(ProgressParserTokenTypes.INDEX_FIELD, null);
while (compAst != null)
{
String fieldName4GL = compAst.getText();
Aast fieldAst = fieldAsts.get(fieldName4GL);
compAst.putAnnotation("refid", fieldAst.getId());
compAst = indexAst.getImmediateChild(ProgressParserTokenTypes.INDEX_FIELD, compAst);
}
indexAst = tableSchema.getImmediateChild(ProgressParserTokenTypes.INDEX, indexAst);
}
return root;
}
/**
* Class which contains information about a dynamically define DMO, including the compiled
* interface and implementation classes, and mappings between the legacy and converted fields.
*/
private static class DMOInfo
{
/** All classes loaded to support this DMO */
private final Collection<Class<?>> loadedClasses;
/** Mapping of legacy, 4GL field names to converted, property names */
private final Map<String, String> legacyToDmoNames;
/** Mapping of converted, property names to legacy, 4GL field names */
private final Map<String, String> dmoToLegacyNames;
/**
* Constructor.
*
* @param loadedClasses
* All classes loaded to support this DMO.
* @param legacyToDmoNames
* Mapping of legacy, 4GL field names to converted, property names.
* @param dmoToLegacyNames
* Mapping of converted, property names to legacy, 4GL field names.
*/
DMOInfo(Collection<Class<?>> loadedClasses,
Map<String, String> legacyToDmoNames,
Map<String, String> dmoToLegacyNames)
{
this.loadedClasses = loadedClasses;
this.legacyToDmoNames = legacyToDmoNames;
this.dmoToLegacyNames = dmoToLegacyNames;
}
/**
* Get all classes loaded to support this DMO.
*
* @return See above.
*/
Collection<Class<?>> getLoadedClasses()
{
return loadedClasses;
}
/**
* Get the mapping of legacy, 4GL field names to converted, property names.
*
* @return See above.
*/
Map<String, String> getLegacyToDmoNames()
{
return legacyToDmoNames;
}
/**
* Get the mapping of converted, property names to legacy, 4GL field names.
*
* @return See above.
*/
Map<String, String> getDmoToLegacyNames()
{
return dmoToLegacyNames;
}
}
/**
* A hashable key that encapsulates the characteristics which uniquely define the structure of
* a dynamic temp-table; namely, its field and index definitions.
*/
private static class CacheKey
{
/** List of field definitions for a dynamic temp-table */
private final List<P2JField> fields;
/** List of index definition wrappers for a dynamic temp-table */
private final List<IndexKey> indexes;
/**
* Constructor.
*
* @param fields
* Collection of field definitions for a dynamic temp-table.
* @param indexes
* Collection of index definitions for a dynamic temp-table.
*/
CacheKey(Collection<P2JField> fields, Collection<P2JIndex> indexes)
{
this.fields = new ArrayList<>(fields);
this.indexes = new ArrayList<>(indexes.size());
for (P2JIndex next : indexes)
{
this.indexes.add(new IndexKey(next));
}
}
/**
* Use the field and index definitions in this key to calculate a hash code. This
* implementation is consistent with this class' implementation of {@link #equals(Object)}.
*
* @return Hash code.
*/
@Override
public int hashCode()
{
int result = 17;
result = 37 * result + fields.hashCode();
result = 37 * result + indexes.hashCode();
return result;
}
/**
* Use the field and index definitions in this key to determine equivalence with the given
* object. This implementation is consistent with this class' implementation of {@link
* #hashCode()}.
*
* @return {@code true} if this object is equivalent with {@code o}, else {@code false}.
*/
@Override
public boolean equals(Object o)
{
if (this == o)
{
return true;
}
if (o instanceof CacheKey)
{
CacheKey that = (CacheKey) o;
return (this.fields.equals(that.fields) && this.indexes.equals(that.indexes));
}
return false;
}
/**
* To string method concatenating the string values of all fields and indexes.
*
* @return string value of this CacheKey object.
*/
@Override
public String toString()
{
String res = "{ fields : [";
for (int i = 0; i < fields.size(); i++)
{
res += fields.get(i).toStringComplete() + ",";
}
res += "], indexes : [";
for (int i = 0; i < indexes.size(); i++)
{
res += indexes.get(i).toString() + ",";
}
res += "]}";
return res;
}
/**
* Instances of this class are used to wrap instances of {@link P2JIndex}, for purposes of
* producing hash codes suitable for using instances of the enclosing class as hash map
* keys. This class provides stricter implementations of <code>hashCode</code> and
* <code>equals</code> than those provided by <code>P2JIndex</code> itself.
*/
private static class IndexKey
{
/** The wrapped instance of an index definition */
private final P2JIndex index;
/**
* Constructor.
*
* @param index
* Index definition to be wrapped.
*/
IndexKey(P2JIndex index)
{
this.index = index;
}
/**
* Use the content of the wrapped index definition to produce a hash code. This is used
* instead of {@link P2JIndex#hashCode()}, because the latter implementation of this
* method is special-purpose, taking only the names of index components into account and
* ignores other index metadata.
*
* @return Hash code.
*/
@Override
public int hashCode()
{
int result = 17;
result = 37 * result + index.getName().hashCode();
result = 37 * result + (index.isPrimary() ? 1 : 0);
result = 37 * result + (index.isUnique() ? 1 : 0);
result = 37 * result + (index.isWord() ? 1 : 0);
ArrayList<P2JIndexComponent> comps = index.components();
for (int i = 0; i < comps.size(); i++)
{
P2JIndexComponent comp = comps.get(i);
result = 37 * result + comp.hashCode();
}
return result;
}
/**
* Overrides the default implementation to provide an implementation consistent with
* {@link #hashCode()}.
*
* @return {@code true} if this object is equivalent with {@code o}, else {@code false}.
*/
@Override
public boolean equals(Object o)
{
if (this == o)
{
return true;
}
if (o instanceof IndexKey)
{
IndexKey that = (IndexKey) o;
if (!this.index.getName().equals(that.index.getName()) ||
this.index.isPrimary() != that.index.isPrimary() ||
this.index.isUnique() != that.index.isUnique() ||
this.index.isWord() != that.index.isWord() ||
this.index.size() != that.index.size())
{
return false;
}
ArrayList<P2JIndexComponent> comps = index.components();
for (int i = 0; i < comps.size(); i++)
{
P2JIndexComponent comp = comps.get(i);
if (!comp.equals(that.index.getComponent(i)))
{
return false;
}
}
return true;
}
return false;
}
/**
* To string method returning the string value of this objects P2JIndex field.
*
* @return string value of this IndexKey object.
*/
@Override
public String toString()
{
return index.toString();
}
}
}
/**
* DMO data to be cached for a dynamically defined temp-table.
*/
private static class CacheData
{
/** DMO info object */
private final DMOInfo dmoInfo;
/** Fully qualified DMO interface name */
private final String dmoIfaceName;
/** Fully qualified DMO implementation class name */
private final String dmoClassName;
/** Database table name */
private final String sqlTableName;
/** List of index definitions using converted names */
private final ArrayList<P2JIndex> fwdIndices;
/** Map of fully qualified class names to classes */
private final Map<String, Class<?>> classes;
/**
* Constructor.
*
* @param dmoInfo
* DMO info object.
* @param dmoIfaceName
* Fully qualified DMO interface name.
* @param dmoClassName
* Fully qualified DMO implementation class name.
* @param sqlTableName
* Database table name.
* @param fwdIndices
* List of index definitions using converted names.
* @param classes
* Map of fully qualified class names to classes.
*/
CacheData(DMOInfo dmoInfo,
String dmoIfaceName,
String dmoClassName,
String sqlTableName,
ArrayList<P2JIndex> fwdIndices,
Map<String, Class<?>> classes)
{
this.dmoInfo = dmoInfo;
this.dmoIfaceName = dmoIfaceName;
this.dmoClassName = dmoClassName;
this.sqlTableName = sqlTableName;
this.fwdIndices = fwdIndices;
this.classes = classes;
}
/**
* Get DMO information.
*
* @return See above.
*/
DMOInfo getDMOInfo()
{
return dmoInfo;
}
/**
* Get fully qualified DMO interface name.
*
* @return See above.
*/
String getDMOIfaceName()
{
return dmoIfaceName;
}
/**
* Get fully qualified DMO implementation class name.
*
* @return See above.
*/
String getDMOClassName()
{
return dmoClassName;
}
/**
* Get database table name.
*
* @return See above.
*/
String getSqlTableName()
{
return sqlTableName;
}
/**
* Get list of index definitions using converted names.
*
* @return See above.
*/
ArrayList<P2JIndex> getFwdIndices()
{
return fwdIndices;
}
/**
* Get map of fully qualified class names to classes.
*
* @return See above.
*/
Map<String, Class<?>> getClasses()
{
return Collections.unmodifiableMap(classes);
}
}
/**
* Context local information, used to clean up resources when the context ends and to respond
* to queries about temp-table resources in use.
* <p>
* TODO: currently, the cleanup code is based on an incorrect assumption that the context's
* state is reset more frequently than it actually is, AND, in some cases, we are cleaning up
* resources prematurely, causing persistence errors when detached objects in an old Hibernate
* session are reattached to a new one. The temporary workaround for this is a high cache
* limit, to prevent the cache from expiring elements.
*/
private class Context
implements Finalizable
{
/** Map of DMO interfaces to sets of users of these interfaces */
private final Map<Class<?>, Set<TempTableBuilder>> dmoUsers = new IdentityHashMap<>();
/** Map of DMO interfaces to DMO information objects */
private final Map<Class<? extends DataModelObject>, DMOInfo> dmoMap = new IdentityHashMap<>();
/** Temp-table builders which are no longer in use in this context */
private List<TempTableBuilder> finishedBuilders = null;
/**
* Constructor. Registers this object as a global finalizable.
*/
private Context()
{
TransactionManager.registerFinalizable(this, true);
}
/**
* Clean up local context - drop tables and DMO definitions.
*/
public void finished()
{
if (dmoUsers.isEmpty())
{
return;
}
if (finishedBuilders == null)
{
finishedBuilders = new ArrayList<>();
}
for (Set<TempTableBuilder> set : dmoUsers.values())
{
finishedBuilders.addAll(set);
}
for (TempTableBuilder builder : finishedBuilders)
{
Class<? extends DataModelObject> dmoIface = builder.getDMOInterface();
Set<TempTableBuilder> users = dmoUsers.get(dmoIface);
if (users == null)
{
continue;
}
users.remove(builder);
}
}
/**
* No-op.
*/
public void deleted()
{
}
/**
* No-op.
*/
public void iterate()
{
}
/**
* No-op.
*/
public void retry()
{
}
/**
* Clean up DMO resources no longer needed by temp-table builders in this context.
*/
private void cleanup()
{
if (finishedBuilders == null)
{
return;
}
// DMO interfaces which already have been cleaned up
Set<Class<?>> processedDMOs = Collections.newSetFromMap(new IdentityHashMap<>());
for (TempTableBuilder builder : finishedBuilders)
{
try
{
Class<? extends DataModelObject> dmoIface = builder.getDMOInterface();
if (dmoIface == null || !processedDMOs.add(dmoIface))
{
continue;
}
dmoUsers.remove(dmoIface);
dmoMap.remove(dmoIface);
}
finally
{
dmoResourceManager.builderInactive(builder);
}
}
finishedBuilders = null;
}
/**
* Register a dynamic temp-table builder in this context.
*
* @param builder
* Temp-table builder to be registered.
*/
private void registerBuilder(TempTableBuilder builder)
{
dmoResourceManager.builderActive(builder);
Class<? extends DataModelObject> dmoIface = builder.getDMOInterface();
Set<TempTableBuilder> users = dmoUsers.computeIfAbsent(dmoIface, k -> new HashSet<>());
users.add(builder);
}
/**
* Get the DMO information object associated with the given DMO interface.
*
* @param dmoIface
* DMO interface.
*
* @return See above.
*/
private DMOInfo getDMOInfo(Class<? extends DataModelObject> dmoIface)
{
return dmoMap.get(dmoIface);
}
/**
* Associate a DMO information object with the given DMO interface.
*
* @param dmoIface
* DMO interface.
* @param dmoInfo
* DMO information object.
*
* @return {@code true} if {@code dmoIface} was not already associated with a
* different object, else {@code false}.
*/
private boolean registerDMOInfo(Class<? extends DataModelObject> dmoIface, DMOInfo dmoInfo)
{
return dmoMap.put(dmoIface, dmoInfo) == null;
}
}
/**
* An instance of this class manages resources requiring cleanup, across all contexts. This
* object is a registered cache listener, receiving notifications when elements are expired
* from the cache. These are tracked until the are no longer actively in use by any context,
* then their associated resources are cleaned/reclaimed.
* <p>
* TODO: this cleanup currently is broken, as it can release resources that are still being
* used by a Hibernate session. The temporary workaround is to define a high cache limit,
* but this needs to be fixed.
*/
private class DMOResourceManager
implements CacheExpiryListener<CacheKey, CacheData>
{
/** Temp-table builders which are currently active across all contexts */
private final Set<TempTableBuilder> activeBuilders = new HashSet<>();
/** Data which has expired from the cache, but has not yet been released */
private final Set<CacheData> expiredCacheData = new HashSet<>();
/**
* Constructor which registers with the dynamic DMO cache.
*/
DMOResourceManager()
{
// cache.addCacheExpiryListener(this);
}
/**
* This method is invoked when the DMO cache has expired one or more elements. We use it to
* know when a DMO's mapping can be deregistered with the Hibernate session factory and the
* associated DMO interface and implementation class(es) can be released.
*
* @param event
* An object containing information about the event.
*/
public void cacheExpired(CacheExpiryEvent<CacheKey, CacheData> event)
{
Map<CacheKey, CacheData> expired = event.getExpiredEntries();
for (CacheData data : expired.values())
{
expiredCacheData.add(data);
}
}
/**
* Track the given temp-table builder as being in use.
*
* @param builder
* Temp-table builder to track.
*/
void builderActive(TempTableBuilder builder)
{
synchronized (cacheLock)
{
activeBuilders.add(builder);
}
}
/**
* Stop tracking the given temp-table builder as being in use.
*
* @param builder
* Temp-table builder to stop tracking.
*/
void builderInactive(TempTableBuilder builder)
{
synchronized (cacheLock)
{
activeBuilders.remove(builder);
}
}
/**
* Release resources which have expired from the dynamic DMO cache, which are no longer in
* active use. This involves deregistering DMO types with Hibernate and releasing the DMO
* interface and class(es) that have been compiled for a dynamic temp-table.
* <p>
* TODO: this cleanup currently is broken, as it can release resources that are still being
* used by a Hibernate session. Also, if there are expired cache data, but they are still
* pinned by active builders, this method could be expensive to execute every time a new
* table is defined. Better to run it periodically in a dedicated thread. The temporary
* workaround is to define a high cache limit, but this needs to be fixed.
*/
void releasePendingResources()
{
// TODO: probably no longer needed, or if needed, needs re-implementation
/*
synchronized (cacheLock)
{
if (expiredCacheData.isEmpty())
{
return;
}
// collect all expired data resources
Map<Class<?>, CacheData> deadCacheData = new HashMap<>();
for (CacheData data : expiredCacheData)
{
Map<String, Class<?>> classMap = data.getClasses();
String dmoIfaceName = data.getDMOIfaceName();
Class<?> dmoIface = classMap.get(dmoIfaceName);
deadCacheData.put(dmoIface, data);
}
// collect all currently active DMO interfaces and PersistenceClass instances
Map<Class<?>, PersistentClass> pinnedResources = new HashMap<>();
for (TempTableBuilder ttb : activeBuilders)
{
pinnedResources.put(ttb.getDMOInterface(), ttb.getPersistentClass());
}
Set<Class<?>> pinnedDMOs = pinnedResources.keySet();
// remove all the expired entries which still are pinned by active builders
deadCacheData.keySet().removeAll(pinnedDMOs);
MutableSessionFactoryImpl sessionFactory = (MutableSessionFactoryImpl)
DatabaseManager.getSessionFactory(DatabaseManager.TEMP_TABLE_DB);
for (CacheData data : deadCacheData.values())
{
// release Hibernate resources
String dmoImplName = data.getDMOClassName();
PersistentClass pc = cfg.getClassMapping(dmoImplName); // caller has cfg lock
cfg.remove(pc);
sessionFactory.remove(pc);
// release class resources
DMOInfo info = data.getDMOInfo();
Collection<Class<?>> loadedClasses = info.getLoadedClasses();
for (Class<?> clz : loadedClasses)
{
// systemInMemoryClassLoader.deregister(clz.getName());
}
// remove this item from the set of expired entries requiring cleanup
expiredCacheData.remove(data);
}
}
*/
}
}
}