public class ImportWorker.Library
extends java.lang.Object
Constructor and Description |
---|
Library() |
Modifier and Type | Method and Description |
---|---|
void |
addSequence(java.lang.String historical,
java.lang.String sqlName)
Adds a sequence to the map of sequences to be initialized from the
sequences export file.
|
private int |
adjustedCounter(int counter,
int badIndex,
int dropCount)
Adjust the given counter to take the specified
badIndex
and dropCount values into account. |
private void |
createIndex(org.hibernate.Session session,
P2JIndex index,
boolean unique)
Create an index based upon the specified definition.
|
private void |
createIndexes(java.util.Collection<P2JIndex> indexes,
boolean unique)
Create all indexes in the specified list which match the uniqueness
specified by
unique . |
private void |
dropIndex(org.hibernate.Session session,
P2JIndex index)
Drop an index based upon the specified definition.
|
private void |
dropIndexes(java.util.Collection<P2JIndex> indexes)
Sequentially drop all of the specified indexes from the database.
|
private void |
generateCreateIndexes(java.io.OutputStream os,
java.lang.String delimiter,
java.util.Collection<P2JIndex> indexes,
boolean unique)
Sequentially create the DDL to create all of the specified indexes from
the database, write this SQL to the given stream.
|
private void |
generateDropIndexes(java.io.OutputStream os,
java.lang.String delimiter,
java.util.Collection<P2JIndex> indexes)
Sequentially create the DDL to drop all of the specified indexes from
the database, write this SQL to the given stream.
|
void |
generateIndexDDL(java.lang.String dialectName,
java.lang.String filename,
java.lang.String delimiter,
java.lang.String table,
java.util.List indexes)
Generate the DDL to drop and recreate the list of indexes for the given table.
|
private void |
importAsync(ImportWorker.ImportBundle bundle,
java.lang.String dateOrder,
java.lang.Long yearOffset)
Import a single table on a dedicated import thread.
|
int |
importTable(ImportWorker.SqlRecordLoader loader,
java.util.List<P2JIndex> indexes)
Import all data for a table into the database, using the given record
loader and current session factory.
|
org.hibernate.SessionFactory |
initialize(org.hibernate.cfg.Configuration config,
java.util.Properties props)
Configure the given Hibernate configuration object using the
information specified in the given configuration file and build
the resulting Hibernate session factory.
|
long |
initializeSequences(java.lang.String path,
java.lang.String seqDumpFile)
Sets the values of the sequences.
|
private void |
messageWithStats(java.lang.String message)
Print a message to stdout, followed by the current import statistics.
|
private ImportWorker.ImportBundle |
nextBundle()
Return the next import bundle which is eligible for import.
|
void |
prepareImport(ImportWorker.SqlRecordLoader loader,
java.util.List indexes)
Store the given objects in preparation for an import run.
|
void |
reindex(java.lang.String table,
java.util.List indexes)
Drop and recreate the list of indexes for the given table.
|
private int |
removeFailingRecord(java.util.List<java.lang.Object> records,
org.hibernate.HibernateException exc,
int counter,
java.io.File dataFile)
Remove a problem record from a batch of records which failed on import, if it can be
determined which record caused the problem.
|
long |
runImport()
Commence an import run, potentially on multiple threads.
|
void |
setMaximumThreads(int maximumThreads)
Set the maximum allowable threads.
|
public org.hibernate.SessionFactory initialize(org.hibernate.cfg.Configuration config, java.util.Properties props) throws java.sql.SQLException
In addition, the names of all text columns are mapped by table name for use during index creation.
config
- Hibernate configuration object.props
- The Hibernate properties.java.sql.SQLException
- if any error occurs collecting database metadata.org.hibernate.HibernateException
- if any error occurs configuring Hibernate or building the
session factory.public void setMaximumThreads(int maximumThreads)
maximumThreads
- Maximum number of import threads which may be active at one
time.public void prepareImport(ImportWorker.SqlRecordLoader loader, java.util.List indexes)
runImport()
method is invoked.loader
- Record loader helper which contains all information
necessary to find the import data and to map each record
in the import file to a DMO of the appropriate class.indexes
- A list of index definitions which must be applied to the
table before (for unique indexes) and after (for non-unique
indexes) data import.setMaximumThreads(int)
public long runImport() throws java.lang.InterruptedException
prepareImport(com.goldencode.p2j.schema.ImportWorker.SqlRecordLoader, java.util.List)
will be imported.
This method organizes the tables into an order which is intended to maximize throughput throughout the import run, by scheduling those tables with the most time-consuming import path to run earliest in the process.
Multiple, parallel threads are used to import multiple tables at once.
This method will block until all prepared tables have been imported.
java.lang.InterruptedException
- if the thread on which this method is invoked is
interrupted.prepareImport(com.goldencode.p2j.schema.ImportWorker.SqlRecordLoader, java.util.List)
,
setMaximumThreads(int)
public void addSequence(java.lang.String historical, java.lang.String sqlName)
historical
- the original name of the sequence.sqlName
- the converted, SQL-compliant name of the sequence.public long initializeSequences(java.lang.String path, java.lang.String seqDumpFile)
seqDumpFile
- The file to which the sequences' values have been exportedpublic int importTable(ImportWorker.SqlRecordLoader loader, java.util.List<P2JIndex> indexes)
Stream.readField(com.goldencode.p2j.util.BaseDataType[])
to scan the input file. Each record's data is stored in a new
instance of a data model object (DMO). Hibernate is used to persist
the DMO to the backing database.
If the DMO refers to any other DMOs, the necessary queries are executed to resolve these references before the object is persisted. This ensures foreign key associations are correctly resolved at import time.
The import uses JDBC batching via Hibernate to send data to the database in batches rather than a single record at a time. Batch size is retrieved from the Hibernate configuration, and is set to a default if not specified in the config.
This method is designed to be fairly fault tolerant. Database errors which are the result of data which does not conform to the expected schema generally are handled by dropping the offending record, adding a log entry to report the nature of the error, and continuing with the next record, if any. Specifically, the following conditions are recoverable:
The following errors are not recoverable:
Errors are logged as configured for the current VM.
loader
- Record loader helper which contains all information
necessary to find the import data and to map each record
in the import file to a DMO of the appropriate class.indexes
- A list of index definitions which must be applied to the
table before (for unique indexes) and after (for non-unique
indexes) data import.public void generateIndexDDL(java.lang.String dialectName, java.lang.String filename, java.lang.String delimiter, java.lang.String table, java.util.List indexes) throws java.io.IOException
table
which are not within the list are ignored. Also, the redundant indexes are
filtered out so only the minimum list of required indexes is written to the given output
filename. The unique indexes are created before other remaining ones.dialectName
- Name of the Hibernate dialect to use for DDL generation.filename
- File to which the DDL should be appended.delimiter
- End of statement delimiter.table
- Table which is to be reindexed.indexes
- List of index definitions.java.io.IOException
public void reindex(java.lang.String table, java.util.List indexes) throws java.sql.SQLException
table
which are not within the list
are ignored.
Indexes to be dropped are identified by the names provided in the list of index definitions; the remainder of the definition is not used to identify indexes to be dropped.
table
- Table which is to be reindexed.indexes
- List of index definitions.java.sql.SQLException
- if any error occurs opening or closing JDBC statements.org.hibernate.HibernateException
- if any error occurs opening or closing sessions, or obtaining database
connections.private ImportWorker.ImportBundle nextBundle() throws java.lang.InterruptedException
null
.
This method will block if the maximum number of active threads has been reached. It will unblock once a slot becomes available.
null
if no bundles remain in the queue.java.lang.InterruptedException
- if the running thread is interrupted while waiting for an
import thread slot to become available.private void importAsync(ImportWorker.ImportBundle bundle, java.lang.String dateOrder, java.lang.Long yearOffset)
Once a thread slot opens up, a new thread is launched to import the table associated
with bundle
. If this table is dependent upon the import of other tables,
and not all of these tables have yet been imported, the new thread will block until such
time as all such dependencies have been resolved. If any tables upon which the target
table is dependent have encountered fatal errors, the import of the target table is
aborted and the target table is added to the set of tables with fatal errors.
bundle
- Import bundle which holds all information necessary to import data into a
single table.dateOrder
- The date format to be used when importing if PSC footer not found in file.yearOffset
- The offset year to be used when importing if PSC footer not found in file.setMaximumThreads(int)
private void messageWithStats(java.lang.String message)
message
- The core message to be printed. May be null
if only statistics are desired.private void generateDropIndexes(java.io.OutputStream os, java.lang.String delimiter, java.util.Collection<P2JIndex> indexes) throws java.io.IOException
os
- Output destination.delimiter
- End of statement delimiter.indexes
- Collection of indexes to be dropped.java.io.IOException
private void dropIndexes(java.util.Collection<P2JIndex> indexes) throws java.sql.SQLException
indexes
- Collection of indexes to be dropped.java.sql.SQLException
- if any error occurs creating or closing a statement.org.hibernate.HibernateException
- if any error occurs opening or closing a session, or
obtaining a database connection.private void dropIndex(org.hibernate.Session session, P2JIndex index) throws org.hibernate.HibernateException
session
- Hibernate session to use.index
- Definition of the index to be dropped.org.hibernate.HibernateException
- if an error occurs at the database during execution of the
index drop statement.private void generateCreateIndexes(java.io.OutputStream os, java.lang.String delimiter, java.util.Collection<P2JIndex> indexes, boolean unique) throws java.io.IOException
os
- Output destination.delimiter
- End of statement delimiter.indexes
- Collection of indexes to be created.java.io.IOException
private void createIndexes(java.util.Collection<P2JIndex> indexes, boolean unique) throws java.sql.SQLException
unique
. If none match, no indexes are
created.indexes
- A list of index definitions.unique
- If true
, all unique indexes in the list are
created; if false
, all non-unique indexes in
the list are created.java.sql.SQLException
- if an error occurs at the database during index creation.private void createIndex(org.hibernate.Session session, P2JIndex index, boolean unique) throws org.hibernate.HibernateException
session
- Hibernate session to use.index
- Definition of the index to be created.unique
- If true
, create the index as a unique index,
false
to omit the unique constraint.org.hibernate.HibernateException
- if an error occurs at the database during index creation.private int adjustedCounter(int counter, int badIndex, int dropCount)
badIndex
and dropCount
values into account.counter
- Number of total records read, but not necessarily imported.badIndex
- Index of the first problem record in an import batch, or
-1 if there was no problem record. The index is zero-based,
relative to the beginning of the batch.dropCount
- Cumulative number of problem records which have been
dropped from the import of the current table.private int removeFailingRecord(java.util.List<java.lang.Object> records, org.hibernate.HibernateException exc, int counter, java.io.File dataFile) throws org.hibernate.HibernateException
exc
until an exception of type
BatchUpdateException
is found. If such a cause is found, the number of
successful updates is extracted. This number represents the index of the first record
which was not successful. It is this record which is removed from the batch.
An error message is logged recording the 1-based index of the problem record, the data
file from which it was read, and the nature of the error as extracted from exc
.
The purpose of this method is to preserve information regarding the failure, while dropping the problem record in order to clean up the current batch of records to enable a retry of the batch insert.
records
- List of records which comprised the failed batch.exc
- Exception which was caught on batch insert failure.counter
- Number of total records read up to this point. This number will be at most
batchSize
higher than the number of records successfully imported to
this point.dataFile
- Data file from which data is being read, for error logging purposes.records
, of the
problem record.org.hibernate.HibernateException
- in the event exc
doesn't contain a chained BatchUpdateException
.
In this case, the index of a problem record cannot be determined, and it is
likely that the error is more severe than a single bad record.