JsonImport.java

/*
** Module   : JsonImport.java
** Abstract : Read JSON content into a temp-table.
**
** Copyright (c) 2017-2024, Golden Code Development Corporation.
**
** -#- -I- --Date-- ---------------------------------------Description---------------------------------------
** 001 ECF 20190123 Created initial version.
** 002 OM  20190327 Renamed DataSource to avoid conflicts with DataSet source.
** 003 OM  20190818 Improve reading and validation support.
** 004 ECF 20200906 New ORM implementation.
**     CA  20200906 Batch error fix.
** 005 OM  20210108 Improve reading compatibility with ABL JSON files.
**     CA  20210510 Fixed JSON and XML serialization when XML-NODE-NAME/SERIALIZE-NAME field options are set.
**                  When reading a temp-table from a JSON file, the JSON may include other fields which do not
**                  map to a temp-table column - these are silently ignored.
**                  Fixed reading a temp-table with extent fields (they were assumed to be a child temp-table).
**     CA  20210709 JSON parse must use a pristine byte stream and use the reader to decode it.
**     OM  20210825 Added support methods for READ-JSON methods.
**     OM  20210922 Fixed default values for XML-NODE-NAME and SERIALIZE-NAME.
**     OM  20211020 Added _DATASOURCE_ROWID property.
**     CA  20220218 Ensure the group and decimal separators are ',' and '.' during record read.
**     OM  20220328 Stop and return false if validation errors occur while reading a table.
**     OM  20220420 Rewrite the read in REPLACE mode.
**     CA  20220420 Reading from JSON allows a table field to not exist in the JSON, which will default to its
**                  initial value.
**     CA  20221012 Fixed JSON import of tables with extent fields, when the source is a JsonObject.
**     TJD 20220504 Upgrade do Java 11 minor changes
**     OM  20220524 Fixed reading a dataset/table with explicit serialize-name attribute.
**     OM  20220525 Fixed case-sensitivity issues related to fields, tables, and datasets.
**     CA  20220526 Fixed JSON deserialization of decimal values.
**     OM  20220914 Use MAX-WIDTH to generate custom sized varchar columns.
**     SVL 20230110 P2JIndex.components() provides direct access to the components array in order to improve
**                  performance.
**     CA  20230116 Avoid using handle.unwrap, handle.getReference or other BDT usage from within FWD runtime.
** 006 IAS 20230325 Fixed source validation. Implemented support for dynamic TEMP-TABLE right 
**                  after CREATE TEMP-TABLE.
**     IAS 20230327 Fixed READ-JSON for static DATA-SET with NESTED/FOREIGN-KEY-HIDDEN relations.
**     IAS 20230414 Added support for inferring empty dynamic DATA-SET from JSON data.   
**     IAS 20230420 Added support for inferring empty dynamic DATA-SET from JSON data w/o root element.   
** 007 GBB 20230512 Logging methods replaced by CentralLogger/ConversionStatus.
** 008 RAA 20230525 Creating a new BaseDataType instance is now done using BaseDataTypeFactory.
** 009 IAS 20230526 Extracted common logic to the 'Importer'.
** 010 CA  20230724 Further reduce context-local usage.
** 011 OM  20231027 Nested child tables are not required to be found after all fields from parent table.
**                  The nested table lookup are done by serialize name.
** 012 EXT 20231122 JSON input must always be UTF-8, regardless of cpinternal (refs #7944) (by Anton)
** 013 CA  20240320 Avoid BDTs within internal FWD runtime.  Replaced iterators with Java 'for', where it 
**                  applies.
** 014 OM  20240502 When reading a table, both the name and the serialization name will match it.
** 015 TJD 20240123 Java 17 compatibility updates and compiler warnings fix
** 016 LS  20240711 Fixed READ-JSON, so that the comparison between a table field and a key from JsonObject
**                  is case-insensitive.
** 017 AP  20240918 Fixed READ-JSON for JsonObjects with SERIALIZE-HIDDEN
** 018 OM  20241128 Multi-tenant runtime support: selected the proper persistence context, eventually
**                  based on [sharedDb] parameter.
**                  Implementation of [setMultiTenantAlternative] method.
** 019 AS  20250305 Added dataset/table name matching for reading operations.
*/

/*
** This program is free software: you can redistribute it and/or modify
** it under the terms of the GNU Affero General Public License as
** published by the Free Software Foundation, either version 3 of the
** License, or (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
** GNU Affero General Public License for more details.
**
** You may find a copy of the GNU Affero GPL version 3 at the following
** location: https://www.gnu.org/licenses/agpl-3.0.en.html
** 
** Additional terms under GNU Affero GPL version 3 section 7:
** 
**   Under Section 7 of the GNU Affero GPL version 3, the following additional
**   terms apply to the works covered under the License.  These additional terms
**   are non-permissive additional terms allowed under Section 7 of the GNU
**   Affero GPL version 3 and may not be removed by you.
** 
**   0. Attribution Requirement.
** 
**     You must preserve all legal notices or author attributions in the covered
**     work or Appropriate Legal Notices displayed by works containing the covered
**     work.  You may not remove from the covered work any author or developer
**     credit already included within the covered work.
** 
**   1. No License To Use Trademarks.
** 
**     This license does not grant any license or rights to use the trademarks
**     Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
**     of Golden Code Development Corporation. You are not authorized to use the
**     name Golden Code, FWD, or the names of any author or contributor, for
**     publicity purposes without written authorization.
** 
**   2. No Misrepresentation of Affiliation.
** 
**     You may not represent yourself as Golden Code Development Corporation or FWD.
** 
**     You may not represent yourself for publicity purposes as associated with
**     Golden Code Development Corporation, FWD, or any author or contributor to
**     the covered work, without written authorization.
** 
**   3. No Misrepresentation of Source or Origin.
** 
**     You may not represent the covered work as solely your work.  All modified
**     versions of the covered work must be marked in a reasonable way to make it
**     clear that the modified work is not originating from Golden Code Development
**     Corporation or FWD.  All modified versions must contain the notices of
**     attribution required in this license.
*/

package com.goldencode.p2j.persist.serial;

import java.io.*;
import java.math.*;
import java.nio.charset.*;
import java.util.*;
import java.util.function.*;
import com.goldencode.p2j.util.logging.*;
import org.apache.commons.lang3.tuple.*;
import com.fasterxml.jackson.core.*;
import com.fasterxml.jackson.core.JsonParser;
import com.fasterxml.jackson.databind.*;
import com.goldencode.p2j.convert.*;
import com.goldencode.p2j.oo.json.*;
import com.goldencode.p2j.oo.json.objectmodel.*;
import com.goldencode.p2j.oo.lang.*;
import com.goldencode.p2j.persist.*;
import com.goldencode.p2j.persist.Record;
import com.goldencode.p2j.persist.orm.*;
import com.goldencode.p2j.util.*;
import static com.fasterxml.jackson.core.JsonToken.*;
import static com.goldencode.p2j.persist.serial.SerializeOptions.*;

/**
 * An implementation of the 4GL READ-JSON method, which parses a JSON stream and reads the content
 * into a temp-table.
 * <p>
 * Runtime support is partial at this time; not all modes and features are supported. PRODATASET
 * is not supported at this time.
 */
public final class JsonImport
extends Importer
implements TableReader
{
   /** Logger. */
   private static final CentralLogger LOG = CentralLogger.get(JsonImport.class);
   
   /** The primary index which drives the duplicate resolution, in case the read mode is REPLACE. */
   private P2JIndex primaryIndex = null;
   
   /** JSON parser */
   private JsonParser parser = null;
   
   /**
    * Constructor which immediately performs the import of a DATASET or a TEMP-TABLE.
    *
    * @param   source
    *          Data source which normalizes access of various media to an input stream.
    * @param   mode
    *          Read mode to determine how records are stored and how non-unique records are
    *          handled. Not currently honored, except temp-table records are deleted if mode
    *          is EMPTY.
    */
   public JsonImport(SourceData source, String mode)
   {
      super(source, mode);
   }
   
   /**
    * Validates the {@code source} and create the dataset structure by building all temp-tables according to
    * source JsonObject.
    * 
    * @param   ds
    *          The {@code Dataset} to be populated with fields and finally prepared.
    * @param   source
    *          The data source.
    *
    * @return  {@code true} if operation is successful and {@code false} otherwise.
    */
   public static boolean createDatasetStructure(DataSet ds, SourceData source)
   {
      String sourceType = source.getType();
      BaseDataType bdtSrc = source.getSource();
      if (!(bdtSrc instanceof object))
      {
         ErrorManager.recordOrShowError(19079, sourceType, "READ-JSON");
         // <argument> argument for <method-name> is not valid. (19079)
         return false;
      }
      
      _BaseObject_ baseObj = ((object<?>) bdtSrc).ref();
      if (!(baseObj instanceof JsonObject))
      {
         ErrorManager.recordOrShowError(19079, sourceType, "READ-JSON");
         // <argument> argument for <method-name> is not valid. (19079)
         return false;
      }
      
      Iterator<Map.Entry<String, Object>> it = ((JsonObject) baseObj).getProperties().iterator();
      if (!it.hasNext())
      {
         return true; // no dataset stored here
      }
      
      Map.Entry<String, Object> next = it.next();
      String dsName = next.getKey();
      
      Object val = next.getValue();
      if (!(val instanceof JsonObject))
      {
         ErrorManager.recordOrShowError(15360, "bracket", "");
         // Error parsing JSON: unexpected token: <token>. (15360)
         return false;
      }
      
      if (it.hasNext())
      {
         ErrorManager.recordOrShowError(15374, "", "");
         //  Unable to infer Temp-Table or dataset schema from JSON Data. (15374)
         return false;
      }
      
      ds.name(dsName);
      for (Map.Entry<String, Object> tableDef : ((JsonObject) val).getProperties())
      {
         String tableName = tableDef.getKey();
         handle tth = ds.bufferHandle(tableName);
         if (tth != null && !tth.isUnknown() && tth._isValid())
         {
            // skip it, this table is already defined
            continue;
         }
         
         Object defValue = tableDef.getValue(); // must be an JsonArray which stores all table's records
         if (!(defValue instanceof JsonArray))
         {
            ErrorManager.recordOrShowError(15374, "", "");
            //  Unable to infer Temp-Table or dataset schema from JSON Data. (15374)
            return false;
         }
         
         JsonArray jsonArray = (JsonArray) defValue;
         int firstElementType = jsonArray.getType(new integer(1)).intValue();
         if (firstElementType != JsonBackend.JSON_OBJECT)
         {
            ErrorManager.recordOrShowError(15358, "bracket", JsonBackend.getJsonDataTypeName(firstElementType));
            // Error parsing JSON: expected <token>, but found <token>. (15358)
            return false;
         }
         
         object<? extends JsonObject> jsonObjectHolder = jsonArray.getJsonObject(new integer(1));
         JsonObject jsonObject = jsonObjectHolder.ref();
         
         tth = new handle(false);
         TempTableBuilder.create(tth);
         TempTableBuilder builder = (TempTableBuilder) tth.getResource();
         
         Iterable<Map.Entry<String, Object>> fields = jsonObject.getProperties();
         for (Map.Entry<String, Object> entry : fields)
         {
            P2JField field = inferFieldType(entry.getKey(), entry.getValue());
            if (field == null)
            {
               ErrorManager.recordOrShowError(15360, "bracket", "");
               // Error parsing JSON: unexpected token: <token>. (15360)
               ErrorManager.recordOrShowError(15374, "", "");
               // Unable to infer Temp-Table or dataset schema from JSON Data. (15374)
               return false;
            }
            builder.addField(field);
         }
         
         if (!builder.tempTablePrepare(tableName).booleanValue())
         {
            ErrorManager.recordOrShowError(15374, "", "");
            //  Unable to infer Temp-Table or dataset schema from JSON Data. (15374)
            return false;
         }
         
         // table buffer ready to be added to dataset
         ds.addBuffer(builder.defaultBufferHandleNative());
      }
      
      return true;
   }
   
   /**
    * Validates the {@code source} and create the table structure for {@code builder} temp-table.
    * 
    * @param   builder
    *          The {@code TempTableBuilder} to be populated with fields and finally prepared.
    * @param   source
    *          The data source.
    *
    * @return  {@code true} if operation is successful, {@code false} on failure, or {@code null}
    *          if source is not JSONOBJECT or JSONARRAY.
    */
   public static Boolean createTableStructure(TempTableBuilder builder, SourceData source)
   {
      String sourceType = source.getType();
      switch (sourceType.toUpperCase())
      {
         case "JSONOBJECT":
         {
            BaseDataType bdtSrc = source.getSource();
            if (!(bdtSrc instanceof object))
            {
               ErrorManager.recordOrShowError(19079, sourceType, "READ-JSON");
               // <argument> argument for <method-name> is not valid. (19079)
               return false;
            }
            
            _BaseObject_ baseObj = ((object<?>) bdtSrc).ref();
            if (!(baseObj instanceof JsonObject))
            {
               ErrorManager.recordOrShowError(19079, sourceType, "READ-JSON");
               // <argument> argument for <method-name> is not valid. (19079)
               return false;
            }
            
            JsonObject jsonObject = (JsonObject) baseObj;
            Iterable<Map.Entry<String, Object>> it = jsonObject.getProperties();
            for (Map.Entry<String, Object> entry : it)
            {
               P2JField field = inferFieldType(entry.getKey(), entry.getValue());
               if (field == null)
               {
                  ErrorManager.recordOrShowError(15360, "bracket", "");
                  // Error parsing JSON: unexpected token: <token>. (15360)
                  ErrorManager.recordOrShowError(15374, "", "");
                  // Unable to infer Temp-Table or dataset schema from JSON Data. (15374)
                  return false;
               }
               builder.addField(field);
            }
            
            return builder.tempTablePrepare("NewTable").booleanValue();
         }
         
         case "JSONARRAY":
         {
            BaseDataType bdtSrc = source.getSource();
            if (!(bdtSrc instanceof object))
            {
               ErrorManager.recordOrShowError(19079, sourceType, "READ-JSON");
               // <argument> argument for <method-name> is not valid. (19079)
               return false;
            }
            
            _BaseObject_ baseObj = ((object<?>) bdtSrc).ref();
            if (!(baseObj instanceof JsonArray))
            {
               ErrorManager.recordOrShowError(19079, sourceType, "READ-JSON");
               // <argument> argument for <method-name> is not valid. (19079)
               return false;
            }
            
            JsonArray jsonArray = (JsonArray) baseObj;
            int firstElementType = jsonArray.getType(new integer(1)).intValue();
            if (firstElementType != JsonBackend.JSON_OBJECT)
            {
               ErrorManager.recordOrShowError(15358, "bracket", JsonBackend.getJsonDataTypeName(firstElementType));
               // Error parsing JSON: expected <token>, but found <token>. (15358)
               return false;
            }
            
            object<? extends JsonObject> jsonObjectHolder = jsonArray.getJsonObject(new integer(1));
            JsonObject jsonObject = jsonObjectHolder.ref();
            
            Iterable<Map.Entry<String, Object>> it = jsonObject.getProperties();
            for (Map.Entry<String, Object> entry : it)
            {
               P2JField field = inferFieldType(entry.getKey(), entry.getValue());
               if (field == null)
               {
                  ErrorManager.recordOrShowError(15360, "bracket", "");
                  // Error parsing JSON: unexpected token: <token>. (15360)
                  ErrorManager.recordOrShowError(15374, "", "");
                  // Unable to infer Temp-Table or dataset schema from JSON Data. (15374)
                  return false;
               }
               builder.addField(field);
            }
            
            return builder.tempTablePrepare("NewTable").booleanValue();
         }
      }
      return null;
//      ErrorManager.recordOrShowError(19079, sourceType, "READ-JSON");
//      // <argument> argument for <method-name> is not valid. (19079)
//      return false;
   }
   
   /** TBA */
   private static P2JField inferFieldType(String name, Object value)
   {
      String typeName = "character";
      if (value instanceof Boolean)
      {
         typeName = "logical";
      }
      else if (value instanceof BigInteger || value instanceof BigDecimal || 
               value instanceof Integer || value instanceof Long)
      {
         typeName = "decimal";
      }
      else if (value instanceof _BaseObject_)
      {
         return null;
      }
      
      return new P2JField(name, TempTableBuilder.validateDatatype(typeName), 0, null, null, null, null,
                          false, null, null, false, null, null, null, null, false, 0, 0);
   }
   
   /**
    * Read JSON data from the input source and store it in the just created empty TEMP-TABLE.
    * @param   table 
    *          TEMP-TABLE to be loaded.
    * @param   loader
    *          A {@code BiConsumer} which helps loading a {@code Record} into the buffer.
    *
    * @return  {@code true} on success.
    * 
    * @throws  PersistenceException
    *          if there is any error reading or storing JSON data.
    */
   public boolean readTable(TempTableBuilder table, BiConsumer<RecordBuffer, Record> loader)
   throws PersistenceException
   {
      this.ctx.tt = table;
      char groupSep = decimal.getGroupSeparator();
      char decSep = decimal.getDecimalSeparator();
      ObjectMapper mapper = new ObjectMapper();
      JsonFactory factory = mapper.getFactory();
      try (Reader reader = new InputStreamReader(source.getStream(), StandardCharsets.UTF_8);
               JsonParser parser = factory.createParser(reader))
      {
         this.parser = parser;
         JsonToken tok = parser.nextToken();
         if (tok != START_OBJECT)
         {
            // TODO: proper message and error reporting
            return false;
         }
         
         this.ctx.ttName = parser.nextFieldName();
         if (ctx.ttName == null || ctx.ttName.length() > 32 || ctx.ttName.trim().isEmpty())
         {
            ErrorManager.recordOrShowError(
                     new int[] {1700, 9044, 15370, 15374},
                     new String[] {
                         "Identifier was left blank or is more than 32 characters",
                         "Unable to prepare TEMP-TABLE fields",
                         String.format("Unable to create table '%s' from JSON Data", ctx.ttName),
                         "Unable to infer Temp-Table or dataset schema from JSON Data"
                     },
                     false, // modal
                     true,  // prefix
                     false, // isError
                     false, // asMsg
                     true   // addDot
            );
            return false;
         }
         boolean array = false;
         while ((tok = parser.nextToken()) != null)
         {
            switch (tok)
            {
               case START_ARRAY:
                  array = true;
                  break;
               case START_OBJECT:
                  if (!(ctx.tt._prepared() ? readRecord(false) : parseRecord(false)))
                  {
                     return false;
                  }
                  break;
               case END_ARRAY:
                  return true;
               case END_OBJECT:
                  if (!array)
                  {
                     return true;
                  }
            }
         }
         if (parser.nextToken() != END_OBJECT)
         {
            return false;
         }
      }
      catch (IOException exc)
      {
         throw new PersistenceException(exc);
      }
      finally
      {
         decimal.setNumericFormat(decSep, groupSep);
      }
      return false;
   }

   /**
    * Read JSON data from the input source and store it in the just created empty TEMP-TABLE 
    * which is a part of a DATA-SET.
    * @param   tableName 
    *          table name
    * @param   table 
    *          TEMP-TABLE to be loaded.
    * @param   loader
    *          A {@code BiConsumer} which helps loading a {@code Record} into the buffer.
    *
    * @return  {@code true} on success.
    * 
    * @throws  PersistenceException
    *          if there is any error reading or storing JSON data.
    */
   @Override
   public boolean readTable(String tableName, TempTableBuilder table, BiConsumer<RecordBuffer, Record> loader)
   throws PersistenceException
   {
      pushContext();
      this.ctx.tt = table;
      this.ctx.ttName = tableName;
      this.tables.put(tableName, table);
      char groupSep = decimal.getGroupSeparator();
      char decSep = decimal.getDecimalSeparator();
      try 
      {
         if (ctx.ttName == null || ctx.ttName.length() > 32 || ctx.ttName.trim().isEmpty())
         {
            ErrorManager.recordOrShowError(
                     new int[] {1700, 9044, 15370, 15374},
                     new String[] {
                         "Identifier was left blank or is more than 32 characters",
                         "Unable to prepare TEMP-TABLE fields",
                         String.format("Unable to create table '%s' from JSON Data", ctx.ttName),
                         "Unable to infer Temp-Table or dataset schema from JSON Data"
                     },
                     false, // modal
                     true,  // prefix
                     false, // isError
                     false, // asMsg
                     true   // addDot
            );
            return false;
         }
         
         boolean array = false;
         JsonToken tok = parser.getCurrentToken();
         do
         {
            switch (tok)
            {
               case START_ARRAY:
                  array = true;
                  break;
               case START_OBJECT:
                  if (!(ctx.tt._prepared() ? readRecord(true) : parseRecord(true)))
                  {
                     return false;
                  }
                  break;
               case END_ARRAY:
                  return true;
               case END_OBJECT:
                  if (!array)
                  {
                     return true;
                  }
            }
         }
         while ((tok = parser.nextToken()) != null);
         
         if (parser.nextToken() != END_OBJECT)
         {
            return false;
         }
      }
      catch (IOException exc)
      {
         throw new PersistenceException(exc);
      }
      finally
      {
         decimal.setNumericFormat(decSep, groupSep);
         popContext();
      }
      return false;
   }
   
   /**
    * Read JSON data from the input source and store it in the temp-table.
    * 
    * @param   buffer
    *          A buffer of a table in which data is stored.
    * @param   loader
    *          A {@code BiConsumer} which helps loading a {@code Record} into the buffer.
    *
    * @return  {@code true} on success.
    * 
    * @throws  PersistenceException
    *          if there is any error reading or storing JSON data.
    */
   public boolean readTable(TemporaryBuffer buffer,
                            BiConsumer<RecordBuffer, Record> loader)
   throws PersistenceException
   {
      this.loader = loader;
      ctx.setBuffer(buffer, true);
      
      if (readMode == Read.EMPTY)
      {
         ctx.proxy.deleteAll();
      }
      
      // JsonObject and JsonArray sources does not have a [stream] in [source]
      if ("JsonObject".equalsIgnoreCase(source.getType()))
      {
         return readRowFromJsonObject(ctx.schema, ((object<JsonObject>) source.getSource()).ref());
      }
      else if ("JsonArray".equalsIgnoreCase(source.getType()))
      {
         return readTableFromJsonArray(ctx.schema, ((object<JsonArray>) source.getSource()).ref());
      }
      
      ObjectMapper mapper = new ObjectMapper();
      JsonFactory factory = mapper.getFactory();
      String tableName = ctx.proxy.name().toJavaType();
      String serializeName = ctx.proxy.getSerializeName().toJavaType();
      try (Reader reader = new InputStreamReader(source.getStream(), StandardCharsets.UTF_8);
           JsonParser parser = factory.createParser(reader))
      {
         this.parser = parser;
         
         JsonToken tok;
         String name;
         while ((tok = parser.nextToken()) != null)
         {
            name = null;
            switch (tok)
            {
               case START_ARRAY:
                  name = serializeName;
               
               case START_OBJECT:
                  name = name == null ? parser.nextFieldName() : name;
                  if (serializeName == null ||
                      (!serializeName.equalsIgnoreCase(name) && !tableName.equalsIgnoreCase(name)))
                  {
                     ErrorManager.recordOrShowError(15376, name, serializeName);
                     // Temp-table name '<name>' in JSON does not match '<name>'. (15376)
                     return false;
                  }
                  
                  if (!readTableContent())
                  {
                     return false;
                  }
                  break;
               
               case END_OBJECT:
                  return true;
            }
         }
      }
      catch (IOException exc)
      {
         throw new PersistenceException(exc);
      }
      
      return true;
   }
   
   /**
    * Read JSON data from the input source and store it in each temp-table of the dataset.
    *
    * @param   ds
    *          The {@code DataSet} in which data is stored.
    *
    * @return  {@code true} on success.
    *
    * @throws  PersistenceException
    *          if there is any error reading or storing JSON data.
    */
   public boolean readDataset(DataSet ds)
   throws PersistenceException
   {
      if (!source.configureSupportedSources(SourceData.SD_READ_JSON, LegacyResource.DATASET + " widget"))
      {
         return false;
      }
      
      processHiddenFk(ds);

      ObjectMapper mapper = new ObjectMapper();
      JsonFactory factory = mapper.getFactory();
      
      if ("JsonArray".equalsIgnoreCase(source.getType()))
      {
         ErrorManager.recordOrShowError(19080, "", "");
         // JsonArray is not a valid source-type for READ-JSON on a ProDataset. (19080)
         return false;
      }
      else if ("JsonObject".equalsIgnoreCase(source.getType()))
      {
         if (/* use 4GL solution */ false)
         {
            longchar target = new longchar();
            JsonConstruct jsonObject = (JsonConstruct) ((object<?>) source.getSource()).ref();
            jsonObject.write(target, new logical(false));
            
            JsonImport jsonImport = new JsonImport(new SourceData("longchar", target), readMode.toString());
            return jsonImport.readDataset(ds);
         }
         else
         {
            if (ds._dynamic())
            {
               if (!createDatasetStructure(ds, source))
               {
                  return false;
               }
            }
            
            this.ds = ds;
            return readDatasetFromJsonObject(((object<JsonObject>) source.getSource()).ref());
         }
      }
      
      try (Reader reader = new InputStreamReader(source.getStream(), StandardCharsets.UTF_8);
           JsonParser parser = factory.createParser(reader))
      {
         this.ds = ds;
         this.parser = parser;
         this.peerMapping = new HashMap<>();
         
         JsonToken tok = parser.nextToken();
         if (tok != START_OBJECT)
         {
            // TODO: proper message and error reporting
            return false;
         }
         
         String name = parser.nextFieldName();
         String dsSerializeName = ds.getSerializeName().toJavaType();
         String dsName = ds._name();
         if (!dsName.isEmpty() && 
             !ds._dynamic()    && 
             !(dsName.equalsIgnoreCase(name) || 
               dsSerializeName.equalsIgnoreCase(name)))
         {
            ErrorManager.recordOrShowError(15375, name, dsName);
            // Dataset name '<name>' in JSON does not match '<name>'.
            return false;
         }
         
         if (ds.getBuffers().isEmpty())
         {
            return readDataSetContent(name);
         }
         if (!readDataSetContent())
         {
            return false;
         }
         if (parser.nextToken() != END_OBJECT)
         {
            return false;
         }
      }
      catch (FileNotFoundException exc)
      {
         ErrorManager.recordOrShowError(293, source.getFileName());
         return false;
      }
      catch (IOException exc)
      {
         throw new PersistenceException(exc);
      }
      finally
      {
         this.ds = null;
         this.peerMapping = null;
      }
      
      return true;
   }

   /**
    * Reads the content of an empty dynamic dataset, by inferring its structure and 
    * populating its whole set of tables.
    *
    * @param name
    *        dataset name.
    * @return  {@code true} on success.
    *
    * @throws  IOException
    *          If an IO error occurred while getting the data.
    * @throws  PersistenceException
    *          If an error occurred while persisting the read data.
    */
   private boolean readDataSetContent(String name)
   throws IOException,
          PersistenceException
   {
      String tableName = null;
      JsonToken tok = parser.nextToken();
      if (tok == JsonToken.START_ARRAY)
      {
         if ((tok = parser.nextToken()) != START_OBJECT)
         {
            return false;
         }
         tableName = name;
         name = null;
         handle tt = TypeFactory.handle();
         TempTableBuilder.create(tt);
         TempTableBuilder ttb = (TempTableBuilder)tt.unwrapJsonData();
         if (!ttb.readTable(tableName, this))
         {
            return false;
         }
      }

      if (tok != START_OBJECT)
      {
         return false;
      }
      
      process:
      while ((tok = parser.nextToken()) != null)
      {
         switch (tok)
         {
            case FIELD_NAME:
               tableName = parser.currentName();
               if ("prods:hasChanges".equals(tableName))
               {
                  String hasChanges = parser.nextValue().asString(); // TODO: maybe remember this?
                  tableName = null;
               }
               else if ("prods:before".equals(tableName))
               {
                  readingBefore = true;
                  if (parser.nextToken() != START_OBJECT)
                  {
                     return false;
                  }
                  tableName = null;
               }
               break;
            case END_OBJECT:
               if (readingBefore)
               {
                  readingBefore = false;
               }
               else 
               {
                  break process;
               }
               break;
         }
         
         if (tableName != null)
         {
            handle tt = TypeFactory.handle();
            TempTableBuilder.create(tt);
            TempTableBuilder ttb = (TempTableBuilder)tt.unwrapJsonData();
            if (!ttb.readTable(tableName, this))
            {
               return false;
            }
         }
      }
      setupDataset(name);
      return true;
   }

   /**
    * Reads the content of a dataset, by populating its whole set of tables.
    *
    * @return  {@code true} on success.
    *
    * @throws  IOException
    *          If an IO error occurred while getting the data.
    * @throws  PersistenceException
    *          If an error occurred while persisting the read data.
    */
   private boolean readDataSetContent()
   throws IOException,
          PersistenceException
   {
      if (parser.nextToken() != START_OBJECT)
      {
         return false;
      }
      
      if (readMode == Read.EMPTY)
      {
         ds.emptyDataset();
      }
      
      String tableName = null;
      JsonToken tok;
      while ((tok = parser.nextToken()) != null)
      {
         switch (tok)
         {
            case FIELD_NAME:
               tableName = parser.currentName();
               if ("prods:hasChanges".equals(tableName))
               {
                  String hasChanges = parser.nextValue().asString(); // TODO: maybe remember this?
                  tableName = null;
               }
               else if ("prods:before".equals(tableName))
               {
                  readingBefore = true;
                  if (parser.nextToken() != START_OBJECT)
                  {
                     return false;
                  }
                  tableName = null;
               }
               break;
            case END_OBJECT:
               if (readingBefore)
               {
                  readingBefore = false;
               }
               else 
               {
                  return true;
               }
               break;
         }
         
         if (tableName != null)
         {
            BufferImpl after = null;
            List<BufferImpl> dsBuffers = ds.getBuffers();
            for (BufferImpl buf : dsBuffers)
            {
               if (tableName.equalsIgnoreCase(buf.getSerializeName().toJavaType()) ||
                   tableName.equalsIgnoreCase(buf._name()))
               {
                  after = buf;
                  break;
               }
            }
            
            if (after == null)
            {
               return false;
            }
            BufferImpl buffer = after;
            if (readingBefore)
            {
               buffer = after.beforeBufferNative();
               if (buffer == null)
               {
                  return false;
               }
               buffer.setUpBeforeBuffer(true);
            }
            
            try
            {
               ctx.setBuffer(buffer.buffer(), true);
               this.primaryIndex = null;
               
               if (readMode == Read.REPLACE && !readingBefore)
               {
                  // the REPLACE mode needs primary unique index declared for the after table
                  this.primaryIndex = buffer.buffer().getDmoInfo().getPrimaryIndex(true);
                  if (this.primaryIndex == null)
                  {
                     ErrorManager.recordOrShowError(13063, ctx.schema.getTableName());
                     // REPLACE mode requires a unique primary index in the target table <table>.
                     return false;
                  }
               }
               
               readTableContent();
            }
            finally
            {
               if (readingBefore)
               {
                  buffer.setUpBeforeBuffer(false);
               }
            }
            
            tableName = null;
         }
      }
      
      return true;
   }
   
   /**
    * Read JSON content from reader and import records into the temp-table.
    * <p>
    * This version ensures that the group and decimal separators are ',' and '.' for the duration of the
    * deserialization.
    * 
    * @throws  IOException
    *          if there is an error reading JSON content from the stream.
    * @throws  PersistenceException
    *          if there is an error storing data in the temp-table.
    */
   private boolean readTableContent()
   throws IOException,
          PersistenceException
   {
      char groupSep = decimal.getGroupSeparator();
      char decSep = decimal.getDecimalSeparator();
      
      try
      {
         decimal.setNumericFormat('.', ',');
         
         JsonToken tok;
         while ((tok = parser.nextToken()) != null)
         {
            switch (tok)
            {
               case START_ARRAY:
                  break;
                  
               case START_OBJECT:
                  if (!readRecord(false))
                  {
                     return false;
                  }
                  break;
                  
               case END_ARRAY:
               case END_OBJECT:
                  return true;
            }
         }
         
         throw new IOException("Unexpected end of JSON data");
      }
      finally
      {
         decimal.setNumericFormat(decSep, groupSep);
      }
   }
   
   /**
    * Parse JSON record to retrieve fields' data, prepare table and populate the first record
    *  
    * @param   addLinkToParent
    *          flag indicating that a link to the parent must be added
    * @return <code>true</code> on success.
    * 
    * @throws IOException
    *         on I/O error
    * @throws PersistenceException
    *         on error.
    */
   private boolean parseRecord(boolean addLinkToParent)
   throws IOException,
          PersistenceException
   {
      FieldMaker maker = this::addField;
      Map<String, Object> readValues = new LinkedHashMap<>();
      String fname = null;
      List<Object> datum = new ArrayList<>();
      boolean extent = false;
      boolean nested = false;
      boolean persisted = false;
      
      JsonToken tok;
      objectLoop: while ((tok = parser.nextToken()) != null)
      {
         // need a look-ahead(1) to decide whether the FIELD_NAME is a field or a NESTED table 
         if (tok == FIELD_NAME)
         {
            fname = parser.currentName();
            tok = parser.nextToken();
         }
         
         switch (tok)
         {
            case START_OBJECT:
               if (!extent)
               {
                  // TODO: error? Should 'never happen' unless the source JSON is badly broken.
                  return false;
               }
               extent = false; 
               if (!datum.isEmpty()) 
               {
                  // TODO: error? should 'never happen' unless the source JSON is badly broken.
                  return false;
               }
               
               nested = true;
               AbstractTempTable att = tables.get(fname);
               if (att == null)
               {
                  handle tt = TypeFactory.handle();
                  TempTableBuilder.create(tt);
                  att = (TempTableBuilder)tt.unwrapJsonData();
                  inferredRelations.add(Pair.of(ctx.ttName, fname));
               }
               if (!readValues.isEmpty() && !persist(maker, readValues, addLinkToParent))
               {
                  return false;
               }
               persisted = true;
               if (!att.readTable(fname, this))
               {
                  return false;
               }
               break;
            case START_ARRAY:
               extent = true;
               break;
            
            case END_ARRAY:
               if (extent && datum.isEmpty())
               {
                  ErrorManager.recordOrShowError(
                           new int[] {15368, 15374},
                           new String[] {
                               "More than one Temp-Table inferred from JSON Data",
                               "Unable to infer Temp-Table or dataset schema from JSON Data"
                           },
                           false, // modal
                           true,  // prefix
                           false, // isError
                           false, // asMsg
                           true   // addDot
                   );
                  return false;
               }
               extent = false;
               break;
            case VALUE_NULL:
               datum.add(null);
               break;
            case VALUE_STRING:
               datum.add(parser.getText());
               break;
            case VALUE_NUMBER_INT:
               datum.add(parser.getLongValue());
               break;
            case VALUE_NUMBER_FLOAT:
               datum.add(parser.getDecimalValue());
               break;
            case VALUE_TRUE:
            case VALUE_FALSE:
               datum.add(parser.getBooleanValue());
               break;
            case END_OBJECT:
               break objectLoop;
         }
         
         if (!extent && !nested)
         {
            Object value = datum.size() == 1 ? datum.get(0) : new ArrayList<>(datum);
            datum.clear();
            if (fname == null)
            {
               throw new PersistenceException("No name for field: '" + value + "'");
            }
            readValues.put(fname, value);
            fname = null;
         }
      }
      
      return persisted || persist(maker, readValues, addLinkToParent);
   }
   
   /**
    * Get an inferred type of the value.
    * 
    * @param   val
    *          The value to be analyzed.
    *
    * @return  the inferred type of the value.
    * 
    * @throws  PersistenceException
    *          when the type cannot be inferred from the provided value.
    */
   @Override
   protected ParmType typeOf(Object val) 
   throws PersistenceException
   {
      if (val == null)
      {
         return ParmType.CHAR;
      }
      if (val instanceof Boolean)
      {
         return ParmType.LOG;
      }
      if (val instanceof BigDecimal || val instanceof Long)
      {
         return ParmType.DEC;
      }
      if (val instanceof String)
      {
         return ParmType.CHAR;
      }
      if (val instanceof List)
      {
         for(Object o: (List<Object>) val)
         {
            if (o != null)
            {
               return typeOf(o);
            }
         }
         return ParmType.CHAR;
      }
      throw new PersistenceException("Unexpected value type:" + val.getClass().getName());
   }
   
   /**
    * Read JSON content from the stream corresponding with a single data record and store it in
    * the temp-table.
    *
    * @param   addLinkToParent
    *          flag indicating that a link to the parent must be added.
    *
    * @throws  IOException
    *          if there is an error reading JSON content from the stream.
    * @throws  PersistenceException
    *          if there is an error storing data in the temp-table.
    */
   private boolean readRecord(boolean addLinkToParent)
   throws IOException,
          PersistenceException
   {
      if (readMode != Read.REPLACE)
      {
         try
         {
            ctx.proxy.create();
         }
         catch (ErrorConditionException ex)
         {
            ErrorManager.recordOrShowError(13050, ctx.proxy.name().toStringMessage());
            // Unable to create record for ''<table> during READ-XML/READ-JSON.
            return false;
         }
      }
      
      String name = null;
      boolean batchError = true;
      boolean returnSuccess = true;
      RecordBuffer recBuffer = ((BufferImpl) ctx.proxy).buffer();
      Map<String, String> readValues = (readMode == Read.REPLACE) ? new HashMap<>() : null;
      
      try
      {
         if (readMode != Read.REPLACE)
         {
            RecordBuffer.startBatch(bufMan, true);
         }
         
         fillFromParent();
         TempTableSchema.Column column = null;
         String value = null;
         Object datum = null;
         boolean extent = false;
         JsonToken tok;
         objectLoop: while ((tok = parser.nextToken()) != null)
         {
            // need a look-ahead(1) to decide whether the FIELD_NAME is a field or a NESTED table 
            if (tok == FIELD_NAME)
            {
               name = parser.currentName();
               tok = parser.nextToken();
               column = getColumn(name);
               // if the column was not found in the columns map,
               // there is a chance that the column key was it's serialize name but, we provided it's name,
               // so we should search iterativelly through all the columns in the schema and check for both
               // the serialize name and column name.
               if (column == null)
               {
                  for (TempTableSchema.Column col : ctx.schema.columns())
                  {
                     if (col.getFieldName().equalsIgnoreCase(name) || col.getSerializeName().equalsIgnoreCase(name))
                     {
                        column = col;
                        break;
                     }
                  }
               }

               if (column == null && tok == START_ARRAY)
               {
                  // if there is no column with this name, then try to read a nested child DataSet table
                  if (!readNested(name))
                  {
                     return false;
                  }
                  
                  continue; // next token
               }
            }
            
            switch (tok)
            {
               case START_ARRAY:
                  extent = true;
                  break;
               
               case END_ARRAY:
                  if (extent)
                  {
                     extent = false;
                     break;
                  }
                  // pass through
               case END_OBJECT:
                  batchError = false;
                  break objectLoop;
               
               case VALUE_NULL:
                  if (column != null)
                  {
                     if (!column.isNillable())
                     {
                        ErrorManager.recordOrThrowError(0,
                                                        "Column not nillable: " + column.getFieldName(),
                                                        false);
                     }
                     
                     if (readMode == Read.REPLACE)
                     {
                        // in REPLACE mode, store the read values temporarily
                        readValues.put(extent ? name + " " + ctx.extentTracker.getNext(name) : name, null);
                     }
                     else
                     {
                        setField(column, name, null, extent);
                     }
                  }
                  break;
               
               case VALUE_STRING:
               case VALUE_NUMBER_INT:
               case VALUE_NUMBER_FLOAT:
               case VALUE_TRUE:
               case VALUE_FALSE:
                  // TODO: this approach seems inefficient in that the parser already has parsed to a useful
                  //       data type, then we convert to a string, but it's necessitated by the use of the
                  //       complex Stream.assignDatum method downstream; we would have to duplicate much of
                  //       the functionality of that method here to do this more efficiently
                  value = null;
                  
                  switch (tok)
                  {
                     case VALUE_STRING:
                        value = parser.getText();
                        break;
                     
                     case VALUE_NUMBER_INT:
                        datum = parser.getLongValue();
                        break;
                     
                     case VALUE_NUMBER_FLOAT:
                        datum = parser.getDecimalValue();
                        // ensure exponent is not used
                        datum = ((BigDecimal) datum).toPlainString();
                        break;
                     
                     case VALUE_TRUE:
                     case VALUE_FALSE:
                        datum = parser.getBooleanValue();
                        break;
                  }
                  
                  if (value == null)
                  {
                     value = datum == null ? null : datum.toString();
                  }
                  
                  if (column != null)
                  {
                     if (readMode == Read.REPLACE)
                     {
                        // in REPLACE mode, store the read values temporarily
                        readValues.put(extent ? name + " " + ctx.extentTracker.getNext(name) : name, value);
                     }
                     else
                     {
                        setField(column, name, value, extent);
                     }
                  }
                  break;
            }
         }
         batchError = false;
      }
      finally
      {
         ctx.extentTracker.reset();
         
         if (readMode == Read.REPLACE)
         {
            boolean batched = false;
            try
            {
               Boolean foundConflict = null;
               DmoMeta dmoInfo = recBuffer.getDmoInfo();
               primaryIndex = dmoInfo.getPrimaryIndex(true);
               
               if (primaryIndex != null && primaryIndex.isUnique())
               {
                  List<Object> params = new ArrayList<>();
                  StringBuilder sb = new StringBuilder();
                  sb.append("from ").append(dmoInfo.getDmoImplInterface().getName()).append(" where ");
                  
                  if (recBuffer.isTemporary()) // always, right?
                  {
                     sb.append("_multiplex=?");
                     params.add(recBuffer.getMultiplexID()); // not null if temp-table
                  }

                  Set<String> indexComps = new HashSet<>();
                  ArrayList<P2JIndexComponent> comps = primaryIndex.components();
                  for (int i = 0; i < comps.size(); i++)
                  {
                     P2JIndexComponent comp = comps.get(i);
                     String fieldName = comp.getLegacyName();
                     if (!readValues.containsKey(fieldName)) // this will not match an extent field
                     {
                        foundConflict = false; // not full field-set for the primary unique index
                        break;
                     }
                     
                     if (!params.isEmpty())
                     {
                        sb.append(" and ");
                     }
                     
                     sb.append(comp.getPropertyName()).append("=?");
                     params.add(readValues.get(fieldName));
                     indexComps.add(fieldName);
                  }
                  
                  if (foundConflict == null)
                  {
                     Query uq = Session.createQuery(sb.toString());
                     for (int i = 0; i < params.size(); i++)
                     {
                        uq.setParameter(i, params.get(i));
                     }
                     
                     Object dmo = uq.uniqueResult(recBuffer.getPersistence().getSession(!dmoInfo.multiTenant));
                     foundConflict = (dmo != null);
                     
                     // check if we have the full index field-set but not a matching record
                     if (foundConflict)
                     {
                        loader.accept(recBuffer, (Record) dmo); // loads the record into the buffer
                        
                        RecordBuffer.startBatch(bufMan, true);
                        batched = true;
                        for (Map.Entry<String, String> entry : readValues.entrySet())
                        {
                           String keyName = entry.getKey();
                           if (!indexComps.contains(keyName))// do not overwrite the key/index component
                           {
                              int k = keyName.indexOf(' ');
                              int extIndex = -1;
                              if (k > 0)
                              {
                                 extIndex = Integer.parseInt(keyName.substring(k + 1));
                                 keyName = keyName.substring(0, k);
                              }
                              
                              setField(getColumn(keyName), keyName, entry.getValue(), extIndex);
                           }
                        }
                     }
                  }
               }
               
               if (foundConflict == null)
               {
                  throw new RuntimeException("Internal error");
               }
               else if (!foundConflict)
               {
                  ctx.proxy.create();
                  
                  RecordBuffer.startBatch(bufMan, true);
                  batched = true;
                  for (Map.Entry<String, String> entry : readValues.entrySet())
                  {
                     String keyName = entry.getKey();
                     int k = keyName.indexOf(' ');
                     int extIndex = -1;
                     if (k > 0)
                     {
                        extIndex = Integer.parseInt(keyName.substring(k + 1));
                        keyName = keyName.substring(0, k - 1);
                     }
                     
                     setField(getColumn(keyName), keyName, entry.getValue(), extIndex);
                  }
               }
            }
            finally
            {
               if (batched)
               {
                  ErrorManager.ErrorHelper eh = ErrorManager.getErrorHelper();
                  boolean wm = eh.isWarningMode();
                  if (!wm)
                  {
                     // this is mostly for APPEND mode, the other ones will not encounter issues here
                     eh.setWarningMode(true);
                  }
                  int validationErrors = RecordBuffer.endBatch(bufMan, batchError);
                  if (!wm)
                  {
                     eh.setWarningMode(false);
                  }
                  // if (ErrorManager.error().toJavaType())
                  if (ErrorManager.isPendingError() || validationErrors != 0)
                  {
                     ((BufferImpl) ctx.proxy).drop();
                     ErrorManager.recordOrShowError(15366, true, ctx.schema.getName(), "");
                     // Unable to update indexes for table '<table>'. (15366)
                     returnSuccess = false;
                  }
               }
               // else some exception occurred preventing the RecordBuffer to start the batch
            }
         }
         else
         {
            if (readMode == Read.MERGE)
            {
               boolean validated = false;
               try
               {
                  validated = recBuffer.validate(false);
               }
               catch (ValidationException e)
               {
                  // ignore, no 'expected' exception will be thrown
               }
               
               // was there an unique conflict with this record?
               if (!validated)
               {
                  // merge mode: ignore it, just skip to next record
                  ctx.proxy.release(); // TODO: not enough 
               }
            }
            
            ErrorManager.ErrorHelper eh = ErrorManager.getErrorHelper();
            boolean wm = eh.isWarningMode();
            if (!wm)
            {
               // this is mostly for APPEND mode, the other ones will not encounter issues here
               eh.setWarningMode(true);
            }
            int validationErrors = RecordBuffer.endBatch(bufMan, batchError);
            if (!wm)
            {
               eh.setWarningMode(false);
            }
            // if (ErrorManager.error().toJavaType())
            if (ErrorManager.isPendingError() || validationErrors != 0)
            {
               ((BufferImpl) ctx.proxy).drop();
               ErrorManager.recordOrShowError(15366, true, ctx.schema.getName(), "");
               // Unable to update indexes for table '<table>'. (15366)
               returnSuccess = false;
            }
         }
      }
      
      return returnSuccess; // if batchError is on, the returnSuccess was also set to false  
   }
   
   /**
    * Read nested table.
    * 
    * @param   name
    *          The name of the table to be read.
    *
    * @return  {@code true} on success and {@code false} on fail.
    *
    * @throws  PersistenceException
    *          on database error.
    * @throws  IOException
    *          on file I/O error.
    */
   private boolean readNested(String name)
   throws PersistenceException, 
          IOException
   {
      boolean rc = true;
      pushContext();
      try
      {
         if (!setParentValues(name))
         {
            return false;
         }
         
         BufferImpl buffer = null;
         if (ds.getBuffers().isEmpty()) // empty dynamic dataset
         {
            AbstractTempTable att = tables.get(name);
            if (att == null)
            {
               handle tt = TypeFactory.handle();
               TempTableBuilder.create(tt);
               att = (TempTableBuilder)tt.getResource();
               inferredRelations.add(Pair.of(ctx.ttName, name));
               if (!att.readTable(name, this))
               {
                  return false;
               }
            }
            else
            {
               buffer = (BufferImpl) att.defaultBufferHandleNative();
            }
         }
         else
         {
            buffer = ds.getBufferBySerializeName(name);
         }
         
         if (buffer != null)
         {
            ctx.setBuffer(buffer.buffer(), true);
            rc = readTableContent();
         }
      }
      finally
      {
         // restore configuration for next row
         popContext();
      }
      
      return rc;
   }
   
   /**
    * Iterates the {@code JsonObject} source and populates the tables matching the properties.
    * 
    * @param   source
    *          The source which contains the {@code JsonObject} source.
    *
    * @return  {@code true} on success and {@code false} otherwise.
    * 
    * @throws  PersistenceException
    *          on unexpected events related to persistence.
    */
   private boolean readDatasetFromJsonObject(JsonObject source)
   throws PersistenceException
   {
      JsonObject jsonObject = source;
      if (!ds.getSerializeHidden().booleanValue())
      {
         Map.Entry<String, Object> dsEntry = source.getProperties().iterator().next();
         // assert dsEntry.getKey() == ds.name()

         jsonObject = (JsonObject) dsEntry.getValue();
      }

      for (Map.Entry<String, Object> entry : jsonObject.getProperties())
      {
         String tableName = entry.getKey();
         handle bufferHandle = ds.bufferHandle(tableName);

         if (bufferHandle == null || bufferHandle.isUnknown() || !bufferHandle._isValid())
         {
            BufferImpl buffer = ds.getBufferBySerializeName(tableName);
            if (buffer == null)
            {
               // there is no such table in this dataset
               continue;
            }
            bufferHandle = new handle(buffer);
         }
         
         RecordBuffer buffer = ((BufferImpl) bufferHandle.getResource()).buffer();
         ctx.setBuffer(buffer, true);
         
         if (readMode == Read.EMPTY)
         {
            ctx.proxy.deleteAll();
         }
         
         if (!readTableFromJsonArray(ctx.schema, (JsonArray) entry.getValue()))
         {
            return false;
         }
      }
      
      // all fine
      return true;
   }
   
   /**
    * Iterates the {@code JsonArray} source and for each element creates a new row in the target temp-table.
    * 
    * @param   schema
    *          The schema of the temp-table.
    * @param   jsonArray
    *          The source which contains the {@code JsonArray} source.
    *
    * @return  {@code true} on success and {@code false} otherwise.
    * 
    * @throws  PersistenceException
    *          on unexpected events related to persistence.
    */
   private boolean readTableFromJsonArray(TempTableSchema schema, JsonArray jsonArray)
   throws PersistenceException
   {
      int length = jsonArray.getLength().intValue();
      
      for (int i = 0; i < length; i++)
      {
         int elementType = jsonArray.getType(new integer(i + 1)).intValue();
         if (elementType != JsonBackend.JSON_OBJECT)
         {
            ErrorManager.recordOrShowError(15358, "bracket", JsonBackend.getJsonDataTypeName(elementType));
            // Error parsing JSON: expected <token>, but found <token>. (15358)
            return false;
         }
         
         if (!readRowFromJsonObject(schema, jsonArray.getJsonObject(new integer(i + 1)).ref()))
         {
            return false;
         }
      }
      
      // all OK
      return true;
   }
   
   /**
    * Add field to the temp-table.
    * 
    * @param   ttb
    *          The destination temp-table builder.
    * @param   name
    *          The name of the field to be added.
    * @param   val
    *          The value of field. It is used for inferring the type of the new field.
    *
    * @throws  PersistenceException
    *          If the type of the field cannot be inferred from the provided value.
    */
   private void addField(TempTableBuilder ttb, String name, Object val) 
   throws PersistenceException
   {
      String type = typeOf(val).toString();
      if (val instanceof List)
      {
         int ext = ((List<Object>) val).size();
         ttb.addNewField(new character(name), new character(type), new integer(ext));
      }
      else
      {
         ttb.addNewField(new character(name), new character(type));
      }
   }
   
   /**
    * Creates a new record in target temp-table, iterates the {@code JsonObject} source and populates the new
    * records with data extracted from the source {@code JsonObject}.
    *
    * @param   schema
    *          The schema of the temp-table.
    * @param   jsonObject
    *          The source which contains the {@code JsonObject} source.
    *
    * @return  {@code true} on success and {@code false} otherwise.
    *
    * @throws  PersistenceException
    *          on unexpected events related to persistence.
    */
   private boolean readRowFromJsonObject(TempTableSchema schema, JsonObject jsonObject)
   throws PersistenceException
   {
      try
      {
         ctx.proxy.create();
      }
      catch (ErrorConditionException ex)
      {
         ErrorManager.recordOrShowError(13050, ctx.proxy.name().toStringMessage());
         // Unable to create record for ''<table> during READ-XML/READ-JSON.
         return false;
      }
      
      boolean res = true;
      for (Map.Entry<String, Object> property : jsonObject.getProperties())
      {
         String key = property.getKey();
         try
         {
            String value = null;
            ArrayList<TempTableSchema.Column> columns = schema.columns();
            TempTableSchema.Column column = null;
            for (int k = 0; k < columns.size(); k++)
            {
               TempTableSchema.Column item = columns.get(k);
               if (key.equalsIgnoreCase(item.getSerializeName()))
               {
                  column = item;
                  break;
               }
            }
            character fieldName = new character(key);
            boolean hasProp = column != null;
            if (hasProp)
            {
               integer type = jsonObject.getType(fieldName);
               if (type.intValue() ==  JsonBackend.JSON_ARRAY)
               {
                  String fname = column.getFieldName();
                  JsonArray arr = jsonObject.getJsonArray(fieldName).ref();
                  Integer extent = column.getExtent();
                  if (extent == null || extent == 0)
                  {
                     ErrorManager.recordOrShowError(new int[] { 15364 },
                                                    new String[] { ErrorManager.replaceTokens(15364, fname) },
                                                    false, // modal
                                                    false, // no prefix
                                                    false, // error
                                                    false, // not as message
                                                    true); // with dot
                     res = false;
                     break;
                  }
                  if (arr.getLength().intValue() > extent)
                  {
                     ErrorManager.recordOrShowError(new int[] { 13054 },
                                                    new String[] { ErrorManager.replaceTokens(13054, fname, schema.getName()) },
                                                    false, // modal
                                                    false, // no prefix
                                                    false, // error
                                                    false, // not as message
                                                    true); // with dot
                     res = false;
                     break;
                  }

                  Iterator<Object> iter = arr.getElements().iterator();
                  integer idx = new integer(1);
                  while (iter.hasNext() && idx.intValue() <= extent)
                  {
                     iter.next(); // we don't get the value directly
                     
                     type = arr.getType(idx);
                     value = null;
                     switch (type.intValue())
                     {
                        case JsonBackend.JSON_STRING:
                           value = arr.getCharacter(idx).getValue();
                           break;
                        case JsonBackend.JSON_NUMBER:
                           value = arr.getDecimal(idx).toStringMessage();
                           break;
                        case JsonBackend.JSON_BOOLEAN:
                           value = arr.getLogical(idx).toStringMessage();
                           break;
                        case JsonBackend.JSON_OBJECT:
                           value = arr.getJsonObject(idx).toStringMessage();
                           break;
                     }
                     
                     setField(column, fname, value, idx.intValue() - 1);

                     idx.assign(idx.intValue() + 1);
                  }
               }
               else
               {
                  switch (type.intValue())
                  {
                     case JsonBackend.JSON_STRING:
                        value = jsonObject.getCharacter(fieldName).getValue();
                        break;
                     case JsonBackend.JSON_NUMBER:
                        value = jsonObject.getDecimal(fieldName).toStringMessage();
                        break;
                     case JsonBackend.JSON_BOOLEAN:
                        value = jsonObject.getLogical(fieldName).toStringMessage();
                        break;
                     case JsonBackend.JSON_OBJECT:
                        value = jsonObject.getJsonObject(fieldName).toStringMessage();
                        break;
                  }
                  
                  BaseDataType datum = BaseDataTypeFactory.instantiate(column.getType());
                  if (value == null)
                  {
                     datum.setUnknown();
                  }
                  else
                  {
                     Stream.assignDatum(datum, value, true);
                  }
                  
                  column.getSetter().invoke(ctx.proxy, datum);
               }
            }
         }
         catch (Exception exc)
         {
            ctx.proxy.deleteRecord();
            throw new PersistenceException(exc);
         }
      }
      
      if (!res)
      {
         ctx.proxy.deleteRecord();
      }
      else
      {
         ctx.proxy.release();
      }
      
      return res;
   }
}