AppServerManager.java

/*
** Module   : AppServerManager.java
** Abstract : Defines APIs to manage the appservers defined by this P2J server and any incoming
**            remote API calls for these appservers.
**
** Copyright (c) 2013-2025, Golden Code Development Corporation.
**
** -#- -I- --Date-- ---------------------------------------Description----------------------------------------
** 001 CA  20130529 Created initial version.
** 002 CA  20130628 Context-local data for this class must not be reset by the agent.  Added a 
**                  listener to automatically disconnect if the remote side has closed its session
**                  while the connection is still active.
** 003 CA  20130705 Reworked the appserver launching.
** 004 CA  20130822 Fixed server disconnection when the P2J client ends abruptly, while still
**                  connected to a local appserver.
** 005 OM  20131018 Updated to conform the new SessionListener API.
** 006 CA  20140326 The remote persistent procedures need to be deleted by the Agent which created
**                  them.
** 007 CA  20140826 Fixed agent context cleanup if the P2J session is terminated before the 
**                  appserver connection is establised.
** 008 CA  20141124 Fixed uniqueId() - the nextId field was not being incremented properly, making
**                  execution of async requests impossible.
** 009 IAS 20160331 Fixed appServers.remove
** 010 CA  20160404 Enhanced the appserver invocations, to allow implicit or explicit termination 
**                  of requests being executing on a certain connection.
** 011 CA  20160405 Fixed H010 - during CONNECT procedure call, activeAgents can not be tracked.
** 012 CA  20180503 Added SESSION:SERVER-CONNECTION-ID support.
** 013 CA  20190612 Added APIs to create remote legacy objects and invoke methods on them.
** 014 CA  20190628 Fixed Agent disconnect in State-Free mode.  Misc improvements.
**     CA  20190703 Added WebHandler service support. 
** 015 CA  20190811 Fixed initRemoteAppServerswhen there are no app_services nodes.
** 016 CA  20200427 Added support for SERVER-CONNECTION-BOUND, SERVER-CONNECTION-CONTEXT,  
**                  SERVER-CONNECTION-BOUND-REQUEST and OERequestInfo.
**     CA  20200514 Fixed error handling for handleWebService.
**                  Fixed error handling when no agent is available.
** 029 IAS 20200722 Improved logging.
** 030 CA  20201027 The remote side can delete the IN HANDLE used by the requester at the RUN statement.
** 031 CA  20211112 Fixed getServerOperatingMode() for multi-session agent appservers.
**     CA  20211214 For State-free appservers, the agents can be bound to remote persistent procedures; this
**                  binding must be made using the agent's ID, as a client can invoke multiple remote 
**                  persistent procedures, and using the procedure's ID for this binding can lead to 
**                  collisions, as the pair (connection ID, procedure ID) is not guaranteed to be unique for
**                  a connection.
**     CA  20211216 Fixed a bug in previous changes, for SINGLE-RUN/SINGLETON invocations - in this case, the
**                  agent must be resolved, as there is no bound agent for the call.
**     CA  20220104 Fixed Stateless appserver mode, which got broken in CA/20211214 - now the agent tracks any
**                  persistent procedure ran over it.
**     CA  20220111 Fixed SESSION:SERVER-CONNECTION-CONTEXT - this value must be shared to any agent running
**                  requests for a connection.
**     CA  20220208 For calls originating from outside the FWD server, do not serialize the legacy OO errors.
**     CA  20220329 Added support for REST services written directly in Java.
**     CA  20220405 Added authentication and authorization for web requests.  When this is enabled, the target
**                  API call will be executed under the authenticated FWD context, and not the agent's context.
**     CA  20220428 Logging improvements.
**     SVL 20230113 Improved performance by replacing some "for-each" loops with indexed "for" loops.
** 030 GBB 20230403 Add LegacyLogManagerConfigs param to startAppServer to be able to store and restore 
**                  LOG-MANAGER configs in State-reset mode.
** 031 GBB 20230512 Logging methods replaced by CentralLogger/ConversionStatus.
** 032 CA  20230620 Initialize the app_services configuration for remote appservers on server startup, which
**                  will prevent the FWD server to start if there are configuration errors.
** 033 HC  20230707 Fixed an NPE when sessionFree app server directory value not defined.
** 034 CA  20230712 Added 'invokeWithArgs', an API which can send any number of arguments to a remote program 
**                  acting as a controller, which can prepare and execute the real target.
** 035 GBB 20230825 SecurityManager csession methods calls updated.
** 036 CA  20240225 Fixed comments in startAppServer to better specify that this is used only on server-side. 
** 037 GBB 20240318 Redundant methods lookupP2JServer, lookupP2JAccount, lookupTargetAppserver deleted.
**                  Class RemoteAppServer made public.
**                  Adding error handling for all cases in lookupRemoteAppServer.
**                  Handling responses from appserver remote proc invocation (return-value, error and stop).
** 038 CA  20240620 Backed out part of H037 - the client needs to handle the response's return-value, error
**                  condition and stop; AppServerManager is executed on the remote side, not on the client-side.
** 039 GBB 20240621 Connection error messages fixed.
**                  Fix for connect without session model for session-managed appserver.
** 040 CA  20240729 Fixed deactivate on procedure delete: the deactivate procedure must be invoked before the
**                  agent gets released.
** 041 DDF 20240923 Filter found servers based on the service name provided.
** 042 CA  20241113 Allow posting a task on the agent running in the current context - this can be used to
**                  post tasks from different threads sharing the agent's context.
** 043 GBB 20250403 Support multi-session appservers in applicable methods.
** 044 CA  20250324 Improvements for remote FWD OpenClient connections: ensure that the remote side uses a
**                  single connection to the FWD server, and also use a client-side pool of worker threads to
**                  send the requests (the pool size is in sync with the maximum agents available to process
**                  requests).
*/
/*
** This program is free software: you can redistribute it and/or modify
** it under the terms of the GNU Affero General Public License as
** published by the Free Software Foundation, either version 3 of the
** License, or (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
** GNU Affero General Public License for more details.
**
** You may find a copy of the GNU Affero GPL version 3 at the following
** location: https://www.gnu.org/licenses/agpl-3.0.en.html
** 
** Additional terms under GNU Affero GPL version 3 section 7:
** 
**   Under Section 7 of the GNU Affero GPL version 3, the following additional
**   terms apply to the works covered under the License.  These additional terms
**   are non-permissive additional terms allowed under Section 7 of the GNU
**   Affero GPL version 3 and may not be removed by you.
** 
**   0. Attribution Requirement.
** 
**     You must preserve all legal notices or author attributions in the covered
**     work or Appropriate Legal Notices displayed by works containing the covered
**     work.  You may not remove from the covered work any author or developer
**     credit already included within the covered work.
** 
**   1. No License To Use Trademarks.
** 
**     This license does not grant any license or rights to use the trademarks
**     Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
**     of Golden Code Development Corporation. You are not authorized to use the
**     name Golden Code, FWD, or the names of any author or contributor, for
**     publicity purposes without written authorization.
** 
**   2. No Misrepresentation of Affiliation.
** 
**     You may not represent yourself as Golden Code Development Corporation or FWD.
** 
**     You may not represent yourself for publicity purposes as associated with
**     Golden Code Development Corporation, FWD, or any author or contributor to
**     the covered work, without written authorization.
** 
**   3. No Misrepresentation of Source or Origin.
** 
**     You may not represent the covered work as solely your work.  All modified
**     versions of the covered work must be marked in a reasonable way to make it
**     clear that the modified work is not originating from Golden Code Development
**     Corporation or FWD.  All modified versions must contain the notices of
**     attribution required in this license.
*/

package com.goldencode.p2j.util;

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import java.util.logging.*;
import java.util.stream.*;

import javax.servlet.http.*;

import com.goldencode.p2j.*;
import com.goldencode.p2j.directory.*;
import com.goldencode.p2j.main.*;
import com.goldencode.p2j.net.*;
import com.goldencode.p2j.net.Session;
import com.goldencode.p2j.oo.web.WebHandler;
import com.goldencode.p2j.security.*;
import com.goldencode.p2j.security.SecurityManager;
import com.goldencode.p2j.util.appserver.*;
import com.goldencode.p2j.util.logging.*;

/**
 * Manage the appservers defined by this P2J server by:
 * <ul>
 * <li>starting the agents when the appserver is started</li>
 * <li>listening and handling all incoming requests for the managed appservers</li>
 * </ul>
 * Also, it manages the appservers known to this P2J server. These known appservers are 
 * disambiguated using the data configured in directory,  under the "server/default/app_services"
 * node.
 */
public class AppServerManager
{
   /** Logger */
   private static final CentralLogger LOG = CentralLogger.get(AppServerManager.class);

   /** Provides a map of default appService names, for each name server. */
   private static Map<String, String> nameServerDefaults = null;
   
   /** The list of the known, remote, appservers. */
   private static ArrayList<RemoteAppServer> remoteAppServers = null;
   
   /** Map of subject IDs to appserver names. */
   private static final Map<String, String> subjectToAppserver = new HashMap<>();

   /** Map of appservers to their associated pool, managed by this P2J server. */
   private static final Map<String, AgentPool> appServers = new HashMap<>();
   
   /** A map of authorized sessions which can access the given connection ID. */
   private static final Map<String, Set<Integer>> authorizedSessions = new HashMap<>();
   
   /** The session listeners registered for each connection. */
   private static final Map<String, SessionListener> sessionListeners = new ConcurrentHashMap<>();

   /** Map of executing requests to their associated Agents. */
   private static final Map<Integer, Agent> requests = new HashMap<>();
   
   /** Set of stopped requests. */
   private static final Set<Integer> stoppedRequests = new HashSet<>();
   
   /**
    * For each connection ID, this map includes the Agents which are currently executing requests.
    */
   private static final Map<String, Set<Agent>> activeAgents = new HashMap<>();
   
   /** Lock used to sync access to {@link #requests} and {@link #stoppedRequests}. */
   private static final Object lock = new Object();
   
   /** The next request ID. */
   private static int nextId = 0;
   
   /**
    * This context-local data is used only to determine which appserver is authenticated and its
    * mode (manager or agent). 
    */
   private static ContextLocal<WorkArea> local = new ContextLocal<WorkArea>()
   {
      protected WorkArea initialValue()
      {
         return new WorkArea();
      };
      
      protected boolean isResetAllowed()
      {
         return false;
      };
   };
   
   /**
    * Init the data related to remote appservers, from the directory. It reads all the appservers
    * configured under the "server/default/app_services" node.
    */
   public static synchronized void initRemoteAppServers()
   {
      // do this only once
      if (nameServerDefaults != null)
      {
         return;
      }

      nameServerDefaults = new HashMap<String, String>();
      remoteAppServers = new ArrayList<RemoteAppServer>();
      // access the directory service
      DirectoryService ds = DirectoryService.getInstance();
      
      if (!ds.bind())
         throw new RuntimeException("directory bind failed");

      String appSrvPath = Utils.findDirectoryNodePath(ds, "app_services", "container", false);
      String[] serviceNodes = appSrvPath == null ? null : ds.enumerateNodes(appSrvPath);
      if (serviceNodes != null)
      {
         for (int i = 0; i < serviceNodes.length; i++)
         {
            String name = serviceNodes[i];
            String path = appSrvPath + "/" + name;
            
            RemoteAppServer data = new RemoteAppServer(ds, appSrvPath, name);
            
            String[] defaults = ds.getNodeStrings(path + "/defaults", "values");
            if (defaults != null)
            {
               for (int j = 0; j < defaults.length; j++)
               {
                  String ns = defaults[j];
   
                  if (nameServerDefaults.containsKey(ns))
                  {
                     throw new RuntimeException("More than one default service specified for nameserver " + 
                                                ns + "!  A namespace must appear as default for a single " +
                                                "'app_services' config.");
                  }
                  
                  nameServerDefaults.put(ns, name);
               }
            }
            
            remoteAppServers.add(data);
         }
      }
      
      // last use of the directory, end our session with it
      ds.unbind();
   }

   /**
    * Obtain an unique ID from the remote side.
    * 
    * @return   An unique ID.
    */
   public synchronized static int uniqueId()
   {
      do
      {
         nextId = nextId + 1;
         if (nextId == Integer.MAX_VALUE)
         {
            nextId = 0;
         }
        
         synchronized (lock)
         {
            if (!requests.containsKey(nextId) && !stoppedRequests.contains(nextId))
            {
               break;
            }
         }
      }
      while (true);
     
      return nextId;
   }

   /**
    * Calculate a random string, to be used as a connection ID for multi-session agent appserver, or as a 
    * context ID for {@link com.goldencode.p2j.oo.lang.OerequestInfo}.
    * 
    * @param    underscore
    *           Flag indicating if the 14 char is an underscore.
    *           
    * @return   A random string.
    */
   static String nextContextId(boolean underscore)
   {
      // 13 8bit chars, underscore (optional), 8bit chars

      String id = "";
      byte[] bytes = new byte[21 + (underscore ? 0 : 1)];
      new Random().nextBytes(bytes);
      for (int i = 0; i < bytes.length; i++)
      {
         // make positive, modulo by 52
         int k = bytes[i];
         k = (k + Byte.MAX_VALUE) % 52;
         
         char next = (char) ('a' + (k % 26));
         if (k >= 26)
         {
            next = Character.toUpperCase(next);
         }
         id = id + next;
         if (underscore && i == 12)
         {
            id = id + "_";
         }
      }
      
      return id;
   }
   
   /**
    * Obtain a proxy for the given external procedure. This will also reserve an agent which will
    * be used for all subsequent invocation (including external procedure initialization).
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    name
    *           A name of a remote external procedure.
    *           
    * @return   A result containing the details about this proxy procedure (resource ID, agent ID).
    */
   public static AppServerInvocationResult obtainProxy(String id, String name)
   {
      AppServerInvocationResult res = new AppServerInvocationResult();

      handle hProc = new handle();

      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         msaManager.obtainProxy(hProc, res, id, name);
      }
      else
      {
         try
         {
            obtainProxy(hProc, res, id, name);
         }
         catch (NumberedException ignored)
         {
            return null;
         }
      }

      ExternalProgramWrapper extProg = (ExternalProgramWrapper) hProc.getResource();
      if (extProg != null && extProg.valid())
      {
         String code = new handle(extProg).toString();
         res.setProcedureId(code);
      }
      else
      {
         res.setInvalidInHandle(true);
      }
      return res;
   }
   
   /**
    * Obtain a proxy for the given external procedure. This will also reserve an agent which will
    * be used for all subsequent invocation (including external procedure initialization).
    *
    * @param    hProc
    *           The procedure handle.
    * @param    res
    *           The invocation result container.
    * @param    id
    *           The appserver connection id. 
    * @param    name
    *           A name of a remote external procedure.
    */
   private static void obtainProxy(handle hProc, AppServerInvocationResult res, String id, String name)
   throws NumberedException
   {
      AgentPool pool = getAgentPool(id);

      // acquire a free agent
      Agent agent = null;

      try
      {
         agent = pool.acquireAgent(id);
         agent.obtainProxy(hProc, res, id, name);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }
   
   /**
    * Initialize the specified proxy procedure, by calling the <code>execute</code> method.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    * @param    code
    *           The code of a persistent external procedure to which the agent is associated. May
    *           be <code>null</code>.
    * @param    agentId
    *           The agent ID where this remote procedure was created.
    * @param    transactionDistinct
    *           Flag indicating if the TRANSACTION DISTINCT clause is in effect.
    * @param    modes
    *           A string representation of the modes of each parameter. May be <code>null</code>
    * @param    args
    *           The procedure's arguments.
    *           
    * @return   The result of the external program initialization.
    */
   public static AppServerInvocationResult initializeProxy(String   id,
                                                           int      requestId,
                                                           String   code,
                                                           int      agentId,
                                                           boolean  transactionDistinct,
                                                           String   modes,
                                                           Object[] args)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         return msaManager.initializeProxy(id, requestId, code, agentId, transactionDistinct, modes, args);
      }
      
      AgentPool pool = getAgentPool(id);

      Agent agent;
      try
      {
         agent = pool.acquireAgent(id, code, agentId);
      }
      catch (NumberedException e)
      {
         AppServerInvocationResult result = new AppServerInvocationResult();
         result.setError(e);
         return result;
      }
      
      try
      {
         // all internal-entries must use the same agent, if this is async and session-free!!!
         return agent.initializeProxy(id, code, requestId, transactionDistinct, modes, args);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }
   
   /**
    * Send the STOP condition to the request with the specified ID.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    */
   public static void sendStop(String id, int requestId)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         msaManager.sendStop(id);
         return;
      }
      
      // security check
      getAgentPool(id);
      
      synchronized (lock)
      {
         Agent a = requests.get(requestId);
         if (a == null)
         {
            // this means the agent didn't get a chance to start, collect this as a stopped 
            // request
            stoppedRequests.add(requestId);
         }
         else
         {
            // terminate the agent
            a.stop();
         }
      }
   }
   
   /**
    * Send a STOP condition to all the requests on the specified connection.
    * 
    * @param    id
    *           The appserver connection id. 
    */
   public static void sendStop(String id)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         msaManager.sendStop(id);
         return;
      }
      
      // security check
      getAgentPool(id);

      // get all requests currently processed for this connection ID and terminate them
      synchronized (lock)
      {
         Set<Agent> active = activeAgents.get(id);
         
         for (Agent a : active)
         {
            a.stop();
         }
      }
   }
   
   /**
    * Method that specifies the remote procedures the client can execute from the AppServer 
    * session.
    * 
    * @param    procList
    *           The list of procedure  and name-patterns separated by comma.
    *         
    * @return   <code>true</code> if the method succeeded.
    */
   public static logical export(String procList)
   {
      return export(new character(procList));
   }

   /**
    * Method that specifies the remote procedures the client can execute from the AppServer 
    * session.
    * 
    * @param    procList
    *           The list of procedure  and name-patterns separated by comma.
    *
    * @return   <code>true</code> if the method succeeded.
    */
   public static logical export(character procList)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaContext())
      {
         return msaManager.export(procList);
      }
      
      AgentPool pool = getAgentPoolBySubject(SecurityManager.getInstance().getUserId());
      if (pool == null)
      {
         // not an AppServer
         return new logical(false);
      }
      pool.setExports(procList);
      
      return new logical(true);
   }

   /**
    * Method implementation for getting the SERVER-OPERATING-MODE attribute of the session handle.
    * <p>
    * Possible values:
    * <ul>
    *    <li>State-reset
    *    <li>State-aware
    *    <li>Stateless
    *    <li>State-free
    * </ul>
    * 
    * @return   The corresponding SERVER-OPERATING-MODE attribute of the session handle.
    */
   public static character getServerOperatingMode()
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaContext())
      {
         return msaManager.isSessionFree() ?
            AppServerSessionModel.SESSION_FREE.toChar() :
            AppServerSessionModel.SESSION_MANAGED.toChar();
      }
      
      AgentPool pool = getAgentPoolBySubject(SecurityManager.getInstance().getUserId());
      if (pool == null)
      {
         return new character("");
      }
      
      String state = pool.getOperatingMode().toString();
      if (pool.isMultiSession())
      {
         state = pool.getOperatingMode().isSessionFree() ?
            AppServerSessionModel.SESSION_FREE.toString() :
            AppServerSessionModel.SESSION_MANAGED.toString();
      }
      
      return new character(state);
   }

   /**
    * Check if the specified web service token is currently in use (there is an active API call on an agent).
    * 
    * @param    token
    *           The web service token.
    *           
    * @return   <code>true</code> if {@link Agent#activeWebServiceTokens} contains this token.
    */
   public static boolean isWebServiceTokenInUse(String token)
   {
      return Agent.isWebServiceTokenInUse(token);
   }
   
   /**
    * Indicates if the session is an AppServer session or not.
    * 
    * @return   <code>true</code> if the current session is an AppServer session or false 
    *           otherwise.
    */
   public static boolean isRemote()
   {
      return MultiSessionAppserverManager.getInstance().isMsaContext() || local.get().appServer != null;
   }
   
   /**
    * Check if this FWD context is running an Agent under MSA compatibility.
    * 
    * @return   See above.
    */
   public static boolean isMultiSession()
   {
      return MultiSessionAppserverManager.getInstance().isMsaContext() ||
         (isRemote() && local.get().agent.isMultiSession());
   }
   
   /**
    * Get the SERVER-CONNECTION-BOUND attribute of this session.
    * 
    * @return   See above.
    */
   public static logical isServerConnectionBound()
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaContext())
      {
         return new logical(msaManager.isConnectionBound());
      }
      
      WorkArea wa = local.get();
      return wa.appServer == null ? new logical() : new logical(wa.agent.isBound());
   }

   /**
    * Get the SERVER-CONNECTION-BOUND-REQUEST attribute of this session.
    * 
    * @return   See above.
    */
   public static logical isServerConnectionBoundRequest()
   {
      if (MultiSessionAppserverManager.getInstance().isMsaContext())
      {
         return new logical(MultiSessionAppserverManager.isServerConnectionBoundRequest());
      }
      
      WorkArea wa = local.get();
      if (wa.appServer == null || wa.agent.getOperatingMode() != AppServerOperatingMode.STATELESS)
      {
         return new logical(false);
      }
      
      return new logical(wa.agent.isBound());
   }

   /**
    * Set the SERVER-CONNECTION-BOUND-REQUEST attribute of this session.
    * 
    * @param   bind
    *          The bound state.
    */
   public static void setServerConnectionBoundRequest(boolean bind)
   {
      if (MultiSessionAppserverManager.getInstance().isMsaContext())
      {
         MultiSessionAppserverManager.setServerConnectionBoundRequest(bind);
         return;
      }
      
      WorkArea wa = local.get();
      if (wa.appServer == null)
      {
         // do nothing
         return;
      }

      AppServerOperatingMode mode = wa.agent.getOperatingMode();
      if (mode != AppServerOperatingMode.STATELESS)
      {
         ErrorManager.recordOrShowError(10849, mode.toString() + " operating mode", 
                                        false, false, false, false);
         return;
      }

      if (bind)
      {
         wa.agent.bind();
      }
      else
      {
         wa.agent.unbind();
      }
   }
   
   /**
    * Get the SERVER-CONNECTION-CONTEXT attribute of this session.
    * 
    * @return   See above.
    */
   public static String getServerConnectionContext()
   {
      if (MultiSessionAppserverManager.getInstance().isMsaContext())
      {
         return MultiSessionAppserverManager.getServerConnectionContext();
      }
      
      WorkArea wa = local.get();
      
      if (wa.appServer == null)
      {
         return "";
      }
      
      return wa.agent.getServerConnectionContext(wa.connectionId);
   }
   
   /**
    * Set the SERVER-CONNECTION-CONTEXT attribute of this session.
    * 
    * @param    value
    *           The attribute's value.
    */
   public static void setServerConnectionContext(String value)
   {
      if (MultiSessionAppserverManager.getInstance().isMsaContext())
      {
         MultiSessionAppserverManager.setServerConnectionContext(value);
         return;
      }
      
      WorkArea wa = local.get();
      
      if (wa.appServer == null)
      {
         return;
      }
      
      // for State-free mode, documentation states that a WARNING is raised, but there is no such
      // proof of this during testing... the value can be read after it was set.
      wa.agent.setServerConnectionContext(wa.connectionId, value);
   }
   
   /**
    * Get the value of SESSION:SERVER-CONNECTION-ID attribute. 
    * 
    * @return  the value of SESSION:SERVER-CONNECTION-ID attribute.
    */
   public static String getServerConnectionId()
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaContext())
      {
         return msaManager.getConnectionId();
      }
      WorkArea wa = local.get();
      return wa.appServer == null ? "" : wa.connectionId;
   }

   /**
    * Get the name of the remote procedure, specified by the resource ID.
    *  
    * @param    id
    *           The appserver connection id. 
    * @param    resId
    *           The ID of a remote procedure.
    * @param    agentId
    *           The agent ID where this remote procedure was created.
    *
    * @return   The procedure's name.
    *
    * @see      #getAgentPool(String)
    */
   public static character getProcedureName(String id, String resId, int agentId)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      
      String procedureName;
      if (msaManager.isMsaConnection(id))
      {
         procedureName = msaManager.getProcedureName(id, resId, agentId);
      }
      else
      {
         AgentPool pool = getAgentPool(id);
         Agent agent = pool.getAgent(agentId);
         procedureName = agent.getProcedureName(resId);
      }
      
      return procedureName == null ? new character() : new character(procedureName);
   }

   /**
    * Delete the remote procedure, specified by the resource id. This must be called at the same
    * time when the local proxy procedure is deleted.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    resId
    *           The ID of a remote procedure which needs to be deleted.
    * @param    agentId
    *           The agent ID where this remote procedure was created.
    * @param    token
    *           When not null, it represents the token of a FWD context created for an authenticated and 
    *           authorized web service call.  The API call will be performed in that context.
    *
    * @return   <code>true</code> if the remote procedure was deleted.
    * 
    * @see      #getAgentPool(String)
    */
   public static boolean deleteProcedure(String id, String resId, int agentId, String token)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         return msaManager.deleteProcedure(id, resId, agentId);
      }
      
      AgentPool pool = getAgentPool(id);
      Agent agent = pool.getAgent(agentId);
      
      if (agent == null)
      {
         // agent ID unknown
         return false;
      }
      
      if (!agent.hasProcedure(resId))
      {
         // not the agent's procedure
         return false;
      }
      
      // this needs to be done using the agent's context
      agent.deleteProcedure(id, resId, token);

      return true;
   }
   
   /**
    * Delete the remote object, specified by the resource ID. This must be called at the same
    * time when the local proxy object is deleted.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    resId
    *           The ID of a remote object which needs to be deleted.
    * @param    agentId
    *           The agent ID where this remote procedure was created.
    *           
    * @return   <code>true</code> if the remote object was deleted.
    */
   public static boolean deleteObject(String id, String resId, int agentId)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         return msaManager.deleteObject(id, resId, agentId);
      }

      AgentPool pool = getAgentPool(id);
      Agent agent = pool.getAgent(agentId);
      
      if (agent == null)
      {
         // agent ID unknown
         return false;
      }
      
      if (!agent.hasProcedure(resId))
      {
         // not the agent's procedure
         return false;
      }
      
      // this needs to be done using the agent's context
      agent.deleteObject(id, resId);

      return true;
   }

   /**
    * Establish a new appserver connection. This will invoke any connect procedure for this 
    * appserver, to which the given <code>user</code>, <code>pwd</code> and <code>serverInfo</code>
    * parameters will be passed.
    * <p>
    * The appserver connection must be configured to start under the given P2J-level subject ID.
    *
    * @param    appServer
    *           The name of the target appserver.
    * @param    sessionFree
    *           Flag indicating if the connection request is for a session-free operating mode.
    * @param    user
    *           The first parameter to pass to the connect procedure, if any.
    * @param    pwd
    *           The second parameter to pass to the connect procedure, if any.
    * @param    serverInfo
    *           The third parameter to pass to the connect procedure, if any.
    * @param    requestInfo
    *           The remote {@link com.goldencode.p2j.oo.lang.OerequestInfo} details, as set by
    *           {@link com.goldencode.p2j.oo.lang.OerequestInfo#toArray()}.
    *
    * @return   An {@link AppServerInvocationResult} instance with any result or caught errors.
    */
   public static AppServerInvocationResult connect(String   appServer,
                                                   boolean  sessionFree,
                                                   String   user,
                                                   String   pwd,
                                                   String   serverInfo,
                                                   Object[] requestInfo)
   {
      return connect(appServer, sessionFree, user, pwd, serverInfo, requestInfo, null);
   }
   
   /**
    * Establish a new appserver connection. This will invoke any connect procedure for this 
    * appserver, to which the given <code>user</code>, <code>pwd</code> and <code>serverInfo</code>
    * parameters will be passed.
    * <p>
    * The appserver connection must be configured to start under the given P2J-level subject ID.
    *  
    * @param    appServer
    *           The name of the target appserver.
    * @param    sessionFree
    *           Flag indicating if the connection request is for a session-free operating mode.
    * @param    user
    *           The first parameter to pass to the connect procedure, if any.
    * @param    pwd
    *           The second parameter to pass to the connect procedure, if any.
    * @param    serverInfo
    *           The third parameter to pass to the connect procedure, if any.
    * @param    requestInfo
    *           The remote {@link com.goldencode.p2j.oo.lang.OerequestInfo} details, as set by
    *           {@link com.goldencode.p2j.oo.lang.OerequestInfo#toArray()}.
    * @param    webServiceType
    *           The type of web service the worker is used for.
    *           
    * @return   An {@link AppServerInvocationResult} instance with any result or caught errors.
    */
   public static AppServerInvocationResult connect(String   appServer,
                                                   boolean  sessionFree,
                                                   String   user,
                                                   String   pwd,
                                                   String   serverInfo,
                                                   Object[] requestInfo,
                                                   String   webServiceType)
   {
      if (MultiSessionAppserverDefinition.get(appServer) != null)
      {
         return MultiSessionAppserverManager.getInstance().connect(appServer,
                                                                   sessionFree,
                                                                   user,
                                                                   pwd,
                                                                   serverInfo,
                                                                   requestInfo,
                                                                   webServiceType);
      }
      
      final AgentPool pool;
      synchronized (appServers)
      {
         pool = appServers.get(appServer);
      }

      AppServerInvocationResult result = null;

      if (pool == null)
      {
         result = new AppServerInvocationResult();
         
         result.setError(new RuntimeException("No appserver named " + appServer + " exists!"));
         
         return result;
      }
      
      // latch to mark when the session listener can process
      final CountDownLatch connectDone = new CountDownLatch(1);
      
      // special var to track session disconnected state
      final AtomicBoolean sessionTerminated = new AtomicBoolean(false);

      // the connect ID - use an array to expose it to the session listener
      final String[] connectId = new String[1];
      
      // the session listener needs to be registered early, in case the the session is terminated
      // while establishing the appserver connection
      SessionListener listener = new SessionListener()
      {
         /**
          * This callback will be called when the P2J client session is ending.
          * 
          * @param    session
          *           The session that is ending.
          */
         @Override
         public void terminate(Session session)
         {
            // notify the session has been terminated
            sessionTerminated.set(true);

            // wait for the connect procedure to process, before going forward
            try
            {
               connectDone.await();
            }
            catch (InterruptedException e)
            {
               // no-op
            }
            
            // 1. the connection was not established: thus nothing to do, as the agent should be
            //    properly disconnect at this time.
            if (connectId[0] == null)
            {
               return;
            }

            // 2. the connection was established, thus disconnect might be needed
            SecurityManager sm = SecurityManager.getInstance();
            synchronized (authorizedSessions)
            {
               Set<Integer> sessions = authorizedSessions.get(connectId[0]);
               
               // only if we are the last authorized session on this connection
               if (sessions != null                     && 
                   sessions.contains(sm.sessionSm.getSessionId()) &&
                   sessions.size() == 1)
               {
                  disconnect(connectId[0]);
               }
            }
            
            synchronized (lock)
            {
               activeAgents.remove(connectId[0]);
            }
         }

         /**
          * This callback will be called after the security context has been set up.
          *
          * @param    session
          *           The session that is starting.
          */
         @Override
         public void initialize(Session session)
         {
            // no ops
         }
      };
      
      final Session session = SessionManager.get().getSession();
      
      // the session listener registration needs to protected, so that the queue can't be stopped
      // while adding it.  This can be done only by obtaining a lock on the Queue object associated
      // with the session, and is done by the specialized API in the Session class.
      
      if (session != null && !session.addSessionListener(listener, true))
      {
         result = new AppServerInvocationResult();
         
         result.setError(new RuntimeException("Client session disconnected before the connect" +
                                              "procedure was started."));
         
         return result;
      }

      // WARNING! From this point forward, no synchronized Queue APIs can be called, else a
      // deadlock might occur.  Also, no other listeners can be added for the same reason.
      try
      {
         result = pool.connect(sessionTerminated, 
                               appServer, 
                               sessionFree, 
                               user, 
                               pwd, 
                               serverInfo, 
                               requestInfo);
      }
      finally
      {
         if (result != null && result.getError() == null)
         {
            connectId[0] = result.getConnectionID();
            
            if (connectId[0] != null)
            {
               sessionListeners.put(connectId[0], listener);
               
               // only if the client socket is still alive
               
               SecurityManager sm = SecurityManager.getInstance();
               
               // add this context to the authorized connection IDs
               synchronized (authorizedSessions)
               {
                  Set<Integer> sessions = authorizedSessions.get(connectId[0]);
                  if (sessions == null)
                  {
                     authorizedSessions.put(connectId[0], sessions = new HashSet<Integer>());
                  }
                  
                  sessions.add(sm.sessionSm.getSessionId());
               }
               
               synchronized (lock)
               {
                  activeAgents.put(connectId[0], new HashSet<>());
               }
            }
         }
         
         // notify the session listener that now is OK to process
         connectDone.countDown();
      }

      return result;
   }

   /**
    * Disconnect the appserver connection, with the given ID.  It will invoke any configured 
    * disconnect procedure, but will not return any errors or values to the caller.
    * 
    * @param    id
    *           The connection ID to disconnect.
    *
    * @see      #getAgentPool(String)
    */
   public static void disconnect(String id)
   {
      disconnect(null, id);
   }

   /**
    * Disconnect the appserver connection, with the given ID.  It will invoke any configured 
    * disconnect procedure, but will not return any errors or values to the caller.
    * 
    * @param    pool
    *           The pool to which the connection belongs (may be <code>null</code>, in which case 
    *           {@link #getAgentPool} is used to resolve it).
    * @param    id
    *           The connection ID to disconnect.
    *
    * @see      #getAgentPool(String)
    */
   public static void disconnect(AgentPool pool, String id)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         msaManager.disconnect(id, false);
         return;
      }

      if (!connected(id))
      {
         return;
      }

      if (pool == null)
      {
         pool = getAgentPool(id);
      }
      
      try
      {
         pool.disconnect(id);
      }
      finally
      {
         SecurityManager sm = SecurityManager.getInstance();
         synchronized (authorizedSessions)
         {
            Set<Integer> sessions = authorizedSessions.get(id);
            sessions.remove(sm.sessionSm.getSessionId());
            
            if (sessions.isEmpty())
            {
               authorizedSessions.remove(id);
            }
         }
         
         SessionListener listener = sessionListeners.remove(id);
         if (listener != null)
         {
            Session session = SessionManager.get().getSession();
            session.removeSessionListener(listener);
         }
      }
   }

   /**
    * Create a new instance of the specified qualified class name.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    * @param    timeout
    *           The maximum allowed time for this invocation to complete.  In milliseconds.
    *           Use <code>0</code> to disable the timeout.
    * @param    name
    *           The legacy qualified class name for the legacy class.
    * @param    modes
    *           A string representation of the modes of each parameter. May be <code>null</code>
    * @param    args
    *           The constructor's arguments.
    *
    * @return   A {@link AppServerInvocationResult} instance, which holds the code for the remote
    *           object, all updated OUTPUT/INPUT-OUTPUT arguments and any returned value.
    */
   public static AppServerInvocationResult newInstance(String    id,
                                                       int       requestId,
                                                       long      timeout,
                                                       character name,
                                                       String    modes,
                                                       Object... args)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         return msaManager.newInstance(id, requestId, timeout, name, modes, args);
      }
      
      AgentPool pool = getAgentPool(id);
      
      // get the agent from the pool, depending on the operating mode
      // ensure all internal-entries must use the same agent, if this is async and session-free
      Agent agent;
      try
      {
         agent = pool.acquireAgent(id);
      }
      catch (NumberedException e)
      {
         AppServerInvocationResult result = new AppServerInvocationResult();
         result.setError(e);
         return result;
      }
      
      try
      {
         // send the invoke command to the agent
         return agent.newInstance(id, requestId, timeout, name, modes, args);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }

   /**
    * Invoke a method in a remotely instantiated class.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    * @param    timeout
    *           The maximum allowed time for this invocation to complete.  In milliseconds.
    *           Use <code>0</code> to disable the timeout.
    * @param    returnValue
    *           Flag indicating if the return value is required.
    * @param    name
    *           The legacy 4GL name for the class method.
    * @param    resId
    *           The code of a remotely instantiated object in which to invoke the method.
    * @param    agentId
    *           The agent ID where this remote procedure was created.
    * @param    modes
    *           A string representation of the modes of each parameter. May be <code>null</code>
    * @param    args
    *           The method's arguments.
    * 
    * @return    A {@link AppServerInvocationResult} instance with all updated OUTPUT/INPUT-OUTPUT
    *            arguments and any returned value.
    */
   public static AppServerInvocationResult invokeMethod(String    id,
                                                        int       requestId,
                                                        long      timeout,
                                                        boolean   returnValue,
                                                        character name,
                                                        String    resId,
                                                        int       agentId,
                                                        String    modes,
                                                        Object... args)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         return msaManager.invokeMethod(id, requestId, timeout, returnValue, name, resId, agentId, modes, args);
      }
      
      AgentPool pool = getAgentPool(id);
      
      // get the agent from the pool, depending on the operating mode
      // ensure all internal-entries must use the same agent, if this is async and session-free
      Agent agent;
      try
      {
         agent = pool.acquireAgent(id, resId, agentId);
      }
      catch (NumberedException e)
      {
         AppServerInvocationResult result = new AppServerInvocationResult();
         result.setError(e);
         return result;
      }
      
      try
      {
         // send the invoke command to the agent
         return agent.invokeMethod(id, requestId, timeout, returnValue, name, resId, modes, args);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }

   /**
    * Invoke a remote external procedure, internal procedure or user-defined function. The 
    * validation for the program name and arguments will be done on the remote side.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    * @param    timeout
    *           The maximum allowed time for this invocation to complete.  In milliseconds.
    *           Use <code>0</code> to disable the timeout.
    * @param    name
    *           The legacy 4GL name for the procedure/function.
    * @param    procID
    *           The code of a remote procedure in which to invoke the procedure or function.
    * @param    agentId
    *           The agent ID where this remote procedure was created.
    * @param    function
    *           <code>true</code> if this is a function call.
    * @param    dynamicFunction
    *           <code>true</code> if this is a DYNAMIC-FUNCTION call.
    * @param    superCall
    *           <code>true</code> if this is a RUN SUPER or SUPER() call.
    * @param    transactionDistinct
    *           Flag indicating if the TRANSACTION DISTINCT clause is in effect.
    * @param    modes
    *           A string representation of the modes of each parameter. May be <code>null</code>
    * @param    args
    *           The procedure's arguments.
    * 
    * @return    A {@link AppServerInvocationResult} instance with all updated OUTPUT/INPUT-OUTPUT
    *            arguments and any returned value.
    *
    * @see      #getAgentPool(String)
    */
   public static AppServerInvocationResult invoke(String    id,
                                                  int       requestId,
                                                  long      timeout,
                                                  character name,
                                                  String    procID,
                                                  int       agentId,
                                                  boolean   function,
                                                  boolean   dynamicFunction,
                                                  boolean   superCall,
                                                  boolean   transactionDistinct,
                                                  String    modes,
                                                  Object... args)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         InvokeConfig cfg = new InvokeConfig();
         cfg.setTarget(name);
         cfg.setDynamicFunction(dynamicFunction);
         cfg.setFunction(function);
         cfg.setSuperCall(superCall);
         cfg.setTransactionDistinct(transactionDistinct);
         cfg.setModes(modes);
         cfg.setArguments(args);

         return msaManager.invoke(id, requestId, cfg, procID, timeout);
      }
      
      AgentPool pool = getAgentPool(id);
      
      // get the agent from the pool, depending on the operating mode
      // ensure all internal-entries must use the same agent, if this is async and session-free
      Agent agent;
      try
      {
         agent = pool.acquireAgent(id, procID, agentId);
      }
      catch (NumberedException e)
      {
         AppServerInvocationResult result = new AppServerInvocationResult();
         result.setError(e);
         return result;
      }
      
      try
      {
         // send the invoke command to the agent
         return agent.invoke(id, requestId, timeout, name, procID,
                             function, dynamicFunction, superCall,
                             transactionDistinct, modes, args);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }

   /**
    * Invoke a remote external procedure, internal procedure or user-defined function. The 
    * validation for the program name and arguments will be done on the remote side. Any errors,
    * result or updated arguments will be sent back to the caller.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    * @param    timeout
    *           The maximum allowed time for this invocation to complete.  In milliseconds.
    *           Use <code>0</code> to disable the timeout.
    * @param    cfg
    *           The invoke configuration.
    * 
    * @return    A {@link AppServerInvocationResult} instance with all updated OUTPUT/INPUT-OUTPUT
    *            arguments and any returned value.
    */
   public static AppServerInvocationResult invoke(String    id,
                                                  int       requestId,
                                                  long      timeout,
                                                  InvokeConfig cfg)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         return msaManager.invoke(id, requestId, cfg, null, timeout);
      }

      AgentPool pool = getAgentPool(id);
      // get the agent from the pool, depending on the operating mode
      // ensure all internal-entries must use the same agent, if this is async and session-free
      Agent agent;
      try
      {
         String procID = null;
         int agentId = -1;
         handle inHandle = cfg.getInHandle();
         if (ProcedureManager.isProcedure(inHandle))
         {
            if (ProcedureManager.isProxy(inHandle))
            {
               procID = ((ProxyProcedureWrapper) inHandle.getResource()).getResourceId();
               agentId = ((ProxyProcedureWrapper) inHandle.getResource()).getAgentId();
            }
            else if (!(inHandle.getResource() instanceof DeferredProgramWrapper))
            {
               // a deferred program means that it is either SINGLETON or SINGLE-RUN, and will be instantiated
               // on the remote side - so no procedure ID can be acquired.

               procID = inHandle.toString();
            }
         }

         agent = pool.acquireAgent(id, procID, agentId);
      }
      catch (NumberedException e)
      {
         AppServerInvocationResult result = new AppServerInvocationResult();
         result.setError(e);
         return result;
      }
      
      try
      {
         // send the invoke command to the agent
         return agent.invoke(id, requestId, timeout, cfg);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }

   /**
    * Invoke a remote external procedure, persistently. The validation of the program name and
    * arguments will be done on the remote side.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    * @param    timeout
    *           The maximum allowed time for this invocation to complete.  In milliseconds.
    *           Use <code>0</code> to disable the timeout.
    * @param    name
    *           The legacy 4GL name for the procedure.
    * @param    transactionDistinct
    *           Flag indicating if the TRANSACTION DISTINCT clause is in effect.
    * @param    modes
    *           A string representation of the modes of each parameter. May be <code>null</code>
    * @param    args
    *           The procedure's arguments.
    *
    * @return    A {@link AppServerInvocationResult} instance, which holds the code for the remote
    *            procedure, all updated OUTPUT/INPUT-OUTPUT arguments and any returned value.
    *
    * @see      #getAgentPool(String)
    */
   public static AppServerInvocationResult invokePersistent(String    id,
                                                            int       requestId,
                                                            long      timeout,
                                                            character name,
                                                            boolean   transactionDistinct,
                                                            String    modes,
                                                            Object... args)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {

         InvokeConfig cfg = new InvokeConfig();
         cfg.setArguments(args);
         cfg.setModes(modes);
         cfg.setProcedureHandle(new handle());
         cfg.setTarget(name);
         cfg.setTransactionDistinct(transactionDistinct);

         return msaManager.invoke(id, requestId, cfg, null, timeout);
      }
      
      AgentPool pool = getAgentPool(id);
      
      // get the agent from the pool, depending on the operating mode
      // ensure all internal-entries must use the same agent, if this is async and session-free
      Agent agent;
      try
      {
         agent = pool.acquireAgent(id);
      }
      catch (NumberedException e)
      {
         AppServerInvocationResult result = new AppServerInvocationResult();
         result.setError(e);
         return result;
      }
      
      try
      {
         // send the invoke command to the agent
         return agent.invokePersistent(id, requestId, timeout, name, transactionDistinct, modes, args);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }
   
   /**
    * Handle a generic web service request.  This exposes the Jetty request and response objects
    * to the Agent which will process it, so it is required for the appserver to be local.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    * @param    timeout
    *           The maximum allowed time for this invocation to complete.  In milliseconds.
    *           Use <code>0</code> to disable the timeout.
    * @param    handler
    *           The {@link com.goldencode.p2j.oo.web.WebHandler} implementation.
    * @param    basepath
    *           The service basepath.
    * @param    paths
    *           The split paths for this handler.
    * @param    token
    *           When not null, it represents the token of a FWD context created for an authenticated and 
    *           authorized web service call.  The API call will be performed in that context.
    * @param    target
    *           The request original target.
    * @param    request
    *           The Jetty request.
    * @param    response
    *           The Jetty response.
    *           
    * @return   A {@link AppServerInvocationResult} instance with any error throw during the web service
    *           processing.
    */
   public static AppServerInvocationResult handleWebService(String                      id,
                                                            int                         requestId,
                                                            long                        timeout,
                                                            Class<? extends WebHandler> handler,
                                                            String                      basepath,
                                                            String[]                    paths,
                                                            String                      token,
                                                            String                      target, 
                                                            HttpServletRequest          request,
                                                            HttpServletResponse         response)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         return msaManager.handleWebService(id, requestId, timeout, handler, 
                                    basepath, paths, token, target, request, response);
      }
      
      AgentPool pool = getAgentPool(id);
      AppServerInvocationResult result = new AppServerInvocationResult();
      
      // get the agent from the pool, depending on the operating mode
      Agent agent;
      try
      {
         agent = pool.acquireAgent(id);
      }
      catch (NumberedException e)
      {
         result.setError(e);
         return result;
      }
      
      try
      {
         // send the invoke command to the agent
         return agent.handleWebService(id, requestId, timeout, handler, 
                                       basepath, paths, token, target, request, response);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }

   /**
    * Invoke the given Java method directly.
    * <p>
    * All arguments are sent by references: if it was marked as an OUTPUT or INPUT-OUTPUT, any change to that
    * instance will be reflected back to the caller.
    * 
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    * @param    timeout
    *           The maximum allowed time for this invocation to complete.  In milliseconds.
    *           Use <code>0</code> to disable the timeout.
    * @param    cfg
    *           The configuration with the Java method to be invoked.
    * 
    * @return   A {@link AppServerInvocationResult} instance with all updated OUTPUT/INPUT-OUTPUT
    *           arguments and any returned value.
    */
   public static AppServerInvocationResult invoke(String           id, 
                                                  int              requestId, 
                                                  int              timeout, 
                                                  JavaInvokeConfig cfg)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         return msaManager.invokeJavaMethod(id, requestId, timeout, cfg);
      }
      
      AgentPool pool = getAgentPool(id);
      
      int agentId = -1;
      
      Agent agent;
      try
      {
         agent = pool.acquireAgent(id, null, agentId);
      }
      catch (NumberedException e)
      {
         AppServerInvocationResult result = new AppServerInvocationResult();
         result.setError(e);
         return result;
      }
      
      try
      {
         // send the invoke command to the agent
         return agent.invoke(id, requestId, timeout, cfg);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }
   
   /**
    * Perform a pseudo-dynamic call where the target procedure doesn't receive the parameter as arguments at
    * the Java method definition, but instead they are managed via {@link LegacyOpenClientCaller} APIs.
    * <p>
    * This allows the target program to act as a controller, where it can prepare the arguments, perform
    * security checks, etc, before dispatching the call to the real target, which can be resolved from the
    * arguments or in some other way.
    *  
    * @param    id
    *           The appserver connection id. 
    * @param    requestId
    *           The ID of the request.
    * @param    timeout
    *           The maximum allowed time for this invocation to complete.  In milliseconds.
    *           Use <code>0</code> to disable the timeout.
    * @param    procedure
    *           The target external program.  Must have no parameters defined.
    * @param    argTypes
    *           The argument types, one of the e.g. {@link LegacyJavaAppserverApi#integer()} values.
    * @param    modes
    *           The argument modes, one of the (I)NPUT, (O)UTPUT or INP(U)T-OUTPUT values.
    * @param    args
    *           The actual arguments.
    * 
    * @return   A {@link AppServerInvocationResult} instance with all updated OUTPUT/INPUT-OUTPUT arguments
    *           and any returned value.
    */
   public static AppServerInvocationResult invokeWithArgs(String    id, 
                                                          int       requestId, 
                                                          long      timeout, 
                                                          String    procedure, 
                                                          int[]     argTypes,
                                                          String    modes, 
                                                          Object... args)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         return msaManager.invokeWithArgs(id, requestId, timeout, procedure, argTypes, modes, args);
      }
      
      AgentPool pool = getAgentPool(id);
      
      int agentId = -1;
      
      Agent agent;
      try
      {
         agent = pool.acquireAgent(id, null, agentId);
      }
      catch (NumberedException e)
      {
         AppServerInvocationResult result = new AppServerInvocationResult();
         result.setError(e);
         return result;
      }
      
      try
      {
         // send the invoke command to the agent
         return agent.invokeWithArgs(id, requestId, timeout, procedure, argTypes, modes, args);
      }
      finally
      {
         pool.releaseAgent(id, agent);
      }
   }

   /**
    * Disable legacy OO errors thrown on this connection.
    * 
    * @param    id
    *           The appserver connection id. 
    */
   public static void disableLegacyErrors(String id)
   {  
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaConnection(id))
      {
         msaManager.disableLegacyErrors(id);
         return;
      }
      
      AgentPool pool = getAgentPool(id);
      pool.disableLegacyErrors(id);
   }
   
   /**
    * Check if the current {@link WorkArea#connectionId connection ID} should suppress legacy OO errors.
    * 
    * @return   See above.
    */
   public static boolean isDisabledLegacyErrors()
   {
      WorkArea wa = local.get();
      AgentPool pool = appServers.get(wa.appServer);
      if (pool != null)
      {
         return pool.isDisabledLegacyErrors(wa.connectionId);
      }
      
      return MultiSessionAppserverManager.getInstance().hasLegacyErrorsDisabled();
   }

   /**
    * Get the number of default threads which can be used on the remote side, to execute appserver calls.
    * This is in sync with the number of maximum agents (or sessions for multi-session mode), being able to
    * process concurrent requests on the FWD server.
    * 
    * @param    appserver
    *           The appserver name.
    *           
    * @return   See above.
    * 
    * @see      RemoteAppServerConnectionPool
    */
   public static int getRemotePoolThreads(String appserver)
   {
      waitForAppserver(appserver);
      
      // get the number of threads which the remote side can use for executing calls; this number is in sync
      // with the maximum number of agents, which can execute remote calls concurrently, without getting into
      // 'no servers available'; assumption is no agents get bound.
      synchronized (appServers)
      {
         AgentPool pool = appServers.get(appserver);
         return pool.getDef().getMaxAgents();
      }
   }
   
   /**
    * Wait for the specific appserver to have at least on agent started.
    * 
    * @param    appserver
    *           The appserver name.
    */
   public static void waitForAppserver(String appserver)
   {
      // wait for the agent pool to have at least one agent started
      while (true)
      {
         int startedAgents = getNumberOfAgentsInPool(appserver);
         if (startedAgents > 0)
         {
            break;
         }
         
         try
         {
            Thread.sleep(50);
         }
         catch (InterruptedException e)
         {
            // ignore
         }
      }
   }
   
   /**
    * Start this FWD context as an appserver agent.  The first established context will initialize the pool 
    * associated for this appserver.
    *
    * @param    legacyLogManagerConfigs
    *           Configs for LOG-MANAGER.
    * 
    * @return   <code>true</code> if this was an appserver. On <code>false</code>, normal P2J
    *           conversation mode will be started.
    */
   public static boolean startAppServer(LegacyLogManagerConfigs legacyLogManagerConfigs)
   {
      // called on the server-side of the appServer connection

      SecurityManager sm = SecurityManager.getInstance();
      SessionManager sessMgr = SessionManager.get();
      String appServer = sm.getAppServer();
      
      if (appServer == null)
      {
         if (LOG.isLoggable(Level.FINE))
         {
            Session session = sessMgr.getSession();
            LOG.log(Level.FINE, 
                   String.format("No app server for the serverId: %s; %d -> %d", sm.getServerId(), 
                                 session.getNodeAddress(), session.getRemoteAddress()));
         }
         // if no appServer defined for this account, then we are a normal client, so fast exit
         return false; 
      }
      
      legacyLogManagerConfigs.setAppServer(appServer);
      LegacyLogOps.appserverNameLogConfigPairs.putIfAbsent(appServer, legacyLogManagerConfigs);
      
      String subject = SecurityManager.getInstance().getUserId();

      // if this is the first connection of this appServer, it will create the pool.
      // all connections authenticated as this appServer will act as agents
      AgentPool pool = null;
      synchronized (appServers)
      {
         pool = appServers.get(appServer);

         if (pool == null)
         {
            // initialize the pool, if this is the first agent to connect
            pool = AgentPool.newPool(appServer);
            appServers.put(appServer, pool);

            synchronized (subjectToAppserver)
            {
               subjectToAppserver.put(subject, appServer);
            }
         }
      }

      WorkArea wa = local.get();
      try
      {
         wa.appServer = appServer;
         
         pool.start();
      }
      finally
      {
         // do not clean up the appserver - let it 'bounce back' if someone else connects to it! 
         /*
         if (pool.isEmpty())
         {
            // the app server was terminated, clean up after it
            synchronized (appServers)
            {
               appServers.remove(pool.getName());
            }
            
            synchronized (subjectToAppserver)
            {
               subjectToAppserver.remove(subject);
            }
            
            AppServerLauncher.terminated(appServer);
         }
         */

         wa.appServer = null;
      }

      return true;
   }
   
   /**
    * Set the current Agent's connection ID.
    * 
    * @param    id
    *           The connection ID.
    */
   static void setConnectionId(String id)
   {
      local.get().connectionId = id;
   }
   
   /**
    * Check if current running appserver knows the given procedure.
    * 
    * @param    referent
    *           A persistent procedure referent.
    *           
    * @return   <code>true</code> if the persistent procedure is known to the current running
    *           appserver.
    */
   static boolean hasProcedure(Object referent)
   {
      MultiSessionAppserverManager msaManager = MultiSessionAppserverManager.getInstance();
      if (msaManager.isMsaContext())
      {
         return msaManager.hasProcedure(referent);
      }
      
      AgentPool pool = getAgentPoolBySubject(SecurityManager.getInstance().getUserId());
      return pool.hasProcedure(referent);
   }

   /**
    * Get the agent pool for the given appserver.
    * 
    * @param    appServer
    *           The appserver name.
    *           
    * @return   See above.
    */
   public static AgentPool getAgentPoolForAppserver(String appServer)
   {
      synchronized (appServers)
      {
         return appServers.get(appServer);
      }
   }
   
   /**
    * Get the number of agents started for that appserver.
    * 
    * @param    appServer
    *           The appserver name.
    *           
    * @return   See above.
    */
   public static int getNumberOfAgentsInPool(String appServer)
   {
      if (MultiSessionAppserverDefinition.get(appServer) != null)
      {
         return MultiSessionAppserverManager.getInstance().getNumberOfAgents(appServer);
      }
      synchronized (appServers)
      {
         AgentPool pool = appServers.get(appServer);
         return pool == null ? 0 : pool.size();
      }
   }

   /**
    * Get the number of ABL sessions started for that appserver.
    *
    * @param    appServer
    *           The appserver name.
    *
    * @return   See above.
    */
   public static int getNumberOfSessions(String appServer)
   {
      if (MultiSessionAppserverDefinition.get(appServer) != null)
      {
         return MultiSessionAppserverManager.getInstance().getNumberOfSessions(appServer);
      }
      synchronized (appServers)
      {
         AgentPool pool = appServers.get(appServer);
         return pool == null ? 0 : pool.size();
      }
   }
   
   /**
    * Post the specific task to be executed in the current agent's request.
    * 
    * @param    work
    *           The task to execute.
    * @param    wait
    *           Flag indicating the method will not return until the task executes; otherwise, the method 
    *           returns immediately.
    *           
    * @throws   IllegalStateException
    *           If the current context is not one for the agent.
    */
   public static void postTask(Runnable work, boolean wait)
   {
      WorkArea wa = local.get();
      if (!isRemote())
      {
         throw new IllegalStateException("Custom tasks can be posted only from an agent's context!");
      }
      
      int requestId = uniqueId();
      wa.agent.postTask(work, requestId, wait);
   }
   
   /**
    * If this is called from an agent's context, check if the agent is still running.
    * 
    * @return   See above.
    */
   public static boolean isAgentRunning()
   {
      WorkArea wa = local.get();
      return wa.appServer != null && wa.agent != null && !wa.agent.terminated(); 
   }
   
   /**
    * Collect an {@link Agent} as is currently executing a task on the specified connection ID.
    * 
    * @param    connectionId
    *           The connection ID.
    * @param    agent
    *           The agent instance.
    */
   static void addActiveAgent(String connectionId, Agent agent)
   {
      if (connectionId != null)
      {
         synchronized (lock)
         {
            if (activeAgents.containsKey(connectionId))
            {
               activeAgents.get(connectionId).add(agent);
            }
         }
      }

      setConnectionId(connectionId);
   }
   
   /**
    * Remove the {@link Agent} as it finished executing a task on the specified connection ID.
    * 
    * @param    connectionId
    *           The connection ID.
    * @param    agent
    *           The agent instance.
    */
   static void removeActiveAgent(String connectionId, Agent agent)
   {
      if (connectionId != null)
      {
         synchronized (lock)
         {
            if (activeAgents.containsKey(connectionId))
            {
               activeAgents.get(connectionId).remove(agent);
            }
         }
      }
   }
   
   /**
    * Associate the specified thread with the specified request, in the {@link #requests} map.
    * 
    * @param    requestId
    *           The request ID.
    * @param    a
    *           The agent executing the request.
    * 
    * @return   <code>true</code> if a STOP was not already received for this request.
    */
   static boolean registerRequest(int requestId, Agent a)
   {
      synchronized (lock)
      {
         // if a STOP was received for this request, then we don't process it
         if (stoppedRequests.remove(requestId))
         {
            return false;
         }
         
         // else, register the agent thread for this request
         requests.put(requestId, a);
         
         return true;
      }
   }

   /**
    * Deregister the specified request from the {@link #requests} map.
    *  
    * @param    requestId
    *           The request ID.
    */
   static void deregisterRequest(int requestId)
   {
      synchronized (lock)
      {
         requests.remove(requestId);
      }
   }

   /**
    * Register the specified agent with this FWD context.
    * 
    * @param    agent
    *           The agent handling this FWD context.
    */
   static void registerAgent(Agent agent)
   {
      local.get().agent = agent;
   }
   
   /**
    * Get the agent pool for the given connection ID.
    * 
    * @param    connectionId
    *           The connection ID.
    *           
    * @return   See above.
    */
   private static AgentPool getAgentPool(String connectionId)
   {
      SecurityManager sm = SecurityManager.getInstance();
      boolean allowed = false;
      synchronized (authorizedSessions)
      {
         Set<Integer> sessions = authorizedSessions.get(connectionId);
         allowed = (sessions != null && sessions.contains(sm.sessionSm.getSessionId()));
      }
      
      if (!allowed)
      {
         final String msg = 
            String.format("Unauthorized usage of connection ID %s from " +
                          "session ID %d by subject %s.",
                          connectionId,
                          sm.sessionSm.getSessionId(),
                          sm.sessionSm.getSessionToken().getSubject());
         if (LOG.isLoggable(Level.WARNING))
         {
            LOG.log(Level.WARNING, msg);
         }
         
         throw new RuntimeException(msg);
      }
      
      String appServer = AgentPool.getAppserver(connectionId);
      synchronized (appServers)
      {
         return appServers.get(appServer);
      }
   }

   /**
    * Check if the given connection ID is still connected.
    * 
    * @param    connectionId
    *           The connection ID.
    * 
    * @return   <code>true</code> if the connection is still alive.
    */
   private static boolean connected(String connectionId)
   {
      String appServer = AgentPool.getAppserver(connectionId);
      return appServer != null;
   }

   /**
    * Search through and determine the remote appserver associated with the given options.
    * 
    * @param    appService
    *           The appserver's service name (the -AppService connect parameter).
    * @param    host
    *           The -H connect parameter.
    * @param    serviceName
    *           The network service name, specified by the -S parameter.
    * @param    port
    *           The network port, specified by the -S parameter.
    * @param    directConnect
    *           Flag indicating this is a direct connect (the -DirectConnect parameter).
    * @param    sessionModelString
    *           The session mode: session-free or session-managed.
    *
    * @return   The resolved appserver configuration.
    */
   static RemoteAppServer lookupRemoteAppServer(String  appService,
                                                String  host,
                                                String  serviceName,
                                                int     port,
                                                boolean directConnect,
                                                String  sessionModelString)
   {
      // handle the defaults
      if (port == 0 && serviceName == null)
      {
         port = 5162;
      }
      if (host == null)
      {
         host = "localhost";
      }

      List<RemoteAppServer> foundServers = null;
      if (directConnect)
      {
         String expectedHost = host;
         foundServers = remoteAppServers.stream()
                                        .filter(server -> server.hosts.contains(expectedHost))
                                        .collect(Collectors.toList());
         if (foundServers.size() == 0)
         {
            // testcases/appsrv/test/connect_errors/host_name.p
            ErrorManager.recordOrShowError(5482, false, expectedHost);
            ErrorManager.recordOrShowError(5468, false);
            return null;
         }
      }
      else if (appService == null)
      {
         // TODO: what about the default: appService = nameServerDefaults.get(host);
         // testcases/appsrv/test/connect_errors/direct_connect.p
         ErrorManager.recordOrShowError(9407, false, host, String.valueOf(port));
         return null;
      }
      else
      {
         foundServers = remoteAppServers.stream()
                                        .filter(server -> server.aliases.contains(appService))
                                        .collect(Collectors.toList());
         if (foundServers.size() == 0)
         {
            // testcases/appsrv/test/connect_errors/service_name.p
            ErrorManager.recordOrShowError(5484, false, appService);
            ErrorManager.recordOrShowError(5468, false);
            return null;
         }
      }
      
      if (serviceName != null && !serviceName.isEmpty())
      {
         foundServers = foundServers.stream()
                                    .filter(server -> server.services.contains(serviceName))
                                    .collect(Collectors.toList());
         if (foundServers.isEmpty())
         {
            ErrorManager.recordOrShowError(5484, 
                                           false,
                                           "Unknown service " + appService + "  transport TCP.");
            ErrorManager.recordOrShowError(5468, false, "Application server connect failure.");
            return null;
         }
      }

      // TODO: fix the case where the port is 0 in the CONNECT string
      if (port != 0)
      {
         int expectedPort = port;
         foundServers = foundServers.stream()
                                    .filter(server -> server.ports.contains(expectedPort))
                                    .collect(Collectors.toList());
         if (foundServers.size() == 0)
         {
            // testcases/appsrv/test/connect_errors/port_number.p
            ErrorManager.recordOrShowError(9407, false, host, String.valueOf(port));
            ErrorManager.recordOrShowError(5468, false);
            return null;
         }
      }

      RemoteAppServer foundServer = foundServers.get(0);
      
      if (MultiSessionAppserverDefinition.get(foundServer.name) != null)
      {
         return foundServer;
      }
      
      boolean isSessionModelDefined = sessionModelString != null && !sessionModelString.isEmpty();
      AppServerSessionModel sessionModel =
         AppServerSessionModel.SESSION_FREE.toString().equalsIgnoreCase(sessionModelString) ?
            AppServerSessionModel.SESSION_FREE :
            AppServerSessionModel.SESSION_MANAGED.toString().equalsIgnoreCase(sessionModelString) ?
               AppServerSessionModel.SESSION_MANAGED :
               null;
         
      if (isSessionModelDefined && sessionModel == null)
      {
         // testcases/appsrv/test/connect_errors/session_model_incorrect.p
         ErrorManager.recordOrShowError(0, "-sessionModel must be either Session-managed or Session-free");
         return null;
      }

      if (foundServer.sessionFree)
      {
         // default or explicit Session-managed
         if (!isSessionModelDefined || sessionModel == AppServerSessionModel.SESSION_MANAGED)
         {
            // testcases/appsrv/test/connect_errors/session_model_mismatch.p
            String errMsg = "Unexpected CONNECT request in state-free mode";
            ControlFlowOps.setReturnValue(errMsg);
            ErrorManager.noRecordOrShowError(errMsg, false);
            ErrorManager.noRecordOrThrowError("Application server connect failure. (5468)");
            return null;
         }
      }
      else
      {
         if (isSessionModelDefined && sessionModel != AppServerSessionModel.SESSION_MANAGED)
         {
            // Stateless / State-aware / State-reset
            ErrorManager.recordOrShowError(13511);
            ErrorManager.recordOrThrowError(5468);
            return null;
         }
      }
      
      return foundServer;
   }
   
   /**
    * Get the agent pool for the given subject ID.
    * 
    * @param    subjectId
    *           The subject ID.
    *           
    * @return   See above.
    */
   private static AgentPool getAgentPoolBySubject(String subjectId)
   {
      String appServer = getAppserverBySubject(subjectId);
      synchronized (appServers)
      {
         return appServers.get(appServer);
      }
   }
   
   /**
    * Get the appserver name started by the given subject ID.
    * 
    * @param    subjectId
    *           The subject ID used to start an appserver.
    *           
    * @return   See above.
    */
   private static String getAppserverBySubject(String subjectId)
   {
      synchronized (subjectToAppserver)
      {
         return subjectToAppserver.get(subjectId);
      }
   }

   /**
    * Container for agent-related info.
    */
   private static class WorkArea
   {
      /** The agent handling this FWD context. */
      private Agent agent;

      /**
       * When the P2J client is connected as an appserver, this indicates the appserver's name. 
       */
      private String appServer = null;
      
      /** The Agent's SERVER-CONNECTION-ID value. */
      private String connectionId = null;
   }

   /**
    * Container for the information needed to identify an appserver, by the data configured at the
    * CONNECT statement's options.
    */
   public static class RemoteAppServer
   {
      /** The name of this known appserver. */
      public final String name;
      
      /** The P2J host identifying this appserver. */
      public final String p2jHost;
      
      /** The P2J port (secure or insecure). */
      public final int p2jPort;
      
      /** The account (configured as an appserver) to authenticate on the remote side.*/
      public final String p2jAccount;
      
      /** The set of aliases defined for this appserver. */
      public final Set<String> aliases = new HashSet<String>();

      /** The list of name servers with which the appserver is registered. */
      public final Set<String> nameServers = new HashSet<String>();
      
      /** The hosts on which this appserver is running. */
      public final Set<String> hosts = new HashSet<String>();
      
      /**
       * The ports used by CONNECT statement to target this appserver.
       * May be nameserver or broker ports.
       */
      public final Set<Integer> ports = new HashSet<Integer>();
      
      /**
       * The network service names used by CONNECT statement to target this appserver.
       */
      public final Set<String> services = new HashSet<String>();
      
      /** Flag indicating if the appserver is session-free or session-managed. */
      public final boolean sessionFree;

      /**
       * Initialize this remote appserver by reading its configuration from the directory.
       * 
       * @param    ds
       *           The {@link DirectoryService} instance.
       * @param    appSrvPath
       *           The directory path where remote appservers are configured.
       * @param    name
       *           The remote appserver's name.
       */
      public RemoteAppServer(DirectoryService ds, String appSrvPath, String name)
      {
         this.name = name;
         
         String path = appSrvPath + "/" + name;
         this.p2jHost = ds.getNodeString(path + "/p2j_host", "value");
         this.p2jPort = ds.getNodeInteger(path + "/p2j_port", "value");
         this.p2jAccount = ds.getNodeString(path + "/p2j_account", "value");
         Boolean sessionFree = ds.getNodeBoolean(path + "/sessionFree", "value");
         this.sessionFree = sessionFree != null && sessionFree;

         String[] aliases = ds.getNodeStrings(path + "/aliases", "values");
         if (aliases != null)
         {
            this.aliases.addAll(Arrays.asList(aliases));
         }

         String[] nameServers = ds.getNodeStrings(path +"/nameservers", "values");
         if (nameServers != null)
         {
            this.nameServers.addAll(Arrays.asList(nameServers));
         }

         String[] hosts = ds.getNodeStrings(path +"/hosts", "values");
         if (hosts != null)
         {
            this.hosts.addAll(Arrays.asList(hosts));
         }

         Integer[] ports = ds.getNodeIntegers(path +"/ports", "values");
         if (ports != null)
         {
            this.ports.addAll(Arrays.asList(ports));
         }

         String[] services = ds.getNodeStrings(path +"/services", "values");
         if (services != null)
         {
            this.services.addAll(Arrays.asList(services));
         }
      }
   }   
}