The idea is that it will be a standard public image that can be executed with a single command line by a customer, and will allow the creation of a new FWD project on the host filesystem. It would be a cleaner, better way than we currently have:

• web based instead of a hacked up prepare command line script
• hide the complexity inside FWD and the docker image instead of forcing the user to deal with all the project mess

Before we do this, I want to clean up the project structure and approach. Just implementing the same approach here is not OK. By this I mean tasks such as #6256, #6320, #5586, #6552.

If we go this way, then the web UI might be primarily about defining the project-level .json file. Then a subsequent docker image can be run to use the results of this created "project". Inside that file, it would define how to find the 4GL code, the .df files and other key configuration.

The "dream scenario" is that our entire project definition is inside a single .json file. The conversion code (in Java) would actually create the destination directories (the directory structure for the intermediate and final output) as needed, so long as it is told the top-level project directory which wouldn't be in the .json so that the same .json can be used with any top-level project dir.

Is there already a task for making directory.xml maintenance more dynamic or "online" rather than static? I have some research to dump related to our prepare_dir.sh, and how it really should only be used in its current state to initialize a directory. From that point on, there should be REST APIs to manipulate it.

Morphing prepare_dir into a runtime Directory REST tool — research & design notes¶

Status: research / pre-design. Captures the investigation done before any code was written. FWD class/method/line references below were gathered from a fast read of the trunk and are a map for orientation — verify exact signatures against the source before coding.

Goal¶

Today, changing FWD directory.xml config means regenerating the whole file and restarting the server (destructive). We want to morph this into a tool that makes targeted modifications to a running environment via REST APIs, where the REST APIs are implemented in the FWD code at ~/projects/fwd/trunk.

Part 1 — How prepare_dir.sh works today (the destructive generator)¶

Location: deploy/server/prepare_dir.sh, driven by prepare_dir.json (samples in deploy/etc/prepare_dir.json and deploy/server/cfgs/*.json).

It is a whole-file generator for FWD's directory.xml:

1. Input: prepare_dir.json, parsed with jq. Structure:
   • top-level scalars: ports (from_port/to_port/as_port), webclient_timeout, pool sizing (client_rcount/client_rlimit, as_rcount/as_rlimit, server_rcount/server_rlimit), memory (webclient_memory, appsrv_memory), etc.
   • optional nested sections: appcds, webcert, soap, restAPI, proxy, sso, dbnames + per-DB objects, brokers + per-broker objects.
2. Base render: the ~132 KB directory.xml.template (full of {token} placeholders) is filled by one large sed invocation → directory_tmp.xml.
3. Fragment merge: each optional section renders its own small template (directory_cert.xml.template, directory_soap.xml.template, directory_restAPI.xml.template, directory_proxy.xml.template, directory_db.xml.template, directory_broker.xml.template) via sed, then is spliced into the base using the FWD Java tool java ... com.goldencode.p2j.directory.DirectoryCopy copy <srcfile> <srcloc> <dstfile> <dstloc> — copies an XML subtree at a directory path such as /security or /server.
4. Crypto / derived values via FWD Java helpers:
   • com.goldencode.p2j.security.HashPassword — admin-console password hash.
   • com.goldencode.p2j.security.EncryptBase64Value — web-cert private key encoding.
   • gencert.sh — per-broker certificate generation.
   • also emits server.xml and each broker's client.xml.
5. Output: the complete file, written atomically (mv directory_tmp.xml $dirfile); refuses to overwrite without -o; optional .backup.

Consequence: to change one value you regenerate the entire file and restart the server.

Part 2 — Why the morph is feasible (FWD trunk capabilities)¶

directory.xml is just the on-disk serialization of FWD's live Directory registry. The running server already exposes nearly everything needed. (Paths under ~/projects/fwd/trunk/src/com/goldencode/p2j/.)

Directory subsystem (directory/)¶

• In-memory model: hierarchical DirNode tree (attributes + children); nodes addressed by absolute paths (/server/default/...). Backends behind a Remapper interface: RamRemapper, XmlRemapper, Redis-backed.
• Runtime READ: DirectoryService / Directory — getString/getInt/getBoolean/getStrings, enumerateNodes, getObjectList.
• Runtime WRITE: addNode(nodeId, nodeClass, Attribute[]), addNodeAttributes(...), setNodeString/Integer/Boolean(...), deleteNode(...), deleteNodeAttribute(...).
• Transactional + lock-safe: openBatch(nodeId) → mutate → closeBatch(disposition) (true=commit, false=rollback). Uses a ShadowRemapper overlay so other sessions don't see partial state; guarded by LockManager (locks RO/RW/RX/WX; enterLock/releaseLock/upgradeLock).
• Persistence: backupDirectory(newFile) / XmlRemapperIO.save(...) write the in-memory tree back to disk. No automatic periodic flush — must be called explicitly.
• Distributed propagation (stateless clustering): commits via the Redis stream (RedisSynchronizer, ShadowRemapper) fan out to every app-server node — so a runtime change on one node reaches the others. (Ties to the wiki's "Stateless Server Clustering" scaling story.)
• DirectoryCopy already has CRUD verbs (offline only today): copy, set <file> <loc> <attr> <value>, remove <file> <loc>, clear, protcopy, selcopy.
• DirectoryDiff.java exists — likely reusable for "desired vs running" diffing.
• Startup-only vs runtime values: some settings are cached at init (e.g. server id, security realm, possibly some DB/pool settings). Must classify each key as apply-live vs needs-restart.

REST framework¶

• Existing admin REST surface: /admin/* via BaseAdminRestHandler (src/com/goldencode/p2j/admin/server/) with handleGet/handlePost/handlePut/handleDelete. Concrete handlers already exist: sessions, users, domains, tenants, certs. No directory/config handler yet — that's the gap.
• Registration pattern: extend BaseAdminRestHandler, set a context path (e.g. super("/admin/directory", auth)), provide a static initialize(WebAuthHandler) that registers with AdminRestHandler.
• Auth: token-based via WebAuthHandler (com/goldencode/p2j/main/); admin-rest enable/timeout read from directory (admin-rest/enabled, admin-rest/timeout).
• JSON: Jackson (com.fasterxml.jackson.databind).
• program-based REST: com/goldencode/p2j/rest/RestHandler.java — the restAPI section in directory.xml configures this; not the right home for admin/config CRUD.

Part 3 — Proposed morph (two coordinated pieces)¶

1. In ~/projects/fwd/trunk: new DirectoryRestHandler extends BaseAdminRestHandler mounted at /admin/directory, exposing CRUD over directory nodes/attributes by wrapping DirectoryService's batch + lock + persist (gaining cluster propagation automatically via the Redis path).
   • e.g. GET /admin/directory/{path} (read node/attr), PUT (set attr), POST (add node), DELETE (remove). Auth via WebAuthHandler.
2. In hotel_docker: evolve prepare_dir.sh (or a successor tool) so it:
   • (a) still does first-time / offline bootstrap (no server running yet) — keep the existing template + DirectoryCopy path; and
   • (b) when a server is running, diffs the desired JSON against the live directory and PATCHes only the deltas over the new REST API — non-destructive, no restart for hot-appliable values. (DirectoryDiff likely reusable here.)

Part 4 — Open questions / decisions before coding¶

• Hot vs. startup-only keys: build a classification of which directory keys take effect live vs require a restart; the tool must report this honestly (don't silently "succeed" on a key the server only reads at init).
• Safety: live config mutation needs strong auth/ACL, an audit trail, and a first-class dry-run / diff mode.
• Verify signatures: confirm exact DirectoryService / BaseAdminRestHandler / DirectoryDiff APIs and line numbers against the trunk source before writing against them.
• Bootstrap defect (resolved): docker/repo/entrypoint/docker-entrypoint_prepare.sh had an unresolved bzr merge conflict — user resolved it.

Key file index¶

hotel_docker (client/generator side):

• deploy/server/prepare_dir.sh — the generator.
• deploy/etc/prepare_dir.json, deploy/server/cfgs/prepare_dir_broker_allinone*.json — sample inputs.
• deploy/server/directory.xml.template (base) + directory_{cert,soap,restAPI,proxy,db,broker,db_connections}.xml.template (fragments).
• deploy/server/{server.xml.template, broker_client.xml.template, gencert.sh}.

fwd trunk (REST API side), under src/com/goldencode/p2j/:

• directory/DirectoryService.java — core read/write/batch/persist.
• directory/DirectoryCopy.java — offline CRUD verbs used by prepare_dir.sh.
• directory/DirectoryDiff.java — diff support.
• directory/{Remapper,RamRemapper,XmlRemapper,XmlRemapperIO,ShadowRemapper,RedisSynchronizer,LockManager}.java.
• admin/server/BaseAdminRestHandler.java + SessionRestHandler.java (example) + AdminRestHandler.java.
• main/WebAuthHandler.java — auth/token.
• rest/RestHandler.java — legacy program-based REST (separate from admin REST).