Project

General

Profile

Feature #11742

implement a fully featured Debug Adapter Protocol (DAP) server for 4GL code (in FWD)

Added by Greg Shah 3 months ago. Updated 5 days ago.

Status:
WIP
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
Due date:
% Done:

0%

billable:
No
vendor_id:
GCD
case_num:
version_reported:
version_resolved:
reviewer:
production:
No
env_name:
topics:

Related issues

Related to Conversion Tools - Feature #11355: VSCode extension New
Related to Conversion Tools - Feature #11741: implement a fully featured Language Server Protocol (LSP) server for 4GL code WIP
Related to Conversion Tools - Feature #11985: implement the DEBUGGER system handle on top of the 4GL DAP server New

History

#1 Updated by Greg Shah 3 months ago

  • Subject changed from implement a fully featured Debug Adapter Protocol server for 4GL code (in FWD) to implement a fully featured Debug Adapter Protocol (DAP) server for 4GL code (in FWD)

The Debug Adapter Protocol is a platform independent protocol for separating the debugging back-end from the debugger UI. As such it allows the same back-end debugging support to be used by multiple IDEs.

It is an open standard, originally created by Microsoft for the VSCode product. It is being adopted by multiple IDEs now and it is a kind of companion to the Language Server Protocol.

It is required to be built for our VSCode extension in #11355.

#2 Updated by Greg Shah 3 months ago

#3 Updated by Greg Shah 3 months ago

  • Related to Feature #11741: implement a fully featured Language Server Protocol (LSP) server for 4GL code added

#4 Updated by Paula Păstrăguș 22 days ago

I have some questions regarding this one:

1. Which source do we debug: the original 4gl file or the preprocessed cache file?
2. Do we debug only the 4GL, or should the converted Java also be visible? For example, a mixed call stack, or a "show Java" link to the companion view
3. May the conversion be changed to produce a 4GL line to Java line map? Today the generated Java carries no line info. If so, in which form, for example a .dbgmap would be a good fit?
4. Should the map be produced always, or only behind a setting in p2j.cfg.xml?
5. Should the debugger control the code from outside, attaching to the server's JVM like a Java debugger (JDI/JDWP)? That means no runtime changes and no cost when not debugging, but 4GL state has to be read out of FWD's Java objects. Or should it control it from inside, through hooks in the FWD runtime?
6. Attach to a running server only, or should the extension also launch the server and client and run a chosen program?
7. OpenEdge compatibility: should DEBUGGER:SET-BREAK and the other DEBUGGER handle features work?
8. What are your expectations for the very first phase of DAP?

#5 Updated by Greg Shah 21 days ago

1. Which source do we debug: the original 4gl file or the preprocessed cache file?

The cache file is the most obvious mapping. It makes the most sense to me. As long as we provide mechanisms (keyboard shortcuts and popup menu items) to navigate to the original sources (we need the ability to jump to the corresponding location in the editable file from which the code came).

2. Do we debug only the 4GL, or should the converted Java also be visible? For example, a mixed call stack, or a "show Java" link to the companion view

Optimally, we would be able to switch back and forth between either the 4GL or the Java debug views. The actual debugging should be a common session. The views just show the correct language and provide the UI for interaction.

3. May the conversion be changed to produce a 4GL line to Java line map? Today the generated Java carries no line info. If so, in which form, for example a .dbgmap would be a good fit?

Yes, we will need something like this. It needs to be bidirectional.

4. Should the map be produced always, or only behind a setting in p2j.cfg.xml?

It should always be produced. I think it can go into the same ./cvt/ directory structure with our other intermediate artifacts.

5. Should the debugger control the code from outside, attaching to the server's JVM like a Java debugger (JDI/JDWP)?

Yes.

That means no runtime changes and no cost when not debugging, but 4GL state has to be read out of FWD's Java objects.

Yes

Or should it control it from inside, through hooks in the FWD runtime?

No

6. Attach to a running server only, or should the extension also launch the server and client and run a chosen program?

As noted in Phase 1 IDE Support, we need the ability to launch servers, agents and clients etc.

7. OpenEdge compatibility: should DEBUGGER:SET-BREAK and the other DEBUGGER handle features work?

Evaluate the effort for this and create a task with some details about the idea and approach. I don't think we need this feature in this task, we will defer it.

8. What are your expectations for the very first phase of DAP?

Proper step-level debugging of 4GL source code with full variable support. It is OK if this must connect to an already running server/clients. Jumping back and forth between Java and 4GL is also pretty important.

#6 Updated by Paula Păstrăguș 21 days ago

  • Related to Feature #11985: implement the DEBUGGER system handle on top of the 4GL DAP server added

#7 Updated by Paula Păstrăguș 6 days ago

  • Status changed from New to WIP

FWD 4GL debugger (DAP) and VS Code extension: what works today (2026-10-29)

Overview

The FWD 4GL debugger debugs converted 4GL programs from VS Code at the level of 4GL statements. It attaches to a running FWD server and works on the preprocessed .cache file of each program in the conversion folder (cvt/…/<program>.p.cache, .w.cache, .cls.cache): breakpoints, stack frames and steps all refer to .cache lines, include files included. A toggle switches the same session to the generated Java at any time, and the Java View shows the generated Java beside its .cache file, kept in sync by the cursor and by debug stops. The FWD runtime is unchanged: the server only needs JDWP enabled.

Getting started

  1. Convert the project with this FWD build and run ant deploy, so every program gets a <program>.dbgmap file next to its .cache.
  2. Build FWD in the project's p2j tree (./gradlew jar); the extension starts the debug adapter from there. The adapter needs a JDK (setting fwd4gl.server.java, default java).
  3. In the extension folder (tools/vscode-4gl): npm install (first time only), npm run compile, then F5 "Run Extension (clean)", and open the converted project folder (the one with cfg/p2j.cfg.xml) in the new window.
  4. Start the server with JDWP: cd deploy/server && ./server.sh -d (JDWP port 2080). The server then runs interpreted, so everything is slower.
  5. Add the attach configuration to launch.json (also offered as the snippet "FWD 4GL: Attach to FWD server"):
{ "type": "fwd-4gl", "request": "attach", "name": "Attach to FWD server",
  "host": "localhost", "port": 2080 }

6. Optional attribute projectRoot points to the converted project when it is not the open folder. Start the debug session, open cvt/…/<program>.cache, set breakpoints, then run the client.

Same FWD build on both sides. The adapter and the server must run the same FWD build. Otherwise the Debug Console shows FWD version mismatch and Buffers, Shared, System, handle attributes and buffer fields show <unavailable: FWD version mismatch> and Set Value is refused (breakpoints, stepping, stack, Locals and Program still work). Fix: rebuild FWD, run ant deploy.prepare (or ant deploy) in the project, restart ./server.sh -d and attach again.

Attach errors say what is wrong, e.g. No JDWP agent at host:port (start the server with -d) or JDWP refused the connection; another debugger may be attached (only one debugger can attach to a server).

Debugging features

Breakpoints

  • Set on lines of .cache files. A breakpoint on a blank, comment, DEFINE or END line moves to the next statement with code in the same block; the dot shows the adjusted line.
  • A breakpoint that cannot be placed stays unverified with the reason (no 4GL map, map stale: …, class not loaded yet — installed when the class loads).
  • Breakpoints work in loops (every iteration), in several programs at once, on include lines, and can be added or removed while stopped.
  • Conditions (Edit Breakpoint → Expression): a 4GL expression; stops only when it is yes. An error or a non-LOGICAL result stops and prints Breakpoint condition error: ….
  • Hit counts, counted over hits whose condition held; they restart when any breakpoint of the file is edited:
Form Stops at
N or ==N the Nth hit only
>N, >=N, <N, <=N hits compared with N
%N every Nth hit
  • Logpoints (Log Message): each {expression} is replaced by its value (or <error: …>), the line goes to the Debug Console and the thread runs on.

Stepping and threads

  • Step Over, Step Into and Step Out move by 4GL statement and follow the real control flow (LEAVE, NEXT, UNDO/RETRY, ON ERROR, CATCH/FINALLY, RETURN).
  • Step Into enters procedures, functions, methods, constructors, property accessors, triggers, external and persistent procedures (RUN, RUN … IN, DYNAMIC-FUNCTION, super procedures, PUBLISH subscribers, RUN VALUE).
  • Step Over runs a called routine at full speed; Step Out returns to the next statement of the caller.
  • DEFINE statements are not stop points; a step stops only once per line (IF x THEN NEXT. stops once).
  • Continue and Pause act on the selected thread only.
  • Each client session is a thread (Session <id> (<account>); appserver workers as <Type> worker #<n>). Only the session that hits a breakpoint stops; the others run on.
  • Disconnect removes the breakpoints, resumes the stopped threads and leaves the server and all clients running.

Call stack

  • In 4GL mode each 4GL routine is one frame on its .cache line, e.g. start.p:42 (FOR EACH) for the main block and process-value (start.p:95) for an internal procedure. FWD runtime frames are hidden; recursion and triggers get their own frames.
  • Selecting an outer frame shows its variables and evaluates expressions in it.
  • A program without a usable map appears as a dimmed Java frame marked (no 4GL map) or (map stale).

4GL ⇄ Java mode

Switch with the debug toolbar button "4GL ⇄ Java" (arrow-swap icon), the status bar item FWD: 4GL / FWD: Java, or the command "4GL ⇄ Java" in the command palette. Stopped threads stay where they are.

4GL mode Java mode
Threads Sessions, workers, stopped threads All JVM threads
Stack 4GL frames on .cache lines Every Java frame on .java lines
Variables 4GL scopes this and Java locals
Stepping By 4GL statement By Java line
Evaluate 4GL expressions Not available

Breakpoints stay on .cache lines in both modes.

Variables

Scope Contents
Locals Parameters (with their mode, e.g. n (INPUT)) and variables of the routine; for the main block, the program variables; the variable of a CATCH block, only while its CATCH runs
Program Program-level (class-level) variables, in internal procedures, functions and methods; STATIC data members with the value of the stopped session
Buffers Buffers in scope, temp-tables (default buffer), BUFFER parameters
Shared Shared and global shared variables, buffers, temp-tables
System ERROR-STATUS, SESSION, THIS-PROCEDURE, SOURCE-PROCEDURE, TARGET-PROCEDURE, SELF
  • Values look like 4GL: "text", 42, yes/no, ?, dates in the session DATE-FORMAT, DECIMAL as STRING() shows it, RAW and MEMPTR as a hex dump.
  • Extents show [n] and expand to their elements (pages of 100).
  • Handles show id and type, e.g. 1002 (PROCEDURE), and expand to their attributes. A handle the program never asked the id of shows only (PROCEDURE). An attribute whose getter reports a 4GL error shows <error: ** … (nnnn)> in place of the value; the debugged session is not affected (no message to the user, ERROR-STATUS unchanged, nothing reaches its CATCH).
  • Objects show their 4GL class name and expand to data members and properties, inherited ones included. STATIC members show the value of the stopped session, or <not set in this session yet> before the session used the class.
  • Buffers show record available, NO-LOCK or no record and expand to the fields of the current record under their 4GL names.
  • Computed properties (GET with a body) are lazy: expanding runs the GET once.
  • THIS-PROCEDURE is the procedure of the selected frame.

Watch, hover, Debug Console

Available in 4GL mode on the selected frame. The expression language (also used by conditions and logpoints) has literals (numbers, strings, yes/no, ?), names, name[i], buffer.field, h:ATTR, obj:Member, the operators OR AND NOT = <> < > <= >= BEGINS + - * / (and EQ NE LT GT LE GE) with 4GL semantics, and parentheses. There are no function calls, assignments, date literals or system handles as names (SESSION:…).

  • Watch and Debug Console results that are extents, handles, objects or buffers expand as in the Variables view.
  • Hover shows the reference under the mouse (name, buffer.field, name[i], h:ATTR chains); keywords, strings and comments give no hover. A hover never runs a computed property's GET; Watch and Debug Console do.

Set Value

"Set Value" in the Variables view takes an expression. Allowed: variables, parameters and extent elements of Locals and Program of type CHARACTER, LONGCHAR, INTEGER, INT64, DECIMAL, LOGICAL; ? sets the unknown value. A STATIC data member is set in the stopped session only. The value is stored as an ordinary 4GL assignment (undo applies). Refused, with a message: other types (DATE, DATETIME, HANDLE, objects, RAW, MEMPTR, RECID, ROWID), properties, buffers, buffer fields, object members, Shared and System, a whole extent, out-of-range numbers, and text into a number.

Java View

Shows the generated Java of a program beside its .cache and keeps both in step, with or without a debug session.

  • Open: command "FWD: Open Java View", Ctrl+Alt+J, or the title-bar button on a .cache editor (from a generated .java it opens the .cache). The two also link by themselves when the .cache and its .java are both visible in two editors.
  • Cursor sync, both ways: the cursor on a 4GL statement highlights all its Java lines, and a Java line highlights its 4GL statement. This includes DEFINE statements (their declaration), PROCEDURE/FUNCTION/method headers (method signature), routine END lines (closing brace) and block END lines of DO/FOR EACH/REPEAT.
  • Debug stops: in 4GL mode the Java shows the statement's lines subtly and the exact line the JVM is at strongly; in Java mode the .cache shows the 4GL statement line. The highlight follows the selected Call Stack frame and clears on Continue or a step.
  • The view follows the program of the active editor and never takes the focus.
  • The Java tab carries a green J badge while the map is current and a yellow S when the .cache or .java changed since conversion; the status bar then shows FWD Java View: map stale — reconvert and no highlights are drawn.
  • The Java file of the pair is read-only while linked.
  • Closing the Java tab ends the link; hiding it behind another tab or moving it to another group does not.

Limitations

  • Attach only: no launch of server or clients from VS Code.
  • Breakpoints only in .cache files, not in the .p/.w/.cls/.i sources or in .java files.
  • No exception, function or data breakpoints; no DEBUGGER system handle (#11985).
  • Expressions: no function calls, assignments, date literals or system handles as names.
  • Set Value: only the types and targets listed above.
  • Step Over a RUN that reads many records is about twice as slow as without the debugger.
  • DO j = 1 TO n revisits its header line at every iteration (REPEAT / FOR EACH do not).
  • SOURCE-PROCEDURE and TARGET-PROCEDURE are shown for the innermost frame only.
  • A handle attribute whose getter reports a 4GL error (e.g. DATA-SOURCE-COMPLETE-MAP of a buffer outside a dataset) shows <error: ** … (nnnn)> in place of the value; the debugged session is not affected. The OpenEdge debugger shows ? there.
  • The variable of a CATCH block is shown only while its CATCH runs; a built-in error object (Progress.Lang.SysError, AppError) does not show its properties yet, and methods such as err:GetMessage(1) cannot be called from a watch.
  • Watches run computed-property GETs again at every stop.
  • A thread stopped by Pause shows no Buffers, Shared, System or handle attributes; step once to inspect it.
  • A value read that takes more than 5 s shows <timed out>; that thread reads no more values until its next stop.
  • Computed EXTENT properties show their backing field.
  • Java View: DEFINE FRAME, widgets (BUTTON, RECTANGLE, IMAGE, BROWSE) and MENU have no Java lines; trigger END, property GET/SET END and a few other END lines are not mapped.
  • Hover reads one line, so a line inside a multi-line comment gives no hover.
  • This extension cannot run together with the #11741 extension yet: both have the same extension id.

Useful tips

  • After a commit or rebuild of FWD: ./gradlew jar, ant deploy.prepare in the project (ant deploy when 4GL code or conversion changed), restart ./server.sh -d, attach again; skipping this gives the version mismatch.
  • After changing a 4GL program, reconvert and redeploy it; until then its breakpoints stay unverified with map stale and the Java View shows the S badge.
  • After an update of the extension: npm run compile and reload the extension window.
  • Output channel FWD Java View gives the details of Java View failures (link failures, unreadable maps, missing Java lines).
  • Debug Console messages explain refused actions and stale maps.

Status

  • All code (debug adapter, .dbgmap generation in conversion, VS Code extension tools/vscode-4gl) is committed in a local git repository (branch 11742a, head 15dc430). Nothing is committed to bazaar yet.
  • Remaining: include the changes in the bazaar branch 11741a, together with the LSP, as several themed commits (being prepared); merge tools/vscode-4gl with the #11741 extension (only extension.ts, package.json and README.md need a manual merge; the debugger files add without clashes).

#8 Updated by Greg Shah 5 days ago

I need to review this further but I wanted to get some initial feedback to you.

1. This is EXCELLENT! I love it. Great work!

2. We can't operate purely on a "single step per line" model. Something like IF x THEN NEXT. is 2 steps because it is 2 statements. Otherwise you can't evaluate x without also executing NEXT. That is not OK.

3. Everything in the debugger should be symmetrical. By this I mean that if a feature works in the 4GL, it must also work in Java and switching back and forth should be natural. For example, if a 4GL variable is in the watchlist, then the Java version of it is in the Java watchlist.

#9 Updated by Greg Shah 5 days ago

Also: please put everything in bzr. Each step of the way needs to be there instead of in a git repo.

#10 Updated by Paula Păstrăguș 5 days ago

Thank you, Greg!

I will update my previous note (note 7), since several improvements were made after it was written.

About your points:

  • One step per statement, not per line. Agreed. Today a step stops at most once per line, so IF x THEN NEXT. evaluates x and executes NEXT in a single step. I will change stepping to stop at every 4GL statement, also when several statements share a line, and to show which statement of the line is the current one, so x can be inspected before NEXT runs.
  • Symmetry between 4GL and Java. Agreed. Every feature which works in 4GL mode should work in Java mode too, and switching should carry the context over: a 4GL variable in the watch list shows as its Java counterpart in Java mode, and back. Today Java mode shows the Java stack and locals and steps by Java line, but has no expression evaluation (so no watches, hovers or conditions), and breakpoints can only be set on .cache lines.

I will try to have as much as possible done by tomorrow, and I will post the result here.

#11 Updated by Greg Shah 5 days ago

All of this needs to be in the same branch as the LSP and all the ANTLR4 changes. Assuming 11741a was branched from 10804a, then I guess combining the DAP and LSP in 11741a will work fine.

Post the proposed package structure here for review and discussion. I want a clean design that puts the LSP, DAP and VS Code extension into its own directory tree.

#12 Updated by Paula Păstrăguș 5 days ago

11742a was not branched from 10804b, neither 11741a. The LSP still lacks auto-complete support since it relies on the ANTLR4 grammar to track valid next tokens after a 'partial' statement. I was planning to defer this past October 30 since 10804b is still awaiting review, and while I was working on the ANTLR deep dive doc, the AI surfaced several issues (which was documented) that we need to resolve first.

#13 Updated by Paula Păstrăguș 5 days ago

Proposed structure: LSP, DAP and the VS Code extension

The language server and the debug adapter each already have their own package tree, and the extension has its own directory. I propose to keep that layout and to bring both into one branch (even branched from 10804b, if needed):

src/com/goldencode/p2j/
   lsp/                    the language server (46 classes), LspLauncher is its main class
      kb/                  knowledge base resources
   dap/                    the debug adapter (38 classes), DapLauncher is its main class
      eval/                the 4GL expressions of watch, hover, conditions, logpoints and set value
      vars/                scopes and variable values, read from the stopped server over JDI
      target/              SilentGetter, the one class which runs inside the FWD server
   dbgmap/                 the .dbgmap model, its JSON reader and writer, and a command-line check

tools/vscode-4gl/          the one VS Code extension, which starts both servers
  • The conversion writes the .dbgmap files: convert/DebugMapWorker.java, hooked in rules/convert/brew.xml and control_flow.rules, with the line counting in io/ByteCountOutputStream and pattern/CommonAstSupport. It uses dbgmap, which the debug adapter reads.
  • The front end serves the language server: in preproc/ the source provider, the supplied includes and the origin hints (#3881); in uast/ the check-only mode of AstGenerator and the annotations progress.g adds in a check-only parse.
  • The FWD server loads dap/target/SilentGetter by name, so it ships in p2j.jar like the rest.

Merging the two branches

  • The Java packages do not overlap: lsp comes from 11741a, dap and dbgmap from 11742a (which is currently a git repo)
  • The extension is merged into one, with one id and one package.json:
    • From 11742a: the debugger and the Java View (debug.ts, dbgmap.ts, expressions.ts, javaView*.ts, javaIndex.ts, javaText.ts, javaBadge.ts).
    • From 11741a: everything else, including the grammar.
    • extension.ts, package.json and README.md are the only files that exists in both, 11742 and 11741.

#14 Updated by Greg Shah 5 days ago

About the directory structure:

  • Add a src/com/goldencode/p2j/ide/ super-package. The dap, lsp... all should be subpackages in that ide package.
  • tools/vscode-4gl should also be a vscode directory in the ide package.

#15 Updated by Paula Păstrăguș 5 days ago

Right. I also updated #11742-7 note with the latest improvements.

#16 Updated by Greg Shah 5 days ago

About branches:

  • For Tuesday's demo, merge 11742a into 11741a and ensure all LSP, DAP and VSCode is in a single branch.
  • After the delivery for Tuesday, rebase 10804a, test it and then merge 11741a into it. There must be only one branch for all IDE and ANTLR4 work.

#17 Updated by Greg Shah 5 days ago

In the meeting this morning, I mentioned that even DEFINE statements need to be able to have a breakpoint. Otherwise one cannot debug resource initialization.

#18 Updated by Greg Shah 5 days ago

Please move #11742-7 into a wiki page. Then edits are versioned and we can refer to it later without reading a task.

Individual commits should have a post here with good details. That way we can follow the work in the task but always find the latest docs in the wiki page.

#19 Updated by Paula Păstrăguș 5 days ago

Would it make sense to have a single wiki page in the Internal Documentation covering both 11741 and 11742, since they are related?

#20 Updated by Greg Shah 5 days ago

Yes.

Also available in: Atom PDF