FWD » Core Development » Conversion Tools

The Language Server Protocol (LSP) is a platform independent protocol for creating language support that can work across various IDEs (VSCode, Eclipse IDE, Eclipse Che, vim...).

It is an open standard, originally created by Microsoft for their internal use and then made more generic through work with Eclipse, IBM and Red Hat.

Why You Should Know the Language Server Protocol

VSCode only works with LSP. Building the LSP is the first step in implementing the full extension in #11355.

A good first step on this task would be to analyis technical approaches, particularly whether or not we should use something like LSP4J as a framework to reduce the effort on writing the LSP. One alternative is to write all the support ourselves, though my instict says that there should be some real value in a framework that handles the guts of the protocol.

• The project needs to be open source and licensed in a manner that we can safely combine with our GPLv3 project. Permissive licenses like MIT, BSD or Apache 2.0 are all OK.
• Is the project active?
• How well supported is the project?
• How mature is the project? (has it been stable for multiple years, is it used in well known projects)
• Are there any critical technical limitations when implementing using this framework?

That last item is the biggest long term concern. If we can't achieve our objectives using it or if it will cause us to do a lot of unnatural implementation to get around its limits, then it isn't a good choice.

Post the analysis here and we can discuss.

Once we have that decision, the next step is to plan the implementation of the LSP itself.

Here is the framework analysis for #11741-10.

What we would be choosing is a binding: a library that handles the protocol layer, the JSON-RPC messages the editor and our server exchange over a pipe, plus the framing, request/response matching and cancellation that go with them, and gives us the protocol's ~300 message types as Java classes. It handles none of the 4GL. That part is ours either way.

The important result of the survey is that there is no real field to compare. Every JVM option is either LSP4J itself or something built on top of LSP4J:

There are good bindings outside the JVM, Microsoft's own in TypeScript, pygls in Python, tower-lsp in Rust, and they are all ruled out by the same thing: our server has to call the conversion front end in-process. Using a non-JVM binding would mean either reimplementing the front end or inventing a second protocol between the binding and our Java, for no gain.

The license is what decides it. LSP4J is dual-licensed, and the second arm is a BSD-3-clause license, which is compatible with GPLv3, so we simply take it under that arm. This matters more than it first appears: EPL-2.0 on its own is not GPL-compatible, which is exactly why Xtext and MagpieBridge are out. LSP4J is the only option in the group that offers a permissive arm at all.

On the rest of the criteria it is clean. Its only runtime dependency is Gson, and FWD already ships the version it wants, so adopting it adds nothing new to our dependency set. It needs Java 11; we are on 17. And the same library implements the Debug Adapter Protocol, the companion protocol for debugging, so it will also serve #11742 instead of forcing a second decision later.

One genuine limitation, from LSP4J's own changelog: breaking API changes between versions are common. The cause is structural, the LSP spec is written in TypeScript, and its types do not map cleanly onto Java, so the binding periodically has to change a boolean to a Boolean, or wrap a plain type once the spec allows an alternative for it. This is an upgrade-time cost rather than a design constraint, and the answer is the one we already use for every FWD dependency: pin a version and upgrade deliberately. It does not force us into unnatural implementation, which was the real concern in #11741-10.

Recommendation¶

Use LSP4J, pinned at 1.0.0. It is the only candidate that passes the license gate, it is active and mature, its reference consumer is a Java language server driving a VS Code extension, it adds no dependencies, and it covers DAP for #11742 as well.

Worth noting for completeness: the difficulties we will actually hit are in our own front end, not in this library, and they would be identical under any of the alternatives. So this is a smaller decision than it looks, and the interesting work is in the implementation plan.

Two things to confirm and then I will move on to #11741-11: agreement on LSP4J, and on pinning 1.0.0.

Two things to confirm and then I will move on to #11741-11: agreement on LSP4J, and on pinning 1.0.0.

Agreed on both.

This is very cool!

Syntax highlighting: keywords, strings, comments, preprocessor directives and macro references.

Please make sure all literals are highlighted. We will want to expand from there to syntax recognized by the parsr, not just the lexer. For example:

• resource names (variables, streams, widgets, frames, ... colored by resource type, linked to their definition)
• program structure (names/references of methods, procedures, functions...)
• data types
• punctuation and block structure
• language statements

The OE devstudio took some short cuts and we probably have a richer AST to leverage here.

The project is found automatically by looking for cfg/p2j.cfg.xml, so there is nothing to configure. A file outside a project is not checked and the reason is reported, though highlighting and folding still work on it.

For now, this is OK. Later, we can put in a "default project cfg" that can work with non-project files. But don't worry about this now.

Also, as in OE, if an include file contains an error, the super file (the file that includes it) is not marked as incorrectly parsed (i.e., in red) until it is edited. As I mentioned before, if a different behavior is desired, please let me know.

Yes, I think this should be different. Every open (in the editor) procedure or class that depends upon a broken include file should report the error, instead of just reporting it at the include.

The colors for highlighting should be the same as in OE? At least for data types I would not consider red, because it makes you think that you just got an error, I would go with green. Also, I don't have access to a dark theme for Open Edge to check the colors due to my old version. Maybe newer versions of OE have this feature?

Also, claude just built a working Outline view for Procedures and Functions, but it relies on a pretty strange regex. It's definitely fast, but I'd rather change it to an AST traversal since our AST already has all the rich context we need. What's your input on this? Should the Outline view be changed in order to gather info from the AST itself, or should we keep using regex's?

Aaand, I'm just exploring some ideas and wanted to also ask, is there any possibility of using the call graph or our reports to gather the Outline view data more easily? Let me know if you think this is an approach worth exploring..

Also, I recall you mentioning a button for incremental project conversion that would display the 4GL and Java views side-by-side. Should we include this feature in the scope of this current task? If so, could you please provide a few more details on how it should look?

The color scheme should be our own, we don't have to follow OE. Let's make it better.

Also, claude just built a working Outline view for Procedures and Functions, but it relies on a pretty strange regex. It's definitely fast, but I'd rather change it to an AST traversal since our AST already has all the rich context we need. What's your input on this? Should the Outline view be changed in order to gather info from the AST itself, or should we keep using regex's?

Do NOT use regexes for anything. We drive it all via AST. That is the point to all of this work. We can do better because we implement a richer AST.

Aaand, I'm just exploring some ideas and wanted to also ask, is there any possibility of using the call graph or our reports to gather the Outline view data more easily? Let me know if you think this is an approach worth exploring..

Not in an initial implementation. We will integrate both facilities though call graph needs configuration hints that will have to be build before it is useful.

Also, I recall you mentioning a button for incremental project conversion that would display the 4GL and Java views side-by-side. Should we include this feature in the scope of this current task? If so, could you please provide a few more details on how it should look?

Not exactly a button. The idea is that every procedure or class will have the ability to have a companion view in Java of its converted code. These will be cross linked so that moving either direction is easy. This isn't just about incremental conversion, but about conversion in general.

But for now, don't focus on this yet.

Greg Shah wrote:

Do NOT use regexes for anything. We drive it all via AST. That is the point to all of this work. We can do better because we

Unfortunately, we can't avoid regexes for syntax highlighting, because VS Code depends on TextMate grammars to apply colors the millisecond a file is opened. TextMate evaluates regular expressions line-by-line, completely independent of our Language Server.
This is actually why syntax highlighting works even if you open a random scratch file that isn't tied to a project.

Here are some thoughts / features we might implement (it also contains ideas from yesterday's meeting):

• Synchronized Cursor Navigation: When a developer switches tabs between the main source file (.p / .w) and the preprocessed cache file, the cursor should automatically jump to the equivalent location in the target file.

• Convert on Save: A project setting that, when enabled, physically regenerates and saves the cache file to disk every time the main program is saved.

• Stale Cache Warning / No Artifacts: If the "Convert on Save" setting is set to false, we shouldn't clutter the project with generated files. However, if the developer opens an outdated cache file, the IDE should display a clear warning badge indicating that the source has been modified and the cache is out of sync.

• Smart Include Refresh Prompt: When saving an include (.i) file, the IDE should detect open parent programs that depend on it and prompt the user: "You have open programs that depend on this include file. Do you want to refresh them?"

• Include-Aware Error Reporting: If a compilation or syntax error originates inside an include file, the error marker (squiggly line) should be placed on the {include.i} reference in the parent file. Since an include can be called multiple times in the same file (sometimes with different parameters), the system should flag all occurrences where that include is called and breaks the build.

• Go to Preprocessor Definition: Ctrl+Click on a {&NAME} reference should navigate directly to its corresponding &SCOPED-DEFINE or &GLOBAL-DEFINE.

• Resolved Preprocessor Peek (Hover): Hovering over a preprocessor reference (like {&WINDOW-NAME}) should display a tooltip showing its final resolved value, saving the developer from having to search for the definition manually.

• Read-Only Cache View with Source Redirection: The preprocessed view should ideally reject direct edits (since they would be overwritten on the next compile). If a developer attempts to type in the cache file, the IDE could offer a quick action to jump to the actual source line that generated that code.

• Peek Expansion (Inline View): Alt+Clicking an include reference should open the expanded text inline (in a peek window) without leaving the file. This provides quick context without forcing the developer to actually open the include file in another TAB.

• Expansion Weight Lens: A small CodeLens annotation above each include reference reporting what it brings in (e.g., "adm2/support.i · 412 lines · 3 nested includes"). This helps developers instantly gauge how much scaffolding a file relies on.

• Origin Ribbon: In the preprocessed view, a colored stripe in the gutter should mark which include each run of lines came from (with darker shades for deeper nesting). The developer's own code would remain unshaded, making it instantly stand out.

• Fold to Source: A command in the preprocessed view that collapses every include-derived region. The file then reads exactly like the original source, allowing the developer to expand only the specific include they are currently investigating.

• Unresolved Include Explained: When an include cannot be found, the error shouldn't just say "File not found." It should explicitly list the PROPATH directories that were searched, turning a dead-end error into an actionable configuration fix.

• Include Path Completion: Typing { should trigger autocomplete suggestions for files based on the PROPATH, filtering the list as the user types to prevent hard-to-spot typos.

• Preprocessor Name Completion: Inside a {& block, the IDE should offer auto-completion for the names actually defined and in-scope at that exact point, along with their values.

• Include Rename and Move Refactoring: Renaming or moving an .i file should trigger a refactoring tool that updates every {...} reference to it across the entire project (this can also be applied for .w / .p files / variable names, procedures and more).

• Circular and Duplicate Include Warnings: The IDE should flag warnings if an include file pulls itself in (circular dependency) or if it is pulled in twice along the same chain, preventing redundant outlines and unexpected compilation errors.

I love all the new ideas! Let's GO!

Are any of these present in OEPDS?

Yes, two of these features are already present in OpenEdge: hovering over a preprocessor reference displays its resolved value, and auto-completion is supported for {&.

As a future enhancement, we could improve this further by explicitly displaying the include filename in the hover tooltip.

Yes, we absolutely should do that. Showing exactly where the definition exists is a very helpful feature.

For this scenario, Ctrl+Click takes you straight to the definition.

We could implement this for the non-local defintion too. That seems like a good idea to me.

Absolutely, both would be very useful for a developer.