FWD » Core Development » User Interface

In the following, the term AJAX is used as a shorthand for a javascript implementation that runs as a "web application", however we are planning to use websockets instead of XmlHttpRequest, so the term AJAX is not technically accurate.

The 4GL is inherently a fat-client environment. Many features of the implementation are designed such that the execution of a 4GL process (pro, mpro, prowin32...) assumes that the processing occurs within the context of a single user's account. For example, 4GL code can:

• launch child processes
• access the file system, printers and other devices
• create buffers in memory and directly manipulate that memory
• make calls to arbitrary shared library (DLL) entry points
• use sockets

In a perfect world, we would be able to virtualize everything and run it in a single server on threads that maintain the proper user context. That is not possible. Each of these items above (and there are more than those listed) must run in the context of the current OS user account. For example, the security and permissions for child processes and file system access cannot be properly duplicated in a shared server environment. Likewise, when the 4GL client starts a "server socket", it might bind that socket to a specific port. If 4GLclients are started on different systems, this would be perfectly possible but it would not be something we could duplicate in an "everything is virtualized in the server" approach. DLL access is similar, since there can only be 1 copy of a DLL in a process at one time and if a DLL was not written to be thread safe/context aware on a per thread basis, then 2 users accessing a non-safe DLL from the same process could corrupt data and cause abends and worse.

The only correct way to implement this is on the client-side. Each P2J client thus has its own process, running in the context of the OS user account and this exactly replicates the execution environment of the 4GL, allowing us to replace all of these client features. Of course, the majority of the processing (business logic, database access...) is still done in the shared server (on threads with dedicated context for that user), but when something client-specific must execute, we remotely access that functionality on the client. More specifically, we call down to the client, from the server, using our remote object protocol.

This means that each user of the system must have their own process in which they run the client-specific portions of the application. To date, this has always meant a Java process, running our P2J client code.

What does all of this have to do with implementing an AJAX client?

The UI for any of our clients (ChUI or GUI) is by nature, a client-side thing. With the move to an AJAX UI, we will be moving the interactive portions (rendering/display of the UI and processing key/mouse input) of the UI into the browser. This is a core objective of the current project and represents a very interesting deployment scenario for virtually all customers with which we have ever spoken.

However, we don't plan to re-implement our entire client in javascript. For one thing, the P2J client code is too complex to easily port to javascript. Secondly, maintaining 2 implementations of the same client is not something we are interested in doing.

With this in mind, in addition to the "every user is a separate process" requirement, we have these other objectives:

• Maximize the use of our current Java client code.
• Maximize the use of common code (in Java) in the future solution.
• Minimize the code on the Javascript side.

The above constraints suggest this approach:

All processing will be HTTPS.

The browser client will open a URL that is serviced by Jetty in the P2J server process. I'll call this the "server jetty". The URL will be something like https://p2jserverhost:jettyport/chui/ (for the AJAX ChUI) or https://p2jserverhost:jettyport/gui/ (for the AJAX GUI).

When the server jetty has an inbound connection for one of these AJAX URLs, it will do the following:

1. dynamically start a new P2J client process (we already do something very similar in starting "appserver instances")
2. this will be done by a special launcher process that is there to create child processes under the control of the P2J server
3. the new client will be started in "browser mode" (ChUI or GUI) and the client code will be the standard P2J java client
4. that P2J client will start up and will make an initial connection to the server's secure port using the P2J protocol (this is the "client jetty")
5. the P2J client will have a Jetty instance in its JVM that is initialized to offer HTTPS support
6. the P2J client will notify the P2J server about the URL for which the client jetty is listening, this will include a custom/dynamic port number
7. the server jetty will respond to the original browser request with a page that redirects the browser to the new client jetty URL
8. the browser will connect to the client jetty, which will send down a page that is the proper AJAX client (ChUI or GUI)
9. that AJAX client will establish a secure websocket connection to the client jetty
10. on the P2J client side, the "upper half" of the AJAX driver will be written in Java and it will map into the ChUI driver interface or the GUI driver interface
11. on the browser side, the javascript client will be the "lower half" of the ChUI driver interface or the GUI driver interface, responsible for the rendering of specific windows, frames, widgets... and handling all user input (keys and mouse)
12. these two halves of the AJAX driver will cooperatively process over the websocket to provide the same kind of UI services that could have been done with an in-process driver in Swing
13. the P2J client will handle the majority of the processing and in fact it can be virtually the same as it is today, including all of the session setup/communications with the P2J server
14. the only difference is that instead of using Swing for the low level drawing and input events, those aspects must be remoted over the websocket
15. this means that the our current client architecture is used exactly as it is designed today, and it can handle all the same client side features like child processes, file system access, DLL calls and so forth
16. those features are just implemented outside of the browser, and potentially on a different physical system, although technically it would be possible to have that client process be local to the same system on which the browser is running (this may be needed in some use cases where the client features in use require interaction with the user or some kind of device access that is not available on the server)

For task #2200, the following must be reviewed/considered:

• The startup process for the ClientDriver and ThinClient will be different with Jetty.
• The AJAX mode for the client is basically a new driver implementation. That means it is going to be mutually exclusive with running any other driver. For example, in AJAX ChUI mode, it would replace the Swing ChUI client AND the JNI ("Console") ChUI client.
• The client will listen on a random port number (above 1024) which will allow the port number to be allocated dynamically by the OS/Java and also for there to be no security restrictions on opening that port.
• The client jetty port will have to be reported back to the server (so that the server jetty can construct the proper redirect page), but that back-channel communication will probably not be handled in #2200. We will have a subsequent task defined to implement the dynamic launch of the AJAX client and that back-channel comms will occur are part of the launch.
• The client jetty must startup in SSL mode and we need to consider what keys are used. The server jetty uses the SSL context already loaded by the server's SecurityManager, which means it is backed by the keys that are configured in the server's directory. At a minimum, we want to enable the same server-configured keys to be utilized in the client jetty. Such a configuration is attractive because it will either allow a properly signed certificate to be re-used or the user's web browser will already have the necessary "certificate exceptions" loaded. In other words, we want to use SSL without having to do any per-client configuration in the common case.
• However, we must also enable the cases where there is client SSL configuration (in our directory as the 1st choice and then possibly overridden through local files/bootstrap config). For example, if the clients are running on a separate host from the P2J server, then there can be certificate issues as a result.

From Marius on November 4, 2013:

Regarding the implementation on P2J I read your specifications.
Here is what I understood so far:

1. The browser client will open a URL that is serviced by Jetty in the P2J server process. I'll call this the "server jetty". The URL will be something like https://p2jserverhost:jettyport/chui/ (for the AJAX ChUI) or https://p2jserverhost:jettyport/gui/ (for the AJAX GUI).

Q: The "server jetty" is in fact the WebServer or another instance, a new design?

2. Dynamically start a new P2J client process (we already do something very similar in starting "appserver instances")
This will be done by a special launcher process that is there to create child processes under the control of the P2J server
The new client will be started in "browser mode" (ChUI or GUI) and the client code will be the standard P2J java client
That P2J client will start up and will make an initial connection to the server's secure port using the P2J protocol (this is the "client jetty")

Q: The "client jetty" is started similar to ServerDriver#launchAppServer?

3. The P2J client will notify the P2J server about the URL for which the client jetty is listening, this will include a custom/dynamic port number
The server jetty will respond to the original browser request with a page that redirects the browser to the new client jetty URL
The browser will connect to the client jetty, which will send down a page that is the proper AJAX client (ChUI or GUI)
That AJAX client will establish a secure websocket connection to the client jetty

Q: The P2J client is a new process allocated per connection having an embedded jetty server which should deliver
Web Sockets Services and the main page?

Q: The port number should be generated by an algorithm randomly or in sequence?

Q: The jetty protocol is wss over https?

Q: The SSL context is the same as the "server jetty" or if in the directory is registered a SSL context
(keystore and credentials) for this purpose use this one as a first option?

From Greg on November 4, 2013:

Q: The "server jetty" is in fact the WebServer or another instance, a new design?

I don't see any important advantage to have another instance. My plan was to just have a different URL (than the admin console) but to extend the same WebServer.

If you have any reasons to do something different, let me know.

Q: The "client jetty" is started similar to ServerDriver#launchAppServer?

Not exactly. That code is related, but I am referring to the server-side code in AppServerLauncher.java. That is the code that really does the launching work. Specifically, look at builder() and launchWorker().

But the development for client launching part will be done in another task than #2200.

Q: The P2J client is a new process allocated per connection having an embedded jetty server which should deliver Web Sockets Services and the main page?

Exactly.

Q: The port number should be generated by an algorithm randomly or in sequence?

I don't want to allocate it at all. Let the operating system allocate it dynamically. We can read the allocated port number from the resulting socket instance and send it back to the server as part of the complete URL.

Q: The jetty protocol is wss over https?

Let's not call it the "jetty protocol", but rather the "web client protocol".

Yes, it is wss from an https session.

Q: The SSL context is the same as the "server jetty" or if in the directory is registered a SSL context (keystore and credentials) for this purpose use this one as a first option?

Yes, exactly.

From Marius on November 5, 2013:

Today I started to design and implement the AJAX Client Driver.
So far I designed the embedded jetty SSL server and a generic
web socket handler used to register web sockets on the server context.

From Greg on November 7, 2013:

Initial Feedback

I realize that this is a work in progress and an early version. Thank you for posting it and the useful description of the approach. I will refrain from a full code review because this is still early and will change a great deal.

I like what you are doing in general. There is some very good stuff here. I have the following questions/comments:

1. WsServer has very little to do with WebSockets. Other than the WsHandler inner class and the addWsHandler() method, the code is otherwise quite generic. In fact, it seems to me that it is very, very similar to our current com/goldencode/p2j/main/WebServer. I prefer to maximize common code here, since the server jetty and client jetty will both have similar configuration requirements and will need to support similar usage patterns.

a. I would like to enhance or extend our current WebServer to be used as both the server jetty and client jetty. This eliminates having a 2nd implementation that is so similar to the code we are already maintaining. To the degree that we need to provide more control over the startup/configuration/shutdown of the WebServer class to make it more flexible, we should do so. It also makes it easier to add WebSockets support to the server jetty, which is something will need in the future as we shift our admin console to AJAX.

b. I'd like to see the WsHandler code moved out of that class and put in its own class. The registration code can be hidden there too (to register itself with the WebServer instance). In addition, please call it WebSocketsHandler because the "ws" abbreviation can mean too many different things.

2. I don't understand the need for the WsClient interface. We can always create a helper class for providing common code for the startup/shutdown of the client jetty. It doesn't need to be in the inheritance hierarchy of the client driver. I prefer that approach because there are less layers of abstraction.

3. If I understand the code properly, all the "static" HTML can be loaded from our p2j.jar, right? That is exactly how we want this to work, since the code will only change when we have a revision of P2J. That avoids maintaining separate files in the filesystem which is very good.

4. I like the handleReplacements() support you have provided. It is useful and will be needed in the future. However, for the specific use case in your test code, I think you can use the javascript location object to get the host, port and so forth.

From Greg on November 8, 2013:

Feedback

This is very good. Some thoughts:

1. Change the "ws" package to "web". This will be the place for generic web server classes.

2. As you get a real client implementation going, you will not need the test package anymore. You can keep it for now, for development purposes, but the final update should not have it.

3. Make sure you are reviewing the ClientDriver, com/goldencode/p2j/ui/chui/ThinClient (in regards to how it integrates with the "drivers" and also how it does much of the startup processing) and the ChUI drivers (com/goldencode/p2j/ui/client/chui/driver/*). These will help you design the changes for how the ChUI web client should be implemented.

4. In regards to coding standards, we now have a 98 character line limit instead of 78 (docs aren't up to date yet). Also, please make sure that all methods and members are javadoc'd. There will be more details on this later, when the code is nearing completion.

Keep going. It is good work.

From Marius, on November 11, 2013:

Today I did a lot of tests regarding the ClientDriver and ThinClient.
I'm able to starts and debug from my IDE both instances, the server and the client.
Here is what I understood from this tests. Please let me know if something are missing.

1. The ClientDriver is a remote client started from a command line which run on separate process having
only one thread, the main thread.

The command line arguments define the ScreenDriver type and parameters.
Also the server host and port are specified as command line parameters.
It is possible to provide the same parameters using a configuration file like client.xml

net:server:host=localhost
net:server:insecure_port=3333

client:driver:type=swing_chui_frame
client:chui:rows=24
client:chui:columns=80
client:chui:background=0x000000
client:chui:foreground=0xFFA500
client:chui:selection=0x0000FF
client:chui:fontname=monospaced
client:chui:fontsize=12

2. From the command line arguments a BootstrapConfig object is constructed used to create, initialize and start
a generic client ThinClient (CoreClient#start).
It's possible to create a remote standalone client running on a separate process like ClientDriver do
or a client running within the server process.

3. Next the UI services are initialized: ThinClient#initializePrep(config, driver, single);
An OutputManager instance wrapping a ScreenDriver object is created.
The OutputManager is used to draw on the screen via an OutputPrimitives interface (ScreenDriver#getPrimitives)
Currently p2j has 2 screen drivers:

- ConsoleDriver - or chui native based on NCURSES package.
- SwingChuiDriver - a Java SWING based client.

I tested both screen drivers and I found to work within ClientDriver.

Also a Java applet ChuiApplet client can be used embedded in an html page loaded by a web browser.

https://localhost:7443/chui

According to my tests this client only draw the main window and a cursor on the top left corner.
It seems that this client is not fully implemented.

4. When the ThinClient instance is created the ClientExports and SessionExports (for static methods) interfaces are exported remotely.
(The SessionExports interface seems to be not used remotely. I didn't found any remote import for this interface).
The registered (exported) ClientExports interface is imported by the LogicalTerminal instance an used to do RMI calls to ThinClient.

5. Next the client is authenticated and a security context for this thread is created.

6. Next finish initializing the thin client: ThinClient.initializePost(session, context)
Imports ServerExports and RemoteErrorData interfaces exported by the remote StandardServer used for RMI calls to server.

7. Finally the MainEntry interface exported (registered) on the StandardServer is imported, a
ClientParameters is constructed having the converted 4gl application
command line parameters and the remote application is started.
In my case Ask#execute is called which is the Java converted version of the 4gl Ask.p

// call the application
running = main.standardEntry(params);

8. The client code waits here until the user will terminate the running application.

9. When the application is terminated by the user, some clean-up statements are executed
and finally the client exits: System.exit(0);

As a conclusion of my tests, I have a question:

I think that we have to design a new ScreenDriver having an embedded server
used to communicate with the JS code which run inside the HTML page on the web
browser via web sockets.
In this case what kind of UI components are used inside the HTML page?

a - HTML specific tags?
b - JS based?
c - java SWING applet?
d - others?

From Greg on November 11, 2013:

Before I review the code, I will provide some comments and answers to note 10.

The ClientDriver is a remote client started from a command line which run on separate process having only one thread, the main thread.

Yes, ClientDriver is a separate Java process that runs in its own JVM, outside of the P2J server.

Yes, it is remote in the sense that it communicates with the P2J server using our own proprietary remote object protocol.

No, it does not only have 1 thread. There is a type-ahead (key reading) thread that supports our requirement to provide asynchronous interrupts for CTRL-C. And the network socket that connects the client to the P2J server has 2 threads (1 reader and 1 writer). The main thread is in fact the primary thread for doing real work. It is enslaved to the "conversation mode" that we implement between the client and server. See the Conversation class in the net package.

FYI, there is a special-purpose mode in which the ServerDriver can be run as both client and server in one JVM. This is not something we use often and no customers use it today.

The command line arguments define the ScreenDriver type and parameters.
Also the server host and port are specified as command line parameters.
It is possible to provide the same parameters using a configuration file like client.xml

Yes, mostly right. I would note that the bootstrap config can be given as a file and any values in it can be overridden on the command line. The idea is to use that for values that are needed for early configuration of the client (or server, since it can be used there too). Generally, we try to push as much configuration into the P2J server's directory.xml (often just referred to as the "directory") as possible. This maximizes central management of the configuration and can make deployments easier.

For programs that use the P2J classes to make their own connection to a P2J server, the bootstrap configuration can be completely built and controlled under program control.

According to my tests this client only draw the main window and a cursor on the top left corner.

That may be a regression or perhaps it is some applet problem on your system/browser. When I put that applet in, it was working without issues. However, if you were trying it with MAJIC, it would not work today because MAJIC is very dependent upon executing child processes (and other direct filesystem access). That is not possible in the applet approach.

The SessionExports interface seems to be not used remotely. I didn't found any remote import for this interface

No, it is in use. Please see com/goldencode/p2j/util/SessionUtils.java. It is used to "push" some 4GL session-level configuration down to the client so that to the extent that the UI can render or otherwise need to use this configuration, it is in synch with the state on the server.

The registered (exported) ClientExports interface is imported by the LogicalTerminal instance an used to do RMI calls to ThinClient.

Almost. I would not call our approach RMI, which is a very specific Java implementation. However, the idea is the same. We have our own "remote object" protocol (sometimes called the "DAP"/distributed application protocol, but mostly just called "the protocol"). This is the equivalent of Java's RMI, but it has better security and it is easier to implement as a developer. It also doesn't need a name registry process to be running.

But otherwise, your description is correct.

Also note that we instantiate some other classes (e.g. FileSystemDaemon, StreamDaemon, ProcessDaemon, MemoryDaemon...) which also register client-side services that can be called from the server. These are the daemons that allow the client-side process launching, file system access and so forth.

The remote object protocol is designed such that either side of the link can export an API. This peer-based or birdirectional RPC is actively used.

Next the client is authenticated and a security context for this thread is created.

Yes. Note that the context is on the server. On the client, there is only one "session" and the security manager is intentionally a null thing. Since all the actual converted code is running on the server, in a session that has a specific context associated with it, this is OK.

The client code waits here until the user will terminate the running application.

Not exactly. The main thread will be unblocked to execute remote API calls through our exported objects (like ClientExports). And with validation expressions and UI triggers, called client-side code can recursively call back to the server, which can then call back down to the client and so forth. In other words, there can be a complex stack of operations where the control flow of execution for the program will shift back and forth between the client and server, possibly on a deeply nested basis. Ultimately, these nested calls will eventually return and pop off the stack until the program exits, returning here for the main thread to exit (or to restart depending on the "stop disposition").

I think that we have to design a new ScreenDriver having an embedded server used to communicate with the JS code which run inside the HTML page on the web browser via web sockets.

Yes.

a - HTML specific tags?

No.

b - JS based?

Yes. But this is not part of this task. The short answer is that we will use the Canvas interfaces to directly draw our own "Window" implementation and everything that is inside of it. There will not be any actual browser components or plugins used.

c - java SWING applet?

No.

d - others?

No.

Feedback 1111a

I like it. I look forward to seeing the client side fill out.

Code Formatting Notes

The following are some general (but minor) problems with the code formatting.

1. Where possible, reduce lines.

This:

   public AjaxMainPageHandler(String targetRoot,
                          String skeletonPage)

can be:

   public AjaxMainPageHandler(String targetRoot, String skeletonPage)

This is preferred and it is OK because it doesn't cross the 98 character length.

2. Don't use more than 1 consecutive blank line (sometimes you are putting 2 lines in between methods).

3. There should always be 1 blank line in between methods (sometimes you don't have any blank lines at all).

4. Don't put a blank line between the end of the last method and the end of the class.

5. Make sure that there is proper spacing when using operators and/or control keywords. For example, if( should be if (.

6. For newly created files, the copyright date should be simply 2013 instead of a range (probably this is just a copy/paste error).

Code Review 1112a

As we are just starting on the path to the web driver(s), we will check something in soon that is just a step along the way. Your work so far is a good start. I will answer some of your questions in a separate note (I have already been doing quite a bit of thinking on these questions, I just haven't documented all of my thinking yet).

Before we move forward with the first version, please consider the following ideas:

1. We will have at least 2 web drivers (ChUI and GUI). I would like to maximize the common code where possible. It seems to me that the embedded web server, web sockets/protocol support and some of the javascript implementation can be common. For example, on the javascript side I would expect the key processing to be common code between both ChUI and GUI. To make common code we might need to put the common portions in a different package and the design might need to be a bit different.

2. We don't need to implement a real driver yet, but stubbing it out is a good idea. However, it is important to know that we don't want to use the websocket in such a granular way that every call to the ScreenDriver or OutputPrimitives interfaces will trigger a "trip" down to the browser. For example, we will want to build up the changes to the virtual screen and only send them to the browser when sync() is called. Please consider how we might rework the driver to minimize the trips to the browser.

3. Although I keep using the name ajax to describe all of this, I don't think it is the best description. I think WebClient is a better description for the common portions and with ChuiWebClient and GuiWebClient being more specific names.

Code Review 1112b

I like the changes.

1. The web socket communications are asynchronous messaging operations.
Here we have to find a solution in case we need a synchronous operation which should wait until an event occur on the browser.

Yes, we will have to solve this when we get to working on the protocol. We don't have to solve this for #2200.

2. The communication between WebSockets on the server and WebSockets on the client (browser) is based on text or binary messages.
We have to define a format for our messages. Because the end client is a JS code, I suggest to use text messages in JSON format.
In this case we have to add support for JSON serialization / deserialization to our implementation for example a Jackson processor which is widely used.

I will post more details in regard to the protocol. But I am pretty convinced that the protocol will have to be binary.

The P2J architecture introduces a logical separation between the business logic and the UI, which does not exist in the original 4GL environment (where everything runs in the same process). We found that we did have to optimize our code in order to solve some performance issues that were due to this architectural decision. In particular, we found that optimizing the bytes transferred (between the P2J server and the P2J clients) was very important for reasonable performance of the UI. Our protocol was already a binary protocol. But our first version used the Java Serializable support. We were able to make major performance improvements by implementing Externalizable instead (and implementing our own read/write functions in all of our objects to be transferred).

The web client introduces yet another separation. It will be well worth our time to make the protocol as efficient as possible. Using a binary approach will very likely be required. I plan for us to use that from the beginning. I do understand that it has costs on the javascript side, but I think this is a price we must pay.

Task #2201 is meant to take the next steps beyond #2200. In #2200 the embedded web server is enabled in the client. In #2201, that client is started based on a request to the server jetty. The client's SSL context will need to be properly setup from the server and the host/port for the client's embedded web server must be passed back to the server so that the web request can be redirected.

Code Review 1113a

This is moving in exactly the direction I was looking for. Very good.

1. I see that there is quite a bit of copied code in ChuiWebSocketSimulator from the Swing ChuiSimulator. It seems to me that it would be better to have an abstract base class that both can derive from, in order to maximize common code. Actually, let's use the following naming:

ui/client/chui/driver/ChuiSimulator - abstract base class
ui/client/chui/driver/web/ChuiWebSimulator
ui/client/chui/driver/swing/ChuiSwingSimulator

I realize that ChuiWebSocketSimulator would no longer be able to extend the WebSocketAdapter. But I think the web sockets part is very small and can be handled external to the simulator functionality.

2. It seems like SwingChuiPrimitives can be moved/renamed ui/client/chui/driver/ChuiPrimitives and it can be made as a common class used by both the swing and web implementations. I don't think there is anything unique about ChuiWebPrimitives, right?

3. I would like to see if ChuiWebSocket can be made generic for both the ChUI and GUI implementations. At a minimum, most functionality can be in an abstract base class, with more specialized subclasses to handle any unique requirements. This would mean that it would live in ui/driver/.

4. I think the same idea as in number 1 above can be done for ChuiWebDriver/SwingChuiDriver. The abstract base class can be ui/client/chui/driver/ChuiScreenDriver and it would implement ScreenDriver.

I found no GUI implementation on the current p2j. The screen driver for GUI should be designed from scratch.

Correct. But that is work for a future task. Right now we will implement: code that can be common for any web driver + chui web driver, while refactoring the swing chui driver to share maximum code between swing chui and web chui.

We should consider using TerminalOptions (and other classes) as common code which may be swing-specific today. It seems to me that we some things like Color instances can be stored in the P2J client as Swing-specific values. But then we would translate this into a value (like #FFFFBF) that can be sent down to the web client, when needed.

Are you intended to use the same pattern design as a CHUI driver, or we have to design something else?

Yes, but the GUI driver interface will by nature be different. I hope that we can have a super-interface that is shared between both chui and gui. For example, the key processing is something that can probably be the same between both driver interfaces.

I also am worried about the optimal place to introduce the driver interface in the GUI environment. There are multiple choices:

1. Provide drawing primitives (lines, boxes...) and allow the widgets to call these directly. The big advantage is that it would be easy to implement these primitives on both Swing and in Javascript canvas and there would be massive common code for the hard parts of the drawing. In particular, the widget-specific drawing logic would be common code. The big downside is that I expect this to perform very poorly when we add the extra split to remotely drive the browser drawing.
2. Provide widget-specific drawing methods in the abstraction layer and then implement widget-specific drawing logic separately in swing and javascript.

Both ideas might be amenable to the optimization of sending a list of the drawing operations down to the javascript client. For example:

- draw a button with this rectangle, colors and text
- draw text in this font at this location
- draw a box of this rectangle and color
...

Then the entire list can be sent at one time and there won't be massive "back and forth" from the P2J client to/from the browser. Many small trips is the thing that really kills performance in this case. One round trip with a larger payload will significantly outperform the many small trips case.

This example is showing the list as a set of widget-level drawing primitives, but we could send a lower-level set of primitive operations instead (draw a line of this color from point a to point b, draw a rectangle with this line color and this fill...).

My intention is to use a ChuiWebClient object as a web client luncher similar to the design in ClientDriver.
Is this a correct design, to implement something like ClientDriver but without a main entry?

Actually, for now let's just extend our current approach. ClientDriver/ThinClient will use different drivers based on bootstrap configuration. Don't add a dedicated client driver.

We can do the redirection in two ways. Which method do you prefer?

Are there any downsides to using HttpServletResponse.sendRedirect()?

1. The server web socket is created only when the remote user will open the communication with him.
On the other hand the simulator MUST be created and is created when the client is lunched.
That's why the simulator does not extends the web socket.
However it is possible to change the web socket creator to return an existing instance
and in this case I think that is possible to extend the web socket in simulator.
I will think to that issue to find the better solution.
When the web socket is open the simulator is notified about this and the simulator send the
screen status down to the web browser in order to be rendered on the web page via JS.
I will try to make the current implementation more abstract as you already suggested.

2. I've started also to work on redirection issue. So far I created a Handler
which create an instance of the ChuiWebDriver, start the embedded server and
redirect the request to the embedded server URI. On error will return a short page
with a message informing the user about this. I did some tests and the redirection
works fine using the response sendRedirect method. I added to the server directory.xml
a new entry chuiWeb like chuiApplet. I found a minor bug on current implementation:
The chuiApplet is using /chui as context root and I want to use /chuiweb
Because the check on chuiApplet is target.startsWith(/chui) is not possible to
handle both /chui and /chuiweb. Finally I used /webchui instead /chuiweb

3. Something remain unclear on my mind at this point. If we aren't use a mechanism
like ClientDriver to start a client how we will connect our screen driver with the
converted 4gl application?

in this case I think that is possible to extend the web socket in simulator

I don't have a problem using the WebSocketAdapter in our solution (or WebSocket directly). I just prefer to separate the use of it from the inheritance hierarchy of the simulator.

The chuiApplet is using /chui as context root and I want to use /chuiweb
Because the check on chuiApplet is target.startsWith(/chui) is not possible to
handle both /chui and /chuiweb. Finally I used /webchui instead /chuiweb

Let's just use /chui and eliminate the ChuiApplet. In practice, there is no good reason to use ChuiApplet when you can use our web client instead. Eliminating the applet is a good thing.

If we aren't use a mechanism like ClientDriver to start a client how we will connect our screen driver with the

converted 4gl application?

The proper driver configuration would be passed in a bootstrap config instance when the client is launched. You will have to modify OutputManager.newInstance(), which is called from OutputManager.init() which is called from ThinClient.initializePrep().

The WebSocketAdapter was used only for notification mechanism because WebSocketAdapter implements WebSocketListener.
A better solution is to design a new interface just for notifications and eliminate the inheritance of WebSocketAdapter.
The class ChuiSimulator from swing based driver extends JComponent and this is a problem because we cannot extends another class like an abstract or base class.

I know about the OutputManager changes that we have to made in order to add our screen driver.
In this case I think that we have to use the ClientCore.start(...) method like ClientDriver finally do.

In this case I think that we have to use the ClientCore.start(...) method like ClientDriver finally do.

I was assuming we would use the same ProcessBuilder mechanism that we use for appserver agents today. It just builds the Java command line up dynamically and then runs the program. There is no reason we can't use ClientDriver from there. Is there some other reason ClientDriver won't work?

Regarding my tests with ClientDriver and "chui_web" screen driver I used the configuration
file client.xml and the key stores from P2J_HOME/testcases/simple/client_strong.
I saw inside the BootstrapConfig informations related to key stores as well as access passwords.

Also I did tests without this configuration file but in this case the BootstrapConfig does not
contains informations regrading SSL.

Code Review 1114a

1. The ChuiWebSocketCreator has nothing that is specific to ChUI. Let's make it generic and put it in the ui/driver/ package so that the functionality can be shared between ChUI and GUI.

2. There are minor code formatting issues in ChuiWebHandler.

3. As mentioned in the previous notes, we don't need the ChuiApplet any more. Please remove that support (including from StandardServer).

4. num should not be set to 1 at the beginning of the initialize() method inside StandardServer.registerDefaultServices().

5. The "Module" name at the top of ChuiScreenDriver (and also ChuiPrimitives) is incorrect.

6. Please duplicate the javadoc for interface methods that are implemented instead of just referencing it using at-see (e.g. in ChuiScreenDriver).

7. I prefer the ChuiScreenDriver.beep() implementation to be moved into SwingChuiDriver. There is no case where that code can be shared and leaving in the base case just raises questions on the part of the reader.

8. ChuiWebHandler seems pretty generic and not specific to ChUI. Let's just rename it WebHandler.

9. About the SSL problem, it is expected.

The server is started and listen for connections but the web browser cannot download the main page.
The browser said the the server does not accept encrypted connection and I think that the embedded server has no SSL certificate.
If you have any hint regarding this issue please let me know.

We need to find a way to send the P2J server's SSL certs/context down to the P2J client so that it can be reused. As for where the server gets it, please see SecurityManager.getSecureSocketContext() and the TransportSecurity class in that same package. Some changes will be needed, but the core idea is that we DO NOT want to have to configure keystores/truststores on each client. Instead, we want to leverage the server's SSL context, but use it on the client. Changes will be needed in SecurityManager and TransportSecurity to make this happen.

Code Review 1114b

It looks good. Please get this runtime regression tested.

I assume that the direct reference to ChuiWebDriver from WebHandler will be eliminated when the proper client spawning is working. Is that correct?

Yes, the reference to ChuiWebDriver from WebHandler will be eliminated and replaced with code to spawn the client.

Marius Gligor wrote:

1. Here are the results of the regression tests. Seems to be expected errors.

Do not update test results here. If you are unsure if you have a real or not error and want us to look over it, send the files via email.

- I tested also the new WebDriver by adding the following node into gso-directory.xml file:

Are these changes required for MAJIC's Admin Console?

The first tests FAILS due to the missing of index.html resource file into p2j.jar

All look like false negatives.

Should I change the build.xml file now or we have to do that when we'll finish the implementation?

Do change it now.

The changes does not affect the the adminConsole applet.

The chuiApplet is no longer used, should I delete the files
related to this applet from project?

The chuiApplet is no longer used, should I delete the files related to this applet from project?

Yes.

Please fix the build.xml and re-do your manual testing. If that works, then you can check in the update and distribute it.

The following files should be deleted:

ChuiAppletHandler.java ChuiApplet.java and chui_applet.html

Also I have to do some changes on OutputManger.java and delete the code related to CuiApplet.java
and in the file build.xml to eliminate the copy of chui_applet.html to build env.

The file OutputManger.java already has some changes that should be commited.

Should I redo the regression tests or is enough to test only locally?

Code Review 1118a

There is some good work here, but this still needs some work. If anything, this is a good start that exposes some core design issues that need to be resolved.

1. The singleton design will not scale properly. The approach you have used will only allow 1 client to be started at a time and it will block until the client has initialized.

2. The current design will only work when run with the server's user.dir as the client's current directory. That cannot be a dependency because most apps have very specific client-side requirements for current directory and they would never let each user have a current directory as the same directory in which the server gets all its configuration.

Please review the client.sh (used in Majic) very carefully. These same features need to be exposed when spawning clients. This will be tricky because there is a "chicken and egg" problem. In our standard client model, the user has already logged into the system and thus provides the context (permissions, home directory and so forth). For this to work in our approach, we may need some scripting help at the OS level. We also may need to know what the OS-level userid should be for each new session. How do we get the OS user context established for each new session?

This opens some major security questions. We definitely cannot run in the same permissions/context as the server, but when we start like this that is exactly what happens.

Right now the implementation assumes the clients will always run on the same system as the server. Many of the above requirements will probably require a remote spawner (one that is not in the server's process).

3. Please plan for the server's SSL context to be sent down to the spawner using some form of serialization. This eliminates the need for direct access to the server's current directory and configuration.

4. The connection setup is hard coded to insecure sockets (from the spawned P2J client to the P2J server) which is not something we want to do. It needs to be configurable, while defaulting to be secure.

5. It is also important to note that any environment that is configured to actually force a login (like MAJIC) will fail in the current ClientCore design. The reason is simple: one must have the URL sent back to the server BEFORE the login can occur because in the "normal" case, the user must be prompted for login credentials. But since you are calling a remote method on the server in order to pass the URL back, you are assuming that the P2J session is already setup. That session is created in the connectDirect() call. Your tests worked because you used the simple server environment that has a neutered security environment.

6. The "client.xml" is also something that should be configurable. The fact that we commonly use that filename is purely a convention, but it should not be a requirement.

7. The user.dir and other JVM parms (heap, parm gen...) are things that will have to be configurable from our directory instead of hard coding them.

8. Code formatting and coding standard problems:

• missing header entries
• missing public keyword on interface methods
• OnRemoteServerListen should be onRemoteServerListen
• wrong sorting of methods by access mode and static/instance

1. I used a singleton design because I cannot export the Spawner interface on multiple instances.
Is any method to export the same interface per instance?

2. What I understood from your description is that the new spawned process MUST run on the OS user environment.
It's not clear for me why and how to do that because the user is virtually a HTTP request that is coming from
the local or from a remote machine on the network. Our WebClient just offer services via web sockets.
The users have no other controls over the running clients.

Should we spawn the process on a remote machine?
   How to do that because the p2j.jar and dependencies in this case must to be on the remote machine?

The web client is running on the same machine where the server is running.
   However the single parameter in CoreClient is false. 

• @param single
• true to startup within the server process which
• must bypass the shared infrastructure initialization. Use
• false for the normal client JVM startup.

Could you please explain me what is the difference between single=true and single=false?

We are using a Java ProcessBuilder on our Spawner and we always create a new process which
    is a separate process from OS perspective. Only the process OS context is the same as the server context.
    Basically instead to use server threads which runs within the server process we use a separate process for each request.
    Why this model is not good for our web client?

3. I have to think about this, because is related to 2.

4,6,7. I will change the design to provide configuration for arguments and not hard codding.

5. Here I have also a lot of questions.
- It is possible to use your RemoteObject design to access the remote server before client authentication?

This snippet of code is from ClientCore class.

// establish or re-establish a session with the P2J server
   // (this handles authentication too)
   session = sessMgr.connectDirect(new InterruptedExceptionHandler(), null);

- connectDirect handles authentication too?.

- should our web client provide an authentication web page too?

Could you please offer me more details because the design of this part is also unclear for me.

Another important design issue.
When a request is made at https://<server_host>:<admin_port>/chui the request is handled by the WebServer instance.
The WebServer calls Handler#handle on each registered handlers and wait until return from handle method.
The base.isHandled() flag is checked after each call. If this flag is true that means that the request
was handled and a response was sent back to the requester and the web server stops this process here.
If all handlers on the chains are called and no one return base.isHandled() as a true value an default error page
is returned as a response by the jetty web server.

Because each handler is a unique instance the access to handle method is serialized.
Even more on our case we have to take a decision before to send a response back to the user.
This means that we always have to wait for a client response in order to take the decision before sending the response back to the user.
On other words waiting on Spawner or waiting on WebHandler has the same effect, serialization access.
Due to this constraints I don't think that is possible to create web clients simultaneously.

1. I used a singleton design because I cannot export the Spawner interface on multiple instances.
Is any method to export the same interface per instance?

You can export an interface using the instance model (RemoteObject.registerNetworkServer()) which registers an instance of an object that implements the interface(s).

You can also export an interface using the "static" model (RemoteObject.registerStaticNetworkServer()) which registers a class that has static methods whose signatures that match the interface(s) exactly (of course we can't use "implements" with static methods, but this is essentially the same thing).

Either way, there is only 1 registration/export of an interface per JVM. Please read through the RemoteObject code to get the idea.

My concern is not so much the use of a singleton itself, but really it is how you are synchronizing on the class and assuming only one new client is launched at a time. Regardless of the exporting method, you will need to change how the spawning and waiting/notification works to eliminate the "one at a time" approach.

What I understood from your description is that the new spawned process MUST run on the OS user environment.

Correct.

It's not clear for me why and how to do that because the user is virtually a HTTP request that is coming from the local or from a remote machine on the network. Our WebClient just offer services via web sockets. The users have no other controls over the running clients.

Yes, I had a hard time sleeping last night because I was thinking about this exact issue. :) But I did come up with an approach.

Consider that in our P2J client, we have native dependencies. Please look in src/native/ to see the level of functionality that is there. This support is needed because of features that the 4GL provides which cannot be exactly duplicated using J2SE features. On Linux/UNIX, our P2J build creates a libp2j.so and on Windows we create a p2j.dll. We access this DLL ONLY on the client (the P2J server does not use it). The access is through JNI.

On Linux/UNIX, the system is inherently multi-user. There are features built in to allow multiple users to login simultaneously. More importantly there are features to allow processes to change their user context. There is the su command and there are APIs such as setuid.

On Windows Server, I suspect that similar facilities exist as provided by the terminal services.

When needed we can and will add native code helpers to this JNI library to allow us to integrate/call such facilities.

I think we should be able to do something like the following:

1. The server jetty + initial web page returned will not redirect immediately. Instead, in the common case it will have to create a dialog with the user to obtain the login credentials in a secure manner and submit them to the P2J server.
2. The P2J server will spawn a new client process which must essentially login/change user context using the given login credentials. We will most likely have to have an alternate approach for guest access (and possibly other scenarios), but we can deal with that later. The result is a process that is running in the proper user's context.
3. Inside that process that has the correct context, the P2J client must be started. It is possible that we might do this the other way around (spawn a new P2J client in the server's context and then shift context via API calls) BUT there are some sensitive security issues that must be handled in such a case. The new context must not have access to sensitive data/environment/configuration AND it must not be possible to shift the effective user context back to the original server's context.
4. From there the client jetty must be initialized and we must send the URL back to the server jetty which will send the redirect back down to the client and the rest works as originally discussed.

On Linux/UNIX there are some potential issues with ensuring that there is a "controlling terminal" or pty (pseudo-terminal) associated with every client process that is spawned. The system will not allow interactive programs (or programs that do more with a UI than just read/write to STDIO) to be launched from parent processes that are not setup up with the proper terminal environment. 4GL apps frequently do start child processes that have such requirements. We must make sure the child processes can handle such requirements.

We also will need to find a way to redirect the I/O to/from such programs and the remote javascript "terminal". I haven't fully gotten my head around how we do that, some research will be needed there too.

We also have some special requirements for how we trap exceptions in the client.sh script (on Linux/UNIX), which we will need to duplicate.

On Windows, I'm not really sure what issues we will encounter.

How to do that because the p2j.jar and dependencies in this case must to be on the remote machine?

Yes, this is the assumption.

In such a case, I think we must have a special "client" process on that remote machine that is connected to the server. This process would be able to do the spawning.

The web client is running on the same machine where the server is running.

Sometimes this will be good. But we will have some customers where this is not acceptable.

However the single parameter in CoreClient is false.

Yes, of course. This is correct.

Could you please explain me what is the difference between single=true and single=false?

single=true means that the server process and client process are the same. This is a special mode where there can only be 1 client, ever. This is not really used in production.

Using single=true is not something we are going to do with the web client.

We are using a Java ProcessBuilder on our Spawner and we always create a new process which is a separate process from OS perspective.

Yes, this is good.

Only the process OS context is the same as the server context.

This is what we must change.

Basically instead to use server threads which runs within the server process we use a separate process for each request.

Not really. We already use a separate process today for the P2J client. This is just a different way to launch it and interface with it.

The user still will have a session with the P2J server (the P2J client will have this session actually) and that session will still have threads running on the server on that user's behalf for all business logic and database access. Only the UI and the client platform dependencies (e.g. accessing files, launching child processes/shell commands, using sockets or direct memory access...) will be done in the P2J client (just as it is done today).

It is possible to use your RemoteObject design to access the remote server before client authentication?

No. One must have a value session before any RemoteObject functionality can be used.

connectDirect handles authentication too?.

Yes.

should our web client provide an authentication web page too?

Yes and no.

As noted above, we will need to do the OS-level authentication in the web interface from the server jetty.

But the P2J authentication is different and would still be handled as it is done today (inside connectDirect()). Please use a debugger to step through the connectDirect() process (that is the P2J client side, you will also have to step through the P2J server side which can be seen in RouterSessionManager.Incoming.run()). It is important that you understand how a P2J session gets setup. The authentication logic in the SecurityManager is quite complex because there are many different ways (batch vs interactive, certificate based auth, userid/password, plugins...) through. You should look at the simple server environment first, where you will see a kind of "null" authentication where we use a plugin to allow anonymous/guest access. But then you should also step through this process for a MAJIC client. That uses a real authentication process against user accounts in the server's directory and each successfully setup session has a specific P2J account context.

The interesting thing here is that we actually use a limited form of the 4GL UI processing to handle those cases (like MAJIC) that use interactive P2J authentication. During the connectDirect() we actually serialize an application-specific plugin class that implements the login UI and send it from the server to the client. It is executed there in our "early" P2J client, interacting with the user to get the userid/password details and then sending the results back to the server. SO: as long as our javascript client can handle our normal 4GL requirements, this support will "just work" without any changes. To see an example of this plugin code, see testing/majic/src/aero/timco/majic/security/LoginClient.java on devsrv01.

Because each handler is a unique instance the access to handle method is serialized.

Do you mean synchronized?

Even more on our case we have to take a decision before to send a response back to the user.
This means that we always have to wait for a client response in order to take the decision before sending the response back to the user.
On other words waiting on Spawner or waiting on WebHandler has the same effect, serialization access.
Due to this constraints I don't think that is possible to create web clients simultaneously.

I suspect we can solve this using:

• thread pools so that there can be multiple incoming sessions at once
• create a separate thread and reply immediately, unblocking the server, then use websockets between a javascript client and that new thread to handle the rest of the process
• something else?

I started to do tests using Linux su and sudo commands.

1. For security reasons su and sudo doesn't allow stdin redirection.
I received the following error messages trying to use this commands with Java ProcessBuilder

su: must be run from a terminal

sudo: no tty present and no askpass program specified

2. As a root I'm able to use both commands. However in this case no ask for password is made!

Another drawback for su command:

Using: su - mag -c java                     // works fine

Using: su - mag -c java -version        // does not work.

we have to use something like this:

su - mag -c ~/script.sh

that is, we have to put the java command and arguments into a bash script and execute this script.

The standard Java process launching does not properly setup the terminal environment. As mentioned before, the terminal environment is very important for the client processes to properly work. As you have found su and sudo won't work without it.

Please note that we do have our own process launching support that does setup the pty properly (see src/native/process.c). But this is only usable from the P2J client.

From a design perspective, we will want the solution to this problem to minimize deployment issues and shell/OS interface usage. This means we want to avoid scripts, avoid special permissions setup, avoid external configuration files, avoid dependence on utility programs and avoid shell processing. If these things can be avoided by writing code to native APIs in our JNI library, then that will be the preferred approach. Of course, at this point we need to find a solution that works at all. But I do know that this is possible (e.g. other programs like sshd are able to do this. It is probably a good idea to investigate OpenSSH Server to see the APIs and techniques they use.

The server KeyStore is exported via a RemoteObject interface not from the command line parameters.

I haven't gotten a chance to look at the posted code. But I think we can't use RemoteObject for transferring the keystore because in the normal case (non-guest access like in MAJIC) there won't be a valid session with the P2J server until the user has done an application login.

This is good work. Some thoughts:

• We will want to limit the scope of the process that is root. In particular, the main P2J server should NOT be run with root because any security flaw or breach will expose the entire system to problems. Instead, we will want to implement a very small "spawning" process that is run with root.
• What is the controlling terminal status of the child process in this case?
• The use of setuid() should be safe in that since the original context is root, there will be no "saved" uid context that can be swapped back in later.
• I am curious to what extent this same behavior can be implemented on Windows. In particular, the process launching approach of fork() followed by exec() is not the design on Windows, so I expect some problems there.

This is really good work. Some thoughts:

1. Each spawned process must have a controlling terminal. You can use the who command to list the current terminals in use. Please look into this topic deeply (see these links) and make sure that the controlling terminal setup is correct for the resulting spawned process.

http://blog.nelhage.com/2009/12/a-brief-introduction-to-termios/
http://blog.nelhage.com/2009/12/a-brief-introduction-to-termios-termios3-and-stty/
http://blog.nelhage.com/2010/01/a-brief-introduction-to-termios-signaling-and-job-control/
http://uw714doc.sco.com/en/SDK_sysprog/_The_Controlling-Terminal_and_Pr.html
http://blog.nelhage.com/2011/02/changing-ctty/

Please also look at initConsole() in src/native/terminal_linux.c to see a bit about how we initialize the terminal settings at startup of the P2J ChUI client.

There is also this text from our P2J Runtime Installation, Configuration and Administration Guide:

Job Control and P2J Client

In environments where interactive shells provide job control, some special care should be taken when providing custom shell scripts that launch P2J clients. An example of such an environment is the bash shell on Linux. The job control feature makes it possible to stop the client process at an arbitrary (userspecified) time by typing CTRL-Z on the terminal. The operating system suspends the P2J client process and the bash shell regains control, which would differ from a typical Progress environment. Below is the explanation of when and why this may happen and how to avoid it.

When CTRL-Z is typed in the terminal, bash, being the parent of the client JVM process, receives a SIGTSTP signal from the tty driver in the kernel, if the controlling terminal is not in “raw” mode. The kernel sends this signal to all processes in the foreground process group in the bash session. bash itself is in the group.

Under normal conditions, the P2J client will have set the controlling terminal into raw mode. When in raw mode, the SIGTSTP signal is not sent and there is no problem. However, every time the JVM launches an interactive child process, the terminal is restored to the regular “cooked” mode. If the user presses CTRL-Z while interacting with this child process, the kernel will generate the SIGTSTP signal.
In response, the JVM safely ignores the signal. Unfortunately, bash still receives the signal and then performs a "service" for us. It reacts to the signal by sending the SIGSTOP signal to the current processes in the foreground job. This latter signal cannot be caught, blocked or ignored. Thanks to bash, the JVM process stops unconditionally and bash displays the command prompt.

Disengaging the JVM from the parent bash won't help, since bash would display the command prompt immediately and the user would be able to interact with the same terminal in which the JVM is operating. This would create a situation where there is concurrent use of the same terminal, quickly resulting in screen corruption or hangs.

The solution is not to allow bash receive the SIGTSTP. One of the two known ways is the TRAP instruction. Another, more efficient way is to use exec to invoke JVM process instead of eval. This causes the parent bash to go away leaving no parent of the JVM process which can process the job control signals.

This issue only affects the CHARVA based CHUI driver, since the swingbased driver does not run inside a terminal or shell environment.

2. The child process MUST NOT have ANY access to state or resources that are available to the parent process. The STDIO must be setup with its own controlling terminal. The current directory should not be the server's current directory. The environment should be fresh and should not have anything from the P2J server's environment. The fact that the p2j.jar was read from your home dir suggests that there is still some work in this area.

3. Please organize the design as follows:

• Minimize what is in spawn.c to only those portions that are absolutely required.
• Put everything else in the P2J code (java or the native code, as needed).
• The P2J native code (accessed via JNI) can launch spawn when all other setup is done.
• Perhaps you can write the password to STDOUT and spawn can read it from STDIN. This should be just as secure as other terminal login code in Linux.

New changes in the current design.

1. In p2j when the user open the following URL https://<host>:<admin_port>/chui a page having a login form is send as response for this request.
The user must enter an user name and a password and press the "Log in" button.
On the server if the user name and the password are valid values the request is redirected to the chui web client page, if not the login page
is returned together with an error message.

2. In the spawner.c the password is read from stdin not from the command line.
Inside the p2j the password is provided via stdin pipe. I've tested and I found that works fine.

On the spawner.c they are still a lot of works to do:

- I saw that the spawned process has no pseudo terminal assigned!

UID        PID  PPID  C STIME TTY       TIME CMD
      bubu     14718 14716 99 18:59 ?     00:11:37 java -Xmx512m -XX:MaxPermSize=6 ...

- The child process should be detached from his parent process.
     In this case the p2j design should be slightly changed to provide paths inside the user account home directory for class path and native libraries path.
     Each user should have the p2j client on his space. Where should be located the p2j client on each user?

I'm going to investigate this issues by reading the links posted by you and other articles related to this topics.

3. When I spawned Java process fro another account first I got the following message:

Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.

Do you ever seen such a message? Is this cause by the pty missing?
  I've fixed this issue by providing the following JVM argument -Djava.awt.headless=true

In this case the p2j design should be slightly changed to provide paths inside the user account home directory for class path and native libraries path.

It is unlikely that each user will have their own copy of P2J. No one would really want that because it would be a nightmare to maintain. And we can't have that version be different code than the code in the server's P2J jar since that can cause problems with Java serialization and our RemoteObject protocol.

The common case will be that the customer will have a shared installation somewhere on the system. I think it is acceptable to have this be part of the configuration in the server's directory. It should be possible to have this point to any valid paths on the system, but it can be shared for all spawned clients. Make sure to provide configuration options for both the classpath and the native library path.

Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.

If I remember properly, this is caused when running a new session using the root account. The X server will fail to initialize by default because running X in a root account opens too many security holes. But our client should never be run until after the root account is dropped.

I don't know if we will be able to run headless or not. One problem will be if our use of any of the swing/AWT classes is allowed. Of course, we can eliminate that dependency by using some new classes to encode colors and rework our hierarchy. It may cause other problems too. The client we run must be as functional as any ChUI client that is run interactively.

This is really good work. It seems very close to what we need. Some feedback:

1. There are a few coding convention issues and mis-spellings (e.g. chek_root() should probably be check_root()).

2. Setting the current dir to the home directory is the correct thing to do as the default. However, it will be very important to allow this to be overridden via configuration in the server's directory.

3. Please integrate the build of this into the primary p2j makefile. If you want to maintain a separate makefile, that is OK, but perhaps it needs to be renamed to makefile.spawn to make it clear it is dedicated for spawn. Both spawn.c and the makefile.spawn should be placed in src/native/ and build when the rest of the native p2j code is built.

4. Since we are adding some tricky permissions processing into our P2J installation/configuration dependencies, I think it is very important and every possible error condition in spawn.c be given its own unique error code. For example, we should be able to tell if setgid() failed or if setegid() failed or if... I know this will cause the code to be expanded, but the future benefits to our admins will be well worth it. Create a set of constant definitions like this:

...
#define  ERR_SPAWN_SETGID_FAILURE   -10
#define  ERR_SPAWN_SETEGID_FAILURE   -11
...

The errors should start at -1 and all be negative numbers.

5. About your question:

On a secure mode both net:server:port and net:server:secure_port should be provided?

You can see how we deal with this in LeafSessionManager.connectDirect(). Basically, if net:connection:secure is true, then we expect there to be a net:server:secure_port specified and we do NOT read or otherwise use the net:server:port.

I will do the code review next. First, I wanted to get some basic answers to you.

When the user open https://<host>:<admin_port>/chui from a web browser a page having an authentication form is send as response. The target root /chui is also configurable. Is this OK or is best to remain hard codded?

It is fine to be hard coded. This will simplify the configuration and documentation. There is no good reason to change the URL so hard coding is OK.

To use only a Linux server to run the p2j server could be a solution?

This is not possible.

Do you have clients that require Windows OS as server host?

Yes, Windows OS support is required.

Please fix these two javadoc errors as part of your update:

  [javadoc] /home/ges/projects/p2j/src/com/goldencode/p2j/ui/client/chui/driver/swing/ChuiSimulator.java:79: warning - Tag @link: reference not found: ChuiApplet
  [javadoc] /home/ges/projects/p2j/src/com/goldencode/p2j/ui/client/chui/driver/swing/ChuiSimulator.java:579: warning - @param argument "opt" is not a parameter name.

About the Windows approach and winspawn.zip:

1. Please do not use the MS VisualStudio or Visual C++ tools. It would bring too many proprietary dependencies to our project. For Windows development, we use mingw. This is an open source, Windows port of gcc and the other common build tools that we use on UNIX/Linux (the linker, make...) that is packaged in a manner that allows full WIN32 development. This is not using the cygwin compatibility environment at runtime, instead we would link to the msvcrt to get the basic C runtime (this DLL is part of the OS so it is OK).

To install:

1. Take one of the full archive: mingw-w32-bin-i686-2013MMDD.7z or mingw-w64-bin-x86_64-2013MMDD.7z. Unpack to \mingw32 or \mingw64.
2. Include the appropriate \mingwXX\bin to the PATH variable. In my systems: C:\mingw64\bin and C:\mingw32\bin depending on Windows architecture.
3. The 64-bit binaries are found at http://www.drangon.org/mingw. The 32-bit binaries and much more info can be found at http://mingw.org.

Please look at the makefile I posted to see the windows specifics. You can make some minor changes and have the same makefile.spawn used for both.

2. Generally, try to use common code between spawn for Linux and spawn for Windows. Where needed, use OS APIs. Where possible use C runtime features that are common. See our src/native/ directory for how we implement platform specific helper files that are called from common/shared source files.

Otherwise, I think you are on the right track.

1. In think that Windows and non-Windows (which are effectively all UNIX/Linux) detection is enough. Os.java has been deleted.

2. TODO. OK I have to do that after the implementation of both spawn projects for Linux and Windows is done.

3. Indeed we can configure sudoers to allow particular command to be executed without any password prompting.
In this case the sudoers file should be edited "by hand" on each workstation where the build is running.
We cannot do that automatically. We can only show how to do that and each member of the team should follow our instructions.
Is this what you are looking for?

4. The Linux/UNIX spawn approach is not integrated with the PAM ("Pluggable Authentication Modules") infrastructure, but will be soon.
Currently I'm working to add PAM authentication support to spawn.c
Using PAM is more portable over Linux/Unix platforms.

5. Done

6. Done. Among error messages an error code is already returned from the main method (see exit(result)).

7. Done.

8. I'm afraid isn't possible to do that on pure Java. Also the PAM implementation require the password unencrypted.
Even more as you can see the crypt API require the clear password and the encrypted password.
crypt(password, correct); The encrypted password contains a "salt" used on encryption according to Linux documentations.
I found also that the algorithm is not the same on all Linux platforms.

9. The working directory could be absolute or relative. With absolute paths the users shares the same working directory.
With relative paths each user has it own working directory inside its home directory.
Another issue here. I tried to access directory services when I prepared to spawn a client. It doesn't work!
According to my debugs something like directory is not bound cause the error.

Location of this error is on DirectoryService.java line 4770, the return value is null, see bellow.

protected Remapper synchBatch(String id, boolean rw)
{
          if (!isBound())
        return null;

The only way to read the directory values was at the moment of StandardServer start-up.
Do you have any idea how to fix that?

10. Done.

11. Done.

12. Done.

13. On SpawnerImpl I'm using a ConcurrentHashMap which doesn't require synchronization.

14. Done with some remarks. When we launch the spawner we have control only over the spawner process.
The spawner will exit after the first fork() and the return code from child processes are no longer under our control.
I can get only the exit code from the spawn process not for it child's.
In conclusion I implemented this feature only for spawner process exit codes (AUTH_ERR_*).

15. I understood. Here we need to use a secure interprocess communication mechanism. Could be JMX, Java RMI, JMS or maybe another solution.
At this moment I have no idea about a best solution. I have to do some "researches" to find a solution.

3. Indeed we can configure sudoers to allow particular command to be executed without any password prompting.
In this case the sudoers file should be edited "by hand" on each workstation where the build is running.
We cannot do that automatically. We can only show how to do that and each member of the team should follow our instructions.
Is this what you are looking for?

Yes, almost. The only additional issue I see is that we don't want to have chown and chmod in the sudoers file as the command. The reason: it is a big security hole to open to solve a small problem like this.

Instead, please create a shell script that handles the exact chown/chmod requirements. Then as part of the instructions, make sure that the ownership and permissions on that shell script are set very tight (maximal limitations).

The only way to read the directory values was at the moment of StandardServer start-up.
Do you have any idea how to fix that?

You have to have a security context defined for any thread that accesses the directory. On such a thread, the following code will provide access:

      DirectoryService ds = DirectoryService.getInstance();

      if (!ds.bind())
         throw new RuntimeException("directory bind failed");

You can see code like that at the top of StandardServer.standardEntry() which is the primary server-side entry point for new client sessions to start running 4GL code.

You may also want to look at the code in StandardServer.bootstrap() which is how the server initializes. That involves creating the directory service and the security manager instances. Notice how we don't read anything from the directory before the security manager is operational.

The WebServer instance is created and initialized on that same bootstrap thread. So reading the directory is OK there. But when the parent class (GenericWebServer) is instantiated, it calls startup() which creates a thread pool for handling requests. These handler threads don't have such a context and cannot access the directory.

For now, can we read the directory during initialization and pass the data into the web server? I prefer not to add security context to the web server handler threads if not necessary, to limit the security risk that may happen if those threads are breached in an attack. Worst case, we will open up some context as needed.

I can get only the exit code from the spawn process not for it child's.
In conclusion I implemented this feature only for spawner process exit codes (AUTH_ERR_*).

We will need to ensure that the child process' return code is written somewhere for logging/diagnosis purposes.

15. I understood. Here we need to use a secure interprocess communication mechanism. Could be JMX, Java RMI, JMS or maybe another solution.
At this moment I have no idea about a best solution. I have to do some "researches" to find a solution.

One way might be to use a pipe. We could open that in the spawner and inherit it in the child. The key store can be read from it and the URI can be written back to it. This has the advantage of being easily accessible from native code and very portable.

A better solution is to implement a temporary P2J session with the server. The idea: create one-time authentication credentials (some kind of random token/key) that can be passed (like our password info) via STDIO to the child process. If we can find a secure way to get it to the P2J client, the P2J client can use this to do a temporary connectDirect() to the server with those credentials, read the key store, initialize the web server and send back the URI. Then it would disconnect and drop into the normal processing. The major advantage of this approach is that it can be extended to the "remote client" scenario, which we will be implementing later. Another major advantage is that we plan to do something similar to allow batch mode P2J clients to be autostarted by the server.

Constantin: do we have anything already made for appserver that can help us here? How do the new appserver "clients" authenticate?

1. Today I succeeded to build the spawner for Windows OS using MinGW compiler.
MinGW does no support main functions for UNICODE applications (_tmain, wmain). This is a well known issue among MinGW Windows developers.
UNICODE is mandatory on spawner because the function CreateProcessWithLogonW is available only for wide characters not for ANSI characters.
Fortunately using a main function wrapper I fixed this issue.

2. I added support for PAM on the Linux version of spawner.
As I told you on Linux version I can get back in Java only the exit code for the master process.
My idea is to use the Linux syslog to audit the execution of child processes.
Please let me know if you agree this solution and I will do the appropriate changes.

3. I started to design a makefile for both Linux and Windows builds.
On Windows we have to use the MinGW compiler and I need references to include and lib folders of MinGW.
I created an environment variable MINGW_HOME instead to set the reference in the makefile.
Also on MinGW the name of the maker is mingw32-make and on Linux gcc is only make.
So far I created 2 ant targets but I have to find if is possible to create properties for maker
depending on platform and finally to have a single ant target for build on Linux and Windows.

My idea is to use the Linux syslog to audit the execution of child processes.
Please let me know if you agree this solution and I will do the appropriate changes.

That is OK.

On Windows we have to use the MinGW compiler and I need references to include and lib folders of MinGW.
I created an environment variable MINGW_HOME instead to set the reference in the makefile.

That is strange. I didn't find the same requirement in src/native/makefile. What is different in the spawn case?

Also on MinGW the name of the maker is mingw32-make and on Linux gcc is only make.

Make sure you are not using the automated installer for mingw. I initially had the same problem but when I "manually" unzipped the archives, everything was named properly (make was make).

So far I created 2 ant targets but I have to find if is possible to create properties for maker
depending on platform and finally to have a single ant target for build on Linux and Windows.

Please call the processing of the spawn makefile from the P2J src/native/makefile instead of from ant.

I guess I should point out that our ant build.xml and src/native/makefile already works for mingw on 32-bit Windows, so you should not have to make too many changes. There must be something different about your environment, from the environment on which we normally run.

Eugenie: do you have any ideas?

Eugenie: do you have any ideas?

I think the best way is to take precompiled binaries(e.g from here: http://code.google.com/p/mingw-w64-dgn/). Then unzip to some location and include path to the BIN subdirectory in Windows PATH variable.

It should work. No other changes required. And "make" will be "make".

The recent binaries are: mingw-w32|64-bin-i686|x86_64-20131018.7z. This works for both 32 and 64 bit Windows(Running inside KVM on Ubuntu).

1. I downloaded and copied the MinGW binaries for both Win 32 and Win 64 bit platforms as follow:

c:\mingw32  (32 bit binaries)
   c:\mingw64  (64 bit binaries)

In PATH I added c:\mingw64\bin Now the maker is make.exe and this is OK.

When the winspawn.c is build following references are needed:

For compile the following include files:
Windows.h
WinBase.h
Userenv.h
Shlwapi.h

For link the following libraries:
libuserenv.a
libshlwapi.a

These references are available from:

c:\mingw32\i686-w64-mingw32\include 
 c:\mingw32\i686-w64-mingw32\lib

and

c:\mingw64\x86_64-w64-mingw32\include 
 c:\mingw64\x86_64-w64-mingw32\lib

To solve this issue I used 2 environment variables:

MINGW32_HOME=c:\mingw32\i686-w64-mingw32
MINGW64_HOME=c:\mingw64\x86_64-w64-mingw32

Inside makefile.spawner I used a place holder MINGW_HOME and depending on detected platform MINGW32_HOME or MINGW64_HOME is assigned to MINGW_HOME.
I did the build on both 32 and 64 platform and it's works.

Do you agree this solution? If not please let me know your proposals.

2. To run the build I created a new target in makefile which is a recursive use of make to build spawn from makefile.spawn
The spawner is build when the native classes are build using ant native target. No ant target for spawn build are available as you suggested.

However I found a little problem here.
If the JRE_HOME variable is located in a directory having spaces in name like JRE_HOME=C:\Program Files\Java\jdk1.7.0_21 the ant native build
fails because the space character between "Program" and "Files" is interpreted as a separator when the INCLUDES is build using foreach (see bellow):

INCLUDEDIRS=${JRE_HOME}/../include ${JRE_HOME}/../include/win32 $(SRCDIR)
 override INCLUDES+=$(foreach d,$(INCLUDEDIRS),-I$(d))

One solution is to use JRE_HOME on a path without spaces like: JRE_HOME=C:\jdk1.7.0_21 and everything works.

libuserenv.a
libshlwapi.a

What are these?

As a general rule, I prefer to dynamically link to DLLs that are part of the base OS rather than using static linking, but the decision can be made on a case by case basis.

Do you agree this solution? If not please let me know your proposals.

Yes, I agree.

If the JRE_HOME variable is located in a directory having spaces in name like JRE_HOME=C:\Program Files\Java\jdk1.7.0_21 the ant native build
fails because the space character between "Program" and "Files" is interpreted as a separator when the INCLUDES is build using foreach (see bellow):

Please rework this makefile code to eliminate this problem. I don't want to force more environment dependencies when we can just use a different technique.

For point 1:

Are you building both 64 bit and 32 bit targets on only 64-bit Windows system?

I have a Windows 7 64bit OS installed on my laptop.
Using ant native target the 64bit platform is detected and the spawner is build using MinGW 64bit tools only.
No 32bit build is done if detected platform is 64bit.
No 64bit build is done if detected platform is 32bit

I did the 32bit build just for testing purposes using a separate makefile without platform detection.

Today I started top test the spawner on Windows OS environment and I found some interesting issues.

1. The font Monospaced is available in both Linux and Windows OS. However ClientDriver in Windwos does not work using this font.
After a short debug I found the cause. At one moment in ChuiSimulator line 627 a check on font is done to be monospaced.
The test is performed by the code from SwingHelper#isMonospaced as follow:

if (!font.canDisplay('\u2500'))
         return false;

// check if this font is monospaced, this is a bit of a hack but should
      // be reliable
      int msize = fm.charWidth('m');
      int isize = fm.charWidth('i');
      int osize = fm.charWidth('1');
      int nsize = fm.charWidth('9');
      int dsize = fm.charWidth('\u2500');

return (msize  isize && msize  nsize &&
              msize  osize && osize  dsize);

According to my tests the following values are returned for Monospaced font size 12:

Linux      Windows
      msize = 7          7
      isize = 7          7
      osize = 7          7
      nsize = 7          7
      dsize = 7         12

As you can see the dsize which is the size of character use to draw boxes is different
   in Linux and in Windows. As a result in Windows the function return false and the font
   is not recognised as monospaced by the ClientDriver

Here I have an ideea how to fix this issue and I want to ask you if is OK to do that:

if (!font.canDisplay('\u2500'))
         return false;

// check if this font is monospaced, this is a bit of a hack but should
      // be reliable
      int msize = fm.charWidth('m');
      int isize = fm.charWidth('i');
      int osize = fm.charWidth('1');
      int nsize = fm.charWidth('9');
      int dsize = fm.charWidth('\u2500');
      int gsize = font.getSize();

return (msize  isize && msize  nsize &&
              msize  osize && (osize  dsize || fsize == dsize));

In other words we can consider also the case on which the size of the box character is equals to font size. 
   I tested this code on Windows and its works.

2. A second issue related to fonts in Windows is when the font name contains spaces.
For example I used the font "Lucida Sans Typewriter" 12 which works on Windows.

Windows
      msize = 7
      isize = 7
      osize = 7
      nsize = 7
      dsize = 7

Here the name should contains quotes in command line.
   Finally I found the solution and I already fixed this issue on Java and now its works.

In other words we can consider also the case on which the size of the box character is equals to font size.
I tested this code on Windows and its works.

I am fine with the change.

Code Review 1206a

The code is moving forward nicely. Keep up the good work.

Before I get into a deep review of these new options, what about my preferred solution as documented in note 74?

A better solution is to implement a temporary P2J session with the server. The idea: create one-time authentication credentials (some kind of random token/key) that can be passed (like our password info) via STDIO to the child process. If we can find a secure way to get it to the P2J client, the P2J client can use this to do a temporary connectDirect() to the server with those credentials, read the key store, initialize the web server and send back the URI. Then it would disconnect and drop into the normal processing. The major advantage of this approach is that it can be extended to the "remote client" scenario, which we will be implementing later. Another major advantage is that we plan to do something similar to allow batch mode P2J clients to be autostarted by the server.

OK. Where I can find such an example in our p2j project?

Eric has done some work with creating an authentication token for remote database access. Perhaps that can help.

The authentication token/certificate should be 1-time use only. It must have a large enough random hash that it has enough entropy that it cannot be reasonably broken by any kind of brute force attack.

Constantin: I'll repeat the previous question:

do we have anything already made for appserver that can help us here? How do the new appserver "clients" authenticate?

Today we also already have an infrastructure for certificate based authentication. If we can enhance that to make the certificates a one-time use thing (generated on the fly?), then the rest of the solution should be an effort in properly configuring the client and server and then calling the connectDirect().

See com.goldencode.util.RandomWordGenerator and its use in com.goldencode.persist.DatabaseManager.createEmbeddedProperties(Database).

do we have anything already made for appserver that can help us here? How do the new appserver "clients" authenticate?

There was nothing special added for the authentication of appserver agents; but, the configuration of an agent requires it to have a passwordless login (i.e. certificate, trusted or something else).

Eric Faulhaber wrote:

See com.goldencode.util.RandomWordGenerator and its use in com.goldencode.persist.DatabaseManager.createEmbeddedProperties(Database).

Please note the performance-related implementation note in RandomWordGenerator.fillRandomSlots. If you need to generate very long words frequently, the algorithm can be improved by decrementing the len value by 1 each time through the loop and walking the array to the generated index, skipping slots already full (also will need to account for elements already in the array from previous passes when determining initial value of len). Not terribly efficient, but possibly better than calling Math.random many times unnecessarily. Another approach might be to add a level of indirection using an additional array to track used indexes in the first array. Another is to simply advance to the next open slot if the slot at the generated index is full, though we lose some randomness with this approach. I'm sure there are other approaches as well...

Anyway, I didn't need this extra efficiency/complexity for the initial use case, which is called infrequently and only generates words of 128 characters, so I didn't complicate the initial implementation.

You can also look on devsrv01 at the ~/testing/majic/run/batch/ directory to find examples of batch client configurations that can be used with run/client/client.sh when you also use the -b option. Of course, these are each also configured already in the server's directory. The trick in this case is to dynamically create all this configuration in a manner that can only ever be used once.

I do prefer the use of a real certificate over an auth token, as it should enable full use of the rest of our batch process support without any changes to the authentication process.

Eric, about optimizing the RandomWordGenerator.fillRandomSlots - take a look at this approach:
- each possible slot in the buffer (from 0 to buffer.length - 1) is added to a list, L. This is passed to each call of fillRandomSlots.
- when determining which slot is next to be assigned, we generate a random number R between 0 and L.size() - 1. The next slot in the buffer will be determined by the value of element at position R in list L.
- after R is determined, we remove the element at position R from list L. Thus, list L will decrease in size each time a buffer slot is assigned and there will be at most buffer.length random calls (for the buffer slot computation).

This is similar to the additional array approach I noted, but I agree a list (specifically linked list) is a better implementation choice, given the removal idea. Thanks for the details.

The current approach is not a bottleneck at all if it is not being called constantly for very large strings, so we should only complicate the implementation if there is a performance need.

Code Review 1209a

I am fine with the changes.

Inside directory.xml I just found:

<node class="authMode" name="auth-mode">
<node-attribute name="mode" value="4"/>
<node-attribute name="retries" value="0"/>
<node-attribute name="plugin" value="com.goldencode.p2j.security.GuestAccess"/>
</node>
<node class="container" name="auth-plugins">
<node class="container" name="guest_login">
<node class="string" name="classname">
<node-attribute name="value" value="com.goldencode.p2j.security.GuestAccess"/>
</node>
<node class="string" name="description">
<node-attribute name="value" value="Non-Interactive Guest Auto-Login"/>
</node>
<node class="string" name="option">
<node-attribute name="value" value="bogus"/>
</node>
</node>

Something similar I found inside Majic server directory.xml
The question here is:

Only a single auth-mode and a single plugin can be defined per server instance?
If yes we cannot create a temporary session for server key store import because the same plugin is always present on all sessions.

1. Could you explain me in a few words what kind of information is stored in this files?
2. Is any server dependency inside this information data?

For details, please see the "Key-Stores and Trust-Stores" chapter of the "Progress 4GL to Java (P2J) Installation, Configuration and Administration Guide".

3. On the client side processalias and useralias are used as aliases on client truststore and keystore resources.
On the server side inside directory I found 2 containers users and processes.
Is any relation between the directory containers and aliases?

Yes, in an indirect manner. In the directory, account information for user accounts is kept in the users container and process accounts in the processes container. The useralias is how the client code finds the certificate to represent a user account and the processalias is how it finds the cert for a process account.

What are the differences between users and processes regarding authentication?

It is an unfortunate design decision that I should have eliminated early in the project. The bottom line: a process account is essentially one that is not directly associated with a specific end-user, but is instead usually representing some non-interactive process like a:

• converted 4GL "batch" program
• remote Java application that is connecting to the P2J server to make remote object calls

The authentication mode is established by server is this true

Yes.

or could be requested by client?

The client must be trying to use one of the modes that are supported by the server. Otherwise this would be a security hole.

Where is specified the authentication mode inside server?

It is configured in the directory (as you have recently found).

6. The TransportSecurity is a singleton as is described by the javadoc. We cannot not call SessionManager#connectDirect twice using 2 different configurations.

As far as I know it is not a singleton. Where do you see this?

7. It is possible to dynamic generate a KeyStore and a X509 certificate self signed at any moment. It is not possible to design such objects for single use.
The expiration date is the single element that is used to validate / invalidate the certificate.
Using an expiration date closest to the issue date a certificate could be used for a short time but not only once.

An idea here would be to remove the certificate once it has been used one time. In other words, if the SecurityManager no longer honors that cert after it has been used once, then it doesn't matter that the expiration time is not over yet.

Please help me with any suggestions.

http://security.stackexchange.com/questions/14000/environment-variable-accessibility-in-linux

On modern systems, it seems that the process' environment is reasonably secure (only accessible by root and by other processes of that same euid). Can we do the following:

1. On the server, generate a one-time use userid and password that is random enough and big enough to be proof against brute force attacks.
2. On the server, the security manager would have to be aware of this special account (it would have to be enabled and then disable itself after being used once).
3. The server would pass this information to the spawn executable.
4. The spawn executable would insert this into the environment and the spawned child process can read it there.
5. The client will use this to establish the temporary session.
6. In the temporary session, the keystore can be read, the embedded web server will be initialized and the URI will be written back.
7. The client will disconnect the temporary session and continue with the normal ClientCore processing.
8. The server will redirect the web browser to that URI.

This avoids the complexity of the certificate processing. We could even pass an encoded keystore down to the P2J client through the environment BUT I don't see a way to get the URI back, so I think some kind of temporary session is still needed.

Only a single auth-mode and a single plugin can be defined per server instance?

No, we allow different auth modes as a default, server-specific default, group-default or account level.

OK Thanks. I will try to do that.
Regarding the singleton instance of TransportSecurity please read the javadoc for SecurityManager#getTransportSecurity at line 6908

Regarding the singleton instance of TransportSecurity please read the javadoc for SecurityManager#getTransportSecurity at line 6908

I see the text now. It is inaccurate. Please fix it as part of this work.

A more portable solution is to send the random generated user-id and password directly as arguments in command line.
To be more secured we could also encrypt the values for this arguments.

The problem is that on Linux/UNIX, command line args are visible to anyone. Using encryption would require both sides know a particular key in order to decrypt the values. That suggests we would have to hard code the key in our code (which is not secure) or we would have to otherwise provide the key as part of configuration (which is the very problem we're trying to solve in the first place).

Today I started to implement the direct connection using a strong random subject id and password passed as environment variables.
Using RandomWordGenerator I generated strong values for both subject id and password.
The values are used to create and add two environment variables using ProcessBuilder#environment interface.
The values are converted to hex strings on server side and converted back to the original values on client side.
I added this conversion steps in order to avoid any problems caused by special characters generated by RandomWordGenerator.

Next I tested on both Linux and Windows OS and I found to works fine. On client now I'm able to get the values with no problems.

On the next step I want to implement the server side code and I have some questions:
I found the AdminServerImpl class in admin package which export a lot of administrative operations.
I saw that is used by the admin console applet.

1. I would like to use this helper class to create and delete temporary users. Is this OK?

2. When we create a new user (account) we have to provide an UserDef structure.
I think that I have to create a protected user (with password) and a custom authentication mode.
I have to specify also a custom authentication plug-in.
Should I design and implement a new custom login plug-in for this purpose?

3. In P2J and Majic I found the following entries registered inside directory

P2J (directory.xml):
<node class="authMode" name="auth-mode">
<node-attribute name="mode" value="4"/>
<node-attribute name="retries" value="0"/>
<node-attribute name="plugin" value="com.goldencode.p2j.security.GuestAccess"/>
</node>

Majic (gso-directory.xml):
<node class="authMode" name="auth-mode">
<node-attribute name="mode" value="4"/>
<node-attribute name="retries" value="-1"/>
<node-attribute name="plugin" value="aero.timco.majic.security.Login"/>
</node>

Now if we have to add a new plug-in for the same mode how we can do that?

1. I would like to use this helper class to create and delete temporary users. Is this OK?

Yes, this should be OK.

You will have to meet some requirements for this to work:

• The thread on which you make these calls must have a proper security context AND that context's account must include sufficient permissions to execute all of the actions required.
• You will have to honor the control flow that would be followed by a "real" admin console user. In particular, you will have to force a "refresh" of the security cache after adding/removing a user.

It may make sense to create multiple accounts, in advance, one time. The idea is to permanently leave these accounts in the system. Then you can just:

1. set the password and enable an account from the pool when you are launching a new client
2. disable that account when the logon has occurred

The only drawback to this approach is that the overall number of simultaneous logons will be limited by the size of this pool. However, in practice, the actual amount of time needed to use this mechanism during the client spawning process will be very small. I suspect that even a relatively small pool will scale pretty well.

a custom authentication mode.
I have to specify also a custom authentication plug-in.
Should I design and implement a new custom login plug-in for this purpose?

I'm not completely sure of this. The permissions of what can be called by this account can be very "locked-down". We can limit this to just the right to call a specific exported remote object interface. Then the client just uses "normal" userid/password authentication mode, obtains the proper remote object, calls that remote object and then disconnects. On the server side, the call to that remote object should be sufficient to notify the server to disable the account and when the logoff occurs, the account must go back into the pool.

There may be some minor changes needed in the security package to enable this, but I don't see the need for a custom auth plugin.

Now if we have to add a new plug-in for the same mode how we can do that?

This specifies a single global DEFAULT auth-mode (and default plugin).

But you can override this at the group account and user account levels (and maybe the server level too, I'm not sure).

On StandardServer start-up the AdminExports interface is registered:

ifaces = new Class[] { AdminExports.class };
RemoteObject.registerStaticNetworkServer(ifaces, AdminServerImpl.class);

On the AdminLogon after a successful logon a remote reference to this interface is obtained and used via a connectDirect call.
I tried to do the same but so far with no success. I'm using the admin account credentials.
The problem is that so far I cannot create a connection to the server using admin credentials and get the AdminExports remote interface.
Is any default session available to get the interface reference without connectDirect?

Is this the right way that we could follow, using such a connection to get access to the AdminExports interface?

Is this the right way that we could follow, using such a connection to get access to the AdminExports interface?

No.

There is no reason for a session or to be using the remote object interface when the code that is running is in the same JVM process. I think this is as simple as calling AdminServerImpl.addUser(), right? You just need to do it from a P2J server thread that has the proper security context.

In addition, please try my idea of creating a pool of pre-existing accounts. This will greatly reduce the effort for each new session AND it should be much faster too. The creation of this pool can be a 1-time thing (if the pool doesn't exist, it gets created) that is persistent (the accounts aren't ever deleted).

I tried to create a temporary account when the StandardSever start-up using AdminServerImpl.addUser() but with no success.
I think that StandardSever runs on P2J server thread because I can read from directory on this thread.
The problem is coming from here:

// access the target
WorkArea wa = workArea.get();
if (!wa.targeted || !wa.sa.adminAccess()) {
return false;
}

wa.targeted is false and wa.sa is null.

I added AdminServerImpl.setTargetLive(true) to solve the previous issue but now It said that I have no admin rights. SystemResource#checkAdminAccess return false.
return systemAccessWorker("admin", SYSTEM_ACCESS);

I think you should bypass AdminServerImpl entirely and just use SecurityAdmin to add a new user.

I added AdminServerImpl.setTargetLive(true) to solve the previous issue but now It said that I have no admin rights.

As this is done using the security context of the server process (i.e. standard for the test directory.xml), you need to make sure that the server process account has the appropriate ACLs; check the ACLs for the "admin" account and "admins" group - I think it could work just by adding "standard" account to the "admins" group.

Greg, about the pool of "generated, pre-existing, accounts": wouldn't adding them to the "live" security cache generation expose them to the Admin Console? I think we will need some separate management for this pool, and not just add them to the global user list.

wouldn't adding them to the "live" security cache generation expose them to the Admin Console

Yes, but that is probably OK. We can create them with verbose descriptions that make it clear these are internal/web accounts. Admins can be trained to leave these alone.

I think we will need some separate management for this pool, and not just add them to the global user list.

I worry that creating "shadow" users would require major changes throughout our security package. I want to keep this as simple as possible. These accounts will normally be disabled anyway, except during the moment we are trying to spawn a client.

I'm using SecurityAdmin instead AdminServerImpl but unfortunately the problem remains the same.
On the server directory I have for groups:

<node class="container" name="groups">
<node class="group" name="everybody">
<node-attribute name="description" value="all users"/>
</node>
<node class="group" name="admins">
<node-attribute name="description" value="all admins"/>
</node>
</node>

The standard is defined as process account.

<node class="container" name="processes">
<node class="process" name="standard">
<node-attribute name="enabled" value="TRUE"/>
<node-attribute name="description" value="P2J server"/>
<node-attribute name="master" value="TRUE"/>
<node-attribute name="server" value="TRUE"/>
<node-attribute name="alias" value="standard"/>
</node>
</node>

I tried to add the standard account to admin group without success.

<node-attribute name="groups" value="admins"/>

This kind of node is not allowed.

I fixed the issue by adding the process standard to the subject list in ACL (see bellow)
Now I'm able to create user accounts from StandardServer thread.

<node class="container" name="acl">
<node class="container" name="system">
<node class="container" name="000100">
<node class="resource" name="resource-instance">
<node-attribute name="reference" value="admin"/>
<node-attribute name="reftype" value="TRUE"/>
</node>
<node class="systemRights" name="rights">
<node-attribute name="check" value="true"/>
</node>
<node class="strings" name="subjects">
<node-attribute name="values" value="admins"/>
<node-attribute name="values" value="standard"/>
</node>
</node>

I started to design and implements the temporary account pool. Finally I was able to create user accounts from StandardServer thread.
I added the "standard" account to the ACL list of subjects in directory.xml for "admin" rights.

- I designed a temporary account pool using a FIFO queue (ConcurrentLinkedQueue) to hold temporary accounts.

- When the pool is created on start-up also a group "temps" is created if the group does not already exists. All temporary accounts created and added to the pool belongs to this group.

- Also the temporary accounts are created if necessary to fill the entire pool. The size of the pool is configured form directory.

- When a temporary account is requested from pool the item form the head of the pool is returned or null if the pool is empty. The item is removed from pool.

- When the item is no longer needed is returned to pool, enqueued at the tail of pool and reused.

This design works fine. However I found a lot of constraints and I want to ask you about a possible solution.

1. The pool is created from the StandardServer thread on start-up when we have enough rights to manipulate user accounts.
Unfortunately after the initial creation it is no longer possible to manipulate user accounts from other threads (jetty).
As a consequence we cannot enable/disable accounts on "the fly". Basically we cannot do any changes to accounts.
Also in my mind was to create new accounts whenever the pool became empty instead to return a null value but this is also not possible.
Do you known any idea on how we could fix that?

2. I saw that the user name is stored always in lower case regarding that the generated string contains upper case characters.
This is just an observation because have no side effects in my implementation.

3. Regarding the password I sow that using the same password for two different users one created form admin console the other
created by me with SecurityAdmin API have different encrypted format!!! Inside the code the decrypted values are OK.
However using the account created from console manager allow me to authenticate remote but using my generated accounts does not.
Do you have any hint for me here? Anyway on the next step I want to do deep investigations on this issue.

1. The pool is created from the StandardServer thread on start-up when we have enough rights to manipulate user accounts.
Unfortunately after the initial creation it is no longer possible to manipulate user accounts from other threads (jetty).
As a consequence we cannot enable/disable accounts on "the fly". Basically we cannot do any changes to accounts.

I think we should create a daemon thread that can handle such tasks. That thread would have to be setup with the proper security context when it is started. In fact, the initial setup of the accounts and so forth can be done there too.

Then it is a matter of establishing the appropriate queueing or IPC mechanism between jetty and this daemon thread. This ensures that the jetty thread doesn't need special permissions.

3. Regarding the password I sow that using the same password for two different users one created form admin console the other created by me with SecurityAdmin API have different encrypted format!!!

This is a surprise to me as well.

In the mean time I found why the password format are different. AdminConsole does not store the clear password encrypted.
Instead first a SHA hash is created using the clear password bytes using HashPassword.hashPassword tool.
The generated hash is encoded and stored in directory as password. This make the password secured, cannot be decrypted ever!
The problem is that we have to store the password like AdminConsole and send the clear password encrypted as environment variable.
On the client side we have to provide the clear password for authentication when we create a server direct connection session.
Because we cannot decrypt the password we have to keep the encoded password on another parameter in user account. Maybe the owner field would be used.
Or another solution should be found.

Also the Base64 encryption/decryption algorithm is used. Basically isn't a strong encryption.

I have an idea. We can use a random string constant like a "salt" and the password should be constructed
from user name also random generated plus the "salt" combined using a specific function.
We have to send remote only the user name and on the client side using the same "salt" and algorithm the password is reconstructed.

Another solution is to delete all accounts from temps group on each server start-up and create new accounts.
The passwords for the new accounts are kept in memory and sent remote. This approach seems to be more secured.

The problem is that we have to store the password like AdminConsole and send the clear password encrypted as environment variable.
On the client side we have to provide the clear password for authentication when we create a server direct connection session.

This is OK.

These accounts will be disabled except during the moment of use. As long as the account has its password changed to a newly created random text just before it is enabled, then it is OK to send the clear version of this down to the client. Only someone with the rights to run a debugger or otherwise inspect the kernel data structures/proc file system can see this information. It is OK.

Also the Base64 encryption/decryption algorithm is used. Basically isn't a strong encryption.

Indeed. It is obscuring the data only and can be reversed so it isn't encryption at all.

We can use a random string constant like a "salt" and the password should be constructed from user name also random generated plus the "salt" combined using a specific function. We have to send remote only the user name and on the client side using the same "salt" and algorithm the password is reconstructed.

We don't need to do to this level. The real security it adds is not enough because anyone with the salt value can still get the password.

Another solution is to delete all accounts from temps group on each server start-up and create new accounts.
The passwords for the new accounts are kept in memory and sent remote. This approach seems to be more secured.

I like this. We still need to disable/enable accounts only when they need to be used, but leaving this data out of the persistent directory does seem cleaner.

I implemented the temporary account pool by deleting all temporary accounts and create new ones on server start-up.
Now it's works fine but without enabled/disable account at this moment.
I will try to fix later this issue because now I have another big issue to fix and I need some helps.

When the ClientDriver is started using chui_native or swing_chui_frame a terminal window is open ant the users are working inside this TTY window.
As I understood that on Majic application an authentication plug-in is used and the user should enter his credentials before start working on the p4g converted application.

In our case however the flaw is different. The web login page is used to authenticate the user against OS.
On success the user is redirected to a web page containing a JS TTY emulator.
No other console windows are open for user. Even more the user should be located somewhere remote on another machine.
In this case the authentication for p4g converted application should be done using JS TTY emulator.
Is this true?

Here I have some remarks and questions:

When I tried first to open a server session using a temporary account always the GuestAccess plug-in is used.
Using this plug-in Regarding the user name and password for temporary account the authentication succeed.
The user is always authenticate using bogus account as is specified inside directory.
I tried to set custom plug-ins to temporary accounts when I created temporary users but it does not work.
It seems the auth-mode is global and unique.

<node class="authMode" name="auth-mode">
<node-attribute name="mode" value="4"/>
<node-attribute name="retries" value="0"/>
<node-attribute name="plugin" value="com.goldencode.p2j.security.GuestAccess"/>
</node>

1. How we can fix this issue?

2. If we change this plug-in how the ClientDriver will work with other screen drivers?

3. Using chui_web screen driver the user authentication should be done using the JS TTY window?
If YES should ClientDriver still have an open session?

In this case the authentication for p4g converted application should be done using JS TTY emulator.
Is this true?

Yes. Of course, we haven't gotten that written yet, but so long as the JS driver works like any other driver, then it will naturally allow the client to operate just like the swing or native client.

1. How we can fix this issue?

I think the problem comes from not knowing the userid until after connectDirect() is called. I think this can be solved using some settings in the bootstrap configuration. Please look at SecurityManager.sendAuthType() which is called from SecurityManager.authenticateClientWorker() which is the main client-side authentication processing method.

2. If we change this plug-in how the ClientDriver will work with other screen drivers?

There is no need to change the plugin.

3. Using chui_web screen driver the user authentication should be done using the JS TTY window?
If YES should ClientDriver still have an open session?

Yes. You are writing a "JS driver" right? It is a kind of plugin that provides UI input and output. In the JS driver case, the driver itself is really split into 2 pieces. The "top half" is written in java and runs in the P2J client JVM process. The "bottom half" is written in javascript and runs in the web browser. But this is no different than the other ChUI drivers that we have. None of them know anything about the authentication process. BUT they all get used by the authentication process when an interactive login is needed.

I think you may need to look at the SecurityManager.authenticateLocal() which is the server-side of the interactive login. Note there that in CUSTOM mode we serialize an Authenticator class and send the class' bytes to the client side, where it is loaded and executed as a special piece of UI client code. That code uses the UI services of the driver (whichever one is there) to display a UI to the user and obtain userid/password credentials. Then it packages these credentials and sends them back to the server. The driver has no idea this has anything to do with login. The driver is just acting like its normal UI service.

To see a real example of this, please look at the code in testing/majic/src/aero/timco/majic/security/LoginClient.java. A simpler version can be found in src/com/goldencode/p2j/security/DefaultLoginPanel.java.

There is no problem to solve here that won't be automatically solved by just properly implementing the JS driver.

Code Review 1213a

The changes continue to move in the right direction. Some minor feedback:

1. I prefer to have a flag at the top of ClientCore.start() named "web". You can set this true if the uuid is in the bootstrap config. If it is set true, only then would you call setWebClientConfig() and startUp(). This allows the reader to understand that these are conditional, without having to read "into" those methods. This is similar to how we have the "dbg" flag.

2. Please rename setWebClientConfig() to addWebClientConfig().

3. Please rename startUp() to initializeWebServer().

4. In WebClientSpawner, I think we may need more than 5 seconds of timeout on systems with a heavy load.

5. In WebClientSpawner, I prefer to wait a reasonable amount of time for an account to become available instead of throwing an exception.

The result is here http://www.gmax.ws/tty.html
It's just an exercise but I think that looks good, isn't it?

Very cool!!! Well done.

Yes, I am really excited to see us get to work on that part.

Marius Gligor wrote:

5. On heavy loaded systems when the pool potentially became empty a new temporary account is created and returned.
This kind of accounts are created enabled and served immediately to caller.
Later when the items are returned to pool the related accounts will be disabled.
Basically all requests for temporary accounts are honoured.

A quick thought: is it possible for the queue size to get out of control? As I understand, this step is before any user authentication is done, thus someone could build a script to send a massive amount of requests, thus OOME-ing the server... I think a static pool size is better; you can limit the amount of time to wait until an account can be retrieved from the pool (this can be made configurable too).

OK. I started to change the design to have a fixed pool of accounts.

Code Review 1217a

1. if(web) in ClientCore should be if (web).

2. In TemporaryAccountPool, the import of LogHelper should be using the wildcard.

3. I don't understand the purpose of this code in TemporaryAccountPool.createUser():

         if (user != null && user.enabled == enabled)
         {
            account = new TemporaryAccount(user.subjectId, password);
         }

4. I prefer if you would rename TemporaryAccountPool.createInstance() back to getInstance(). I think it better describes what is being done AND it is more consistent with other singleton classes in P2J (like SecurityManager...).

5. In TemporaryAccountPool.poll(), there may be a logic problem. In the item == null (pool empty) case, it seems that if an account is found before the timeout expires, the account is returned back to the caller without being enabled.

Shouldn't this code:

            // Change user status
            Runnable core = new UpdateAccountTask(item.getSubject(), true);
            TemporaryAccountPool.poolTask.execute(core);

execute no matter how the "item" is found?

6. I would like you to make a change to the build process to bypass the execution of postbuild.sh by default. In other words, for the default build, no postbuild.sh would get run. This is the most common case (where people don't need that to run) and I don't want to make all users force a new sudoers configuration change or a requirement to enter the password. If you can make this something that can be added optionally from the command line, that may be best.

After #2201, you will next be working on #2212. I will add more details/guidance tomorrow. If you run out of work before then, you can start by analyzing the kinds of communication flows that will be needed to support the ChUI driver. Make sure you consider the needs of both directions of flow. Document your findings/thoughts here.

• in makefile.spawn, you are using tabs for indenting. More, in postbuild.sh, spawn.c and winspawn.c, the indenting is done at 4 chars, not 3.
• ChuiWebDriver.startupServer is missing javadoc
• WebHandler - the header is indented one-space to the right...
• WebHandler.handle/doGet/doPost - check the formatting of the method definition
• Clientcore.initializeWebServer - you are checking for the driver to be a ChuiWebDriver instance. Isn't this somehow restricting? I think there should be a WebDriver interface, to mark the web drivers.
• For the new added classes, the class javadoc is minimal. Please improve the javadoc to explain what these classes do and how they should be used (this will be helpful later on, when we will write documentation for all of this).
• TemporaryAccountPool.createUser - I think the certificate alias for the new user should be null, not hard-coded to shared.
• in UpdateAccountTask.run, we are calling AdminServerImpl.targetRefresh();. This is a very expensive operation (as it generates a new security cache generation on each call) and I'm not sure how this will behave in high-load moments, when a high number of users are authenticating simultaneously.

in UpdateAccountTask.run, we are calling AdminServerImpl.targetRefresh();. This is a very expensive operation (as it generates a new security cache generation on each call) and I'm not sure how this will behave in high-load moments, when a high number of users are authenticating simultaneously.

This is a very good point. The security generations will grow with no bound in this case AND it can have performance implications.

Please look into ways that we can solve this:

1. dynamically edit all existing security cache(s) in which the temp accounts exist, without using a refresh or creating a new generation
2. make the security and admin code aware of the concept of truly temporary account that is not backed by the directory (automatically goes away at server shutdown) and is kept separate from the generations of security cache, allowing these accounts to be managed as a single pool that is not subject to the refresh requirements

I prefer option 2 if it is reasonable to implement.

According to my tests without a "refresh" the account changes (enable/disable) are not "visible" and when the client try
to do a remote authentication an error "account is disabled" is throw and the authentication fails.

On the other hand the account pool is mandatory because we need to keep the clear password in memory.
This means that we have always to generate this accounts on server start-up when we know the clear passwords.
The pool is used just to serve the account subject and a clear password used for authentication.
The authentication process it self has nothings to do with our pool.

In my opinion authentication is done relative rarely and isn't used intensively by users.
That's why we assert that a small size pool should works fine.
Anyway I'll try to find a solution as you suggested.

In my opinion authentication is done relative rarely and isn't used intensively by users.

In comparison to the runtime of a session, yes. In other words, each session is commonly expected to be long-lived. But on any given day there will be hundreds or even thousands of logins to a particular server instance. Server instances that are left running for months will have a very large number of security generations that may start to cause problems.

That's why we assert that a small size pool should works fine.

The small pool is OK so long as there are not too many simultaneous logins. Except for companies that have shift operations where a large number of users login within a short interval, a small pool will work OK.

But that doesn't mean that this processing can't have an effect. When the security cache is refreshed, it is a costly activity in the security code. It will affect the performance of all other ongoing security operations, such as ACL checks. This isn't only about login processing.

According to my tests without a "refresh" the account changes (enable/disable) are not "visible" and when the client try to do a remote authentication an error "account is disabled" is throw and the authentication fails.

Yes, this is expected and correct.

There is one other thing, that I should have noticed before. Please change the password every time you re-enable the account. I don't want the passwords to be re-used. The accounts themselves can be reused. But otherwise, someone that breaks a password can just keep trying it in a tight loop while spawning new sessions and eventually they will get in. It is perfectly possible to login using the same account more than 1 time simultaneously. That is by design because our users must be able to have multiple sessions. But it exposes a security hole in the scenario where we re-use passwords.

Code Review 1218a

Really good improvements. Yes, we are close now.

5. For security reasons when an account is (re)enabled the password is also changed, that is, a new random password is generated.
This is also related to issue 4 because is a change of a user account which is synchronized with directory but not updated inside the security cache.

I understand. All changes to the account are subject to this limit. It is by design.

I think that doing a short work around the SecurityManager and SecurityCache it is possible to add a method to update (refresh) a user account inside the SecurityCache instead to refresh the entire context.
We have to search on the list of accounts stored inside the SecurityCache instance and update only one account.
Do you think that this scenario could be a solution?

Yes, I do think this can work. This is what I was suggesting with my "option 1".

BUT I do think we must update all instances of that account. I believe there may be separate instances of those accounts in every cache generation since the account was created. This would mean that you must edit a list of instances of that account, otherwise when the latest generation is popped, the previous generation may have a different state and cause problems. If I am mis-remembering how we do this, let me know.

Code Review 1219a

Everything looks good. Please get this regression tested (only runtime testing is needed).

I agree that my idea of having truly non-persistent accounts would require much more significant change to the security package. You did the right thing in choosing to implement in the simplest manner.

I've started to do the regression tests. When I executed this step:

run_regression.sh build && run_regression.sh save-generated -Dsave.candidate=y

the build of project fails due to the missing of PAM libraries when spawn.c is build.
I need this libraries to be installed on devsvr01 but I need root permissions for sudo apt-get install
Another option is to change the makefile.spawn and compile without PAM options.

In order to install PAM libraries we need to install this package:

sudo apt-get install libpam0g-dev

Another option is to define a build variable like -Dpam=yes in order to build spawn.c with PAM option.
By default the build will be without PAM options.
This is useful because other members of team should not install the libpam0g-dev on their workstations.

Another option is to define a build variable like -Dpam=yes in order to build spawn.c with PAM option.
By default the build will be without PAM options.

This is a good idea. Please do make this change.

I have also just installed the PAM libraries on devsrv01, so that we can test both with and without PAM support.

OK. Now the project was successfully build on devsvr01.
Nevertheless I'm going to add the PAM variable for conditional build similar with skip "postbuild" target already fixed.

Code Review 1219b

The changes look good.

-------- Original Message --------
Subject:     regression tests
Date:     Fri, 20 Dec 2013 12:38:35 +0200
From:     Marius Gligor <mag@goldencode.com>
Reply-To:     mag@goldencode.com
To:     Constantin Asofiei <ca@goldencode.com>, Greg Shah <ges@goldencode.com>

Constantin,

The package that I started to test require some changes inside directory.
Do you think that missing this values from directory could cause a failure?

Thanks
Marius

1. The standard subject should be added  to admin ACL in order to create temporary account on start-up:

 <node class="container" name="system">
          <node class="container" name="000100">
            <node class="resource" name="resource-instance">
              <node-attribute name="reference" value="admin"/>
              <node-attribute name="reftype" value="TRUE"/>
            </node>
            <node class="systemRights" name="rights">
              <node-attribute name="check" value="true"/>
            </node>
            <node class="strings" name="subjects">
              <node-attribute name="values" value="admins"/>
              <node-attribute name="values" value="standard"/>
            </node>
          </node>

2. The node chuiWeb should be also added to directory used on client spawn.

   <node class="container" name="server">
      <node class="container" name="default">
        <node class="boolean" name="adminEnabled">
          <node-attribute name="value" value="TRUE"/>
        </node>
        <node class="container" name="chuiWeb">
          <node class="boolean" name="enabled">
            <node-attribute name="value" value="TRUE"/>
          </node>
          <node class="boolean" name="secure">
            <node-attribute name="value" value="TRUE"/>
          </node>
          <node class="string" name="spawner">
            <node-attribute name="value" value="/usr/share/p2j/spawn"/>
          </node>
          <node class="string" name="jvmArgs">
            <node-attribute name="value" value="-Xmx512m -XX:MaxPermSize=64m -Djava.awt.headless=true"/>
          </node>
          <node class="string" name="classpath">
            <node-attribute name="value" value="/usr/share/p2j/lib/p2j.jar"/>
          </node>
          <node class="string" name="libPath">
            <node-attribute name="value" value="/usr/share/p2j/lib"/>
          </node>
          <node class="string" name="workingDir">
            <node-attribute name="value" value="./"/>
          </node>
          <node class="string" name="configFile">
            <node-attribute name="value" value="config.xml"/>
          </node>
          <node class="integer" name="rows">
            <node-attribute name="value" value="24"/>
          </node>
          <node class="integer" name="columns">
            <node-attribute name="value" value="80"/>
          </node>
          <node class="string" name="background">
            <node-attribute name="value" value="0x000000"/>
          </node>
          <node class="string" name="foreground">
            <node-attribute name="value" value="0xFFA500"/>
          </node>
          <node class="string" name="selection">
            <node-attribute name="value" value="0x0000FF"/>
          </node>
          <node class="string" name="fontname">
            <node-attribute name="value" value="monospaced"/>
          </node>
          <node class="integer" name="fontsize">
            <node-attribute name="value" value="12"/>
          </node>
        </node>

In regard to this, please setup our code so that most of this has sensible default values when the backing node is missing from the directory. Perhaps everything except for chuiWeb/enabled can be defaulted.

This will greatly reduce the amount of configuration needed. The only tricky part is where there are platform-specific values (e.g. /usr/...), in such cases we will need different defaults by platform.

I provided default values for all this parameters (see ClientBuilderOptions).

Now the server starts without errors if directory settings are not in place:
- If node chuiWeb is missing the http://server_address:admin_port/chui returns HTTP 404 page not fond because no handler is registered for /chui target.
- If the server standard subject is not added to admin ACL the pool is empty and no client could be spawned.

The changes are currently in regression tests. At the end of tests if no errors I have to download the Majic build to my workstation add appropriate directory configuration
and do a test to spawn web clients.

If the server standard subject is not added to admin ACL the pool is empty and no client could be spawned.

I'm fine with this, but I would like you to log a diagnostic message that reports that the web chui is enabled but the sufficient rights have not been granted to the standard subject such that the web interface can function properly.

Please upload the new code here. You can keep going with testing.

Also I could provide platform specific (Linux/Windows) default values and do only runtime tests with Majic build on my workstation without to add chuiWeb node to Majic directory.
The server standard subject MUST be added on tests.

I still prefer you to test on devsrv01 with the chui web active. This will make sure that we have a real chui environment (MAJIC) to test the web chui driver when that is ready. TIMCO will also be very interested in using the chui web driver, so we need to make sure it works well for them.

Please work with Constantin to figure out how to make the changes to the directory template files that are being used for the devsrv01 environment.

Finally, it will be important to make sure that the code can optionally use relative pathing for all the jars, classpath, libpath etc... which will allow the same configuration to work for multple users in our testing environment. So instead of absolute paths /usr/... it would be relative to the user's home dir.

Classpath, native libraries path and spawner location should be shared by all web clients.
We cannot provide relative paths for this values we have to specify absolute paths.
With relative paths we have to replicate the jar's on all users home directory.

I think the solution for devsrv01 is not about setting relative paths, but computing the full paths dynamically for each user. Inside the directory template, we can set the i.e. /usr/share/p2j/lib/p2j.jar path to something like P2J_JAR and change the majic/run/server/prepare_dir.sh script so that these values are replaced with the proper full path for the current user. Keep in mind that each user on devsrv01 has a ~/testing/majic/p2j folder.

We cannot provide relative paths for this values we have to specify absolute paths.

Most production environments would follow this approach, that is true.

With relative paths we have to replicate the jar's on all users home directory.

This is exactly what we already do in our testing environments. So this is OK and in fact it is desired, because each user may have different changes to test.

Keep in mind that each user on devsrv01 has a ~/testing/majic/p2j folder.

This is why I prefer to use relative paths on devsrv01. It reduces the changes to be made in the directory template and it should still work.

Actually, I think we should be able to determine the library path and the classpath dynamically. See the code in AppServerLauncher.builder - there I determine the jar which stores the ClientDriver class and this will be added to the classpath; also, the assumption is that the library path is the parent folder of this jar.

What remains is the spawner folder - can you remind me what this is used for?

I think we should be able to determine the library path and the classpath dynamically.

This will work as a default. But I do want to make sure that we can specify a completely independent installation. The idea: the server and clients may not even be on the same machine and/or the clients may not be allowed access to the directory structure in which the server runs.

We can detect the location of p2j.jar as you already mention and I use this method in the previous snapshots but I think this should be used
as a default value in case no other value is specified inside directory.
In other words we have to find a set of default values which are relative to user and override this values in directory if we need absolute or other values.
Please let me know if this is what you are looking for and if we can use as default values the following:

- p2j.jar location for classpath native libraries directory and spawner location.
- user (current) directory for default working directory.

They are also many other issues that should be solved related to spawner build.
In Linux the postbuild target is mandatory and require sudo permissions.
Are all users (developers) on devsvr01 permissions tu use sudo if they want to do tests?

Also keep in mind that the p2j.jar location detection is done on the server instance
and is absolute to the user on which the server is running.
For other users this method of location detection does not work.

Here are some details on our approach in regard to javascript frameworks.

1. Cross-browser compatibility is still a problem. Primarily I refer here to issues caused by browsers implementing the same features in incompatible ways (e.g. the IE event model versus everyone else).
2. Javascript standards move slowly and many problems that have yet to be standardized are already solved cleanly by existing frameworks.
3. Uneven implementation of existing javascript standards across browsers.
4. The event-driven (asynchronous) + single-threaded nature of javascript can lead to a great deal of boilerplate code being needed for things that in other languages would be shorter and simpler. Often javascript frameworks provide mechanisms to reduce this.
5. There are common javascript problems that can be solved with helper functions and other utility features which enhance productivity and simplify code.

I think it is reasonable to use a javascript framework to reduce or eliminate the above issues.

I have taken a close look at jQuery and Dojo. I've taken a more cursory look at Prototype, GWT and a few others. Based on what I have seen, Dojo seems to be the best option for "web applications". It can also be used for simpler web pages, wherever javascript is used. But it seems to really shine when one is developing an entire web app in javascript. In other words, when the user is not moving from page to page, but instead there is some technique (e.g. AJAX, websockets...) that is used to communicate to the back end server while the javascript drives a dynamic interface on a single page load.

The open source licensing, breadth of features and depth of features of Dojo are all very good. It is also moving forward at a fast pace. As a technology it is vibrant and healthy.

The plan is to include the necessary Dojo files from our own local installation instead of from a content network. Where we can do something simply in javascript that isn't enhanced or made more simple by using Dojo, then we should use our own javascript. But we don't need to be worried about trying Dojo functionality if it makes sense.

Also keep in mind that the p2j.jar location detection is done on the server instance and is absolute to the user on which the server is running.

This will be fine for our use on devsrv01. It may also be good for other simple test environments. Production environments can configure more explicit choices, so that is OK.

Are all users (developers) on devsvr01 permissions tu use sudo if they want to do tests?

This hasn't been done yet. Unfortunately, I have to step out for a couple of hours.

Constantin: can you make the sudoers file changes? See the postbuild.sh for details.

Greg Shah wrote:

Constantin: can you make the sudoers file changes? See the postbuild.sh for details.

Yes, I can make the change. Just confirm that this line needs to be added (i.e. for user mag):

mag ALL=(ALL) NOPASSWD: /home/mag/testing/majic/p2j/src/native/postbuild.sh

Marius Gligor wrote:

Monday I want to do runtime tests first on my workstation and then on devsvr01.

Regression testing can be ran only on devsrv01; what is possible on your machine is to run a local MAJIC server. For this, you need to download the MAJIC jars and other setup (without converted and generated sources) to your machine. If MAJIC is starting OK and clients can login, then you can move to regression testing on devsrv01.

Constatin please let me know if is this OK. Are you available on Monday to help me if necessary?

Yes, I'll be available on Monday.

BTW, you need sudo rights too, beside the line at note 168, right? I'll make the sudo-related changes on Monday (if Greg doesn't get a chance to make them).

I did something similar with AdminConsole if you remember.
After the local tests you suggest me to repeat the regression tests?
What I want to do is to start the Majic server on devsrv01 and try to access https://localhost:7443/chui from my web browser.
But I'm afraid will not work 100% due to the redirection on a random port which require a ssh tunnel.

Marius Gligor wrote:

After the local tests you suggest me to repeat the regression tests?

Yes

What I want to do is to start the Majic server on devsrv01 and try to access https://localhost:7443/chui from my web browser.
But I'm afraid will not work 100% due to the redirection on a random port which require a ssh tunnel.

For this reason is best to start the Majic server on your machine (that's why we allow remote connections to each user's dedicated postgres cluster on devsrv01). Have you done this before?

Yes I did that when I tested Admin Console on my local station. So I have to do tests on my local station and after that redo regression tests with the latest changes.

Related to JS frameworks. I have some experience using jQuery but no experience using other frameworks.

Marius: I've made the sudo-related changes for your account on devsrv01. After regression testing passes, make sure the /etc/sudoers.d/sudoers-ext contains the settings for all other users.

Go ahead with regression testing on devsrv01 without making changes to enable the web chui. That will allow us to get this code checked in sooner.

Meanwhile, Constantin and I will make the devsrv01 changes needed to enable the web chui in the near future. In regard to the sudo approach, I have been thinking about it. The current plan needs to be changed because it opens a security hole.

The core issue is the path to the postbuild.sh and to the target spawn executible are both under user control. That means that the script can be edited to do anything at all. Likewise, the spawn source code is under the control of the user leading to the same problem. Although we could go with an approach that puts the specific commands (chown, chmod including their specific arguments so that the commands are very limited in scope) into sudoers intead of the script, that only protects from the modification of postbuild.sh.

Just as our customers would do, I think we will manually install a prebuilt spawn to a common location. Then we will not need sudo rights for each user. The downside is that changes can't be automatically applied.

Constantin: I think that is no need to do that. This issue should be tested only on a local copy of Majic not on devsvr01.
The currently regressions tests scripts does not test the spawn of web client.
Greg: Should I repeat the regressions tests on devsvr01 using the current changes?

Marius Gligor wrote:

Constantin: I think that is no need to do that. This issue should be tested only on a local copy of Majic not on devsvr01.

OK, I've backed out the sudo changes. But, the problem is: you need to change the "native" task so that sudo rights are not required; with your current build, you need sudo rights on each machine were P2J needs to be built (and the native task is executed by default - maybe make a separate task to build the spawner?).

Actually the build does not require sudo permissions.
You have to use some command line arguments to execute the postbuild.sh

-Dhave.pam=yes // used to compile with PAM options
-Dpost.build=yes // to execute the postbuild target which in turn run postbuild.sh script

The normal ant task
ant native
will build the native artifacts without ask for sudo permissions.

The following build (manual)
ant -Dhave.pam=yes -Dpost.build=yes native
will ask for sudo password and you shoul have PAM libraries installed on machine.

Marius Gligor wrote:

Actually the build does not require sudo permissions. You have to use some command line arguments to execute the postbuild.sh

Ahh, I missed that, is OK then.

Code Review 1223a

I am fine with the changes.

Please merge up to the latest bzr revision (10426) and then run runtime regression testing on devsrv01. Do not test the web chui on devsrv01, just get it past regression testing so that we can get it checked in.

We have to design the protocol used by websockets.
We have two options: JSON or binary payloads. As I remember you said that you prefer a binary protocol.

Yes, for network efficiency it is important this is binary.

Are you already designed such a protocol?

No.

Additional Details on the Protocol

1. The protocol will need to be full-duplex. It must be possible to have both sides sending/recving independently.

One big reason for this is that even when the normal client processing is blocked (because the flow of control for the app is executing on the "server" side), the end-user can still execute a CTRL-C which should cause an interruption of the server. It is important that you review the CTRL-C processing which we implement today. It is a bit complex because we have split a single threaded process into 2 processes that are connected via a socket for interprocess communications (IPC). We use the "conversation" thread on both sides of the socket to force this to behave like a single-threaded environment even though we are inherently multi-threaded. In other words, we make all the primary control flow paths operate synchronously. But since the CTRL-C is an asynchronous event, it must interrupt the flow of control wherever it is executing (locally in the P2J client or remotely on the P2J server). This is where it is tricky. Please review the following files to understand this better:

com/goldencode/p2j/net/Control.java
com/goldencode/p2j/net/Conversation.java
com/goldencode/p2j/net/Dispatcher.java (search on RoutingKey.INTERRUPT_SESSION)
com/goldencode/p2j/net/InterruptHandler.java
com/goldencode/p2j/util/InterruptedExceptionHandler
com/goldencode/p2j/ui/chui/ThinClient.java (see watchCtrlC())
com/goldencode/p2j/ui/client/TypeAhead.java (search on Key.VK_CTRL_C)
com/goldencode/p2j/ui/client/Watcher.java

I hope that you can simply deliver the CTRL-C event to the P2J client side and let the rest of the infrastructure execute as is coded.

2. It is unknown if we need a full remote procedure call (RPC) implementation. I would like to avoid it if possible, BUT I don't want to make a poor solution just to avoid such an implementation. Our standard P2J protocol (see the net package for the Protocol, Queue, Dispatcher and Message classes) is message based and we implement both synchronous (RPC) and asynchronous (events) using the same transport.

3. As mentioned above, the message encoding must be very efficient and we already know we want to go binary. There are a couple of existing open source options that can mix JSON with a binary transport (UBJSON and messagepack).

4. On the javascript side, we may want (or even need) to use Shared Workers (background threads) and MessagePorts/MessageChannel to implement the CTRL-C notification. Or maybe that is not necessary.

Start by focusing on the requirements for the protocol (based on what services the java driver will need and what services the javascript side will need). Document your findings here. You should also investigate the binary JSON options. If those are not useful/appropriate, we an write our own.

Something else to consider with the protocol is that the GUI driver/javascript will have more significant requirements (and a different driver interface) than can be seen right now with the ChUI driver.

Mouse events (movement, clicks...), fonts, images, drag and drop and a whole different level of drawing will be needed at a minimum.

Code Review 0107a

I am fine with the changes. When it passes regression testing, please check it in and distribute it.

Ref. #2201 implement the web client process spawning and server redirect.
Passed the regression tests.
Committed revision 10429.

added src/com/goldencode/p2j/main/ClientBuilder.java
added src/com/goldencode/p2j/main/ClientBuilderOptions.java
added src/com/goldencode/p2j/main/ClientBuilderParameters.java
modified src/com/goldencode/p2j/main/ClientCore.java
added src/com/goldencode/p2j/main/ServerKeyStore.java
added src/com/goldencode/p2j/main/Spawner.java
added src/com/goldencode/p2j/main/SpawnerImpl.java
added src/com/goldencode/p2j/main/SpawnerListener.java
modified src/com/goldencode/p2j/main/StandardServer.java
added src/com/goldencode/p2j/main/TemporaryAccount.java
added src/com/goldencode/p2j/main/TemporaryAccountPool.java
added src/com/goldencode/p2j/main/TemporaryAccountWorker.java
added src/com/goldencode/p2j/main/UpdateAccountTask.java
added src/com/goldencode/p2j/main/WebClientSpawner.java
modified src/com/goldencode/p2j/main/WebHandler.java
modified src/com/goldencode/p2j/security/SecurityCache.java
modified src/com/goldencode/p2j/security/SecurityManager.java
modified src/com/goldencode/p2j/ui/client/chui/driver/swing/ChuiSimulator.java
modified src/com/goldencode/p2j/ui/client/chui/driver/web/ChuiWebDriver.java
added src/com/goldencode/p2j/ui/client/chui/driver/web/WebScreenDriver.java
added src/com/goldencode/p2j/ui/client/chui/driver/web/chui_web.html
modified src/com/goldencode/p2j/ui/client/swing/SwingHelper.java
modified src/com/goldencode/p2j/web/GenericWebServer.java
modified src/native/makefile
added src/native/makefile.spawn
added src/native/postbuild.sh
added src/native/spawn.c
added src/native/winspawn.c
modified build.xml

1. WebSockets are full-duplex communication channels over TCP. So we have to implements a communication protocol on server side in Java and on the client side using JS.
Maybe a short research on how to use GZIP with embedded jetty could be useful.
Do you think that a JSON and GZIP could be an alternative to binary compressed JSON?

2. At this stage I have a general overview over the implementation issues.
On the client side the protocol should looks like:

- The input events from keyboard and mouse are send over the network using the WebSockets and binary protocols toward chui web driver.
- The commands from chui driver on server side are send over the network toward the client page and rendered on the screen.
- The java script code on client side should manage a video memory array to render the output commands on a canvas, a screen cursor, the keyboard
and the mouse events as input commands send to server.

On the server side:
- The output commands from chui screen driver are send to client where are rendered on the screen or on the speaker (beep command)
- The inputs coming from client consisting in keystrokes and mouse events should be enqueued on the server side and dequeued on screen driver requests.

Regarding the keystrokes and CTRL-C handle in particular, according to my researches could be easily implemented to use the existing flaw with no changes in code.
  The ScreenDriver define a readKey() method used to get keystrokes from a type-ahead buffer using a ChuiSimulator instance. 
  Our ChuiWebDriver use an instance of ChuiWebSimulator. ChuiWebSimulator extends ChuiSimulator and override the readKey() method. 
  So far we have a simple implementation for readKey() method which should be improved if necessary.

3. Dojo toolkit is distributed as a zip file containing a lot of Dojo modules (short java script files)
The latest release is 1.9.2 which is around 13 Mb in size. (the compressed format) A new release 2.0 is expected earlier this year.
The best way to handle the Dojo modules inside p2j project is to put the zip file on the classpath (add to the list of dependencies inside the manifest file)
and access modules as classpath resources.
In order to do that I have to design a separate jetty Handler class which handle all Dojo targets and add this handler to the embedded server handlers collection.
I think that is better to use the Dojo toolkit wherever is possible in our java script code for example in WebSockets communication implementation, DOM manipulation etc.

4. I think also that we have to use canvas HTML5 API which in turn restricts the use of application only on web browsers HTML5 compliant.

Maybe a short research on how to use GZIP with embedded jetty could be useful.
Do you think that a JSON and GZIP could be an alternative to binary compressed JSON?

I am willing to consider this. However, I am generally concerned with any approach that encodes more data than necessary and uses compression to "fix" the issue. Ultimately, the network traffic is reduced but by trading CPU cycles.

We should also consider just implementing our own non-JSON binary message passing protocol where we handle the encoding/decoding ourselves.

2. At this stage I have a general overview over the implementation issues.

Yes, this all sounds like you are going in the right direction.

Dojo toolkit is distributed as a zip file containing a lot of Dojo modules (short java script files)
The latest release is 1.9.2 which is around 13 Mb in size. (the compressed format) A new release 2.0 is expected earlier this year.

We should not need the full distribution, especially for the ChUI support.

Dojo has 2 things that may help us:

http://dojotoolkit.org/documentation/tutorials/1.9/build/ - a build toolset to create a custom dojo distribution (that can be much smaller)
http://dojotoolkit.org/documentation/tutorials/1.9/modules/ - asynchronous module definition support to make code modular in a standard way

The best way to handle the Dojo modules inside p2j project is to put the zip file on the classpath (add to the list of dependencies inside the manifest file) and access modules as classpath resources.

I can see this may be needed. But I wonder if we can't customize/minimize our distribution using the Dojo build tools and get a simpler result. Please review and respond.

I think that is better to use the Dojo toolkit wherever is possible in our java script code for example in WebSockets communication implementation, DOM manipulation etc.

If we can write simple non-Dojo javascript to do a job, I prefer that option except for the following cases:

1. The javascript must be browser-specific and/or would have compatibility issues and Dojo has a clean solution.
2. The Dojo approach provides a much cleaner result because it has functionality that would have to be otherwise duplicated ourselves.

This means that the Dojo code may indeed be used often. But I don't want to default to writing Dojo code. Here is why:

1. Dojo changes rapidly and the more dependent we are upon it, the more effort there is to keep up.
2. High levels of dependency on Dojo will make it harder or more costly to get rid of it/replace it later, if ever the time comes where that is desired or necessary.
3. More code and complexity will be added to the project than may be strictly needed. With more code and complexity comes more bugs and a harder to maintain/understand code base.

I want to find the right balance. So the first time you plan to use Dojo for some particular feature, please consider the above and document (here) why it makes sense to use it, based on the criteria above.

4. I think also that we have to use canvas HTML5 API which in turn restricts the use of application only on web browsers HTML5 compliant.

Indeed, this is certainly true. It is not a problem.

1. I think that is better to start by designing our custom binary protocol. This means that we have to define the structure
of binary packages for each communication between web server and browser JS code.

2. The Dojo toolkit SDK allow us to do the following:

- To build a custom toolkit minimized and optimized containing only the used dependencies.
     The result is a zip file containing only an subset of Dojo modules. The size of the zip file and the modules
     could be reduced which is useful for us. I did a custom build today and it works fine.
     However because we are using an embedded web server all the resources (web pages, css files, js files, image files, etc.)
     should be delivered to browsers using jetty Handlers which are designed as Java classes.
     So far I started to design such a Handler for Dojo toolkit using a full build Dojo toolkit minimized (13 Mb) as a zip file resource.
     The requested JS modules are get from the zip file on request and serialized as HTTP response back to requester.
     Finally I have to substitute this Dojo toolkit with a custom build containing only the used dependencies.

- To integrate the user applications with Dojo toolkit. Because we decided to avoid Dojo dependencies as much as possible 
     this option is not in our interest.

3. For CHUI application we have to manage the following resources using WebSockets and JS code:

- Screen and cursor rendering. This could be done using HTML5 Canvas API without using any Dojo toolkit modules.

- Keyboard and mouse events. Here the Dojo toolkit could help us because this part depend on browser implementation so my suggestion is to use it.

- WebSocket communication. Here the Dojo toolkit added JS code (Long-Polling) to emulate WebSockets communications on web browsers (or servers)
    which doesn't support the standard WebSockets API. Because our jetty server support standard WebSockets API and
    the API is simple I think that we could design our custom JS standard module. (no other JS frameworks like Dojo).
    Also I think that we will use only WebSockets API no other protocols supported by Dojo toolkit.

3. For a GUI application Dojo toolkit modules could help us but for the moment I did no investigations on this area.

1. I think that is better to start by designing our custom binary protocol.

OK, good.

2. The Dojo toolkit SDK allow us to do the following:

OK.

Screen and cursor rendering. This could be done using HTML5 Canvas API without using any Dojo toolkit modules.

Yes.

Keyboard and mouse events. Here the Dojo toolkit could help us because this part depend on browser implementation so my suggestion is to use it.

Agreed, use Dojo here.

Because our jetty server support standard WebSockets API and the API is simple I think that we could design our custom JS standard module. (no other JS frameworks like Dojo). Also I think that we will use only WebSockets API no other protocols supported by Dojo toolkit.

Just like with HTML5 canvas, we are making it a requirement that only browsers that support websockets will be supported in our ajax driver. So it is OK to code our own approach directly using the standard websockets API instead of Dojo.

3. For a GUI application Dojo toolkit modules could help us but for the moment I did no investigations on this area.

This is a question we will answer later.

Code Review 0110a

I am fine with the general approach and how the code is progressing.

I only have 3 comments:

1. Although we don't have coding standards for Javascript, please consider that all Java coding standards are basically the same standards we should use for Javascript.

2. Please think about breaking the JS code into smaller modules. I'm not sure if we are at that point of needing it yet with tty.js, but I can see that we probably will need it before the chui JS client is done.

3. The HTML formatting should follow most of the same Java standards concepts. I know this may sound weird. But I think many of the standards make sense in general. For example, your HTML usually has no indents, but I would expect us to use indents of 3 spaces for each nested level like:

<!DOCTYPE html>
<html>
   <head>
      <script language="javascript" type="text/javascript" src="tty.js">
      </script>
      <script>
         window.onload=function()
         {
            tty.initSound('beep');
            tty.initScreen('screen', 25, 80, 'monospace', 20);
            tty.initWebSocket('wss://' + window.location.host + '${context}/ajax');
         };
      </script>
   </head>
   <body>
   </body>
</html>

The following classes are used to emulate a terminal:

ChuiWebDriver extends (ChuiScreenDriver implements ScreenDriver)
ChuiPrimitives extends DriverPrimitives
ChuiWebSimulator extends (ChuiSimulator extends JComponent implements Printable)

The following methods are used by the terminal driver:

ChuiScreenDriver: ================
O beep() -> sim.beep()
O clear() -> sim.clear();
I readKey() -> sim.readKey();
O setCursorStatus(boolean on) -> sim.setCursorStatus(on);

ChuiPrimitives: ===============
O append(String str, Color color) -> sim.append(data);
O cursorAt(int x, int y) -> sim.setCursor(x, y);
O cursorStay(int x, int y) -> sim.setCursor(x, y);
I getX() -> sim.getCursorX();
I getY() -> sim.getCursorY();
I screenHeight() -> sim.getRows();
I screenWidth() -> sim.getColumns();
O sync() -> sim.triggerRepaint(true);
O updateScreen(ScreenData screen) -> sim.replace(screen.asArray(), screen.getCursorX(), screen.getCursorY());

Translation of driver calls into simulator calls:

ChuiScreenDriver,ChuiPrimitives ChuiSimulator I/O Remarks
(ChuiWebDriver) (ChuiWebSimulator) ===============================================================================================
beep() beep() O
clear() clear() O
readKey() readKey() I
setCursorStatus(on) setCursorStatus(on) O
append(str, color) append(data) O
cursorAt(x, y) setCursor(x, y) 0
cursorStay(x, y) setCursor(x, y) 0
getX() getCursorX() I (update by mouse event)
getY() getCursorY() I (update by mouse event)
screenHeight getRows() I (from configuration)
screenWidth getColumns() I (from configuration)
sync() triggerRepaint(true) O
updateScreen(screen) replace(arry, x, y) O

A possible binary protocol might looks like:

(O)utput Protocol Packet ===============================================================================================
beep() 0x80
clear() 0x81
setCursorStatus(on) 0x82 // cursor on
0x83 // cursor off
setCursor(x, y) 0x83 x y // new x, y cursor position
setCursor(x, y) 0x83 x y // new x, y cursor position
triggerRepaint(full) 0x84 size cell cell cell ... cell // synchronize screen memory
// cell = [x, y, colour, character]
// should be efficient implemented
// the cell list should contains only the changed cells.

(I)nput Protocol Packet ===============================================================================================
KeyEvent 0x00 meta key // meta = CTRL, ALT, SHIFT etc.
// key = key code
MouseEvent 0x01 meta x y // meta = click, contextmenu, dblclick
// x, y = mouse cursor position (row and col not pixels)

It looks good (both the code in 0113a and the proposed protocol).

The code changes in 0114a look good. There is some missing javadoc, but otherwise things seem pretty much on target.

According to my researches the Color is not used to render the character, in other words the current implementation is for a monochrome terminal.
Please let me know if my assertion is true or not.

It is true for today. But Hynek is working on proper COLOR support in #2037 and so this is about to change (in the next 2 weeks).

Finally I decided to use a JSON format for payload because is more easily to manage in JS.
Binary payloads require more processing in JS which it's a very slow process.

I know that other implementations have found it possible to improve performance using binary transfers. Have you reviewed the contents of UBJSON (http://http://ubjson.org/)?

We can go text for now, but we may need to shift this approach to binary in the future.

Marius Gligor wrote:

So I tried to do a new conversion but unfortunately without success.

What command do you use for conversion? When converting directly under the uast/ folder, is best to pass an explicit file list to the ConversionDriver.

Should I do an update for testcases project from the remote repository?

Yes, you should update it (there are changes in some scripts, like server.sh, not just tests).

According to conversion handbook:

export P2J=/mag/p2j/to/p2j
export CLASSPATH=$P2J/build/lib/p2j.jar

java com.goldencode.p2j.convert.ConversionDriver -d2 f2+m0+cb ask.p primes.p

And what error do you get?

Marius Gligor wrote:

According to conversion handbook:

export P2J=/home/mag/p2j
export CLASSPATH=$P2J/build/lib/p2j.jar

java com.goldencode.p2j.convert.ConversionDriver -d2 f2+m0+cb ask.p primes.p

Make sure you are using the latest revision (and P2J is built). From the log, you have an outdated rules/annotations/record_scoping_prep.rules file.

My p2j project is updated and the build all is OK, no errors. However the conversion does not work.
The rules/annotations/record_scoping_prep.rules file size is 24.326 bytes and the date is 6 December 2013 13:58

Then I think there is some other p2j/ installation on the path; and I can't be of much help with that. Why I say this: from the conversion log, the failure is at this expression:

buf.registerBuffer(uniqueName, bufName, schemaName, javaName, getNoteString("dbname"), (level == -1), upPath("TRIGGER_PROCEDURE/KW_OLD") or upPath("STATEMENT/KW_ON/KW_OLD/"))

               buf.registerBuffer(uniqueName,
                                  bufName,
                                  getNoteString("force_dmo_alias"),
                                  schemaName,
                                  javaName,
                                  getNoteString("dbname"),
                                  (level == -1),
                                  upPath("TRIGGER_PROCEDURE/KW_OLD") or
                                     upPath("STATEMENT/KW_ON/KW_OLD/"),
                                  isNote("is_dynamic_tables") and
                                     getNoteBoolean("is_dynamic_tables"))

Search your drive for the record_scoping_prep.rules files and see which P2J has the incorrect file.

Thanks I fixed the problem.
I just copied the rules directory from p2j on testcases/uast/p2j/rules and the conversion was done.

Please note that you should always have a symbolic link in your project home (testcases/uast/) to the p2j/ directory that you are using.

I've started to implements the protocol for keyboard management.
I have good news! So far It's works fine see the attached picture.

That is AWESOME! I am very excited to see this come together. Great work!

Take the testcases/simple/server/server.sh from the repository. The problem is caused by the fact that the server is missing this JVM argument:

-Djava.system.class.loader=com.goldencode.p2j.classloader.MultiClassLoader

Now seems to be a serialization problem.

WARNING: {Reader} failure in reading loop
java.io.InvalidClassException: com.goldencode.p2j.util.NumberType; local class incompatible: stream classdesc serialVersionUID = -505777196529990035, local class serialVersionUID = 4677557220506711006
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at com.goldencode.p2j.ui.client.WidgetEntry.readExternal(WidgetEntry.java:116)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1704)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1342)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at com.goldencode.p2j.ui.ScreenBuffer.readExternal(ScreenBuffer.java:929)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1704)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1342)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at com.goldencode.p2j.net.Message.readExternal(Message.java:367)
at com.goldencode.p2j.net.SyncMessage.readExternal(SyncMessage.java:102)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at com.goldencode.p2j.net.Protocol$Reader.byteArrayToObj(Protocol.java:436)
at com.goldencode.p2j.net.Protocol$Reader.run(Protocol.java:347)
at java.lang.Thread.run(Thread.java:724)

com.goldencode.p2j.net.SilentUnwindException: Connection ended abnormally
at com.goldencode.p2j.net.Queue.transactImpl(Queue.java:1140)
at com.goldencode.p2j.net.Queue.transact(Queue.java:583)
at com.goldencode.p2j.net.BaseSession.transact(BaseSession.java:179)
at com.goldencode.p2j.net.HighLevelObject.transact(HighLevelObject.java:163)
at com.goldencode.p2j.net.RemoteObject$RemoteAccess.invokeCore(RemoteObject.java:1416)
at com.goldencode.p2j.net.InvocationStub.invoke(InvocationStub.java:97)
at com.sun.proxy.$Proxy4.standardEntry(Unknown Source)
at com.goldencode.p2j.main.ClientCore.start(ClientCore.java:260)
at com.goldencode.p2j.main.ClientCore.start(ClientCore.java:80)
at com.goldencode.p2j.main.ClientDriver.start(ClientDriver.java:223)
at com.goldencode.p2j.main.CommonDriver.process(CommonDriver.java:423)
at com.goldencode.p2j.main.ClientDriver.process(ClientDriver.java:1)
at com.goldencode.p2j.main.ClientDriver.main(ClientDriver.java:241)
Caused by: com.goldencode.p2j.net.ApplicationRequestedStop: Queue stop is requested by application
at com.goldencode.p2j.net.Conversation.block(Conversation.java:304)
at com.goldencode.p2j.net.Conversation.waitMessage(Conversation.java:257)
at com.goldencode.p2j.net.Queue.transactImpl(Queue.java:1126)
... 12 more

The problem is you have two distinct P2J builds - one for the server, and one for the client. And the serialVersionUID in the two builds (For a certain class) are not the same. You should always have a single P2J build and symlink to it as necessary, as the P2J server and the P2J client must use the same compiled sources.

OK Thanks. Now the SWING driver works fine.
I have configurations on my IDE to start the server and the client which helps on development.
I have to find a solution to change also my IDE run configurations.

• working dir: testcases/simple/client
• main class: com.goldencode.p2j.main.ClientDriver
• program args: client.xml; its content is (change the ports accordingly):
  

  <?xml version="1.0"?>
  <node type="client">
     <net>
        <server secure="false" />
        <server host="localhost" />
        <server secure_port="3334" />
        <server insecure_port="3434" />
     </net>
     <client>
        <driver type="swing_chui_frame"/>
        <chui rows="24"/>
        <chui columns="80"/>
        <chui background="0x000000"/>
        <chui foreground="0xFFA500"/>
        <chui selection="0x0000FF"/>
        <chui fontname="monospaced"/>
        <chui fontsize="12"/>
     </client>
  </node>
  

• JVM args: -Dfile.encoding="UTF-8" -XX:+HeapDumpOnOutOfMemoryError -Xdebug -Xnoagent -Djava.compiler=NONE -Djava.library.path=../../../build/lib/ (library path can be changed as needed)

Very cool stuff. Please use testcases/uast/keys.p to test out your key processing. Compare the results to running the Swing client and the NCURSES client.

By the way, our current ChUI client (swing and ncurses) does not support mouse events. You don't need to add support for mouse to the AJAX client at this time.

Marius: what happens when the browser is closed? Does the P2J client process get shut down properly?

This feature isn't yet implemented but will be. We have a notification on web sockets when the connection is closed on browser.
My intention is to implements here the code to terminate the client process.
Also other changes I have to made which are currently in my short list.
For example on CTRL-C the ClientCore#start loop is executed again and we should avoid to reinitialize the embedded server.
I did some temporary changes inside ClientCore class which allow me to run the client directly.
Also temporary the server embedded port is fixed to 5555 instead a random value.
That's why the ClientCore in not yet in my attached archive.
Finally I have to tests the entire flow by eliminating all temporary changes.

Code Review 0115a

I am generally fine with the changes. Minor feedback:

1. Some imports need to be shifted to *.

2. Please put a comment in tty.js that the socket module must be initialized before the keyboard module.

3. You can put the mouse code back into screen.js. I don't think it hurts anything to have it.

On Chrome seems to be a bug when secured connection (WSS) with self signed certificate is used.
https://code.google.com/p/chromium/issues/detail?id=119793

Code Review 0116a

Generally, I'm OK with the code. My only concern: the use of Runtime.getRuntime().halt(statusCode) seems a bit "heavy handed". What is the reason it is needed?

Using monospaced font results in an ugly screen so in JS I changed the monospaced with monospace font (see monospaced.png).

It is fine to have different defaults for the JS and Swing clients.

In regard to the cross-browser compatibility issues, please limit the effort you put in on that right now. For the near term, we are only going to support very recent versions of:

Firefox
IE
Safari
Chrome

Opera, mobile browsers and older browsers will be out of scope for now.

As to the versions of the above which we will support, we will pick those based on our requirements. So long as we limit our requirements to pretty common features (html5 canvas, html5 audio tag, websockets...), then we will be able to pick reasonable versions that already exist. But it may be needed to use very recent versions of these browsers (e.g. IE 10).

Our customers won't be rolling this out til late this year or next year at the earliest. I don't want to spend time on support for browsers that won't even be heavily installed at that time. Our customer has already agreed to this.

Marius Gligor wrote:

1. I found another way to terminate the client process on connection close. Just simple put an END-ERROR key (code 304) on the keyboard queue.
Doing so I force the ClientCore.start loop to exit and finally to terminate the client process.

I don't think this will work. The application logic can catch the raised ERROR condition and choose to ignore it. The fact that the client ends in your case is just a coincidence specific to that program. To catch the ERROR condition, use something like:

do on error undo, next:
  /* main logic goes here */
end.

I checked in the TerminalMenu class and we have this code for the Exit menu item:

      if (container.canExit())
      {
         exit.addActionListener(new ActionListener()
         {
            public void actionPerformed(ActionEvent evt) 
            {
               sim.quit();
               System.exit(0);
            }
         });
      }

Go ahead and use System.exit() to cause the process to halt.

Code Review 0117a

The changes look fine except for the END-ERROR approach already discussed above.

The use of System.exit() should result in the session termination anyway, so I don't think you need the session object to be there.

Obviously, there is still some work to go with getting the restart behavior right on a CTRL-C, but it is shaping up nicely.

Good stuff!

Code Review 0116b

The code looks good. Does ChuiWebSimulator really need to call super.quit()?

1. I did more investigations regarding the Google Chrome web browser.
Both Google Chrome and jetty teams works currently to implements WebSockets extensions.
Google Chrome is ahead and the latest release is not compatible with the current jetty implementation.
Here is a snapshot representing the handshake between browser and server.

Request URL:wss://192.168.1.15:5555/ajax
Request Method:GET
Status Code:101 Switching Protocols
Request Headers CAUTION: Provisional headers are shown.
Cache-Control:no-cache
Connection:Upgrade
Host:192.168.1.15:5555
Origin:https://192.168.1.15:5555
Pragma:no-cache
Sec-WebSocket-Extensions:permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
Sec-WebSocket-Key:J23OCuuwjQ4i/evCnDnNog==
Sec-WebSocket-Version:13
Upgrade:websocket
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36

Response Headersview source
Connection:Upgrade
Sec-WebSocket-Accept:S/YgcUNrTi05FKJdi+l0aO6PkJo=
Sec-WebSocket-Extensions:permessage-deflate, x-webkit-deflate-frame
Upgrade:WebSocket

As you can see Google Chrome already implemented the client_max_window_bits WebSocket extension but jetty not.
So we have to wait until all WebSockets extensions will be implemented by jetyy and than upgrade our library.

2. I did some small changes inside the java script screen module. I fixed all issues related to canvas rendering.
Fortunately the old Chrome 31 release remains on my Windows installation and can be used for tests.
Now the applications looks good on all major web browser Chrome, Firefox, IE, Opera.
I cannot do tests using Safari on Windows because the version installed on my laptop does not work like the new Google release.

3. Currently I'm working to implements the keyboard interface in JS.
In JS we have 3 events related to keystrokes which are fired in the following order:

keydown
 keypress
 keyup

The keypress event is fired only for keys mapped to printable characters not for modifier keys (CTRL, ALT, SHIFT).
This event provide the printable character code for the key pressed.
Also the event object does not contains any information related to CTRL ALT and SHIFT keys
because if keys like CTRL, ALT or SHIFT are pressed the keypress event is not fired at all.

The keydown event is fired on all keys but unfortunately we receive a key scan code not the mapped printable character code.
Even more not all the key scan codes are the same across the web browsers. (see http://www.javascripter.net/faq/keycodes.htm)
The mapping between scan codes and character codes depends on the keyboard layout (US, EN, RO etc.)
So I think that we have to design and implements also a key mapper for a US keyboard layout inside the JS code.
Please let me know your opinion related to this issue.

Please document the differences between Java/Swing key processing and the Javascript equivalent. It was my understanding that the two models were pretty close. And I thought that the event.shiftKey, event.ctrlKey, event.altKey and event.metaKey properties were defined on the keypress event object. Is that incorrect?

If those modifier properties are there, then it seems we could leverage our SwingKeyboardReader code. Perhaps we can simulate the Java events using the JS input and let the SwingKeyboardReader code handle the processing on the Java side.

I have not gotten to look at your latest code yet.

However, I think you have come to the wrong conclusion in regard to the keypress event. You can definitely get the state of the modifier keys. Please try out this code:

<html>
   <head>
      <title>keypress Event Tester</title>

      <script type="text/javascript">
         keyHandler = function(event)
         {
            text = "keypress event:<br>";

            for (attr in event)
            {
               text = text + attr + " = " + event[attr] + "<br>" 
            }

            element.innerHTML = text;
         }

         var element;

         if (window.addEventListener)
         {
            window.addEventListener("load", function()
            {
               document.addEventListener("keypress", keyHandler, false);
               element = document.getElementById("something");
            }
            , false);
         }
      </script>

   </head>
   <body>
      <div id="something">Type keys!</div>
   </body>
</html>

You can see that there are Boolean properties for each of the modifier keys. As far as I can tell, we should be able to use keypressed for the printable keys (eliminating the layout issues) and use keydown for non-printable stuff, which is just like we do in Java.

Perhaps the Dojo event processing is obscuring this? I deliberately just used basic JS for this example. I have only tested it in Firefox, but it works great there.

By the way, you re-uploaded mag_upd20140117b.zip when I think you intended to upload mag_upd20140121b.zip.

I Tested your code and here are my conclusions:

1. The code does not work at all on IE (Windows) web browser.
The event keypress is fired only for keys mapped to printable characters.
For example if CTRL or ALT key is pressed the event keypress isn't fired.

2. The code does works partial on Chrome (Linux and Windows).
- If CTRL key is pressed works only if event.defaultPrevented = true. For example it's works for CTRL-Q but does not work for CTRL-A, CTRL-B, CTRL-S etc.
- If ALT key is pressed the event is not fired.

3. The code does not work on Safari (Windows).
The event keypress is fired only for keys mapped to printable characters.
Foe example if we press CTRL-Q the browser is closed because on CTRL-Q is mapped to exit command.

4. On Firefox on the other hand it's works indeed but with a small change.
In the original code when you press CTRL-S for example the keystroke also open the browser "Save AS" dialog.
In order to prevent this behaviour we have to avoid the browser action execution associated with the event by adding the following line: event.preventDefault()
With this changes in place the code works as we expected.

<!DOCTYPE html>
<html>
<head>
<title>keypress Event Tester</title>

<script type="text/javascript">
         keyHandler = function(event)
      {
            event.preventDefault();

text = "keypress event:<br>";

for (attr in event)
{
               text = text + attr + " = " + event[attr] + "<br>" 
            }

element.innerHTML = text;
         }

var element;

if (window.addEventListener)
{
            window.addEventListener("load", function()
   {
               document.addEventListener("keypress", keyHandler, true);
               element = document.getElementById("something");
            }
            , false);
         }
      </script>

</head>
   <body>
      <div id="something">Type keys!</div>
   </body>
</html>

5. The event object structure is different on each web browsers. It contains common fields as well as custom fields specific to each web browser.
Another important difference between JS and Java: CTRL key in not processed in JS.
We have only the key code or char code for the key and a flag which store the CTRL key status (true - pressed, false - not pressed)

6. In conclusion the Firefox behaviour on keypress event seems to be a really useful solution for our purposes.
Unfortunately cannot be used for a cross web browsers solution.

OK, these are good findings.

Please review the DOM level 3 events specification: http://www.w3.org/TR/DOM-Level-3-Events/#event-type-input

I wonder if the keydown can be used for most processing but then for the printable chars the corresponding input event can be used.

The issue here is simply that we will constantly be having to add support for new keyboard layouts if we cannot find a better solution. I assume the customer uses a UK English layout. We have other potential customers around the world that will certainly have other layouts in use. Now if the time to see if we can find a more elegant solution. Obviously, I prefer this to be completely in JS.

But certainly we do know that the Java Swing code has already solved this issue. I don't just mean our code in SwingKeyboardReader. I mean that the Java platform solves this problem. Is there a way to leverage this? For example, if we detect the locale/layout in JS and then match that with the same locale in Java, we can force it into a Swing component using component.getInputContext().selectInputMethod(locale). From there we can potentially use that component to process the low level events that we create OR we can dynamically create a map (of low-level to high-level transformations) which can be sent down to the JS code.

Code Review 0123a

This is really, really good! I'm glad that you were able to work out an approach that is much more generic. Well done.

My only question: Should mapUSkeys() be renamed to something else, since this is no longer a US-only solution?

Answers to your questions:

The question here is: Does p4g process Unicode UTF-16 characters?

It can. But more importantly, when we are operating in Java we DO process using Unicode. So it is perfectly valid to have the JS code deal entirely in Unicode.

Where we output something that can have different codepages, we have to write code to handle that (to convert the Unicode data into the codepage-specific output). Likewise, where we read input that can come in multiple codepages, we must have code that is aware (so that it can translate from the input codepage into Unicode). If everything is already in Unicode, no translation is needed.

The only problem that I found is in Safari which does not set the location field on event object.
The last release 5.1.7 for Windows Safari is dated May 9 2012 and think that we could consider that Safari for Windows is obsolete.
The same observation for Opera, 2012 is the year of the latest release!

Old releases are not a problem. We aren't planning to support Opera at all right now and for Safari, we can choose a much newer version that does work.

Code Review 0124a

All the changes look good. I like the more modular approach.

Code Review 0124b

I prefer the non-Dojo version. My only question: does the document.addEventListener() work properly on IE (at least older versions didn't have it). If it is working properly there, then let's go with the non-Dojo version for now. Good stuff.

Can you please post a list of what is left to do on #2212?

1. We have to solve the CTRL-C issue to initialize the converted application without a new server authentication if possible.
2. I have to do some tests with IE and will be back to you.

It's works in IE but I have to fix a small issue. Event preventDefault() does not work properly with IE.
I tested both versions Dojo/Native and the problem is the same.
I'm going to fix this issue.

To disable default behavior you may have to do 2 things:

1. Prevent the default behavior. In browsers that support addEventListener() there is the event.preventDefault() which does this. IE in versions prior to IE9 used event.returnValue = false to cause the same result.

2. Stop the event from "bubbling" or propagation. The idea is that there may be other event handlers registered on the current DOM element or on one of the parent DOM elements. Unless you take some action, these other event handlers will be executed. In browsers that support addEventListener() there is the event.stopPropagation() which stops invocation of any event handlers registered on the PARENT DOM elements. Prior to IE9, IE had event.cancelBubble = true to achieve the same thing.

But the new DOM Level 3 Events supports a event.stopImmediatePropagation() which is even better because it stops invocation of additional handlers that are registered at the current DOM element (not just those registered at the parent DOM element).
Have you tried event.stopImmediatePropagation()?

One other idea: instead of registering for key events at the document level, I wonder if registering at the canvas may be better for cancellation purposes. On the other hand, it may create conditions where some events are not delivered if the keyboard focus is somehow placed outside of the canvas (e.g. with a mouse click).

Code Review 0124c

My only thought here is that we probably need to always preventDefault() and stopImmediatePropagation() for every event, since these events are meant to be consumed by us. For now, you only do this when you detect the modifier keys.

Starting from IE 9 preventDefault() is supported. We are interested on IE 10 and IE 11 because IE 9 does not support WebSockets.
In my code I'm using both preventDefault() and stopImmediatePropagation() but with no effect.
Anyway during this weekend I will try to find a solution.
One idea is to use a pop-up window for our JS simulator.
I saw that when a pop-up window is created we can eliminate menu bars and toll bars.
Do you believe that could be a solution?

I saw that when a pop-up window is created we can eliminate menu bars and toll bars.
Do you believe that could be a solution?

Possibly.

I have been considering this approach in general, as it has some other advantages. But I am worried about the disadvantages that may come along with this. Popup blocking configurations in the browser can interfere and there may be other issues. Please analyze the pros/cons of this approach and document them here.

Code Review 0125a

I don't want to remove all the server-side Dojo support that you already provided. Put that work back in so that when the time comes for us to use Dojo features, we will be able to easily do so.

About the unavailable keys, don't worry. It was on my list as a potential issue that might not be resolved. We will document this set of limitations and customer's can choose browsers knowing the limitations they will have to deal with.

With enough effort, we could solve this problem. But it would require a non-trivial bit of native code for each platform we would support (and probably some native code for each browser too). For example, most platforms provide an architected hook interface to intercept messages before they are delivered to the application's specific queue. On Windows:

http://msdn.microsoft.com/en-us/library/windows/desktop/ms632589%28v=vs.85%29.aspx

Since most (nearly all) GUI implementations are based on a message-passing design, this trick is something that is commonly available. But it is NASTY and FRAGILE. And in the end it is not our responsibility to make up for the design decisions of the browser developers.

At this time, just make sure you document the list of browser-specific key handling findings so that we don't have to do more research later.

Code Review 0125b

I like this version better from the perspective of how you made the code better separated.

But since the canvas approach doesn't solve any real problems for us AND it does add a new focus problem, I think that it is best to go back to registering the handlers on the document.

Code Review 0125c

I like it. The only thing I don't understand is why the tty.keymap.js is missing.

It is good, thanks.

B. Another solution is to use the existing RemoteObject to get a new account and do a re-logon. This scenario should be implemented.

How about a slight variation on this? We already return a boolean from the standardEntry() method. Let's just return back an object that contains both the boolean value and any re-logon/authentication info needed to re-establish the next session. That way, it minimizes changes.

Is any time-out value set for RemotObjects sessions?

The "original" temporary session was disconnected, so it no longer will work. But at the time standardEntry() returns, there is still a valid session and the data we would obtain via environment var can be just returned back using that existing session.

Code Review 0127a

I am fine with the code. It looks very good.

My only request is for tty.keyboard.js, tty.keymap.js, tty.screen.js and tty.socket.js. Please put back (or add if not already there) blank lines that separate "member" variables. For example:

   /* exports object */
   var my = {};    
   /* track escape status */
   var escape = false;                     
   /* track alt char status */
   var altChar = false;

should be:

   /* exports object */
   var my = {};    

   /* track escape status */
   var escape = false;                     

   /* track alt char status */
   var altChar = false;

Code Review 0128a

1. In ChuiWebSimulator, I would like to shift to an idle timeout (maxIdleTime) that is set programmatically instead of via an annotation. Using the 15 minute as a default is fine. Please add a mechanism to override the default using configuration in the directory.

2. The Result class is missing from the update.

3. In StandardServer, I am not convinced that the use of a ConcurrentHashMap for lockedAccounts is sufficient to make this code thread safe. There is too much editing of the contents from multiple locations for this to be safe. Please use synchronization to make this safe.

4. The TemporaryAccount now has state that is edited in multiple places. That class needs to be properly synchronized.

5. The MainEntry javadoc should specify that the uuid parameter is optional (and thus it may be safely passed as null) unless this is a web client.

The ConcurrentHashMap is fully thread safe by design and is internally synchronized.
The Java concurrent package is special designed to avoid the external synchronization.

Yes, I'm aware of this. My point is that it is only sufficient for some use cases. Where you are potentially adding/removing/editing contents from multiuple locations on multiple threads, it is not always safe. In this case, the things that make me nervous:

1. lockAccount() and unlockAccounts() are not atomic.
2. There are multiple remove points for the map.
3. The uuid is provided by the client, if there is ever a scenario where the same uuid is accidentially (or maliciously) reused within a short interval of time, the results would probably break the logic since it would be possible to replace a TemporaryAccount instance that is already there and which is not closed. If that were to happen, that temporary account would never be disabled and it wouldn't timeout. It could be used over and over to access the server.

I think this is a case for proper synchronization.

Sorry, but I don't find any place in this class to use synchronization.
The open and close methods are synchronized in TemporaryAccountPool class.
Other places?? Could you help me please with a short hint?

TemporaryAccount instances are created on one thread (in TemporaryAccountPool), updated on other threads (in UpdateAccountTask) and read on other threads (from StandardServer)...

This means it is unsafe to share the 3 members of the class (subject, password and timeout) without synchronization. Make sure that any access to those members are done from synchronized methods. It doesn't solve all access problems, but it ensures that the data being read/written is done atomically.

Code Review 0128c

The code looks good.

but just for safety is OK.

Exactly.

1. We are not use Dojo at this stage. Should I provide the Dojo jar dependencies on lib folder or we will add this jar later?

Go ahead and add this now so that it is easy to use later.

I have a new build for jetty 9 the latest available. Should we update the jetty jar file or we have to wait until the final release.

Update it now.

Code Review 0129a

There are no concerns. Please note that I don't plan to check in the Dojo build, into bzr. I assume that it can be present or not without any errors (unless we try to use it and it isn't there). Is that correct?

Please continue getting this tested.

For #2238: the following are some potential issues that we need to consider/research. Document your thoughts and findings here. If they are in fact real issues, then work to resolve them.

• What happens when the user forces the browser to exit (without exiting the converted app first)? We would want to make sure that we get a notification when this happens, so that we can tell the server-side to exit.
• Can we disable the browser's context menu?
• How do we handle the use of the browser's FWD and BACK buttons (and history in general)? Can we disable those?

The Dojo dependencies zip file is always in the CLASSPATH because is in the list of dependencies inside the manifest file.
The build of p2j is not affected if the presence of Dojo zip file in lib directory.
Also at runtime as long as Dojo dependencies are not use no problems occur.

When the page having the JS code running is loaded a WebSocket connection is open.
If the user close the browser the WebSocket connection is closed and the server is notified.
On close notification the client exit and the process is terminated.
Also once the WebSocket connection is open we have an idle time out defined.
On time out also the client force an exit and the process is terminated.
However if from vary reasons the WebSocket is not open (the page is not loaded, WebSocket are not supported, JS disabled, others) the process could remains on running state for an indefinite period of time.
Maybe for safety it's a good idea to design and implements something like a "watchdog" timer used to
force an exit and terminate the client if no WebSocket connection is open during a specified amount of time.
What do you think about this?

Maybe for safety it's a good idea to design and implements something like a "watchdog" timer used to force an exit and terminate the client if no WebSocket connection is open during a specified amount of time.

It is an excellent idea. Implement this, but don't include it in the current update. I want to commit the 0129a update as it is (assuming it passes testing).

The Dojo dependencies zip file is always in the CLASSPATH because is in the list of dependencies inside the manifest file.

You mean it is implicitly in the CLASSPATH because of the manifest file? But there are no negative effects for the actual file being missing (unless we change our code to use Dojo).

To avoid any confusions maybe is better to eliminate the Dojo zip from CLASSPATH (from manifest file).
We could add this later when we effectively use the Dojo dependencies.

To avoid any confusions maybe is better to eliminate the Dojo zip from CLASSPATH (from manifest file).

No, it is OK to leave it. I just wanted to confirm that my understanding was correct.

Regarding browser session and JS code.
The web browser are designed to navigate over the Internet and for security reasons have a lot of restrictions.
Most of them are related to JS code which directly interact with browser environment.

1. The browser context menu can be controlled from JS on all major web browsers IE, Firefox and Chrome:

// disable browser context menu
  document.oncontextmenu = function () 
{ 
    return false;
  };

Using this piece of JS code the context menu will be completed disabled.

2. The navigation buttons on the other hand cannot be controlled from JS to be hidden or disabled.
However it is possible to control the navigation from JS in many ways:

a). By using the following JS code on the page which is the target of the back navigation:

// prevent back navigation
  function noBack() 
{ 
    window.history.forward(); 
  }
  noBack();
  window.onload = noBack;
  window.onpageshow = function(evt) 
{ 
    if (evt.persisted) 
      noBack(); 
  }
  window.onunload = function() 
{ 
    void (0); 
  }

With this JS code in place when the user press the back button the back operation is executed
  but the JS code forward the request back to the page. (from B page back to A page and than from A page forward to B page)

b). A better solution is to open our TTY JS on a new page (pop-up window).
      This require an intermediate html page something like a "splash" window from which the pop-up window is created and open.
      Here we have also 2 options:

- Use JS and automatically create the pop-up window when the "splash" page is loaded.
        In this case the browser pop-up blocker blocks the pop-up window.
        The user should confirm that the pop-up window could be open.

- In order to avoid the pop-up blocker action we can put a link inside the "splash" window and when the user click the link
        open the pop-up window. In this case the pop-up blocker is inactive, no action is take. 

The problem on both cases is that we cannot close the "splash" window after the pop-up window is open. It's works
        on Chrome but IE ask for a confirmation and Firefox do nothings. Nevertheless we can keep both windows opened and hide the link on the "splash" window.
        On the pop-up window we can disable or hide a lot of objects like menu, context menu, scroll bars, etc.
        The only object that cannot be eliminated from the pop-up window is the address bar.
        I have also to do some tests in order to find out if it's possible to use short-cuts keys in pop-up windows to navigate back.

It is possible also to combine pop-up windows with JS code to avoid back navigation.

Currently whenever the user try to navigate back or reload (refresh) the page the web socket connection is closed and the client process is terminated.
The user should go to the authentication page and restart a new session.
A good idea is not to terminate the process when the user refresh the page and close the socket connection but instead we should use the "watchdog" timer.

On web sockets close - start (arm) the watchdog timer.
On web socket open - stop the watchdog timer.

In other words the exit statement should be executed only when the converted application is ended (F4 for example)
or by the watchdog timer if the web socket connection is not open after a specified amount of time.

Please check in and distribute the 0129a update. Make sure to post the bzr revision number here.

The changes inside the mag_upd20140129a.zip archive has been committed to bzr revision 10457

1. user accesses the https://localhost:<port>/chui, where the username/password for an existing account on the machine is needed
2. if these credentials are valid on the machine, it will run the command to spawn a new P2J Client.
3. the P2J Client will use the credentials read from the P2J_SUBJECT and P2J_PASSWORD environment variables (this will point to the temporary account) to authenticate. After control gets back to P2J Client, it will compute the redirect URI and return it back to the server, to redirect the user to that page.

1. in case there is a custom authentication hook, this will be bypassed.
2. after step 3, the Reader, Writer and Conversation threads remain on the context for the temporary account. This doesn't look right to me. Even when using the testcases project, the Conversation thread is still on the context for the temporary account. The correct account should have been the bogus account, specified at the configuration for the com.goldencode.p2j.security.GuestAccess auth plugin.

What we are missing is the implementation for this part from note 74 from Greg (can't tell from the comments if this is still WIP or not):

A better solution is to implement a temporary P2J session with the server. The idea: create one-time authentication credentials (some kind of random token/key) that can be passed (like our password info) via STDIO to the child process. If we can find a secure way to get it to the P2J client, the P2J client can use this to do a temporary connectDirect() to the server with those credentials, read the key store, initialize the web server and send back the URI. Then it would disconnect and drop into the normal processing. The major advantage of this approach is that it can be extended to the "remote client" scenario, which we will be implementing later. Another major advantage is that we plan to do something similar to allow batch mode P2J clients to be autostarted by the server.

1. using the temporary credentials, to start the web server. This needs to use a copy of the original BootstrapConfig instance, to not lose any explicit user/passwd passed via access:subject:id and access:password:user configurations (or other configuration).
2. if web server was started, connect again. This time, fallback using the normal processing: use auth plugin as necessary, let the user input credentials, etc.

Finally, when you get a chance, please document all the rights needed for the server account (i.e. standard account for the test server or gso for MAJIC). Currently, I had to remove the NetResource, AdminResource and DireectorResource security plugins to be able to populate the temporary account pool. The default security plugins should be:

       <node class="strings" name="resource-plugins">
         <node-attribute name="values" value="com.goldencode.p2j.security.SystemResource"/>
          <node-attribute name="values" value="com.goldencode.p2j.security.AdminResource"/>
          <node-attribute name="values" value="com.goldencode.p2j.net.NetResource"/>
          <node-attribute name="values" value="com.goldencode.p2j.directory.DirectoryResource"/>
      </node>

OK. I have to think to this issue.

At this moment what I understood is as follow:

1. Use the temporary account to connect to the server, import the server key, start server and redirect user.
2. Close the temporary account session.
3. Create a new session this time using the credentials provided by the user. In other words the ThinClient
should be initialized using this session and the MainEntry should be executed also using this session.

What it's not clear for me. In case of a custom authentication plug-in where the user should enter the credentials?
The plug-in is like a p4g program which provide a login form on the terminal, in this case in the web page?

Marius Gligor wrote:

At this moment what I understood is as follow:
1. Use the temporary account to connect to the server, import the server key, start server and redirect user.
2. Close the temporary account session.
3. Create a new session this time using the credentials provided by the user. In other words the ThinClient
should be initialized using this session and the MainEntry should be executed also using this session.

You understood correct.

What it's not clear for me. In case of a custom authentication plug-in where the user should enter the credentials?
The plug-in is like a p4g program which provide a login form on the terminal, in this case in the web page?

Yes, in the web page; in MAJIC case, the first thing the user will see will be the MAJIC login screen. If the system credentials are the same as the P2J credentials (when user/pass is used), we could reuse that, but this would require changes in the authentication code (and the custom auth plugin probably).

OK. In this case I have to do some changes on the ClientCore and potentially on other places.
I think that using the temporary accounts implementation is OK to authenticate for server key import and user redirection.
I have to use a copy of the original bootstrap configuration file for this purpose.
On CTRL-C in this case we don't need a temporary account credentials, I think that the Majic login plug-in should occur on the client web page.
Is this correct? If yes we have to revert also the changes form StandardServer.

Marius Gligor wrote:

I think that using the temporary accounts implementation is OK to authenticate for server key import and user redirection.

Correct, this is their purpose.

I have to use a copy of the original bootstrap configuration file for this purpose.

Correct.

On CTRL-C in this case we don't need a temporary account credentials, I think that the Majic login plug-in should occur on the client web page.

Correct, the custom login plug-in is invoked and must be shown on the client web page.

Is this correct?

CTRL-C in MAJIC doesn't unwind all the way back to the login screen - is caught and handled in the business logic, to keep the user in the main menu. In normal cases, when CTRL-C is not caught, it will need to act as the client is restarted - so yes, this will unwind all the way to the while (running) loop in ClientCore.start.

If yes we have to revert also the changes form StandardServer.

If the changes are no longer needed, they should be removed.

OK Constatin thanks. Tomorrow I will try to fix this issue.
Are you did tests with the web client? How it's works?

1. The browser context menu can be controlled from JS on all major web browsers IE, Firefox and Chrome:

Go ahead and implement this.

2. The navigation buttons on the other hand cannot be controlled from JS to be hidden or disabled.

For now, I want to avoid the problems that come with a popup window.

We could put an implementation of 2a (JS approach) in ui/client/chui/driver/web/chui_web.html. This is the previous page in our case, right?

Another approach is to edit/control the browser's history. If the previous entry in the history is the current page, this may be a solution to the back/forward problem. Please see:

https://developer.mozilla.org/en-US/docs/Web/Guide/API/DOM/Manipulating_the_browser_history

I have also to do some tests in order to find out if it's possible to use short-cuts keys in pop-up windows to navigate back.

Please do let me know the results of your tests here.

Currently whenever the user try to navigate back or reload (refresh) the page the web socket connection is closed and the client process is terminated.
The user should go to the authentication page and restart a new session.
A good idea is not to terminate the process when the user refresh the page and close the socket connection but instead we should use the "watchdog" timer.

Hmmm. This is a good find on the refresh issue. I had not thought of that.

Some questions:

1. Does the use of the 2a JS "solution" still cause the back to sever the web socket connection?
2. Can the browser history manipulation help here?
3. Can we get a notification on refresh? If so, can we cancel it?
4. Can we make the websocket survive by using a separate worker thread (and separate window object) as documented in note 184 point number 4?

1. Put screen color message statement isn't implemented on the web client. I have to check how to do that. Could you please attach a picture for this issue?
2. The cursor could be "solid" or "line" blinking and not blinking, visible or hidden according to the cursor settings (see the JSON used for initialization inside index.html).
3. Greg told me to keep the mouse events but it is not a problem to cut off.
4. Rendering is another issue that is browser and platform dependent. In this case we have to do some small changes in the drawing code to fix the problem.
5. I think that we can use the client log file. Maybe a good idea is to spawn the client and activate the log at this time.
6. I have no idea about this issue, maybe you can offer me more details like where is the file located, how is loaded etc.
I'm working only for 3 moths on this complex system and so far I have no enough knowledges about all aspects regarding the system capabilities.
7. This message cannot be eliminated as long as we use self signed certificates.

1. The redirection to a log file is not possible from ProcessBuilder because we redirect the stderr for spawner not for web client process.
Form spawner is also difficult and we have to found solutions for Linux and Windows.
However I found a better way which works like magic!
- I did the redirection from ClientDriver.main function the entry point of the client process. If the process is a web client
(if found the following string client:driver:type=chui_web in args list) than I constructed a file name like client_$user_$time.log
and I redirected the stderr to this file. The log file is created into client working directory.

2. The socket timeout is configurable. Nevertheless we have to improve this part (see watchdog proposals)