Project

General

Profile

Feature #1811

implement the AJAX client driver

Added by Greg Shah over 8 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Start date:
11/01/2013
Due date:
03/06/2014
% Done:

100%

Estimated time:
(Total: 308.00 h)
billable:
No
vendor_id:
GCD

mag_upd20131107a.zip (10.2 KB) Greg Shah, 11/11/2013 03:57 PM

mag_upd20131108a.zip (13 KB) Greg Shah, 11/11/2013 04:00 PM

mag_upd20131111a.zip (9.09 KB) Greg Shah, 11/11/2013 04:02 PM

mag_upd20131112a.zip (18.9 KB) Marius Gligor, 11/12/2013 12:09 PM

mag_upd20131112b.zip (18.8 KB) Marius Gligor, 11/12/2013 01:21 PM

mag_upd20131113a.zip (22.7 KB) Marius Gligor, 11/13/2013 10:06 AM

mag_upd20131114a.zip (52.8 KB) Marius Gligor, 11/14/2013 10:23 AM

problem_loading_page.png (41.1 KB) Marius Gligor, 11/14/2013 10:23 AM

problem_loading_page1.png (55.7 KB) Marius Gligor, 11/14/2013 10:58 AM

mag_upd20131114b.zip (54 KB) Marius Gligor, 11/14/2013 12:20 PM

mag_upd20131116a.zip (62.7 KB) Marius Gligor, 11/18/2013 11:11 AM

mag_upd20131118a.zip (11.9 KB) Marius Gligor, 11/18/2013 01:14 PM

mag_upd20131120a.zip (19 KB) Marius Gligor, 11/20/2013 10:10 AM

spawn.c Magnifier (3.42 KB) Marius Gligor, 11/21/2013 01:22 PM

spawn.c Magnifier (4.15 KB) Marius Gligor, 11/22/2013 11:38 AM

mag_upd20131127a.zip (3.46 KB) Marius Gligor, 11/27/2013 01:08 PM

mag_upd20131127b.zip (3.48 KB) Marius Gligor, 11/27/2013 01:18 PM

mag_upd20131128a.zip (3.5 KB) Marius Gligor, 11/28/2013 05:28 AM

mag_upd20131129a.zip (49.4 KB) Marius Gligor, 11/29/2013 10:28 AM

mag_upd20131202a.zip (49.5 KB) Marius Gligor, 12/02/2013 03:50 AM

mag_upd20131202b.zip (58.6 KB) Marius Gligor, 12/02/2013 12:39 PM

winspawn.zip (13.5 KB) Marius Gligor, 12/02/2013 01:04 PM

build.xml Magnifier (34 KB) Greg Shah, 12/02/2013 08:23 PM

makefile (5.22 KB) Greg Shah, 12/02/2013 08:23 PM

mag_upd20131205a.zip (64.7 KB) Marius Gligor, 12/05/2013 12:18 PM

mag_upd20131206a.zip (65.1 KB) Marius Gligor, 12/06/2013 05:18 AM

mag_upd20131206b.zip (68.1 KB) Marius Gligor, 12/06/2013 12:44 PM

mag_upd20131209a.zip (68.4 KB) Marius Gligor, 12/09/2013 09:26 AM

netbeans-7.4.desktop (265 Bytes) Marius Gligor, 12/10/2013 11:10 AM

mag_upd20131213a.zip (72 KB) Marius Gligor, 12/13/2013 11:20 AM

mag_upd20131216a.zip (74.1 KB) Marius Gligor, 12/16/2013 11:17 AM

mag_upd20131216b.zip (74.8 KB) Marius Gligor, 12/16/2013 01:24 PM

mag_upd20131217a.zip (75.1 KB) Marius Gligor, 12/17/2013 06:46 AM

mag_upd20131218a.zip (78.5 KB) Marius Gligor, 12/18/2013 12:20 PM

mag_upd20131218b.zip (78.3 KB) Marius Gligor, 12/18/2013 01:22 PM

mag_upd20131219a.zip (147 KB) Marius Gligor, 12/19/2013 08:07 AM

mag_upd20131219b.zip (147 KB) Marius Gligor, 12/19/2013 01:03 PM

mag_upd20131220a.zip (148 KB) Marius Gligor, 12/20/2013 01:48 PM

majic_1811.txt Magnifier (9.38 KB) Marius Gligor, 12/23/2013 08:36 AM

mag_upd20131223a.zip (148 KB) Marius Gligor, 12/23/2013 08:36 AM

mag_upd20140107a.zip (148 KB) Marius Gligor, 01/07/2014 11:42 AM

mag_upd20140110a.zip (28.4 KB) Marius Gligor, 01/10/2014 01:15 PM

mag_upd20140113a.zip (31.5 KB) Marius Gligor, 01/13/2014 12:28 PM

protocol.txt Magnifier (2.66 KB) Marius Gligor, 01/13/2014 12:33 PM

tty.png - screen capture ow web tty simulator (37.9 KB) Marius Gligor, 01/14/2014 12:07 PM

mag_upd20140114a.zip (44.1 KB) Marius Gligor, 01/14/2014 12:07 PM

primes.png (43.1 KB) Marius Gligor, 01/15/2014 06:57 AM

conversion.txt Magnifier (23.5 KB) Marius Gligor, 01/15/2014 07:15 AM

client_mag_6832.log Magnifier (4.63 KB) Marius Gligor, 01/15/2014 09:03 AM

server.log Magnifier (57.5 KB) Marius Gligor, 01/15/2014 09:03 AM

primes.zip (228 KB) Marius Gligor, 01/15/2014 10:54 AM

mag_upd20140115a.zip (52.7 KB) Marius Gligor, 01/15/2014 12:32 PM

monospaced.png (5.26 KB) Marius Gligor, 01/16/2014 01:36 PM

chromium.png (33.1 KB) Marius Gligor, 01/16/2014 01:36 PM

mag_upd20140116a.zip (54.1 KB) Marius Gligor, 01/16/2014 01:36 PM

mag_upd20140117a.zip (53.7 KB) Marius Gligor, 01/17/2014 12:10 PM

mag_upd20140117b.zip (53.7 KB) Marius Gligor, 01/17/2014 01:29 PM

mag_upd20140121a.zip (54.4 KB) Marius Gligor, 01/21/2014 10:31 AM

mag_upd20140121b.zip (54.6 KB) Marius Gligor, 01/22/2014 02:37 AM

mag_upd20140122a.zip - JS Dojo code (54.6 KB) Marius Gligor, 01/22/2014 07:51 AM

mag_upd20140122b.zip - JS native code (54.8 KB) Marius Gligor, 01/22/2014 07:51 AM

keyevents.zip (57.2 KB) Marius Gligor, 01/23/2014 12:01 PM

mag_upd20140123a.zip (54.7 KB) Marius Gligor, 01/23/2014 12:01 PM

mag_upd20140124a.zip - Dojo (55.2 KB) Marius Gligor, 01/24/2014 10:40 AM

mag_upd20140124b.zip - Native (55.2 KB) Marius Gligor, 01/24/2014 10:40 AM

mag_upd20140124c.zip (55.2 KB) Marius Gligor, 01/24/2014 01:48 PM

mag_upd20140125a.zip (51.8 KB) Marius Gligor, 01/25/2014 06:31 AM

mag_upd20140125b.zip (51.8 KB) Marius Gligor, 01/25/2014 09:46 AM

mag_upd20140125c.zip (55.2 KB) Marius Gligor, 01/25/2014 10:43 AM

mag_upd20140127a.zip (60 KB) Marius Gligor, 01/27/2014 08:30 AM

mag_upd20140128a.zip (78.3 KB) Marius Gligor, 01/28/2014 11:44 AM

mag_upd20140128b.zip (81.7 KB) Marius Gligor, 01/28/2014 01:13 PM

mag_upd20140128c.zip (81.7 KB) Marius Gligor, 01/28/2014 01:47 PM

mag_upd20140129a.zip (2.44 MB) Marius Gligor, 01/29/2014 02:05 PM

ctrl-c.zip (313 KB) Marius Gligor, 01/30/2014 11:08 AM

mag_upd20140131a.zip (23.1 KB) Marius Gligor, 01/31/2014 03:07 AM

artifacts.png (5.44 KB) Constantin Asofiei, 01/31/2014 05:33 AM

mag_upd20140131b.zip (23 KB) Marius Gligor, 01/31/2014 06:45 AM

menu.png (20.2 KB) Constantin Asofiei, 01/31/2014 07:40 AM

mag_upd20140131c.zip (26.7 KB) Marius Gligor, 01/31/2014 08:28 AM

mag_upd20140203a.zip (25.2 KB) Marius Gligor, 02/03/2014 09:45 AM

mag_upd20140203b.zip (2.68 KB) Marius Gligor, 02/03/2014 12:05 PM

menu.p_ok.png (56.9 KB) Marius Gligor, 02/03/2014 12:05 PM

mag_upd20140204a.zip (18.2 KB) Marius Gligor, 02/04/2014 09:40 AM

mag_upd20140205a.zip (22.7 KB) Marius Gligor, 02/05/2014 10:29 AM

mag_upd20140206a.zip (38.5 KB) Marius Gligor, 02/06/2014 11:45 AM

client_mag_18704.log Magnifier (240 Bytes) Marius Gligor, 02/07/2014 12:13 PM

color.txt Magnifier (3.75 KB) Marius Gligor, 02/10/2014 10:15 AM

chui_swing_color.zip (46.3 KB) Marius Gligor, 02/11/2014 11:56 AM

mag_upd20140211a.zip (11 KB) Marius Gligor, 02/11/2014 11:56 AM

mag_upd20140212a.zip (23.4 KB) Marius Gligor, 02/12/2014 12:42 PM

chui_web_color.zip (178 KB) Marius Gligor, 02/12/2014 12:42 PM

mag_upd20140213a.zip (25.5 KB) Marius Gligor, 02/13/2014 09:06 AM

client_mag_1392289900215.log Magnifier (1.29 KB) Marius Gligor, 02/13/2014 09:06 AM

mag_upd20140213b.zip (25.6 KB) Marius Gligor, 02/13/2014 11:22 AM

mag_upd20140213c.zip (25.6 KB) Marius Gligor, 02/14/2014 11:49 AM

mag_upd20140215a.zip (2.3 MB) Marius Gligor, 02/15/2014 01:55 AM

mag_upd20140217a.zip (2.34 MB) Marius Gligor, 02/17/2014 11:03 AM

mag_upd20140217b.zip (2.34 MB) Marius Gligor, 02/17/2014 01:18 PM

mag_upd20140219c.zip (2.35 MB) Marius Gligor, 02/19/2014 12:59 PM

mag_upd20140220a.zip (25.1 KB) Marius Gligor, 02/20/2014 09:54 AM

mag_upd20140221a.zip (26 KB) Marius Gligor, 02/21/2014 12:00 PM

mag_upd20140221b.zip (25.6 KB) Marius Gligor, 02/21/2014 02:26 PM

mag_upd20140224a.zip (26 KB) Marius Gligor, 02/24/2014 08:25 AM

font.html Magnifier (3.34 KB) Marius Gligor, 02/24/2014 08:25 AM

mag_upd20140225a.zip (19.5 KB) Marius Gligor, 02/25/2014 03:38 AM

server_20140225.log Magnifier (46.9 KB) Marius Gligor, 02/25/2014 12:44 PM

mag_upd20140225b.zip (23.1 KB) Marius Gligor, 02/25/2014 01:43 PM

client_logs_20140225.zip (970 Bytes) Marius Gligor, 02/25/2014 01:43 PM

mag_upd20140226a.zip (23.2 KB) Marius Gligor, 02/26/2014 06:17 AM

test_batch_stdout_no_redirect.zip (5.18 KB) Marius Gligor, 02/26/2014 06:17 AM

test_batch_stdout_redirect.zip (5.57 KB) Marius Gligor, 02/26/2014 06:17 AM

mag_upd20140226c.zip (24.2 KB) Marius Gligor, 02/26/2014 01:01 PM

mag_upd20140227a.zip (26.4 KB) Marius Gligor, 02/27/2014 09:02 AM

mag_upd20140226b.zip (23.2 KB) Marius Gligor, 02/27/2014 10:15 AM

mag_upd20140228a.zip (34 KB) Marius Gligor, 02/28/2014 11:16 AM

mag_upd20140228b.zip (34 KB) Marius Gligor, 02/28/2014 01:10 PM

mag_upd20140303a.zip (41.5 KB) Marius Gligor, 03/03/2014 11:51 AM

mag_upd20140305a.zip (45.6 KB) Marius Gligor, 03/05/2014 07:15 AM

mag_upd20140305b.zip (44 KB) Marius Gligor, 03/05/2014 12:56 PM

mag_upd20140306a.zip (44 KB) Marius Gligor, 03/06/2014 08:48 AM

mag_upd20140306b.zip (44 KB) Marius Gligor, 03/07/2014 04:10 AM

mag_upd20140310a.zip (85.3 KB) Marius Gligor, 03/10/2014 12:10 PM

mag_upd20140311a.zip (85.5 KB) Marius Gligor, 03/11/2014 05:18 AM

mag_upd20140311b.zip (85.4 KB) Marius Gligor, 03/11/2014 12:57 PM

mag_upd20140312a.zip (81.6 KB) Marius Gligor, 03/12/2014 03:24 AM

mag_upd20140312b.zip (81.9 KB) Marius Gligor, 03/12/2014 09:13 AM

mag_upd20140312c.zip (82.9 KB) Marius Gligor, 03/12/2014 12:04 PM

ansi.c Magnifier (71.8 KB) Marius Gligor, 03/17/2014 11:11 AM

mag_upd20140324a.zip (30.6 KB) Marius Gligor, 03/24/2014 12:33 PM

mag_upd20140324b.zip (30.6 KB) Marius Gligor, 03/24/2014 01:21 PM

mag_upd20140325a.zip (30.8 KB) Marius Gligor, 03/25/2014 06:02 AM

mag_upd20140326a.zip (49.4 KB) Marius Gligor, 03/26/2014 02:29 PM

mag_upd20140328a.zip (54.4 KB) Marius Gligor, 03/28/2014 01:50 PM

mag_upd20140402a.zip (65.7 KB) Marius Gligor, 04/02/2014 01:20 PM

mag_upd20140417a.zip (65.8 KB) Marius Gligor, 04/17/2014 07:53 AM

getting_started_with_p2j_development.html Magnifier (14.6 KB) Marius Gligor, 04/22/2014 01:01 PM

mag_upd20140424a.zip (63.3 KB) Marius Gligor, 04/24/2014 10:46 AM

vt100_test1.png (19 KB) Marius Gligor, 04/24/2014 10:46 AM

mag_upd20140425a.zip (64.6 KB) Marius Gligor, 04/25/2014 09:51 AM

mag_upd20140428a.zip (65.7 KB) Marius Gligor, 04/28/2014 01:14 PM

mag_upd20140429a.zip (65.8 KB) Marius Gligor, 04/29/2014 11:27 AM

mag_upd20140429b.zip (65.9 KB) Marius Gligor, 04/29/2014 11:48 AM

mag_upd20140429c.zip (66 KB) Marius Gligor, 04/29/2014 01:14 PM

mag_upd20140507a.zip (9.11 KB) Marius Gligor, 05/07/2014 05:46 AM

swing-process.png (10.3 KB) Constantin Asofiei, 05/07/2014 06:19 AM

mag_upd20140507b.zip (11.8 KB) Marius Gligor, 05/07/2014 06:57 AM

mag_upd20140507c.zip (11.9 KB) Marius Gligor, 05/07/2014 09:19 AM

mag_upd20140508a.zip (18.4 KB) Marius Gligor, 05/08/2014 01:47 AM

mag_upd20140508b.zip (23 KB) Marius Gligor, 05/08/2014 03:46 AM

mag_upd20140702a.zip (18.6 KB) Marius Gligor, 07/02/2014 12:05 PM

mag_upd20140703a.zip (19 KB) Marius Gligor, 07/03/2014 08:16 AM

paste_example.html Magnifier (1.81 KB) Marius Gligor, 07/04/2014 09:26 AM

copy_paste_example.html Magnifier (1.78 KB) Marius Gligor, 07/08/2014 12:51 PM

dialog.html Magnifier (633 Bytes) Marius Gligor, 07/08/2014 01:06 PM

mag_upd20140709a.zip (22.1 KB) Marius Gligor, 07/09/2014 11:46 AM

mag_upd20140711a.zip (22.3 KB) Marius Gligor, 07/11/2014 03:57 AM

mag_upd20140714a.zip (28.9 KB) Marius Gligor, 07/14/2014 05:05 AM

mag_upd20140714b.zip (28.9 KB) Marius Gligor, 07/14/2014 01:34 PM

ca_upd20141206d.zip (13.8 KB) Constantin Asofiei, 12/06/2014 03:28 PM

ges_upd20150507a.zip (579 KB) Greg Shah, 05/07/2015 06:48 PM

compare.sh Magnifier (22.2 KB) Greg Shah, 05/07/2015 06:48 PM

ges_upd20150508a.zip (580 KB) Greg Shah, 05/08/2015 09:21 AM

compare.sh Magnifier (22.4 KB) Greg Shah, 05/08/2015 10:05 AM

ges_upd20150508b.zip (582 KB) Greg Shah, 05/08/2015 10:05 AM

ges_upd20150508c.zip (582 KB) Greg Shah, 05/08/2015 10:52 AM

ges_upd20150508d.zip (594 KB) Greg Shah, 05/09/2015 12:06 AM

compare.sh Magnifier (22.8 KB) Greg Shah, 05/09/2015 12:09 AM

ges_upd20150509a.zip (594 KB) Greg Shah, 05/09/2015 07:03 AM

ges_upd20150515a.zip (596 KB) Greg Shah, 05/15/2015 12:22 PM

ges_upd20150515b.zip (596 KB) Greg Shah, 05/15/2015 02:30 PM

ab_upd20150516a.zip (12.4 KB) Anton Breaur, 05/16/2015 09:20 AM

ges_upd20150516a.zip (599 KB) Greg Shah, 05/16/2015 10:29 AM

compare.sh Magnifier - Use this to code review the update. (23 KB) Greg Shah, 05/16/2015 11:35 AM

rem.sh Magnifier - This deletes everything needed from a p2j directory so that the update can be applied. (3.17 KB) Greg Shah, 05/16/2015 11:35 AM

ca_upd20150516a.zip (4.97 KB) Constantin Asofiei, 05/16/2015 01:57 PM

ges_upd20150517a.zip (599 KB) Greg Shah, 05/17/2015 11:07 AM

ges_upd20150518a.zip (594 KB) Greg Shah, 05/18/2015 12:10 PM

compare.sh Magnifier (22.8 KB) Greg Shah, 05/18/2015 12:10 PM

ab_upd20150518a.zip (7.72 KB) Anton Breaur, 05/18/2015 02:01 PM

compare.sh Magnifier (339 Bytes) Anton Breaur, 05/18/2015 02:01 PM

ges_upd20150518b.zip (594 KB) Greg Shah, 05/18/2015 04:21 PM

fonts_js.zip - partial demo work for javascript string drawing (244 KB) Anton Breaur, 05/19/2015 07:06 PM

fonts_js.zip - Here is a bundle with the new approach (56.3 KB) Anton Breaur, 05/22/2015 07:54 AM

fonts_js.zip (57.6 KB) Anton Breaur, 05/25/2015 03:28 PM

copy_paste_example_20150713.html Magnifier (3.6 KB) Greg Shah, 07/13/2015 05:20 PM

ParentChildFrames.java Magnifier (1.26 KB) Greg Shah, 07/14/2015 02:37 PM

canvas_drawing_example_20150721.html Magnifier (4.87 KB) Greg Shah, 07/21/2015 06:20 PM

canvas_drawing_output_comparison.jpg (89.6 KB) Greg Shah, 07/21/2015 06:20 PM

GraphicsOpsExamples.java Magnifier (1.72 KB) Greg Shah, 07/21/2015 06:20 PM

WhyUseStrictDirective.pdf - "JavaScript’s Strict Mode and Why You Should Use It" http://cjihrig.com/blog/javascripts-strict-mode-and-why-you-should-use-it/ gives analysis of why "use strict" is safe to do. (80.1 KB) Sergey Ivanovskiy, 07/23/2015 02:26 PM

canvas_drawing_example_20150723.html Magnifier (13.2 KB) Greg Shah, 07/23/2015 06:08 PM

canvas_drawing_output_comparison_20150723.jpg (126 KB) Greg Shah, 07/23/2015 06:08 PM

GraphicsOpsExamples.java Magnifier (2.48 KB) Greg Shah, 07/23/2015 06:09 PM

example_of_javascript_canvas_anti_aliasing_effects_compared_to_java2d.png (7.69 KB) Greg Shah, 07/24/2015 10:01 AM

canvas_drawing_output_comparison_offset_tick_marks_20150724a.jpg (90.4 KB) Greg Shah, 07/24/2015 10:01 AM

canvas_drawing_output_comparison_tick_marks_with_translate_by_0.5_20150724b.jpg (89.7 KB) Greg Shah, 07/24/2015 10:01 AM

example_of_javascript_canvas_with_translate_0.5_anti_aliasing_effects_compared_to_java2d.png (8.52 KB) Greg Shah, 07/24/2015 10:01 AM

canvas_drawing_output_comparison_manual_pixel_overwriting_20150724.jpg (157 KB) Greg Shah, 07/24/2015 06:31 PM

canvas_drawing_example_20150724.html Magnifier (17.8 KB) Greg Shah, 07/24/2015 06:31 PM

1811p_diff.txt Magnifier (8.97 KB) Sergey Ivanovskiy, 07/27/2015 10:30 AM

canvas_drawing_example_20150727.html Magnifier (19.6 KB) Greg Shah, 07/27/2015 05:35 PM

canvas_drawing_output_comparison_20150727.jpg (161 KB) Greg Shah, 07/27/2015 05:35 PM

1811p_all_changes.txt Magnifier (15.5 KB) Sergey Ivanovskiy, 07/31/2015 08:29 AM

1811p_all_changes_1.txt Magnifier (16.3 KB) Sergey Ivanovskiy, 07/31/2015 01:54 PM

test.zip (37.7 KB) Sergey Ivanovskiy, 08/04/2015 06:03 PM

test_20150805.zip (40.6 KB) Sergey Ivanovskiy, 08/05/2015 12:54 PM

test_20150806.zip (22.2 KB) Sergey Ivanovskiy, 08/06/2015 05:33 PM

1811o_20150807_1.txt Magnifier (23.1 KB) Sergey Ivanovskiy, 08/07/2015 12:20 PM

1811o_20150807_2.txt Magnifier (21.9 KB) Sergey Ivanovskiy, 08/07/2015 04:52 PM

1811o_20150810_1.txt Magnifier (56.6 KB) Sergey Ivanovskiy, 08/10/2015 05:01 PM

1811o_web_chui_client.txt Magnifier (2.05 KB) Sergey Ivanovskiy, 08/11/2015 08:11 AM

1811o_fix_web_chui_client_1.txt Magnifier (1.77 KB) Sergey Ivanovskiy, 08/11/2015 10:29 AM

1811o_20150811_1.txt Magnifier (9.11 KB) Sergey Ivanovskiy, 08/11/2015 06:17 PM

1811o_20150812.txt Magnifier (2.9 KB) Sergey Ivanovskiy, 08/12/2015 12:40 PM

canvas_resize_copy_bits_test_20150812.html Magnifier (3.4 KB) Greg Shah, 08/12/2015 01:38 PM

clipboard_basic.p Magnifier (459 Bytes) Sergey Ivanovskiy, 08/13/2015 11:00 AM

simple_sm.png (14.7 KB) Sergey Ivanovskiy, 08/13/2015 11:36 AM

combo_box.combo_box9_1.p.png (15.5 KB) Sergey Ivanovskiy, 08/13/2015 12:22 PM

rectangle.rect_test2.p.png (14.2 KB) Sergey Ivanovskiy, 08/13/2015 12:22 PM

rectangle.rect_test2.p_done.png (14.2 KB) Sergey Ivanovskiy, 08/13/2015 12:22 PM

rectangle.rect_test6.p.png (13.7 KB) Sergey Ivanovskiy, 08/13/2015 12:22 PM

rectangle.rect_test7_1.p.png (13.1 KB) Sergey Ivanovskiy, 08/13/2015 12:22 PM

simple_alert_box.p.png (13.2 KB) Sergey Ivanovskiy, 08/13/2015 12:22 PM

toggle_box.gui.tbx_present.p.png (14.7 KB) Sergey Ivanovskiy, 08/13/2015 12:22 PM

toggle_box.gui.tbx_present.p_done.png (14.8 KB) Sergey Ivanovskiy, 08/13/2015 12:22 PM

tests_suite_conversion_trunc.txt Magnifier (66.5 KB) Sergey Ivanovskiy, 08/13/2015 12:51 PM

test_list (280 Bytes) Sergey Ivanovskiy, 08/13/2015 12:51 PM

font_face_load.zip (2.29 MB) Constantin Asofiei, 08/13/2015 01:09 PM

1811o_20150813_1.txt Magnifier (10.2 KB) Sergey Ivanovskiy, 08/13/2015 04:37 PM

1811q_20150814.txt Magnifier (10.2 KB) Sergey Ivanovskiy, 08/14/2015 08:41 AM

canvas_resize_move_zorder_test_20150814.html Magnifier (4.59 KB) Greg Shah, 08/14/2015 02:39 PM

canvas_resize_move_zorder_test_20150814.png (7.58 KB) Greg Shah, 08/14/2015 02:39 PM

paragraph_drawing.zip (719 KB) Constantin Asofiei, 08/15/2015 05:14 PM

p2j.strokes.js (4.57 KB) Sergey Ivanovskiy, 08/17/2015 10:22 AM

LineStrokeTest_20150818.zip (5.13 KB) Sergey Ivanovskiy, 08/18/2015 09:02 AM

LineStrokeTest_20150818.zip (6.11 KB) Sergey Ivanovskiy, 08/18/2015 12:49 PM

J2DWithStrokes.png (31.5 KB) Sergey Ivanovskiy, 08/18/2015 09:10 PM

BrowserWithStrokesNativeImpl.png (51.8 KB) Sergey Ivanovskiy, 08/18/2015 09:10 PM

BrowserWithStrokesCustomImpl.png (35.4 KB) Sergey Ivanovskiy, 08/18/2015 09:10 PM

GraphicsOpsExamples2.java Magnifier (5.07 KB) Sergey Ivanovskiy, 08/18/2015 09:10 PM

LineStrokeTest_20150819.zip (8.4 KB) Sergey Ivanovskiy, 08/19/2015 09:33 AM

BrowserWithStrokesCustomImpl.png (36.5 KB) Sergey Ivanovskiy, 08/19/2015 09:33 AM

BrowserWithStrokesNativeImpl.png (48.3 KB) Sergey Ivanovskiy, 08/19/2015 09:33 AM

J2DWithStrokes.png (31.5 KB) Sergey Ivanovskiy, 08/19/2015 09:33 AM

LineStrokeTest_20150819_2.zip (8.36 KB) Sergey Ivanovskiy, 08/19/2015 03:01 PM

1811q_strokes_2.txt Magnifier (39.4 KB) Sergey Ivanovskiy, 08/20/2015 10:42 AM

gui_web_client_first_almost_working_attempt_20150820.png (23.9 KB) Greg Shah, 08/20/2015 05:22 PM

better_drawing.png (18 KB) Constantin Asofiei, 08/21/2015 06:16 AM

save_translated_origin.zip (16.3 KB) Constantin Asofiei, 08/21/2015 06:17 AM

better_drawing_polygons.png (7.3 KB) Greg Shah, 08/21/2015 08:14 AM

hello_world_swing.png (6.73 KB) Greg Shah, 08/21/2015 08:25 AM

FixToResizeImages.png (11.7 KB) Sergey Ivanovskiy, 08/21/2015 11:53 AM

fix_to_resize_images_1.txt Magnifier (1.42 KB) Sergey Ivanovskiy, 08/21/2015 11:53 AM

overdrawing_in_the_gui_client.png (6.56 KB) Greg Shah, 08/21/2015 01:06 PM

colors_interpolations_1.txt Magnifier (2 KB) Sergey Ivanovskiy, 08/21/2015 04:22 PM

colors_interpolations_2.txt Magnifier (4.8 KB) Sergey Ivanovskiy, 08/22/2015 03:55 PM

Int32ReadWriteTest.js (4.95 KB) Sergey Ivanovskiy, 08/22/2015 04:14 PM

1811q_strokes_4.txt Magnifier (6.4 KB) Sergey Ivanovskiy, 08/24/2015 03:04 PM

hello_world_web_gui_vs_swing_gui_20150825.png (11.5 KB) Greg Shah, 08/25/2015 02:30 PM

log4jsConsole.png (227 KB) Sergey Ivanovskiy, 08/25/2015 03:49 PM

log4javascript_diff_1.txt Magnifier (3.46 KB) Sergey Ivanovskiy, 08/25/2015 04:08 PM

canvas_drawing_example_20150826.html Magnifier (19.3 KB) Greg Shah, 08/26/2015 09:21 AM

canvas_drawing_output_comparison_blue_20150826.jpg (161 KB) Greg Shah, 08/26/2015 09:21 AM

GraphicsOpsExamples.java Magnifier (4.81 KB) Greg Shah, 08/26/2015 09:21 AM

hello_world_web_gui_vs_swing_gui_20150826a.png (11.5 KB) Greg Shah, 08/26/2015 09:21 AM

web_gui_js.log Magnifier (19.8 KB) Greg Shah, 08/26/2015 01:14 PM

web_gui_java_stderr.log Magnifier (23 KB) Greg Shah, 08/26/2015 01:14 PM

swing_gui_java_stderr.log Magnifier (26.9 KB) Greg Shah, 08/26/2015 01:14 PM

p2j.screen_diff_1.txt Magnifier (7.81 KB) Sergey Ivanovskiy, 08/26/2015 05:39 PM

hello_world_web_gui_vs_swing_gui_20150826b.png (11.4 KB) Greg Shah, 08/26/2015 06:47 PM

canvas_drawing_example_20150826.html Magnifier (19.6 KB) Greg Shah, 08/26/2015 06:47 PM

p2j.screen_diff_2.txt Magnifier (8.47 KB) Sergey Ivanovskiy, 08/26/2015 07:29 PM

webClient.png (18.5 KB) Sergey Ivanovskiy, 08/26/2015 07:29 PM

1811q_diff_20150827.txt Magnifier (8.48 KB) Sergey Ivanovskiy, 08/27/2015 03:56 AM

1811q_diff_20150827_2.txt Magnifier (12.9 KB) Sergey Ivanovskiy, 08/27/2015 09:58 AM

javascript_test_of_96_dpi_assumptions.png (5.9 KB) Greg Shah, 08/27/2015 01:18 PM

java2d_test_of_96_dpi_assumptions.png (8.87 KB) Greg Shah, 08/27/2015 01:18 PM

canvas_w_96px_square.html Magnifier (3.2 KB) Greg Shah, 08/27/2015 01:19 PM

FontSizeExample.java Magnifier (1.59 KB) Greg Shah, 08/27/2015 01:19 PM

hello_world_web_gui_vs_swing_gui_20150827a.png (11.1 KB) Greg Shah, 08/27/2015 03:00 PM

close_button_corruption.png (1.48 KB) Greg Shah, 08/27/2015 03:00 PM

message_area_sticks_out_1_px_in_swing.png (3.27 KB) Greg Shah, 08/27/2015 03:00 PM

fonts_web_vs_swing.png (16.4 KB) Constantin Asofiei, 08/28/2015 09:20 AM

drawCloseButton.js (1.63 KB) Sergey Ivanovskiy, 08/28/2015 10:35 AM

GraphicsOpsExamples2.java Magnifier (9.79 KB) Sergey Ivanovskiy, 08/28/2015 03:55 PM

strokesJava2DTest.png (17 KB) Sergey Ivanovskiy, 08/28/2015 03:55 PM

strokesJava2DTest_antialias_on_off.png (19.1 KB) Sergey Ivanovskiy, 08/29/2015 04:10 PM

GraphicsOpsExamples2.java Magnifier (11.7 KB) Sergey Ivanovskiy, 08/29/2015 04:10 PM

caption_buttons.png (721 Bytes) Hynek Cihlar, 08/31/2015 12:13 PM

GraphicsOpsExamples2.java Magnifier (13.1 KB) Sergey Ivanovskiy, 08/31/2015 01:08 PM

CaptionButton_diff.txt Magnifier (1.25 KB) Sergey Ivanovskiy, 08/31/2015 01:08 PM

StrokeControl&LineCoveredWithSquares.png (20.3 KB) Sergey Ivanovskiy, 08/31/2015 01:19 PM

gui&webClients.png (24.4 KB) Sergey Ivanovskiy, 08/31/2015 05:28 PM

1811q_diff_20150901.txt Magnifier (13.9 KB) Sergey Ivanovskiy, 09/01/2015 08:50 AM

1811q_diff_20150901_2.txt Magnifier (10.6 KB) Sergey Ivanovskiy, 09/01/2015 02:39 PM

1811q_diff_20150901_3.txt Magnifier (7.83 KB) Sergey Ivanovskiy, 09/01/2015 04:18 PM

1811q_diff_20150901_4.txt Magnifier (3.32 KB) Sergey Ivanovskiy, 09/01/2015 04:58 PM

SetImageIconTest.java Magnifier (2.4 KB) Sergey Ivanovskiy, 09/02/2015 02:19 PM

SetImageIconTest.java Magnifier (2.84 KB) Sergey Ivanovskiy, 09/02/2015 03:43 PM

SetImageIconTest.java Magnifier (2.26 KB) Sergey Ivanovskiy, 09/03/2015 07:32 AM

1811q_20150906_1.txt Magnifier (24.8 KB) Sergey Ivanovskiy, 09/06/2015 07:51 PM

1811q_20150907_2.txt Magnifier (3.51 KB) Sergey Ivanovskiy, 09/07/2015 05:31 PM

1811q_201509008_1.txt Magnifier (12 KB) Sergey Ivanovskiy, 09/08/2015 06:21 AM

1811q_201509008_2.txt Magnifier (17.5 KB) Sergey Ivanovskiy, 09/08/2015 01:09 PM

1811q_set_title.txt Magnifier (2.33 KB) Sergey Ivanovskiy, 09/08/2015 06:04 PM

1811q_20150909_1.txt Magnifier (22.4 KB) Sergey Ivanovskiy, 09/09/2015 11:59 AM

TaskBarHorizontal.png (52.6 KB) Sergey Ivanovskiy, 09/09/2015 12:14 PM

taskBarVertical.png (61.8 KB) Sergey Ivanovskiy, 09/09/2015 12:14 PM

1811q_20150909_2.txt Magnifier (14.8 KB) Sergey Ivanovskiy, 09/09/2015 06:04 PM

window1_visible.png (28.2 KB) Sergey Ivanovskiy, 09/10/2015 05:25 AM

window1_hidden.png (25.5 KB) Sergey Ivanovskiy, 09/10/2015 05:25 AM

P2JGUIClient.png (53.5 KB) Sergey Ivanovskiy, 09/10/2015 09:47 AM

VirtualDesktopTest.png (36.6 KB) Sergey Ivanovskiy, 09/10/2015 09:47 AM

TaskBarChangedColours.png (44.8 KB) Sergey Ivanovskiy, 09/10/2015 10:10 AM

VirtualDesktop.zip (24.9 KB) Sergey Ivanovskiy, 09/10/2015 10:10 AM

1811q_20150910_1.txt Magnifier (12.8 KB) Sergey Ivanovskiy, 09/10/2015 12:59 PM

visibility_test.p Magnifier (2.32 KB) Sergey Ivanovskiy, 09/10/2015 03:53 PM

images_artifacts.png (10 KB) Sergey Ivanovskiy, 09/11/2015 08:02 AM

1811q_20150911_1.txt Magnifier (3.64 KB) Sergey Ivanovskiy, 09/11/2015 08:17 AM

combo_box_button_mostly_works.png (20.8 KB) Greg Shah, 09/11/2015 08:53 AM

1811q_20150911_2.txt Magnifier (5.19 KB) Sergey Ivanovskiy, 09/11/2015 03:34 PM

1811q_20150913_1.txt Magnifier (6.82 KB) Sergey Ivanovskiy, 09/13/2015 07:09 PM

1811q_20150914_1.txt Magnifier (2.28 KB) Sergey Ivanovskiy, 09/14/2015 10:07 AM

1811q_20150914_3.txt Magnifier (3.39 KB) Sergey Ivanovskiy, 09/14/2015 02:42 PM

1811q_20150915_1.txt Magnifier (1.14 KB) Sergey Ivanovskiy, 09/15/2015 12:01 PM

logs_MenuGuiImpl_217.txt Magnifier (3.42 KB) Sergey Ivanovskiy, 09/18/2015 11:00 AM

set_window_bounds_diff.txt Magnifier (6.43 KB) Sergey Ivanovskiy, 09/18/2015 01:07 PM

web_client_draw_logs.txt Magnifier (8.07 KB) Sergey Ivanovskiy, 09/18/2015 01:07 PM

web_server_draw_logs.txt Magnifier (156 KB) Sergey Ivanovskiy, 09/18/2015 01:07 PM

hello_world_p2j_swing_broken_20150918.png (7.5 KB) Greg Shah, 09/18/2015 01:36 PM

hello_world_4gl_windows_screen_capture.png (3.66 KB) Greg Shah, 09/18/2015 01:36 PM

keyboards.zip (14.4 KB) Sergey Ivanovskiy, 09/20/2015 04:56 PM

ExceptionListForTheSecondCondition.txt Magnifier (7.08 KB) Sergey Ivanovskiy, 09/21/2015 11:04 AM

editorWidget.p Magnifier (971 Bytes) Sergey Ivanovskiy, 09/22/2015 01:42 PM

keyboards.zip (38.7 KB) Sergey Ivanovskiy, 09/23/2015 05:55 PM

1811q_20150924.txt Magnifier (5.63 KB) Sergey Ivanovskiy, 09/23/2015 07:55 PM

1811q_20150924_2.txt Magnifier (14.6 KB) Sergey Ivanovskiy, 09/24/2015 10:22 AM

test.html Magnifier (11.7 KB) Sergey Ivanovskiy, 09/24/2015 10:22 AM

clip_test2.html Magnifier (822 Bytes) Constantin Asofiei, 09/25/2015 06:02 AM

test.html Magnifier (21.1 KB) Sergey Ivanovskiy, 09/25/2015 02:40 PM

keyboards_20150928.zip (6.76 KB) Sergey Ivanovskiy, 09/28/2015 02:49 PM

dll_build_test_windev01.png (71.9 KB) Sergey Ivanovskiy, 10/02/2015 04:30 PM

couldnt_load_dll_procedure.png (72.7 KB) Sergey Ivanovskiy, 10/02/2015 04:30 PM

Success.png (51.6 KB) Sergey Ivanovskiy, 10/03/2015 07:00 PM

Window_3D_border_fix_20151007.jpg - Window 3D border fix (25.8 KB) Eugenie Lyzenko, 10/07/2015 05:20 PM

status_line_fix_20151008.jpg - Fixed status line painting (16.3 KB) Eugenie Lyzenko, 10/08/2015 02:10 PM

caching_images_20151010.txt Magnifier (15 KB) Sergey Ivanovskiy, 10/10/2015 05:40 AM

caching_images_20151011.txt Magnifier (20.4 KB) Sergey Ivanovskiy, 10/11/2015 05:11 PM

caching_images_20151012.txt Magnifier (22.5 KB) Sergey Ivanovskiy, 10/12/2015 12:44 PM

clipboard_20151014.txt Magnifier (10.6 KB) Sergey Ivanovskiy, 10/13/2015 07:18 PM

clipboard_20151014_2.txt Magnifier (8.34 KB) Sergey Ivanovskiy, 10/14/2015 12:35 PM

demo_widgets.png (14.5 KB) Sergey Ivanovskiy, 10/14/2015 04:52 PM

p2j_poor_title_font_20151014.jpg - New font(not easy to read) (6.1 KB) Eugenie Lyzenko, 10/14/2015 07:58 PM

p2j_good_title_font_20151014.jpg - Old and nice font (6.51 KB) Eugenie Lyzenko, 10/14/2015 07:58 PM

exp_results.txt Magnifier (29.6 KB) Sergey Ivanovskiy, 10/15/2015 10:58 AM

keys_measure_test_diff.txt Magnifier (2.17 KB) Sergey Ivanovskiy, 10/15/2015 11:07 AM

clients_result.txt Magnifier (3.89 KB) Sergey Ivanovskiy, 10/15/2015 12:11 PM

server_result.txt Magnifier (1.65 KB) Sergey Ivanovskiy, 10/15/2015 12:11 PM

keys_measure_java_test_diff.txt Magnifier (5.18 KB) Sergey Ivanovskiy, 10/15/2015 12:15 PM

demo_widget_regression.png (17.8 KB) Sergey Ivanovskiy, 10/15/2015 03:35 PM

demo_widgets.p_p2j_20151015_1811s_v10969.jpg - GUI regression (61.7 KB) Eugenie Lyzenko, 10/15/2015 03:53 PM

putImagData_1.txt Magnifier (14.7 KB) Sergey Ivanovskiy, 10/16/2015 07:44 AM

first_step_1.png (24.5 KB) Sergey Ivanovskiy, 10/16/2015 07:57 AM

select_item_step.png (30.6 KB) Sergey Ivanovskiy, 10/16/2015 07:57 AM

fill_in_foreground_regression_10977.jpg - Regression in 10977 (7.76 KB) Eugenie Lyzenko, 10/16/2015 01:17 PM

mouse_20151019_1.txt Magnifier (16.2 KB) Sergey Ivanovskiy, 10/19/2015 08:41 AM

AskGui.png (24.1 KB) Sergey Ivanovskiy, 10/19/2015 10:38 AM

calc-static-chars.png (19.8 KB) Sergey Ivanovskiy, 10/23/2015 09:38 AM

web-calc-static-chars.png (23 KB) Sergey Ivanovskiy, 10/23/2015 09:38 AM


Subtasks

Feature #2200: enable a Jetty web server to be initialized and used within a client processClosedMarius Gligor

Feature #2201: implement the web client process spawning and server redirectClosedMarius Gligor

Feature #2212: design websockets protocol for bidirectional communication between the javascript presentation engine and the java-based JS UI driver (ChUI and GUI versions)ClosedMarius Gligor

Feature #2238: investigate and resolve issues related to browser "session management"ClosedMarius Gligor

Feature #2240: implement proper color support in the Swing ChUI and AJAX ChUI clientsClosedMarius Gligor

Feature #2247: interactive child process supportClosedMarius Gligor

Feature #2306: add copy/paste support to the web clientClosedMarius Gligor

Feature #2308: remote client launchClosedMarius Gligor

Bug #2929: test chrome browser supportClosedSergey Ivanovskiy

Bug #2930: test IE browser supportClosedSergey Ivanovskiy


Related issues

Related to User Interface - Bug #2244: web client performance Closed 02/17/2014 02/26/2014
Related to Runtime Infrastructure - Feature #2243: integrate spawn.c changes into winspawn.c Closed
Related to User Interface - Feature #2298: temporary account creation New 05/05/2014
Related to User Interface - Feature #2342: fix cross-domain cookie for web client on remote launch Closed 07/22/2014
Related to User Interface - Feature #1807: implement clipboard support Closed
Related to User Interface - Bug #2664: vim doesn't work as an interactive child process in either Swing ChUI or the web ChUI client New 08/28/2015
Related to User Interface - Feature #1794: implement font support Closed
Related to User Interface - Bug #2956: When switching windows with Web GUI task bar the source window is not deactivated Closed
Related to User Interface - Bug #2964: fix key processing for web chui in Firefox, Chrome, IE and Safari New 01/21/2016
Related to User Interface - Bug #2969: websocket keep-alive Closed
Related to User Interface - Bug #2968: disconnect/reconnect and manual reload must re-establish the websocket connection, needs to update the websock references in all GuiWebEmulatedWindows Closed
Related to User Interface - Feature #2967: web socket message sizes are limited to 32K, make sure that drawing ops and images larger than this are handled properly Closed
Related to User Interface - Support #2672: GUI web client performance improvements New
Related to User Interface - Feature #2673: support a kind of virtual desktop in the GUI web client WIP
Related to User Interface - Support #2674: list of HTML5, CSS and Javascript features in use in the web client (both ChUI and GUI) + minimum browser levels New
Related to User Interface - Support #2675: list of differences between the web GUI client and the original 4GL Windows GUI environment New
Related to User Interface - Bug #2931: test Safari browser support Closed
Related to User Interface - Feature #2666: web client login page improvements New
Related to User Interface - Feature #2683: reverse proxy implementation Closed
Related to User Interface - Bug #3250: win_spawn failure cannot create process Closed

History

#1 Updated by Greg Shah over 8 years ago

  • Target version set to Milestone 12

#2 Updated by Greg Shah over 7 years ago

In the following, the term AJAX is used as a shorthand for a javascript implementation that runs as a "web application", however we are planning to use websockets instead of XmlHttpRequest, so the term AJAX is not technically accurate.

The 4GL is inherently a fat-client environment. Many features of the implementation are designed such that the execution of a 4GL process (pro, mpro, prowin32...) assumes that the processing occurs within the context of a single user's account. For example, 4GL code can:

  • launch child processes
  • access the file system, printers and other devices
  • create buffers in memory and directly manipulate that memory
  • make calls to arbitrary shared library (DLL) entry points
  • use sockets

In a perfect world, we would be able to virtualize everything and run it in a single server on threads that maintain the proper user context. That is not possible. Each of these items above (and there are more than those listed) must run in the context of the current OS user account. For example, the security and permissions for child processes and file system access cannot be properly duplicated in a shared server environment. Likewise, when the 4GL client starts a "server socket", it might bind that socket to a specific port. If 4GLclients are started on different systems, this would be perfectly possible but it would not be something we could duplicate in an "everything is virtualized in the server" approach. DLL access is similar, since there can only be 1 copy of a DLL in a process at one time and if a DLL was not written to be thread safe/context aware on a per thread basis, then 2 users accessing a non-safe DLL from the same process could corrupt data and cause abends and worse.

The only correct way to implement this is on the client-side. Each P2J client thus has its own process, running in the context of the OS user account and this exactly replicates the execution environment of the 4GL, allowing us to replace all of these client features. Of course, the majority of the processing (business logic, database access...) is still done in the shared server (on threads with dedicated context for that user), but when something client-specific must execute, we remotely access that functionality on the client. More specifically, we call down to the client, from the server, using our remote object protocol.

This means that each user of the system must have their own process in which they run the client-specific portions of the application. To date, this has always meant a Java process, running our P2J client code.

What does all of this have to do with implementing an AJAX client?

The UI for any of our clients (ChUI or GUI) is by nature, a client-side thing. With the move to an AJAX UI, we will be moving the interactive portions (rendering/display of the UI and processing key/mouse input) of the UI into the browser. This is a core objective of the current project and represents a very interesting deployment scenario for virtually all customers with which we have ever spoken.

However, we don't plan to re-implement our entire client in javascript. For one thing, the P2J client code is too complex to easily port to javascript. Secondly, maintaining 2 implementations of the same client is not something we are interested in doing.

With this in mind, in addition to the "every user is a separate process" requirement, we have these other objectives:

  • Maximize the use of our current Java client code.
  • Maximize the use of common code (in Java) in the future solution.
  • Minimize the code on the Javascript side.

The above constraints suggest this approach:

All processing will be HTTPS.

The browser client will open a URL that is serviced by Jetty in the P2J server process. I'll call this the "server jetty". The URL will be something like https://p2jserverhost:jettyport/chui/ (for the AJAX ChUI) or https://p2jserverhost:jettyport/gui/ (for the AJAX GUI).

When the server jetty has an inbound connection for one of these AJAX URLs, it will do the following:

  1. dynamically start a new P2J client process (we already do something very similar in starting "appserver instances")
  2. this will be done by a special launcher process that is there to create child processes under the control of the P2J server
  3. the new client will be started in "browser mode" (ChUI or GUI) and the client code will be the standard P2J java client
  4. that P2J client will start up and will make an initial connection to the server's secure port using the P2J protocol (this is the "client jetty")
  5. the P2J client will have a Jetty instance in its JVM that is initialized to offer HTTPS support
  6. the P2J client will notify the P2J server about the URL for which the client jetty is listening, this will include a custom/dynamic port number
  7. the server jetty will respond to the original browser request with a page that redirects the browser to the new client jetty URL
  8. the browser will connect to the client jetty, which will send down a page that is the proper AJAX client (ChUI or GUI)
  9. that AJAX client will establish a secure websocket connection to the client jetty
  10. on the P2J client side, the "upper half" of the AJAX driver will be written in Java and it will map into the ChUI driver interface or the GUI driver interface
  11. on the browser side, the javascript client will be the "lower half" of the ChUI driver interface or the GUI driver interface, responsible for the rendering of specific windows, frames, widgets... and handling all user input (keys and mouse)
  12. these two halves of the AJAX driver will cooperatively process over the websocket to provide the same kind of UI services that could have been done with an in-process driver in Swing
  13. the P2J client will handle the majority of the processing and in fact it can be virtually the same as it is today, including all of the session setup/communications with the P2J server
  14. the only difference is that instead of using Swing for the low level drawing and input events, those aspects must be remoted over the websocket
  15. this means that the our current client architecture is used exactly as it is designed today, and it can handle all the same client side features like child processes, file system access, DLL calls and so forth
  16. those features are just implemented outside of the browser, and potentially on a different physical system, although technically it would be possible to have that client process be local to the same system on which the browser is running (this may be needed in some use cases where the client features in use require interaction with the user or some kind of device access that is not available on the server)

#3 Updated by Greg Shah over 7 years ago

For task #2200, the following must be reviewed/considered:

  • The startup process for the ClientDriver and ThinClient will be different with Jetty.
  • The AJAX mode for the client is basically a new driver implementation. That means it is going to be mutually exclusive with running any other driver. For example, in AJAX ChUI mode, it would replace the Swing ChUI client AND the JNI ("Console") ChUI client.
  • The client will listen on a random port number (above 1024) which will allow the port number to be allocated dynamically by the OS/Java and also for there to be no security restrictions on opening that port.
  • The client jetty port will have to be reported back to the server (so that the server jetty can construct the proper redirect page), but that back-channel communication will probably not be handled in #2200. We will have a subsequent task defined to implement the dynamic launch of the AJAX client and that back-channel comms will occur are part of the launch.
  • The client jetty must startup in SSL mode and we need to consider what keys are used. The server jetty uses the SSL context already loaded by the server's SecurityManager, which means it is backed by the keys that are configured in the server's directory. At a minimum, we want to enable the same server-configured keys to be utilized in the client jetty. Such a configuration is attractive because it will either allow a properly signed certificate to be re-used or the user's web browser will already have the necessary "certificate exceptions" loaded. In other words, we want to use SSL without having to do any per-client configuration in the common case.
  • However, we must also enable the cases where there is client SSL configuration (in our directory as the 1st choice and then possibly overridden through local files/bootstrap config). For example, if the clients are running on a separate host from the P2J server, then there can be certificate issues as a result.

#4 Updated by Greg Shah over 7 years ago

From Marius on November 4, 2013:

Regarding the implementation on P2J I read your specifications.
Here is what I understood so far:

1. The browser client will open a URL that is serviced by Jetty in the P2J server process. I'll call this the "server jetty". The URL will be something like https://p2jserverhost:jettyport/chui/ (for the AJAX ChUI) or https://p2jserverhost:jettyport/gui/ (for the AJAX GUI).

Q: The "server jetty" is in fact the WebServer or another instance, a new design?

2. Dynamically start a new P2J client process (we already do something very similar in starting "appserver instances")
This will be done by a special launcher process that is there to create child processes under the control of the P2J server
The new client will be started in "browser mode" (ChUI or GUI) and the client code will be the standard P2J java client
That P2J client will start up and will make an initial connection to the server's secure port using the P2J protocol (this is the "client jetty")

Q: The "client jetty" is started similar to ServerDriver#launchAppServer?

3. The P2J client will notify the P2J server about the URL for which the client jetty is listening, this will include a custom/dynamic port number
The server jetty will respond to the original browser request with a page that redirects the browser to the new client jetty URL
The browser will connect to the client jetty, which will send down a page that is the proper AJAX client (ChUI or GUI)
That AJAX client will establish a secure websocket connection to the client jetty

Q: The P2J client is a new process allocated per connection having an embedded jetty server which should deliver
Web Sockets Services and the main page?

Q: The port number should be generated by an algorithm randomly or in sequence?

Q: The jetty protocol is wss over https?

Q: The SSL context is the same as the "server jetty" or if in the directory is registered a SSL context
(keystore and credentials) for this purpose use this one as a first option?

#5 Updated by Greg Shah over 7 years ago

From Greg on November 4, 2013:

Q: The "server jetty" is in fact the WebServer or another instance, a new design?

I don't see any important advantage to have another instance. My plan was to just have a different URL (than the admin console) but to extend the same WebServer.

If you have any reasons to do something different, let me know.

Q: The "client jetty" is started similar to ServerDriver#launchAppServer?

Not exactly. That code is related, but I am referring to the server-side code in AppServerLauncher.java. That is the code that really does the launching work. Specifically, look at builder() and launchWorker().

But the development for client launching part will be done in another task than #2200.

Q: The P2J client is a new process allocated per connection having an embedded jetty server which should deliver Web Sockets Services and the main page?

Exactly.

Q: The port number should be generated by an algorithm randomly or in sequence?

I don't want to allocate it at all. Let the operating system allocate it dynamically. We can read the allocated port number from the resulting socket instance and send it back to the server as part of the complete URL.

Q: The jetty protocol is wss over https?

Let's not call it the "jetty protocol", but rather the "web client protocol".

Yes, it is wss from an https session.

Q: The SSL context is the same as the "server jetty" or if in the directory is registered a SSL context (keystore and credentials) for this purpose use this one as a first option?

Yes, exactly.

#6 Updated by Greg Shah over 7 years ago

From Marius on November 5, 2013:

Today I started to design and implement the AJAX Client Driver.
So far I designed the embedded jetty SSL server and a generic
web socket handler used to register web sockets on the server context.

#7 Updated by Greg Shah over 7 years ago

From Marius on November 7, 2013:

Web Sockets support - the current design

Each client running on server has an embedded jetty server which offer web sockets support.

package com.goldencode.p2j.ws;

1. WsServer - implements the embedded jetty server.
- The server is running on the default SSL context but if a BootstrapConfig structure is provided the SSL
context will be constructed based on this bootstrap configuration.
- The port number could be specified explicit or could be be dynamically allocated by the OS when the port = 0.
- The host name could be null case on which the server will listen on all available addresses.
- On each server a context path (default /) could be defined.
Example:
https://localhost:8443/
https://localhost:8443/contextPath
- The server provide a method to add Handlers for web resources like HTML pages.
- The server provide a method to register web sockets based on a target string and the web socket class.
- All Handlers are registered into the embedded server context.

2. WsClient - is a contract (Interface) that should be implemented by each client.
He provide two methods for start-up and shut-down the embedded server.
It is possible to transform WsClient into an abstract class that must be extended by the client.
This is not a flexible design if the client implementation should extend another class.

3. TestPageHandler - is an implementation of a Jetty Handler for delivering HTML files resources.
This class does not provide placeholder substitution. If we need placeholder substitutions
just extended this class and override the method handleReplacements in order to provide a custom
code for substitution. (see the TestPageHandler class). Each handler define a target string
which is part of the URL path.

package com.goldencode.p2j.ws.test;

This package provide a simple example of a client which expose web sockets services via an embedded server.

1. TestWsClient is a very simple client implementing the WsClient interface.
An embedded WsServer instance is created, configured and started.

2. TestPageHandler is a Handler for index.html file. The file is parsed and some placeholders are
substituted before to deliver the page. This is basically the main page.

3. TestWsServlet is a Web Socket simple class.
Jetty offer the following methods to create Web Sockets:

a. POJO - Using WebSocket annotations.
b. Implements WebSocketListener interface.
c. Extends WebSocketAdapter class.

This design has been tested by creating an instance of TestWsClient inside the WebServer
class after the web server start statement:

new TestWsClient().startup(null);

and found to work properly.

On the web browser open: https://localhost:8443/client/page and click on the link inside the page.

I did also tests having two web sockets registered and found to work properly.

#8 Updated by Greg Shah over 7 years ago

From Greg on November 7, 2013:

Initial Feedback

I realize that this is a work in progress and an early version. Thank you for posting it and the useful description of the approach. I will refrain from a full code review because this is still early and will change a great deal.

I like what you are doing in general. There is some very good stuff here. I have the following questions/comments:

1. WsServer has very little to do with WebSockets. Other than the WsHandler inner class and the addWsHandler() method, the code is otherwise quite generic. In fact, it seems to me that it is very, very similar to our current com/goldencode/p2j/main/WebServer. I prefer to maximize common code here, since the server jetty and client jetty will both have similar configuration requirements and will need to support similar usage patterns.

a. I would like to enhance or extend our current WebServer to be used as both the server jetty and client jetty. This eliminates having a 2nd implementation that is so similar to the code we are already maintaining. To the degree that we need to provide more control over the startup/configuration/shutdown of the WebServer class to make it more flexible, we should do so. It also makes it easier to add WebSockets support to the server jetty, which is something will need in the future as we shift our admin console to AJAX.

b. I'd like to see the WsHandler code moved out of that class and put in its own class. The registration code can be hidden there too (to register itself with the WebServer instance). In addition, please call it WebSocketsHandler because the "ws" abbreviation can mean too many different things.

2. I don't understand the need for the WsClient interface. We can always create a helper class for providing common code for the startup/shutdown of the client jetty. It doesn't need to be in the inheritance hierarchy of the client driver. I prefer that approach because there are less layers of abstraction.

3. If I understand the code properly, all the "static" HTML can be loaded from our p2j.jar, right? That is exactly how we want this to work, since the code will only change when we have a revision of P2J. That avoids maintaining separate files in the filesystem which is very good.

4. I like the handleReplacements() support you have provided. It is useful and will be needed in the future. However, for the specific use case in your test code, I think you can use the javascript location object to get the host, port and so forth.

#9 Updated by Greg Shah over 7 years ago

From Marius on November 8, 2013:

Release candidate after your feedback.

1.a The embedded server was renamed as GenericeWebServer. This class could be extended or embedded.
I changed the old WebServer class to benefit from the new design. Now the WebServer extends
the GenericWebServer and can expose web sockets services.

1.b The Web Socket generic handler is no longer an inner class. The name of the class is PojoWebSocketHandler
I preferred this name because PojoWebSocketHandler already extends the jetty WebSocketHandler class.

2. I called the "jetty client" server Agents. I understood that an Agent is lunched by an instance of a
ProcessBuilder class which is designed to create OS processes. I our case an Agent is a Java class
having a main method and the ProcessBuilder will construct and lunch a command like:

java -cp bin;lib/lib1.jar;lib/lib2.jar com.example.AgentClass
The agent will run as a separate process having its own JVM isolated from other processes.
Using the command line arguments passed (server ports and others) as parameters on the main method the Agent
will be able to build an interface in order to communicate with the server via p2j RMI.
For testing purposes I created a WebSocketAgent which extends the GenericWebServer and have a main method.
Until the luncher of agents will be designed and implemented I tested this agent in the context of the server.

3. The handler for static pages has been designed to load the html pages for the jar file
(a resource on the classpath). Nevertheless other handlers could be desigen to serve
other resource from different URI's.

4. I'm using window.location.host in JS instead placeholders, thanks for this hint!
Only the ${context} placeholder is resolved on the handleReplacements().

Please let me know if you agree the new design.
Also I would like to ask you if I should move the classes from my ws package
to the main package and perhaps the ws.test package to ws.

#10 Updated by Greg Shah over 7 years ago

From Greg on November 8, 2013:

Feedback

This is very good. Some thoughts:

1. Change the "ws" package to "web". This will be the place for generic web server classes.

2. As you get a real client implementation going, you will not need the test package anymore. You can keep it for now, for development purposes, but the final update should not have it.

3. Make sure you are reviewing the ClientDriver, com/goldencode/p2j/ui/chui/ThinClient (in regards to how it integrates with the "drivers" and also how it does much of the startup processing) and the ChUI drivers (com/goldencode/p2j/ui/client/chui/driver/*). These will help you design the changes for how the ChUI web client should be implemented.

4. In regards to coding standards, we now have a 98 character line limit instead of 78 (docs aren't up to date yet). Also, please make sure that all methods and members are javadoc'd. There will be more details on this later, when the code is nearing completion.

Keep going. It is good work.

#11 Updated by Greg Shah over 7 years ago

From Marius, on November 11, 2013:

Today I did a lot of tests regarding the ClientDriver and ThinClient.
I'm able to starts and debug from my IDE both instances, the server and the client.
Here is what I understood from this tests. Please let me know if something are missing.

1. The ClientDriver is a remote client started from a command line which run on separate process having
only one thread, the main thread.

The command line arguments define the ScreenDriver type and parameters.
Also the server host and port are specified as command line parameters.
It is possible to provide the same parameters using a configuration file like client.xml

net:server:host=localhost
net:server:insecure_port=3333

client:driver:type=swing_chui_frame
client:chui:rows=24
client:chui:columns=80
client:chui:background=0x000000
client:chui:foreground=0xFFA500
client:chui:selection=0x0000FF
client:chui:fontname=monospaced
client:chui:fontsize=12

2. From the command line arguments a BootstrapConfig object is constructed used to create, initialize and start
a generic client ThinClient (CoreClient#start).
It's possible to create a remote standalone client running on a separate process like ClientDriver do
or a client running within the server process.

3. Next the UI services are initialized: ThinClient#initializePrep(config, driver, single);
An OutputManager instance wrapping a ScreenDriver object is created.
The OutputManager is used to draw on the screen via an OutputPrimitives interface (ScreenDriver#getPrimitives)
Currently p2j has 2 screen drivers:

- ConsoleDriver - or chui native based on NCURSES package.
- SwingChuiDriver - a Java SWING based client.

I tested both screen drivers and I found to work within ClientDriver.

Also a Java applet ChuiApplet client can be used embedded in an html page loaded by a web browser.

https://localhost:7443/chui

According to my tests this client only draw the main window and a cursor on the top left corner.
It seems that this client is not fully implemented.

4. When the ThinClient instance is created the ClientExports and SessionExports (for static methods) interfaces are exported remotely.
(The SessionExports interface seems to be not used remotely. I didn't found any remote import for this interface).
The registered (exported) ClientExports interface is imported by the LogicalTerminal instance an used to do RMI calls to ThinClient.

5. Next the client is authenticated and a security context for this thread is created.

6. Next finish initializing the thin client: ThinClient.initializePost(session, context)
Imports ServerExports and RemoteErrorData interfaces exported by the remote StandardServer used for RMI calls to server.

7. Finally the MainEntry interface exported (registered) on the StandardServer is imported, a
ClientParameters is constructed having the converted 4gl application
command line parameters and the remote application is started.
In my case Ask#execute is called which is the Java converted version of the 4gl Ask.p

// call the application
running = main.standardEntry(params);

8. The client code waits here until the user will terminate the running application.

9. When the application is terminated by the user, some clean-up statements are executed
and finally the client exits: System.exit(0);

As a conclusion of my tests, I have a question:

I think that we have to design a new ScreenDriver having an embedded server
used to communicate with the JS code which run inside the HTML page on the web
browser via web sockets.
In this case what kind of UI components are used inside the HTML page?

a - HTML specific tags?
b - JS based?
c - java SWING applet?
d - others?

#12 Updated by Greg Shah over 7 years ago

From Marius, on November 11, 2013.

#13 Updated by Greg Shah over 7 years ago

From Greg on November 11, 2013:

Before I review the code, I will provide some comments and answers to note 10.

The ClientDriver is a remote client started from a command line which run on separate process having only one thread, the main thread.

Yes, ClientDriver is a separate Java process that runs in its own JVM, outside of the P2J server.

Yes, it is remote in the sense that it communicates with the P2J server using our own proprietary remote object protocol.

No, it does not only have 1 thread. There is a type-ahead (key reading) thread that supports our requirement to provide asynchronous interrupts for CTRL-C. And the network socket that connects the client to the P2J server has 2 threads (1 reader and 1 writer). The main thread is in fact the primary thread for doing real work. It is enslaved to the "conversation mode" that we implement between the client and server. See the Conversation class in the net package.

FYI, there is a special-purpose mode in which the ServerDriver can be run as both client and server in one JVM. This is not something we use often and no customers use it today.

The command line arguments define the ScreenDriver type and parameters.
Also the server host and port are specified as command line parameters.
It is possible to provide the same parameters using a configuration file like client.xml

Yes, mostly right. I would note that the bootstrap config can be given as a file and any values in it can be overridden on the command line. The idea is to use that for values that are needed for early configuration of the client (or server, since it can be used there too). Generally, we try to push as much configuration into the P2J server's directory.xml (often just referred to as the "directory") as possible. This maximizes central management of the configuration and can make deployments easier.

For programs that use the P2J classes to make their own connection to a P2J server, the bootstrap configuration can be completely built and controlled under program control.

According to my tests this client only draw the main window and a cursor on the top left corner.

That may be a regression or perhaps it is some applet problem on your system/browser. When I put that applet in, it was working without issues. However, if you were trying it with MAJIC, it would not work today because MAJIC is very dependent upon executing child processes (and other direct filesystem access). That is not possible in the applet approach.

The SessionExports interface seems to be not used remotely. I didn't found any remote import for this interface

No, it is in use. Please see com/goldencode/p2j/util/SessionUtils.java. It is used to "push" some 4GL session-level configuration down to the client so that to the extent that the UI can render or otherwise need to use this configuration, it is in synch with the state on the server.

The registered (exported) ClientExports interface is imported by the LogicalTerminal instance an used to do RMI calls to ThinClient.

Almost. I would not call our approach RMI, which is a very specific Java implementation. However, the idea is the same. We have our own "remote object" protocol (sometimes called the "DAP"/distributed application protocol, but mostly just called "the protocol"). This is the equivalent of Java's RMI, but it has better security and it is easier to implement as a developer. It also doesn't need a name registry process to be running.

But otherwise, your description is correct.

Also note that we instantiate some other classes (e.g. FileSystemDaemon, StreamDaemon, ProcessDaemon, MemoryDaemon...) which also register client-side services that can be called from the server. These are the daemons that allow the client-side process launching, file system access and so forth.

The remote object protocol is designed such that either side of the link can export an API. This peer-based or birdirectional RPC is actively used.

Next the client is authenticated and a security context for this thread is created.

Yes. Note that the context is on the server. On the client, there is only one "session" and the security manager is intentionally a null thing. Since all the actual converted code is running on the server, in a session that has a specific context associated with it, this is OK.

The client code waits here until the user will terminate the running application.

Not exactly. The main thread will be unblocked to execute remote API calls through our exported objects (like ClientExports). And with validation expressions and UI triggers, called client-side code can recursively call back to the server, which can then call back down to the client and so forth. In other words, there can be a complex stack of operations where the control flow of execution for the program will shift back and forth between the client and server, possibly on a deeply nested basis. Ultimately, these nested calls will eventually return and pop off the stack until the program exits, returning here for the main thread to exit (or to restart depending on the "stop disposition").

I think that we have to design a new ScreenDriver having an embedded server used to communicate with the JS code which run inside the HTML page on the web browser via web sockets.

Yes.

a - HTML specific tags?

No.

b - JS based?

Yes. But this is not part of this task. The short answer is that we will use the Canvas interfaces to directly draw our own "Window" implementation and everything that is inside of it. There will not be any actual browser components or plugins used.

c - java SWING applet?

No.

d - others?

No.

#14 Updated by Greg Shah over 7 years ago

Feedback 1111a

I like it. I look forward to seeing the client side fill out.

#15 Updated by Marius Gligor over 7 years ago

The archive mag_upd20131112a.zip contains the first implementation for the Ajax screen driver.

AjaxScreenDriver has an embedded web server inside used to deliver a main HTML page and offer support for web socket services.
Web sockets are created by a custom WebSocketCreator object which allow the injection of a screen driver instance inside the created web socket.
The AjaxScreenDriver follow the SwingChuiDriver model but instead to use a ChuiSimulator it use an AjaxWebSocket object.

All the API calls on AjaxScreenDriver and AjaxOutputPrimitives are delegated to the AjaxWebSocket.
Here we have to implement the interface between the screen driver and the JS client via web sockets.

However the Ajax screen driver has a different paradigm.

1. The web socket communications are asynchronous messaging operations.
Here we have to find a solution in case we need a synchronous operation which should wait until an event occur on the browser.

2. The communication between WebSockets on the server and WebSockets on the client (browser) is based on text or binary messages.
We have to define a format for our messages. Because the end client is a JS code, I suggest to use text messages in JSON format.
In this case we have to add support for JSON serialization / deserialization to our implementation for example a Jackson processor which is widely used.

#16 Updated by Greg Shah over 7 years ago

Code Formatting Notes

The following are some general (but minor) problems with the code formatting.

1. Where possible, reduce lines.

This:

   public AjaxMainPageHandler(String targetRoot,
                          String skeletonPage)

can be:

   public AjaxMainPageHandler(String targetRoot, String skeletonPage)

This is preferred and it is OK because it doesn't cross the 98 character length.

2. Don't use more than 1 consecutive blank line (sometimes you are putting 2 lines in between methods).

3. There should always be 1 blank line in between methods (sometimes you don't have any blank lines at all).

4. Don't put a blank line between the end of the last method and the end of the class.

5. Make sure that there is proper spacing when using operators and/or control keywords. For example, if( should be if (.

6. For newly created files, the copyright date should be simply 2013 instead of a range (probably this is just a copy/paste error).

#17 Updated by Marius Gligor over 7 years ago

code formatting

#18 Updated by Greg Shah over 7 years ago

Code Review 1112a

As we are just starting on the path to the web driver(s), we will check something in soon that is just a step along the way. Your work so far is a good start. I will answer some of your questions in a separate note (I have already been doing quite a bit of thinking on these questions, I just haven't documented all of my thinking yet).

Before we move forward with the first version, please consider the following ideas:

1. We will have at least 2 web drivers (ChUI and GUI). I would like to maximize the common code where possible. It seems to me that the embedded web server, web sockets/protocol support and some of the javascript implementation can be common. For example, on the javascript side I would expect the key processing to be common code between both ChUI and GUI. To make common code we might need to put the common portions in a different package and the design might need to be a bit different.

2. We don't need to implement a real driver yet, but stubbing it out is a good idea. However, it is important to know that we don't want to use the websocket in such a granular way that every call to the ScreenDriver or OutputPrimitives interfaces will trigger a "trip" down to the browser. For example, we will want to build up the changes to the virtual screen and only send them to the browser when sync() is called. Please consider how we might rework the driver to minimize the trips to the browser.

3. Although I keep using the name ajax to describe all of this, I don't think it is the best description. I think WebClient is a better description for the common portions and with ChuiWebClient and GuiWebClient being more specific names.

#19 Updated by Greg Shah over 7 years ago

Code Review 1112b

I like the changes.

#20 Updated by Greg Shah over 7 years ago

1. The web socket communications are asynchronous messaging operations.
Here we have to find a solution in case we need a synchronous operation which should wait until an event occur on the browser.

Yes, we will have to solve this when we get to working on the protocol. We don't have to solve this for #2200.

2. The communication between WebSockets on the server and WebSockets on the client (browser) is based on text or binary messages.
We have to define a format for our messages. Because the end client is a JS code, I suggest to use text messages in JSON format.
In this case we have to add support for JSON serialization / deserialization to our implementation for example a Jackson processor which is widely used.

I will post more details in regard to the protocol. But I am pretty convinced that the protocol will have to be binary.

The P2J architecture introduces a logical separation between the business logic and the UI, which does not exist in the original 4GL environment (where everything runs in the same process). We found that we did have to optimize our code in order to solve some performance issues that were due to this architectural decision. In particular, we found that optimizing the bytes transferred (between the P2J server and the P2J clients) was very important for reasonable performance of the UI. Our protocol was already a binary protocol. But our first version used the Java Serializable support. We were able to make major performance improvements by implementing Externalizable instead (and implementing our own read/write functions in all of our objects to be transferred).

The web client introduces yet another separation. It will be well worth our time to make the protocol as efficient as possible. Using a binary approach will very likely be required. I plan for us to use that from the beginning. I do understand that it has costs on the javascript side, but I think this is a price we must pay.

#21 Updated by Greg Shah over 7 years ago

Task #2201 is meant to take the next steps beyond #2200. In #2200 the embedded web server is enabled in the client. In #2201, that client is started based on a request to the server jetty. The client's SSL context will need to be properly setup from the server and the host/port for the client's embedded web server must be passed back to the server so that the web request can be redirected.

#22 Updated by Marius Gligor over 7 years ago

1. I've created a new Chui screen simulator which use the web socket for remote communication.
Basically the simulator is an in memory controller for a character terminal video memory (similar with ChuiSimulator on swing client).
The memory is organized as a 2d array having a number of cells equals to number of rows multiplied by the number columns.
Each cell holds the character that should be displayed and the other attributes like colour.
The Chui screen driver send commands to this simulator for simple draw operations on the video memory.
The screen driver is UI widgets aware, in sense that he only draw text (characters) not UI widgets.
Almost all draws operations are done only in memory. When a full screen repaint ( sync() ) is compulsory
the screen will be send over the WebSockets to the remote peer (web browser JS code)
Also the implementation of other operations like beep() should be delegated to the remote peer.
At this stage the server web socket has as a minimum a FIFO key buffer (Queue) on which the key code pressed by user are stored.
The key events are send over the network from the remote peer to server.
Obviously a real implementation will refine the current design.

2. I found no GUI implementation on the current p2j. The screen driver for GUI should be designed from scratch.
Are you intended to use the same pattern design as a CHUI driver, or we have to design something else?

3. The name of the web screen driver is ChuiWebDriver because the other drivers has similar names.
My intention is to use a ChuiWebClient object as a web client luncher similar to the design in ClientDriver.
Is this a correct design, to implement something like ClientDriver but without a main entry?

4. Regarding the issue #2201 I have to design a new jetty Handler for a request like https://<server>:<admin_port>/chuiweb
This Handler should start a ChuiWebClient, get the embedded server URI and redirect the request to this URI.
We can do the redirection in two ways. Which method do you prefer?

a) By returning a web page having this <meta http-equiv="refresh" content="0; url=https://<server>:<port>/<context>/<page>" /> 
tag inside the head section. The page could be an html file or could be constructed in memory.
b) By calling HttpServletResponse#sendRedirect(String URL) inside the Handler#handle(...) method.
This method does not use a html page for redirection.
However in case of an error on both cases we have to return an error page in order to inform the user
about the error. This page should be also constructed in memory.

#23 Updated by Greg Shah over 7 years ago

Code Review 1113a

This is moving in exactly the direction I was looking for. Very good.

1. I see that there is quite a bit of copied code in ChuiWebSocketSimulator from the Swing ChuiSimulator. It seems to me that it would be better to have an abstract base class that both can derive from, in order to maximize common code. Actually, let's use the following naming:

ui/client/chui/driver/ChuiSimulator - abstract base class
ui/client/chui/driver/web/ChuiWebSimulator
ui/client/chui/driver/swing/ChuiSwingSimulator

I realize that ChuiWebSocketSimulator would no longer be able to extend the WebSocketAdapter. But I think the web sockets part is very small and can be handled external to the simulator functionality.

2. It seems like SwingChuiPrimitives can be moved/renamed ui/client/chui/driver/ChuiPrimitives and it can be made as a common class used by both the swing and web implementations. I don't think there is anything unique about ChuiWebPrimitives, right?

3. I would like to see if ChuiWebSocket can be made generic for both the ChUI and GUI implementations. At a minimum, most functionality can be in an abstract base class, with more specialized subclasses to handle any unique requirements. This would mean that it would live in ui/driver/.

4. I think the same idea as in number 1 above can be done for ChuiWebDriver/SwingChuiDriver. The abstract base class can be ui/client/chui/driver/ChuiScreenDriver and it would implement ScreenDriver.

#24 Updated by Greg Shah over 7 years ago

I found no GUI implementation on the current p2j. The screen driver for GUI should be designed from scratch.

Correct. But that is work for a future task. Right now we will implement: code that can be common for any web driver + chui web driver, while refactoring the swing chui driver to share maximum code between swing chui and web chui.

We should consider using TerminalOptions (and other classes) as common code which may be swing-specific today. It seems to me that we some things like Color instances can be stored in the P2J client as Swing-specific values. But then we would translate this into a value (like #FFFFBF) that can be sent down to the web client, when needed.

Are you intended to use the same pattern design as a CHUI driver, or we have to design something else?

Yes, but the GUI driver interface will by nature be different. I hope that we can have a super-interface that is shared between both chui and gui. For example, the key processing is something that can probably be the same between both driver interfaces.

I also am worried about the optimal place to introduce the driver interface in the GUI environment. There are multiple choices:

  1. Provide drawing primitives (lines, boxes...) and allow the widgets to call these directly. The big advantage is that it would be easy to implement these primitives on both Swing and in Javascript canvas and there would be massive common code for the hard parts of the drawing. In particular, the widget-specific drawing logic would be common code. The big downside is that I expect this to perform very poorly when we add the extra split to remotely drive the browser drawing.
  2. Provide widget-specific drawing methods in the abstraction layer and then implement widget-specific drawing logic separately in swing and javascript.

Both ideas might be amenable to the optimization of sending a list of the drawing operations down to the javascript client. For example:

- draw a button with this rectangle, colors and text
- draw text in this font at this location
- draw a box of this rectangle and color
...

Then the entire list can be sent at one time and there won't be massive "back and forth" from the P2J client to/from the browser. Many small trips is the thing that really kills performance in this case. One round trip with a larger payload will significantly outperform the many small trips case.

This example is showing the list as a set of widget-level drawing primitives, but we could send a lower-level set of primitive operations instead (draw a line of this color from point a to point b, draw a rectangle with this line color and this fill...).

My intention is to use a ChuiWebClient object as a web client luncher similar to the design in ClientDriver.
Is this a correct design, to implement something like ClientDriver but without a main entry?

Actually, for now let's just extend our current approach. ClientDriver/ThinClient will use different drivers based on bootstrap configuration. Don't add a dedicated client driver.

We can do the redirection in two ways. Which method do you prefer?

Are there any downsides to using HttpServletResponse.sendRedirect()?

#25 Updated by Marius Gligor over 7 years ago

1. The server web socket is created only when the remote user will open the communication with him.
On the other hand the simulator MUST be created and is created when the client is lunched.
That's why the simulator does not extends the web socket.
However it is possible to change the web socket creator to return an existing instance
and in this case I think that is possible to extend the web socket in simulator.
I will think to that issue to find the better solution.
When the web socket is open the simulator is notified about this and the simulator send the
screen status down to the web browser in order to be rendered on the web page via JS.
I will try to make the current implementation more abstract as you already suggested.

2. I've started also to work on redirection issue. So far I created a Handler
which create an instance of the ChuiWebDriver, start the embedded server and
redirect the request to the embedded server URI. On error will return a short page
with a message informing the user about this. I did some tests and the redirection
works fine using the response sendRedirect method. I added to the server directory.xml
a new entry chuiWeb like chuiApplet. I found a minor bug on current implementation:
The chuiApplet is using /chui as context root and I want to use /chuiweb
Because the check on chuiApplet is target.startsWith(/chui) is not possible to
handle both /chui and /chuiweb. Finally I used /webchui instead /chuiweb

3. Something remain unclear on my mind at this point. If we aren't use a mechanism
like ClientDriver to start a client how we will connect our screen driver with the
converted 4gl application?

#26 Updated by Greg Shah over 7 years ago

in this case I think that is possible to extend the web socket in simulator

I don't have a problem using the WebSocketAdapter in our solution (or WebSocket directly). I just prefer to separate the use of it from the inheritance hierarchy of the simulator.

The chuiApplet is using /chui as context root and I want to use /chuiweb
Because the check on chuiApplet is target.startsWith(/chui) is not possible to
handle both /chui and /chuiweb. Finally I used /webchui instead /chuiweb

Let's just use /chui and eliminate the ChuiApplet. In practice, there is no good reason to use ChuiApplet when you can use our web client instead. Eliminating the applet is a good thing.

If we aren't use a mechanism like ClientDriver to start a client how we will connect our screen driver with the

converted 4gl application?

The proper driver configuration would be passed in a bootstrap config instance when the client is launched. You will have to modify OutputManager.newInstance(), which is called from OutputManager.init() which is called from ThinClient.initializePrep().

#27 Updated by Marius Gligor over 7 years ago

The WebSocketAdapter was used only for notification mechanism because WebSocketAdapter implements WebSocketListener.
A better solution is to design a new interface just for notifications and eliminate the inheritance of WebSocketAdapter.
The class ChuiSimulator from swing based driver extends JComponent and this is a problem because we cannot extends another class like an abstract or base class.

I know about the OutputManager changes that we have to made in order to add our screen driver.
In this case I think that we have to use the ClientCore.start(...) method like ClientDriver finally do.

#28 Updated by Greg Shah over 7 years ago

In this case I think that we have to use the ClientCore.start(...) method like ClientDriver finally do.

I was assuming we would use the same ProcessBuilder mechanism that we use for appserver agents today. It just builds the Java command line up dynamically and then runs the program. There is no reason we can't use ClientDriver from there. Is there some other reason ClientDriver won't work?

#29 Updated by Marius Gligor over 7 years ago

I did more changes in the structure of the project to make the implementation more generic and abstract.

1. The ChuiSimulator already extends JComponent and cannot extend another abstract class.
However I used the ChuiSimulator as a base class for ChuiWebSimulator.

2. Having a generic ChuiSimulater the ChuiPrimitives became also generic
and common for both swing and web screen drivers. The SwingChuiPrimitives is no longer needed and has been deleted.

3. The ChuiWebSimulator is also decorated with WebSocket annotations, that is, he play also the role of a web socket.
The ChuiWebSocketCreator has been changed to return an existing instance and not to create a new one because the
ChuiWebSimulator must be created earlier and exists at the moment when the jetty server call the WebSocketCreator
asking for a WebSocket instance.

4. A ChuiScreenDriver has been created as a base class for both SwingChuiDriver and ChuiWebDriver.

5. I changed also the OutputManager and I added code to use the new web screen driver as having "chui_web" name.

6. ChuiWebHandler class is jetty Handler used for start the web client and redirect the HTTP request to the client embedded server URI.
At this moment here is a simple test code used just for testing purposes and should replaced with the process spawn code.

7. I tested also the ClientDriver from command line using chui_web driver and I found that the the web client work properly,
the connection with the remote 4gl application is done and the client waits for ended of the converted application.
However I found also that the embedded web server does not work.
The server is started and listen for connections but the web browser cannot download the main page.
The browser said the the server does not accept encrypted connection and I think that the embedded server has no SSL certificate.
If you have any hint regarding this issue please let me know.

8. After all this changes I tested also the swing console terminal and so far I found no errors.

#30 Updated by Marius Gligor over 7 years ago

Regarding my tests with ClientDriver and "chui_web" screen driver I used the configuration
file client.xml and the key stores from P2J_HOME/testcases/simple/client_strong.
I saw inside the BootstrapConfig informations related to key stores as well as access passwords.

Also I did tests without this configuration file but in this case the BootstrapConfig does not
contains informations regrading SSL.

#31 Updated by Marius Gligor over 7 years ago

#32 Updated by Greg Shah over 7 years ago

Code Review 1114a

1. The ChuiWebSocketCreator has nothing that is specific to ChUI. Let's make it generic and put it in the ui/driver/ package so that the functionality can be shared between ChUI and GUI.

2. There are minor code formatting issues in ChuiWebHandler.

3. As mentioned in the previous notes, we don't need the ChuiApplet any more. Please remove that support (including from StandardServer).

4. num should not be set to 1 at the beginning of the initialize() method inside StandardServer.registerDefaultServices().

5. The "Module" name at the top of ChuiScreenDriver (and also ChuiPrimitives) is incorrect.

6. Please duplicate the javadoc for interface methods that are implemented instead of just referencing it using at-see (e.g. in ChuiScreenDriver).

7. I prefer the ChuiScreenDriver.beep() implementation to be moved into SwingChuiDriver. There is no case where that code can be shared and leaving in the base case just raises questions on the part of the reader.

8. ChuiWebHandler seems pretty generic and not specific to ChUI. Let's just rename it WebHandler.

9. About the SSL problem, it is expected.

The server is started and listen for connections but the web browser cannot download the main page.
The browser said the the server does not accept encrypted connection and I think that the embedded server has no SSL certificate.
If you have any hint regarding this issue please let me know.

We need to find a way to send the P2J server's SSL certs/context down to the P2J client so that it can be reused. As for where the server gets it, please see SecurityManager.getSecureSocketContext() and the TransportSecurity class in that same package. Some changes will be needed, but the core idea is that we DO NOT want to have to configure keystores/truststores on each client. Instead, we want to leverage the server's SSL context, but use it on the client. Changes will be needed in SecurityManager and TransportSecurity to make this happen.

#33 Updated by Marius Gligor over 7 years ago

Changes after Code Review 1114a.

#34 Updated by Greg Shah over 7 years ago

Code Review 1114b

It looks good. Please get this runtime regression tested.

I assume that the direct reference to ChuiWebDriver from WebHandler will be eliminated when the proper client spawning is working. Is that correct?

#35 Updated by Marius Gligor over 7 years ago

Yes, the reference to ChuiWebDriver from WebHandler will be eliminated and replaced with code to spawn the client.

#36 Updated by Marius Gligor over 7 years ago

  • File regression_tests_reports_01.zip added
  • File regression_tests_reports_02.zip added

1. Here are the results of the regression tests. Seems to be expected errors.

2. I did also tests on my workstation like on the previous regression tests.

- Admin console works properly on my workstation.
- I tested also the new WebDriver by adding the following node into gso-directory.xml file:
<node class="container" name="chuiWeb">
<node class="boolean" name="enabled">
<node-attribute name="value" value="TRUE"/>
</node>
<node class="integer" name="rows">
<node-attribute name="value" value="24"/>
</node>
<node class="integer" name="columns">
<node-attribute name="value" value="80"/>
</node>
<node class="string" name="background">
<node-attribute name="value" value="0x000000"/>
</node>
<node class="string" name="foreground">
<node-attribute name="value" value="0xFFA500"/>
</node>
<node class="string" name="selection">
<node-attribute name="value" value="0x0000FF"/>
</node>
<node class="string" name="fontname">
<node-attribute name="value" value="monospaced"/>
</node>
<node class="integer" name="fontsize">
<node-attribute name="value" value="12"/>
</node>
</node>

The first tests FAILS due to the missing of index.html resource file into p2j.jar
The ant script (build.xml) should be updated to copy this file resource into the build
and finally into the jar file.

I added the file index.html to p2j.jar "by hand" and finally it's works fine.
Should I change the build.xml file now or we have to do that when we'll finish the implementation?

#37 Updated by Constantin Asofiei over 7 years ago

Marius Gligor wrote:

1. Here are the results of the regression tests. Seems to be expected errors.

Do not update test results here. If you are unsure if you have a real or not error and want us to look over it, send the files via email.

- I tested also the new WebDriver by adding the following node into gso-directory.xml file:

Are these changes required for MAJIC's Admin Console?

The first tests FAILS due to the missing of index.html resource file into p2j.jar

All look like false negatives.

Should I change the build.xml file now or we have to do that when we'll finish the implementation?

Do change it now.

#38 Updated by Constantin Asofiei over 7 years ago

  • File deleted (regression_tests_reports_01.zip)

#39 Updated by Constantin Asofiei over 7 years ago

  • File deleted (regression_tests_reports_02.zip)

#40 Updated by Marius Gligor over 7 years ago

The changes does not affect the the adminConsole applet.

The chuiApplet is no longer used, should I delete the files
related to this applet from project?

#41 Updated by Greg Shah over 7 years ago

The chuiApplet is no longer used, should I delete the files related to this applet from project?

Yes.

#42 Updated by Greg Shah over 7 years ago

Please fix the build.xml and re-do your manual testing. If that works, then you can check in the update and distribute it.

#43 Updated by Marius Gligor over 7 years ago

The following files should be deleted:

ChuiAppletHandler.java ChuiApplet.java and chui_applet.html

Also I have to do some changes on OutputManger.java and delete the code related to CuiApplet.java
and in the file build.xml to eliminate the copy of chui_applet.html to build env.

The file OutputManger.java already has some changes that should be commited.

Should I redo the regression tests or is enough to test only locally?

#44 Updated by Marius Gligor over 7 years ago

The archive contains the ChuiWebDriver implementation based on jetty web sockets.
Passed regression testing, committed to bzr revision 10412. Refs #1811.

NOTE: The SWING applet is no longer used and the files related to the applet has been deleted.

The following node should be added to the server directory.xml
Basically is the same as the "chuiApplet" node.
Since the applet is no longer used you should rename the "chuiApplet" node to "chuiWeb" instead to add a new node.
If your directory.xml does not contains a "chuiApplet" node add the "chuiWeb" node.

<node class="container" name="chuiWeb">
<node class="boolean" name="enabled">
<node-attribute name="value" value="TRUE"/>
</node>
<node class="integer" name="rows">
<node-attribute name="value" value="24"/>
</node>
<node class="integer" name="columns">
<node-attribute name="value" value="80"/>
</node>
<node class="string" name="background">
<node-attribute name="value" value="0x000000"/>
</node>
<node class="string" name="foreground">
<node-attribute name="value" value="0xFFA500"/>
</node>
<node class="string" name="selection">
<node-attribute name="value" value="0x0000FF"/>
</node>
<node class="string" name="fontname">
<node-attribute name="value" value="monospaced"/>
</node>
<node class="integer" name="fontsize">
<node-attribute name="value" value="12"/>
</node>
</node>

#45 Updated by Marius Gligor over 7 years ago

Here is the first implementation of our process spawner. It was tested and found to work properly.
Bellow are some details related to the most important solved issues.

1. The server security context is exported via command line by introducing 3 new parameters:

ssl:keystore:filename
ssl:password:keystore
ssl:password:keyentry
These parameters are constructed from the server BootstrapConfig structure (SpawnerImpl#exportSslContext).
On the client side using the client BootstrapConfig object a SslContextFactory instance is constructed
(see GenericWebServer#getSslContextFactory) using the exported server SSL context parameters.

2. The Spawner is a singleton design pattern.
The Spawner register a Remote Object Network Server used to receive a notification from client
which must to inform about the embedded server URL on which the client listen via a notification.

When a new web client is spawned a command line for a Java application is constructed.
Next a ProcessBuilder object is used to start the process and wait an amount of time for client notification.
When a notification is received or at the end of the waiting time (time out) the spawner
return the remote embedded server URI or null if no notification is coming.
Also if no notification is coming the spawned process is destroyed.

IMPORTANT: We have to implement also a mechanism to terminate a spawned process when the user close the browser
without to explicit terminate the remote application. In this case the spawned process remain active in memory
and could be accessed by any other user which guess the URI of the embedded server.

#46 Updated by Greg Shah over 7 years ago

Code Review 1118a

There is some good work here, but this still needs some work. If anything, this is a good start that exposes some core design issues that need to be resolved.

1. The singleton design will not scale properly. The approach you have used will only allow 1 client to be started at a time and it will block until the client has initialized.

2. The current design will only work when run with the server's user.dir as the client's current directory. That cannot be a dependency because most apps have very specific client-side requirements for current directory and they would never let each user have a current directory as the same directory in which the server gets all its configuration.

Please review the client.sh (used in Majic) very carefully. These same features need to be exposed when spawning clients. This will be tricky because there is a "chicken and egg" problem. In our standard client model, the user has already logged into the system and thus provides the context (permissions, home directory and so forth). For this to work in our approach, we may need some scripting help at the OS level. We also may need to know what the OS-level userid should be for each new session. How do we get the OS user context established for each new session?

This opens some major security questions. We definitely cannot run in the same permissions/context as the server, but when we start like this that is exactly what happens.

Right now the implementation assumes the clients will always run on the same system as the server. Many of the above requirements will probably require a remote spawner (one that is not in the server's process).

3. Please plan for the server's SSL context to be sent down to the spawner using some form of serialization. This eliminates the need for direct access to the server's current directory and configuration.

4. The connection setup is hard coded to insecure sockets (from the spawned P2J client to the P2J server) which is not something we want to do. It needs to be configurable, while defaulting to be secure.

5. It is also important to note that any environment that is configured to actually force a login (like MAJIC) will fail in the current ClientCore design. The reason is simple: one must have the URL sent back to the server BEFORE the login can occur because in the "normal" case, the user must be prompted for login credentials. But since you are calling a remote method on the server in order to pass the URL back, you are assuming that the P2J session is already setup. That session is created in the connectDirect() call. Your tests worked because you used the simple server environment that has a neutered security environment.

6. The "client.xml" is also something that should be configurable. The fact that we commonly use that filename is purely a convention, but it should not be a requirement.

7. The user.dir and other JVM parms (heap, parm gen...) are things that will have to be configurable from our directory instead of hard coding them.

8. Code formatting and coding standard problems:

  • missing header entries
  • missing public keyword on interface methods
  • OnRemoteServerListen should be onRemoteServerListen
  • wrong sorting of methods by access mode and static/instance

#47 Updated by Marius Gligor over 7 years ago

1. I used a singleton design because I cannot export the Spawner interface on multiple instances.
Is any method to export the same interface per instance?

2. What I understood from your description is that the new spawned process MUST run on the OS user environment.
It's not clear for me why and how to do that because the user is virtually a HTTP request that is coming from
the local or from a remote machine on the network. Our WebClient just offer services via web sockets.
The users have no other controls over the running clients.

Should we spawn the process on a remote machine?
How to do that because the p2j.jar and dependencies in this case must to be on the remote machine?
The web client is running on the same machine where the server is running.
However the single parameter in CoreClient is false.
  • @param single
  • true to startup within the server process which
  • must bypass the shared infrastructure initialization. Use
  • false for the normal client JVM startup.
Could you please explain me what is the difference between single=true and single=false?
We are using a Java ProcessBuilder on our Spawner and we always create a new process which
is a separate process from OS perspective. Only the process OS context is the same as the server context.
Basically instead to use server threads which runs within the server process we use a separate process for each request.
Why this model is not good for our web client?

3. I have to think about this, because is related to 2.

4,6,7. I will change the design to provide configuration for arguments and not hard codding.

5. Here I have also a lot of questions.
- It is possible to use your RemoteObject design to access the remote server before client authentication?

This snippet of code is from ClientCore class.
// establish or re-establish a session with the P2J server
// (this handles authentication too)
session = sessMgr.connectDirect(new InterruptedExceptionHandler(), null);
- connectDirect handles authentication too?.
- should our web client provide an authentication web page too?
Could you please offer me more details because the design of this part is also unclear for me.

#48 Updated by Marius Gligor over 7 years ago

Another important design issue.
When a request is made at https://&lt;server_host&gt;:&lt;admin_port&gt;/chui the request is handled by the WebServer instance.
The WebServer calls Handler#handle on each registered handlers and wait until return from handle method.
The base.isHandled() flag is checked after each call. If this flag is true that means that the request
was handled and a response was sent back to the requester and the web server stops this process here.
If all handlers on the chains are called and no one return base.isHandled() as a true value an default error page
is returned as a response by the jetty web server.

Because each handler is a unique instance the access to handle method is serialized.
Even more on our case we have to take a decision before to send a response back to the user.
This means that we always have to wait for a client response in order to take the decision before sending the response back to the user.
On other words waiting on Spawner or waiting on WebHandler has the same effect, serialization access.
Due to this constraints I don't think that is possible to create web clients simultaneously.

#49 Updated by Greg Shah over 7 years ago

1. I used a singleton design because I cannot export the Spawner interface on multiple instances.
Is any method to export the same interface per instance?

You can export an interface using the instance model (RemoteObject.registerNetworkServer()) which registers an instance of an object that implements the interface(s).

You can also export an interface using the "static" model (RemoteObject.registerStaticNetworkServer()) which registers a class that has static methods whose signatures that match the interface(s) exactly (of course we can't use "implements" with static methods, but this is essentially the same thing).

Either way, there is only 1 registration/export of an interface per JVM. Please read through the RemoteObject code to get the idea.

My concern is not so much the use of a singleton itself, but really it is how you are synchronizing on the class and assuming only one new client is launched at a time. Regardless of the exporting method, you will need to change how the spawning and waiting/notification works to eliminate the "one at a time" approach.

What I understood from your description is that the new spawned process MUST run on the OS user environment.

Correct.

It's not clear for me why and how to do that because the user is virtually a HTTP request that is coming from the local or from a remote machine on the network. Our WebClient just offer services via web sockets. The users have no other controls over the running clients.

Yes, I had a hard time sleeping last night because I was thinking about this exact issue. :) But I did come up with an approach.

Consider that in our P2J client, we have native dependencies. Please look in src/native/ to see the level of functionality that is there. This support is needed because of features that the 4GL provides which cannot be exactly duplicated using J2SE features. On Linux/UNIX, our P2J build creates a libp2j.so and on Windows we create a p2j.dll. We access this DLL ONLY on the client (the P2J server does not use it). The access is through JNI.

On Linux/UNIX, the system is inherently multi-user. There are features built in to allow multiple users to login simultaneously. More importantly there are features to allow processes to change their user context. There is the su command and there are APIs such as setuid.

On Windows Server, I suspect that similar facilities exist as provided by the terminal services.

When needed we can and will add native code helpers to this JNI library to allow us to integrate/call such facilities.

I think we should be able to do something like the following:

  1. The server jetty + initial web page returned will not redirect immediately. Instead, in the common case it will have to create a dialog with the user to obtain the login credentials in a secure manner and submit them to the P2J server.
  2. The P2J server will spawn a new client process which must essentially login/change user context using the given login credentials. We will most likely have to have an alternate approach for guest access (and possibly other scenarios), but we can deal with that later. The result is a process that is running in the proper user's context.
  3. Inside that process that has the correct context, the P2J client must be started. It is possible that we might do this the other way around (spawn a new P2J client in the server's context and then shift context via API calls) BUT there are some sensitive security issues that must be handled in such a case. The new context must not have access to sensitive data/environment/configuration AND it must not be possible to shift the effective user context back to the original server's context.
  4. From there the client jetty must be initialized and we must send the URL back to the server jetty which will send the redirect back down to the client and the rest works as originally discussed.

On Linux/UNIX there are some potential issues with ensuring that there is a "controlling terminal" or pty (pseudo-terminal) associated with every client process that is spawned. The system will not allow interactive programs (or programs that do more with a UI than just read/write to STDIO) to be launched from parent processes that are not setup up with the proper terminal environment. 4GL apps frequently do start child processes that have such requirements. We must make sure the child processes can handle such requirements.

We also will need to find a way to redirect the I/O to/from such programs and the remote javascript "terminal". I haven't fully gotten my head around how we do that, some research will be needed there too.

We also have some special requirements for how we trap exceptions in the client.sh script (on Linux/UNIX), which we will need to duplicate.

On Windows, I'm not really sure what issues we will encounter.

How to do that because the p2j.jar and dependencies in this case must to be on the remote machine?

Yes, this is the assumption.

In such a case, I think we must have a special "client" process on that remote machine that is connected to the server. This process would be able to do the spawning.

The web client is running on the same machine where the server is running.

Sometimes this will be good. But we will have some customers where this is not acceptable.

However the single parameter in CoreClient is false.

Yes, of course. This is correct.

Could you please explain me what is the difference between single=true and single=false?

single=true means that the server process and client process are the same. This is a special mode where there can only be 1 client, ever. This is not really used in production.

Using single=true is not something we are going to do with the web client.

We are using a Java ProcessBuilder on our Spawner and we always create a new process which is a separate process from OS perspective.

Yes, this is good.

Only the process OS context is the same as the server context.

This is what we must change.

Basically instead to use server threads which runs within the server process we use a separate process for each request.

Not really. We already use a separate process today for the P2J client. This is just a different way to launch it and interface with it.

The user still will have a session with the P2J server (the P2J client will have this session actually) and that session will still have threads running on the server on that user's behalf for all business logic and database access. Only the UI and the client platform dependencies (e.g. accessing files, launching child processes/shell commands, using sockets or direct memory access...) will be done in the P2J client (just as it is done today).

It is possible to use your RemoteObject design to access the remote server before client authentication?

No. One must have a value session before any RemoteObject functionality can be used.

connectDirect handles authentication too?.

Yes.

should our web client provide an authentication web page too?

Yes and no.

As noted above, we will need to do the OS-level authentication in the web interface from the server jetty.

But the P2J authentication is different and would still be handled as it is done today (inside connectDirect()). Please use a debugger to step through the connectDirect() process (that is the P2J client side, you will also have to step through the P2J server side which can be seen in RouterSessionManager.Incoming.run()). It is important that you understand how a P2J session gets setup. The authentication logic in the SecurityManager is quite complex because there are many different ways (batch vs interactive, certificate based auth, userid/password, plugins...) through. You should look at the simple server environment first, where you will see a kind of "null" authentication where we use a plugin to allow anonymous/guest access. But then you should also step through this process for a MAJIC client. That uses a real authentication process against user accounts in the server's directory and each successfully setup session has a specific P2J account context.

The interesting thing here is that we actually use a limited form of the 4GL UI processing to handle those cases (like MAJIC) that use interactive P2J authentication. During the connectDirect() we actually serialize an application-specific plugin class that implements the login UI and send it from the server to the client. It is executed there in our "early" P2J client, interacting with the user to get the userid/password details and then sending the results back to the server. SO: as long as our javascript client can handle our normal 4GL requirements, this support will "just work" without any changes. To see an example of this plugin code, see testing/majic/src/aero/timco/majic/security/LoginClient.java on devsrv01.

Because each handler is a unique instance the access to handle method is serialized.

Do you mean synchronized?

Even more on our case we have to take a decision before to send a response back to the user.
This means that we always have to wait for a client response in order to take the decision before sending the response back to the user.
On other words waiting on Spawner or waiting on WebHandler has the same effect, serialization access.
Due to this constraints I don't think that is possible to create web clients simultaneously.

I suspect we can solve this using:

  • thread pools so that there can be multiple incoming sessions at once
  • create a separate thread and reply immediately, unblocking the server, then use websockets between a javascript client and that new thread to handle the rest of the process
  • something else?

#50 Updated by Marius Gligor over 7 years ago

Here is a new design which is able to spawn the WebClient into the server context.

a) I designed a generic ClientBuilder class and a ClientBuilderParameters to store client parameters and avoid hard code values.

b) The client lunched was moved on WebHandler which is also a SpawnerListenr registered with the Spawner in order to receive a notifications.

c) The Spawner is used only to receive notification from the spawned process and to notify the local registered listeners about the remote server URI.

d) I assigned also a unique id (UUID) to each spawned process. When the process will notify the spawner the UUID and the URI are send over the network.
The Spawner listeners is a Map<UUID, Listener> and USING the remote UUID we are able to know precisely which listener should be notified.

e) The server KeyStore is exported via a RemoteObject interface not from the command line parameters.

2. In order to run the client on a specific user context (account) I think that we have a way to do that.

a) On Linux we have the command: su - &lt;user&gt; -c &lt;command&gt;.
b) On Windows we have the command: runas /user:&lt;user&gt; &lt;command&gt;

Both do the same, run the command using the <user> account.
Both asks for a user password.
Both seems to use the stdin to get the password.

The Process object in Java allow us to have control over stdin, stdout and stderr streams.
We can try to set the password by controling the stdin.

Also we have to change also the login scenario as follow.

a) The user open the browser and make a request to https://&lt;server_address&gt;:&lt;admin_port&gt;/chui
This is basically a HTTP/GET method. The response is a HTML page having a form which allow
user to send a user name and a password. This should be the user account on the machine
where the client should be spawned.

b) When the user press "Login" button the form is submitted. This is HTTP/POST request.

c) Using the user credentials we will try to spawn our client on the user account context.
If succeeded redirect the user to the page having the JS Terminal.
If error a page containing an error message will inform the user about the fails.

#51 Updated by Marius Gligor over 7 years ago

I started to do tests using Linux su and sudo commands.

1. For security reasons su and sudo doesn't allow stdin redirection.
I received the following error messages trying to use this commands with Java ProcessBuilder

su: must be run from a terminal
sudo: no tty present and no askpass program specified

2. As a root I'm able to use both commands. However in this case no ask for password is made!

Another drawback for su command:
Using: su - mag -c java                     // works fine
Using: su - mag -c java -version        // does not work.
we have to use something like this:
su - mag -c ~/script.sh
that is, we have to put the java command and arguments into a bash script and execute this script.

#52 Updated by Greg Shah over 7 years ago

The standard Java process launching does not properly setup the terminal environment. As mentioned before, the terminal environment is very important for the client processes to properly work. As you have found su and sudo won't work without it.

Please note that we do have our own process launching support that does setup the pty properly (see src/native/process.c). But this is only usable from the P2J client.

From a design perspective, we will want the solution to this problem to minimize deployment issues and shell/OS interface usage. This means we want to avoid scripts, avoid special permissions setup, avoid external configuration files, avoid dependence on utility programs and avoid shell processing. If these things can be avoided by writing code to native APIs in our JNI library, then that will be the preferred approach. Of course, at this point we need to find a solution that works at all. But I do know that this is possible (e.g. other programs like sshd are able to do this. It is probably a good idea to investigate OpenSSH Server to see the APIs and techniques they use.

The server KeyStore is exported via a RemoteObject interface not from the command line parameters.

I haven't gotten a chance to look at the posted code. But I think we can't use RemoteObject for transferring the keystore because in the normal case (non-guest access like in MAJIC) there won't be a valid session with the P2J server until the user has done an application login.

#53 Updated by Marius Gligor over 7 years ago

  • File spawn.c added

A. I did a lot of researches to find out a way to execute a command on Linux on the environment of other user,
Here are the results of my today work:

1. For security reasons it is obviously that a normal user cannot gain root permissions or access the
environment of other users directly. Only using sudo or su it is possible to do that.
The user MUST provide the root or the user password he otherwise cannot do sudo and su commands.
In other words only downgrade user permissions are allowed.
Even at low level using API calls it's not possible to do that as normal user.
2. Running at root (having root privileges) however it is possible to execute a command on the context of an existing user.
The steps to do that are:
- Having a user name and a password get the user informations (uid, password, etc.)
- If the user exists check if password match. Basically this is the authentication step.
- If user exists and the password match create a child process via fork(). The child process is still running as root.
On child process:
- Downgrade the child process permissions by changing the process uid to user uid. Set also the user environment.
- Execute the desire command.
On parent process:
- Wait for child process to exit.
3. The root mode execution is a MUST constraint.
4. The attached c source code is a simple implementation of this steps.
Of course this is a minimal implementation.
A real implementation could be more complex depending on the UNIX / Linux OS on which the application is running.
On my workstation was tested and found to work.

#54 Updated by Marius Gligor over 7 years ago

  • File deleted (spawn.c)

#55 Updated by Marius Gligor over 7 years ago

#56 Updated by Greg Shah over 7 years ago

This is good work. Some thoughts:

  • We will want to limit the scope of the process that is root. In particular, the main P2J server should NOT be run with root because any security flaw or breach will expose the entire system to problems. Instead, we will want to implement a very small "spawning" process that is run with root.
  • What is the controlling terminal status of the child process in this case?
  • The use of setuid() should be safe in that since the original context is root, there will be no "saved" uid context that can be swapped back in later.
  • I am curious to what extent this same behavior can be implemented on Windows. In particular, the process launching approach of fork() followed by exec() is not the design on Windows, so I expect some problems there.

#57 Updated by Marius Gligor over 7 years ago

Today I started to do tests using the spawn command written in C.
Here are the results.

I did some changes inside the C code. After the spawn application was build I did the following:

$sudo -s
$chown -v root:root spawn
$chmode -v 4755 spawn
Now the spawn command can be started by a normal user but run with root privileges!
I moved the spawn file into my ~/bin directory.
Due to my changes on the C code when spawn is used we should provide the entire path: ~/bin/spawn or /home/mag/bin/spawn
Next I created a new user on my system named bubu.
Than I tried:
mag@mag:~$ ~/bin/spawn mag &lt;mypassword&gt; ls /home/bubu
ls: cannot open directory /home/bubu: Permission denied
So far so good, I have no access on the bubu home directory.
Next I tried:
mag@mag:~$ ~/bin/spawn bubu &lt;bubu_password&gt; ls la /home/bubu  
total 24
drwxr-x--
2 bubu bubu 4096 nov 22 18:15 .
drwxr-xr-x 5 root root 4096 nov 22 17:16 ..
rw------ 1 bubu bubu 8 nov 22 17:17 .bash_history
rw-r--r- 1 bubu bubu 220 mar 30 2013 .bash_logout
rw-r--r- 1 bubu bubu 3637 mar 30 2013 .bashrc
rw-r--r- 1 bubu bubu 675 mar 30 2013 .profile
Works good, I have access on /home/bubu.
Next I tried to use spawn in the p2j to lunch the web client.
I added the following lines before java command on ClientBuilder.java:
...
command.add("/home/mag/bin/spawn");
command.add("bubu");
command.add(&lt;bubu_password&gt;);
// Build command
command.add("java");
...
I started the sever, I opened the Mozilla Firefox browser and I typed.
https://localhost:7443/chui
Unbelievable, It works!
I checked, if is real:
mag@mag:~$ ps -fu bubu
UID PID PPID C STIME TTY TIME CMD
bubu 20343 20341 99 17:54 ? 00:04:06 java -Xmx512m -XX:MaxPermSize=64m -Djava.library.path=/home/mag/p2j/build/lib -classp
mag@mag:~$ ps -fu root
root 20341 1 0 17:54 ? 00:00:00 /home/mag/bin/spawn bubu &lt;bubu_password&gt; java -Xmx512m -XX:MaxPermSize=64m -Djava.library.pa
Yes is real, it works!
1. However the password on the command line is in clear and this is not a good think.
Maybe en encryption is mandatory to hide the real password.
2. A big issue that I still not understand. My expectation was for java to require the -classpath to be set somewhere inside the bubu user environment.
In this case the p2j.jar and his dependencies must exists on each user home directory.
Surprisingly it works without this constraint, is seems that the spawned process is still able to access the jar files from my home directory (mag)!
3. I think that we can use this small tool in our project with minor changes regarding the password and potentially to limit the usage only for our purposes.
In this case I have to do some changes in Java cod to integrate the spawn command.
4. For Windows OS I have to do another research.

#58 Updated by Greg Shah over 7 years ago

This is really good work. Some thoughts:

1. Each spawned process must have a controlling terminal. You can use the who command to list the current terminals in use. Please look into this topic deeply (see these links) and make sure that the controlling terminal setup is correct for the resulting spawned process.

http://blog.nelhage.com/2009/12/a-brief-introduction-to-termios/
http://blog.nelhage.com/2009/12/a-brief-introduction-to-termios-termios3-and-stty/
http://blog.nelhage.com/2010/01/a-brief-introduction-to-termios-signaling-and-job-control/
http://uw714doc.sco.com/en/SDK_sysprog/_The_Controlling-Terminal_and_Pr.html
http://blog.nelhage.com/2011/02/changing-ctty/

Please also look at initConsole() in src/native/terminal_linux.c to see a bit about how we initialize the terminal settings at startup of the P2J ChUI client.

There is also this text from our P2J Runtime Installation, Configuration and Administration Guide:


Job Control and P2J Client

In environments where interactive shells provide job control, some special care should be taken when providing custom shell scripts that launch P2J clients. An example of such an environment is the bash shell on Linux. The job control feature makes it possible to stop the client process at an arbitrary (userspecified) time by typing CTRL-Z on the terminal. The operating system suspends the P2J client process and the bash shell regains control, which would differ from a typical Progress environment. Below is the explanation of when and why this may happen and how to avoid it.

When CTRL-Z is typed in the terminal, bash, being the parent of the client JVM process, receives a SIGTSTP signal from the tty driver in the kernel, if the controlling terminal is not in “raw” mode. The kernel sends this signal to all processes in the foreground process group in the bash session. bash itself is in the group.

Under normal conditions, the P2J client will have set the controlling terminal into raw mode. When in raw mode, the SIGTSTP signal is not sent and there is no problem. However, every time the JVM launches an interactive child process, the terminal is restored to the regular “cooked” mode. If the user presses CTRL-Z while interacting with this child process, the kernel will generate the SIGTSTP signal.
In response, the JVM safely ignores the signal. Unfortunately, bash still receives the signal and then performs a "service" for us. It reacts to the signal by sending the SIGSTOP signal to the current processes in the foreground job. This latter signal cannot be caught, blocked or ignored. Thanks to bash, the JVM process stops unconditionally and bash displays the command prompt.

Disengaging the JVM from the parent bash won't help, since bash would display the command prompt immediately and the user would be able to interact with the same terminal in which the JVM is operating. This would create a situation where there is concurrent use of the same terminal, quickly resulting in screen corruption or hangs.

The solution is not to allow bash receive the SIGTSTP. One of the two known ways is the TRAP instruction. Another, more efficient way is to use exec to invoke JVM process instead of eval. This causes the parent bash to go away leaving no parent of the JVM process which can process the job control signals.

This issue only affects the CHARVA based CHUI driver, since the swingbased driver does not run inside a terminal or shell environment.

2. The child process MUST NOT have ANY access to state or resources that are available to the parent process. The STDIO must be setup with its own controlling terminal. The current directory should not be the server's current directory. The environment should be fresh and should not have anything from the P2J server's environment. The fact that the p2j.jar was read from your home dir suggests that there is still some work in this area.

3. Please organize the design as follows:

  • Minimize what is in spawn.c to only those portions that are absolutely required.
  • Put everything else in the P2J code (java or the native code, as needed).
  • The P2J native code (accessed via JNI) can launch spawn when all other setup is done.
  • Perhaps you can write the password to STDOUT and spawn can read it from STDIN. This should be just as secure as other terminal login code in Linux.

#59 Updated by Marius Gligor over 7 years ago

New changes in the current design.

1. In p2j when the user open the following URL https://&lt;host&gt;:&lt;admin_port&gt;/chui a page having a login form is send as response for this request.
The user must enter an user name and a password and press the "Log in" button.
On the server if the user name and the password are valid values the request is redirected to the chui web client page, if not the login page
is returned together with an error message.

2. In the spawner.c the password is read from stdin not from the command line.
Inside the p2j the password is provided via stdin pipe. I've tested and I found that works fine.

On the spawner.c they are still a lot of works to do:
- I saw that the spawned process has no pseudo terminal assigned!
UID        PID  PPID  C STIME TTY       TIME CMD
bubu 14718 14716 99 18:59 ? 00:11:37 java -Xmx512m -XX:MaxPermSize=6 ...
- The child process should be detached from his parent process.
In this case the p2j design should be slightly changed to provide paths inside the user account home directory for class path and native libraries path.
Each user should have the p2j client on his space. Where should be located the p2j client on each user?
I'm going to investigate this issues by reading the links posted by you and other articles related to this topics.

3. When I spawned Java process fro another account first I got the following message:

Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.
Do you ever seen such a message? Is this cause by the pty missing?
I've fixed this issue by providing the following JVM argument -Djava.awt.headless=true

#60 Updated by Greg Shah over 7 years ago

In this case the p2j design should be slightly changed to provide paths inside the user account home directory for class path and native libraries path.

It is unlikely that each user will have their own copy of P2J. No one would really want that because it would be a nightmare to maintain. And we can't have that version be different code than the code in the server's P2J jar since that can cause problems with Java serialization and our RemoteObject protocol.

The common case will be that the customer will have a shared installation somewhere on the system. I think it is acceptable to have this be part of the configuration in the server's directory. It should be possible to have this point to any valid paths on the system, but it can be shared for all spawned clients. Make sure to provide configuration options for both the classpath and the native library path.

Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.

If I remember properly, this is caused when running a new session using the root account. The X server will fail to initialize by default because running X in a root account opens too many security holes. But our client should never be run until after the root account is dropped.

I don't know if we will be able to run headless or not. One problem will be if our use of any of the swing/AWT classes is allowed. Of course, we can eliminate that dependency by using some new classes to encode colors and rework our hierarchy. It may cause other problems too. The client we run must be as functional as any ChUI client that is run interactively.

#61 Updated by Marius Gligor over 7 years ago

Today I finished to design and implements a spawner tool working like "su" command but allow us to specify the user password by redirecting the stdin.

- In the current implementation the users rights are restricted to their rights on the system.
For example the normal user "foo" has no rights to read the "mag" files on "mag" home directory.
As a result the Java code should be adapted to this behaviour by specifying shared locations for classpath and native libraries path.
Also we have to assign a working directory for our client. The spawner set as working directory the user home directory.
If you have a proposal regarding this topic please let me know.

- I fixed the issue regarding the controlling terminal. When the spawn starts a child process is created.
The child process create a new session and became a session leader (PPID=1).
A session leader is decoupled from his parent and the parent can exit without to wait for child exit.
However a session leader has no controlling terminal.
Next a new pseudo terminal (pts) is assigned to session leader process.
A new child process is created to execute the command. The session leader than wait for his child to exit.
Bellow is an example of a spawned client for user "mag":

UID PID PPID C STIME TTY TIME CMD
mag 7728 1 0 17:46 pts/10 00:00:00 /home/mag/bin/spawn mag java -Xmx512m -XX:MaxPermSize=64m -Djava.awt.headless=true -D
mag 7729 7728 99 17:46 pts/10 00:00:16 java -Xmx512m -XX:MaxPermSize=64m -Djava.awt.headless=true -Djava.library.path=/home/

mag 6876 1 0 17:34 ? 00:00:03 /usr/bin/konsole
mag 7075 6876 0 17:38 pts/5 00:00:00 /bin/bash
mag 7227 7075 0 17:39 pts/5 00:00:00 mc
mag 7229 7227 0 17:39 pts/9 00:00:00 bash -rcfile .bashrc
mag 7650 7229 0 17:46 pts/9 00:00:00 /bin/bash ./server.sh
mag 7651 7650 8 17:46 pts/9 00:00:02 java -XX:MaxPermSize=256m -Xmx1024m -server -classpath ../../../build/lib/p2jadmin.ja

- I created also a simple makefile to build the spawn tool.

- Another question please. On a secure mode both net:server:port and net:server:secure_port should be provided?

The next step is to adapt the Java code on P2J to define and use shared directories in order to spawn chui clients
other users than the user which runs the server.

#62 Updated by Marius Gligor over 7 years ago

Add GCC flags on makefile for a RELEASE build not a DEBUG one.

#63 Updated by Marius Gligor over 7 years ago

Makefile improvements.

#64 Updated by Greg Shah over 7 years ago

This is really good work. It seems very close to what we need. Some feedback:

1. There are a few coding convention issues and mis-spellings (e.g. chek_root() should probably be check_root()).

2. Setting the current dir to the home directory is the correct thing to do as the default. However, it will be very important to allow this to be overridden via configuration in the server's directory.

3. Please integrate the build of this into the primary p2j makefile. If you want to maintain a separate makefile, that is OK, but perhaps it needs to be renamed to makefile.spawn to make it clear it is dedicated for spawn. Both spawn.c and the makefile.spawn should be placed in src/native/ and build when the rest of the native p2j code is built.

4. Since we are adding some tricky permissions processing into our P2J installation/configuration dependencies, I think it is very important and every possible error condition in spawn.c be given its own unique error code. For example, we should be able to tell if setgid() failed or if setegid() failed or if... I know this will cause the code to be expanded, but the future benefits to our admins will be well worth it. Create a set of constant definitions like this:

...
#define  ERR_SPAWN_SETGID_FAILURE   -10
#define  ERR_SPAWN_SETEGID_FAILURE   -11
...

The errors should start at -1 and all be negative numbers.

5. About your question:

On a secure mode both net:server:port and net:server:secure_port should be provided?

You can see how we deal with this in LeafSessionManager.connectDirect(). Basically, if net:connection:secure is true, then we expect there to be a net:server:secure_port specified and we do NOT read or otherwise use the net:server:port.

#65 Updated by Marius Gligor over 7 years ago

1. Spawner is now integrated with p2j project.

- I did minor changes into spawn.c to manage (close) the file descriptors properly.
- I provided separate exit codes on each API calls.
- On build.xml I created 3 new targets:
spawn         // build spawn
spawn_root // build spawn and apply chown and chmode
spawn_clean // delete spawn build.
Both spawn_root and spawn_clean require root password to operate properly.
Basically spawn_root should be execute for a complete build.
I added the spawn_root target to the dependencies list of native target.
When the native target is executed prior the spawn_root target is executed which ask for root password.
If you want other behaviour please let me know.

2. I did changes on the p2j project in order to adapt the code for per client environment spawn.
The configuration vlaues are read from server directory. Bellow are my new directory settings:

<node class="container" name="chuiWeb">
<node class="boolean" name="enabled">
<node-attribute name="value" value="TRUE"/>
</node>
<node class="boolean" name="secure">
<node-attribute name="value" value="FALSE"/>
</node>
<node class="string" name="target">
<node-attribute name="value" value="/chui"/>
</node>
<node class="string" name="spawner">
<node-attribute name="value" value="/usr/share/p2j/spawn"/>
</node>
<node class="string" name="jvmArgs">
<node-attribute name="value" value="-Xmx512m -XX:MaxPermSize=64m -Djava.awt.headless=true"/>
</node>
<node class="string" name="classpath">
<node-attribute name="value" value="/usr/share/p2j/lib/p2j.jar"/>
</node>
<node class="string" name="libPath">
<node-attribute name="value" value="/usr/share/p2j/lib"/>
</node>
<node class="string" name="workingDir">
<node-attribute name="value" value="./"/>
</node>
<node class="string" name="configFile">
<node-attribute name="value" value="config.xml"/>
</node>
<node class="integer" name="rows">
<node-attribute name="value" value="24"/>
</node>
<node class="integer" name="columns">
<node-attribute name="value" value="80"/>
</node>
<node class="string" name="background">
<node-attribute name="value" value="0x000000"/>
</node>
<node class="string" name="foreground">
<node-attribute name="value" value="0xFFA500"/>
</node>
<node class="string" name="selection">
<node-attribute name="value" value="0x0000FF"/>
</node>
<node class="string" name="fontname">
<node-attribute name="value" value="monospaced"/>
</node>
<node class="integer" name="fontsize">
<node-attribute name="value" value="12"/>
</node>
</node>

I used /usr/share/p2j as a location to shared spawn, native library, p2j.jar and dependencies.
Users should have rights to read files and scan directory from this location.
When the user open https://&amp;lt;host&amp;gt;:&amp;lt;admin_port&amp;gt;/chui from a web browser a page having an authentication form is send as response.
The target root /chui is also configurable. Is this OK or is best to remain hard codded?
The user should enter his account credentials (user name and password) an press the "Log in" button.
If the credentials match the user account a new web client is spawn using his account environment and the
original request is redirected to a new URI, the "chui web client" JS WebSocket application.
On any error the login page having an error message is returned and the user could try again.
This spawner is build on Linux OS and my expectations are to work properly on almost all Linux distributions with few or no changes.
I'm not sure how it works on Unix OS like Solaris because I have no conditions here to do tests.
Mac OS is also build on top of an Unix kernel, but I think that nobody is using a Mac machine as a server.
For Windows OS we have to think to a solution. I will try to see if runas command supports stdin redirection for password.
Usually the Windows Servers OS are more restrictive on security aspects than an trivial Windows Workstation OS.
So a best idea is to finally test the Windows spawn solution on a Windows Server OS.
Anyway this constraints are for the OS on which the p2j server is running. The user could use any OS since the application
is a web one and he need only a web browser Chrome, Firefox, Safari, Opera or IE.
To use only a Linux server to run the p2j server could be a solution?
Do you have clients that require Windows OS as server host?

#66 Updated by Greg Shah over 7 years ago

I will do the code review next. First, I wanted to get some basic answers to you.

When the user open https://&lt;host&gt;:&lt;admin_port&gt;/chui from a web browser a page having an authentication form is send as response. The target root /chui is also configurable. Is this OK or is best to remain hard codded?

It is fine to be hard coded. This will simplify the configuration and documentation. There is no good reason to change the URL so hard coding is OK.

To use only a Linux server to run the p2j server could be a solution?

This is not possible.

Do you have clients that require Windows OS as server host?

Yes, Windows OS support is required.

#67 Updated by Marius Gligor over 7 years ago

1. I did some minor changes on the current implementation:
- The target root /chui is now hard coded.
- On the spawner I did a change to fix the set-up of the working directory for spawned process.

2. For Windows OS using "runas" command is useless, like using "su" command on Linux.
However I found Win API calls which allow us to design and implements a small tool that looks like Linux spawn.
Basically the tool will have the same arguments on the command line and the behaviour will be the same obviously on Windows environment.
Currently I'm working to design and implements the spawn tool for Windows OS.
Until the end of the day I will send you a first release of this tool.

#68 Updated by Greg Shah over 7 years ago

Please fix these two javadoc errors as part of your update:

  [javadoc] /home/ges/projects/p2j/src/com/goldencode/p2j/ui/client/chui/driver/swing/ChuiSimulator.java:79: warning - Tag @link: reference not found: ChuiApplet
  [javadoc] /home/ges/projects/p2j/src/com/goldencode/p2j/ui/client/chui/driver/swing/ChuiSimulator.java:579: warning - @param argument "opt" is not a parameter name.

#69 Updated by Marius Gligor over 7 years ago

Fixed javadoc errors on ChuiSimulator.java

#70 Updated by Marius Gligor over 7 years ago

This is the first Windows OS version of our spawner. Its looks and works like the Linux spawner.
I'm using Microsoft Visual C++ 2010 Express Edition and Microsoft Windows 7 SDK to create and build this solution.
I tested on my laptop having a Windows 7 OS and found to work properly.

Useful informations:

- Users MUST have a password, blank password is not allowed.
- According to MSDN this API's are available since _WIN32_WINNT as 0x0500 (Windows 2000).
- Due to some privileges requirements we can improve this version by combining CreateProcessAsUser call
with CreateProcessWithLogonW call as MSDN recommended (see bellow). I will try to do that.

MSDN:
Windows XP with SP2 and Windows Server 2003: You cannot call CreateProcessWithLogonW from a process that is running under the "LocalSystem" account, because the function uses the logon SID in the caller token, and the token for the "LocalSystem" account does not contain this SID. As an alternative, use the CreateProcessAsUser and LogonUser functions.
To compile an application that uses this function, define _WIN32_WINNT as 0x0500 or later.

MSDN:
Typically, the process that calls the CreateProcessAsUser function must have the SE_INCREASE_QUOTA_NAME privilege and may require the SE_ASSIGNPRIMARYTOKEN_NAME privilege if the token is not assignable. If this function fails with ERROR_PRIVILEGE_NOT_HELD (1314), use the CreateProcessWithLogonW function instead. CreateProcessWithLogonW requires no special privileges, but the specified user account must be allowed to log on interactively. Generally, it is best to use CreateProcessWithLogonW to create a process with alternate credentials.

#71 Updated by Greg Shah over 7 years ago

Code Review 1202b

This is really good work. Some feedback:

1. Please look at src/com/goldencode/p2j/util/EnvironmentOps.getOperatingSystem() and src/com/goldencode/util/PlatformHelper.java. I prefer to enhance one of these instead of implementing another approach to OS detection like Os.java. The EnvironmentOps approach is 4GL compatible, so it is probably more correct to modify PlatformHelper if necessary. Today, we simply treat things as Windows and non-Windows (which are effectively all UNIX/Linux). This will work fine for all platforms that we anticipate working with in the near future. Let me know what changes might be needed. I want to make sure we don't break the current functionality.

2. Instead of calling the native build of spawn from build.xml, I prefer if you call it from src/native/makefile. I am also attaching some recent changes that I have made to build.xml and makefile. This won't work without additional files that I am editing, but it will give you can idea of the changes that are in play here. You will see that I have moved much of the OS detection and other complexity from build.xml to makefile, in an effort to centralize the native stuff and make it as independent as possible. It is straightforward to call the spawn build from the main makefile.

3. Please consider how we can configure things to allow the builds to be automated. You can edit the sudoers file and setup a particular command to be executed as root without any password prompting. The idea here is that we need to ensure that we can enable things like our automated regression testing or continuous integration servers to build without needing a human to authorize the chown/chmod processing.

4. Does the Linux/UNIX spawn approach integrate with the PAM ("Pluggable Authentication Modules") infrastructure? I assume it does, but we should know one way or the other.

5. Please change the spawn.c to have if(, switch( and check_user ( to be coded as if (, switch ( and check_user(.

6. When the authentication fails (AUTH_ERR_*), shouldn't we return a specific code via exit() instead of writing using puts()? Otherwise, how will the user (at the web browser) see the error?

7. In comments and javadoc, "lunch" or "Lunch" should be "launch"/"Launch". ;)

8. Can we easily crypt the plain text password in Java (using J2SE features) that is compatible with the Linux/UNIX crypt? The idea is to do this as soon as the input is made via the POST so that we don't store the plain text in memory. Then we can remove the crypting in spawn.c and never have to worry about the plain text being intercepted. In a perfect world it would alreayd be crypted in javascript before sending it over the wire to the P2J server, but that may be too much work to do quickly.

9. I wonder if some of the settings in ClientBuilderOptions should not be static. For example, in the common case it is likely that the working directory will not be something that is shared between users. Of course, the most common case is to have the working dir be set to the home dir, but I can see other cases too. Please consider which of these may be useful to split up on a per user basis and/or how to deal with this issue.

10. In ClientBuilder, the use of System.out.println() should probably be replaced with proper logging. I can see this as being useful long term (but only when verbose logging is requested).

11. Let's default to not having the client.xml bootstrap file. It is not required generally and we want to avoid using it unless absolutely necessary. It looks to me like we always try use it today and warn when it is not around. It should be rare to need it. In fact, I would prefer to add directory options to allow the customer to put all the needed settings in the directory (and we would just construct the proper command line with all those options), eliminating the need for the bootstrap config. For now, leave it as a possibility but if it is not specified in the directory, don't try to add it.

12. Please rename onRemoteServerListen() to embeddedWebClientIsReady().

13. I think SpawnerImpl needs to be properly synchronized.

14. I'd like to see the error codes from spawn failures (and appropriate error text messages) propagate back to the error page returned by WebHandler. At a minimum, the AUTH_ERR_* issues must be shown to the user. But I think it makes sense to show all errors rather than forcing the admin to look at some other location.

15. We still have the "chicken and egg" problem I mentioned before. This current code only works because the simple server environment is configured for guest access. As soon as a normal P2J authentication process is required, the user won't be able to access the login screen from the web browser, because during connectDirect(), the client's embedded web server is not yet started (AND the server's key store has not yet been obtained AND the remote URI has not yet been returned back to the server's WebHandler). Today, the connectDirect() is able to establish a session without any user interaction because of the special guest access authentication plugin that is in use in your testing environment. That is not a normal configuration. The trick here: how do we move the key store and URI transfer to happen BEFORE the connectDirect() AND still keep strong security/privacy?

#72 Updated by Greg Shah over 7 years ago

About the Windows approach and winspawn.zip:

1. Please do not use the MS VisualStudio or Visual C++ tools. It would bring too many proprietary dependencies to our project. For Windows development, we use mingw. This is an open source, Windows port of gcc and the other common build tools that we use on UNIX/Linux (the linker, make...) that is packaged in a manner that allows full WIN32 development. This is not using the cygwin compatibility environment at runtime, instead we would link to the msvcrt to get the basic C runtime (this DLL is part of the OS so it is OK).

To install:

1. Take one of the full archive: mingw-w32-bin-i686-2013MMDD.7z or mingw-w64-bin-x86_64-2013MMDD.7z. Unpack to \mingw32 or \mingw64.
2. Include the appropriate \mingwXX\bin to the PATH variable. In my systems: C:\mingw64\bin and C:\mingw32\bin depending on Windows architecture.
3. The 64-bit binaries are found at http://www.drangon.org/mingw. The 32-bit binaries and much more info can be found at http://mingw.org.

Please look at the makefile I posted to see the windows specifics. You can make some minor changes and have the same makefile.spawn used for both.

2. Generally, try to use common code between spawn for Linux and spawn for Windows. Where needed, use OS APIs. Where possible use C runtime features that are common. See our src/native/ directory for how we implement platform specific helper files that are called from common/shared source files.

Otherwise, I think you are on the right track.

#73 Updated by Marius Gligor over 7 years ago

1. In think that Windows and non-Windows (which are effectively all UNIX/Linux) detection is enough. Os.java has been deleted.

2. TODO. OK I have to do that after the implementation of both spawn projects for Linux and Windows is done.

3. Indeed we can configure sudoers to allow particular command to be executed without any password prompting.
In this case the sudoers file should be edited "by hand" on each workstation where the build is running.
We cannot do that automatically. We can only show how to do that and each member of the team should follow our instructions.
Is this what you are looking for?

4. The Linux/UNIX spawn approach is not integrated with the PAM ("Pluggable Authentication Modules") infrastructure, but will be soon.
Currently I'm working to add PAM authentication support to spawn.c
Using PAM is more portable over Linux/Unix platforms.

5. Done

6. Done. Among error messages an error code is already returned from the main method (see exit(result)).

7. Done.

8. I'm afraid isn't possible to do that on pure Java. Also the PAM implementation require the password unencrypted.
Even more as you can see the crypt API require the clear password and the encrypted password.
crypt(password, correct); The encrypted password contains a "salt" used on encryption according to Linux documentations.
I found also that the algorithm is not the same on all Linux platforms.

9. The working directory could be absolute or relative. With absolute paths the users shares the same working directory.
With relative paths each user has it own working directory inside its home directory.
Another issue here. I tried to access directory services when I prepared to spawn a client. It doesn't work!
According to my debugs something like directory is not bound cause the error.

Location of this error is on DirectoryService.java line 4770, the return value is null, see bellow.
protected Remapper synchBatch(String id, boolean rw)
{
if (!isBound())
return null;

The only way to read the directory values was at the moment of StandardServer start-up.
Do you have any idea how to fix that?

10. Done.

11. Done.

12. Done.

13. On SpawnerImpl I'm using a ConcurrentHashMap which doesn't require synchronization.

14. Done with some remarks. When we launch the spawner we have control only over the spawner process.
The spawner will exit after the first fork() and the return code from child processes are no longer under our control.
I can get only the exit code from the spawn process not for it child's.
In conclusion I implemented this feature only for spawner process exit codes (AUTH_ERR_*).

15. I understood. Here we need to use a secure interprocess communication mechanism. Could be JMX, Java RMI, JMS or maybe another solution.
At this moment I have no idea about a best solution. I have to do some "researches" to find a solution.

#74 Updated by Greg Shah over 7 years ago

3. Indeed we can configure sudoers to allow particular command to be executed without any password prompting.
In this case the sudoers file should be edited "by hand" on each workstation where the build is running.
We cannot do that automatically. We can only show how to do that and each member of the team should follow our instructions.
Is this what you are looking for?

Yes, almost. The only additional issue I see is that we don't want to have chown and chmod in the sudoers file as the command. The reason: it is a big security hole to open to solve a small problem like this.

Instead, please create a shell script that handles the exact chown/chmod requirements. Then as part of the instructions, make sure that the ownership and permissions on that shell script are set very tight (maximal limitations).

The only way to read the directory values was at the moment of StandardServer start-up.
Do you have any idea how to fix that?

You have to have a security context defined for any thread that accesses the directory. On such a thread, the following code will provide access:

      DirectoryService ds = DirectoryService.getInstance();

      if (!ds.bind())
         throw new RuntimeException("directory bind failed");

You can see code like that at the top of StandardServer.standardEntry() which is the primary server-side entry point for new client sessions to start running 4GL code.

You may also want to look at the code in StandardServer.bootstrap() which is how the server initializes. That involves creating the directory service and the security manager instances. Notice how we don't read anything from the directory before the security manager is operational.

The WebServer instance is created and initialized on that same bootstrap thread. So reading the directory is OK there. But when the parent class (GenericWebServer) is instantiated, it calls startup() which creates a thread pool for handling requests. These handler threads don't have such a context and cannot access the directory.

For now, can we read the directory during initialization and pass the data into the web server? I prefer not to add security context to the web server handler threads if not necessary, to limit the security risk that may happen if those threads are breached in an attack. Worst case, we will open up some context as needed.

I can get only the exit code from the spawn process not for it child's.
In conclusion I implemented this feature only for spawner process exit codes (AUTH_ERR_*).

We will need to ensure that the child process' return code is written somewhere for logging/diagnosis purposes.

15. I understood. Here we need to use a secure interprocess communication mechanism. Could be JMX, Java RMI, JMS or maybe another solution.
At this moment I have no idea about a best solution. I have to do some "researches" to find a solution.

One way might be to use a pipe. We could open that in the spawner and inherit it in the child. The key store can be read from it and the URI can be written back to it. This has the advantage of being easily accessible from native code and very portable.

A better solution is to implement a temporary P2J session with the server. The idea: create one-time authentication credentials (some kind of random token/key) that can be passed (like our password info) via STDIO to the child process. If we can find a secure way to get it to the P2J client, the P2J client can use this to do a temporary connectDirect() to the server with those credentials, read the key store, initialize the web server and send back the URI. Then it would disconnect and drop into the normal processing. The major advantage of this approach is that it can be extended to the "remote client" scenario, which we will be implementing later. Another major advantage is that we plan to do something similar to allow batch mode P2J clients to be autostarted by the server.

Constantin: do we have anything already made for appserver that can help us here? How do the new appserver "clients" authenticate?

#75 Updated by Marius Gligor over 7 years ago

1. Today I succeeded to build the spawner for Windows OS using MinGW compiler.
MinGW does no support main functions for UNICODE applications (_tmain, wmain). This is a well known issue among MinGW Windows developers.
UNICODE is mandatory on spawner because the function CreateProcessWithLogonW is available only for wide characters not for ANSI characters.
Fortunately using a main function wrapper I fixed this issue.

2. I added support for PAM on the Linux version of spawner.
As I told you on Linux version I can get back in Java only the exit code for the master process.
My idea is to use the Linux syslog to audit the execution of child processes.
Please let me know if you agree this solution and I will do the appropriate changes.

3. I started to design a makefile for both Linux and Windows builds.
On Windows we have to use the MinGW compiler and I need references to include and lib folders of MinGW.
I created an environment variable MINGW_HOME instead to set the reference in the makefile.
Also on MinGW the name of the maker is mingw32-make and on Linux gcc is only make.
So far I created 2 ant targets but I have to find if is possible to create properties for maker
depending on platform and finally to have a single ant target for build on Linux and Windows.

#76 Updated by Greg Shah over 7 years ago

My idea is to use the Linux syslog to audit the execution of child processes.
Please let me know if you agree this solution and I will do the appropriate changes.

That is OK.

On Windows we have to use the MinGW compiler and I need references to include and lib folders of MinGW.
I created an environment variable MINGW_HOME instead to set the reference in the makefile.

That is strange. I didn't find the same requirement in src/native/makefile. What is different in the spawn case?

Also on MinGW the name of the maker is mingw32-make and on Linux gcc is only make.

Make sure you are not using the automated installer for mingw. I initially had the same problem but when I "manually" unzipped the archives, everything was named properly (make was make).

So far I created 2 ant targets but I have to find if is possible to create properties for maker
depending on platform and finally to have a single ant target for build on Linux and Windows.

Please call the processing of the spawn makefile from the P2J src/native/makefile instead of from ant.

#77 Updated by Greg Shah over 7 years ago

I guess I should point out that our ant build.xml and src/native/makefile already works for mingw on 32-bit Windows, so you should not have to make too many changes. There must be something different about your environment, from the environment on which we normally run.

Eugenie: do you have any ideas?

#78 Updated by Eugenie Lyzenko over 7 years ago

Eugenie: do you have any ideas?

I think the best way is to take precompiled binaries(e.g from here: http://code.google.com/p/mingw-w64-dgn/). Then unzip to some location and include path to the BIN subdirectory in Windows PATH variable.

It should work. No other changes required. And "make" will be "make".

The recent binaries are: mingw-w32|64-bin-i686|x86_64-20131018.7z. This works for both 32 and 64 bit Windows(Running inside KVM on Ubuntu).

#79 Updated by Marius Gligor over 7 years ago

1. I downloaded and copied the MinGW binaries for both Win 32 and Win 64 bit platforms as follow:

c:\mingw32  (32 bit binaries)
c:\mingw64 (64 bit binaries)

In PATH I added c:\mingw64\bin Now the maker is make.exe and this is OK.

When the winspawn.c is build following references are needed:

For compile the following include files:
Windows.h
WinBase.h
Userenv.h
Shlwapi.h

For link the following libraries:
libuserenv.a
libshlwapi.a

These references are available from:

c:\mingw32\i686-w64-mingw32\include 
c:\mingw32\i686-w64-mingw32\lib
and
c:\mingw64\x86_64-w64-mingw32\include 
c:\mingw64\x86_64-w64-mingw32\lib

To solve this issue I used 2 environment variables:

MINGW32_HOME=c:\mingw32\i686-w64-mingw32
MINGW64_HOME=c:\mingw64\x86_64-w64-mingw32

Inside makefile.spawner I used a place holder MINGW_HOME and depending on detected platform MINGW32_HOME or MINGW64_HOME is assigned to MINGW_HOME.
I did the build on both 32 and 64 platform and it's works.

Do you agree this solution? If not please let me know your proposals.

2. To run the build I created a new target in makefile which is a recursive use of make to build spawn from makefile.spawn
The spawner is build when the native classes are build using ant native target. No ant target for spawn build are available as you suggested.

However I found a little problem here.
If the JRE_HOME variable is located in a directory having spaces in name like JRE_HOME=C:\Program Files\Java\jdk1.7.0_21 the ant native build
fails because the space character between "Program" and "Files" is interpreted as a separator when the INCLUDES is build using foreach (see bellow):

INCLUDEDIRS=${JRE_HOME}/../include ${JRE_HOME}/../include/win32 $(SRCDIR)
override INCLUDES+=$(foreach d,$(INCLUDEDIRS),-I$(d))

One solution is to use JRE_HOME on a path without spaces like: JRE_HOME=C:\jdk1.7.0_21 and everything works.

#80 Updated by Greg Shah over 7 years ago

libuserenv.a
libshlwapi.a

What are these?

As a general rule, I prefer to dynamically link to DLLs that are part of the base OS rather than using static linking, but the decision can be made on a case by case basis.

Do you agree this solution? If not please let me know your proposals.

Yes, I agree.

If the JRE_HOME variable is located in a directory having spaces in name like JRE_HOME=C:\Program Files\Java\jdk1.7.0_21 the ant native build
fails because the space character between "Program" and "Files" is interpreted as a separator when the INCLUDES is build using foreach (see bellow):

Please rework this makefile code to eliminate this problem. I don't want to force more environment dependencies when we can just use a different technique.

#81 Updated by Eugenie Lyzenko over 7 years ago

For point 1:

Are you building both 64 bit and 32 bit targets on only 64-bit Windows system?

#82 Updated by Marius Gligor over 7 years ago

I have a Windows 7 64bit OS installed on my laptop.
Using ant native target the 64bit platform is detected and the spawner is build using MinGW 64bit tools only.
No 32bit build is done if detected platform is 64bit.
No 64bit build is done if detected platform is 32bit

I did the 32bit build just for testing purposes using a separate makefile without platform detection.

#83 Updated by Marius Gligor over 7 years ago

1. For Linux spawner I added PAM support and syslog audit.
Basically the spawner can be build with or without PAM support depending if PAM is defined or not in the build script.
By default the PAM is defined and spawn is PAW aware. To check you could do that:

ldd spawn | grep libpam

to see if spwan depends on libpam.

2. To build the spawner for Linux and Windows a single makefile target is used.
On the makefile for build java native libraries I added a new target spawn which in turn will start the spawner make build script from makefile.spawn
When the native artifacts are build the spawn artifact is also build. No ant targets are used to build the spawner.
I have a postbuild target defined in the makefile.spawn script used to do the sudo commands on Linux/Unix.
On Windows postbuild target is used only to copy the spawn.exe file to /build/lib folder.
In Linux postbuild target run a bash script postbuild.sh which should be added to sudoers in order to skip enter the sudo password.
How to add to sudoers is described inside the postbuild.sh script file.
Unfortunately I cannot test the build script for Solaris OS because I have no such OS installed on my machines.
Searching on Internet I found that Solaris does not have sudo command but could be installed.
Could you help me here with some suggestions?

3. Regarding the space characters in file path I didn't found any solution.
All that I found so far says to avoid using space characters in file names and paths when working with GNU make.
GNU make does not support spaces in file names or paths.

#84 Updated by Marius Gligor over 7 years ago

I changed the winspawn.c to use dynamic DLL link instead a static link.
As a result of this changes the makefile build script has been simplified. The reference MINGW_HOME is no longer needed.
The MAX_PATH (260) is not enough to hold the command line and has been increased.
I done also a minor improvement in postbuild.sh

#85 Updated by Marius Gligor over 7 years ago

Today I started top test the spawner on Windows OS environment and I found some interesting issues.

1. The font Monospaced is available in both Linux and Windows OS. However ClientDriver in Windwos does not work using this font.
After a short debug I found the cause. At one moment in ChuiSimulator line 627 a check on font is done to be monospaced.
The test is performed by the code from SwingHelper#isMonospaced as follow:

if (!font.canDisplay('\u2500'))
return false;
// check if this font is monospaced, this is a bit of a hack but should
// be reliable
int msize = fm.charWidth('m');
int isize = fm.charWidth('i');
int osize = fm.charWidth('1');
int nsize = fm.charWidth('9');
int dsize = fm.charWidth('\u2500');
return (msize  isize && msize  nsize &&
msize osize && osize dsize);
According to my tests the following values are returned for Monospaced font size 12:
Linux      Windows
msize = 7 7
isize = 7 7
osize = 7 7
nsize = 7 7
dsize = 7 12
As you can see the dsize which is the size of character use to draw boxes is different
in Linux and in Windows. As a result in Windows the function return false and the font
is not recognised as monospaced by the ClientDriver
Here I have an ideea how to fix this issue and I want to ask you if is OK to do that:
if (!font.canDisplay('\u2500'))
return false;
// check if this font is monospaced, this is a bit of a hack but should
// be reliable
int msize = fm.charWidth('m');
int isize = fm.charWidth('i');
int osize = fm.charWidth('1');
int nsize = fm.charWidth('9');
int dsize = fm.charWidth('\u2500');
int gsize = font.getSize();
return (msize  isize && msize  nsize &&
msize osize && (osize dsize || fsize == dsize));
In other words we can consider also the case on which the size of the box character is equals to font size. 
I tested this code on Windows and its works.

2. A second issue related to fonts in Windows is when the font name contains spaces.
For example I used the font "Lucida Sans Typewriter" 12 which works on Windows.

Windows
msize = 7
isize = 7
osize = 7
nsize = 7
dsize = 7
Here the name should contains quotes in command line.
Finally I found the solution and I already fixed this issue on Java and now its works.

#86 Updated by Greg Shah over 7 years ago

In other words we can consider also the case on which the size of the box character is equals to font size.
I tested this code on Windows and its works.

I am fine with the change.

#87 Updated by Greg Shah over 7 years ago

Code Review 1206a

The code is moving forward nicely. Keep up the good work.

#88 Updated by Marius Gligor over 7 years ago

I tested the Windows spawner and found to work properly.
I fixed also some issues related to fonts, monospace fonts detection and space characters in font names.

#89 Updated by Marius Gligor over 7 years ago

1. The archive contains some small improvements. I extended the use of quoted string on file name and path strings as well.

2. I understand that we cannot use the p2j RemoteObject to export the server KeyStore as currently is implemented
an we have to do that earlier before the p2j authentication when RemoteObject is not available.
So I did a research to find a cross platform solution for server KeyStore imports / exports. Bellow are my conclusions:

a). Named pipes are available on both Linux/Unix and Windows platforms.
However Java does not provide support for named pipes.
The implementation of named pipes is different on Linux and Windows platforms.
In Linux named pipes are implemented as files but on Windows not.
On Windows it's possible to create and use named pipes only by using Win API.
In order to use named pipes a JNI solution should be designed and implemented.

b). The best solution remains the use of a network socket solution.
This solution is cross platform, portable, less error prone and easier to implements.
On this area I have 2 proposals:

- First proposal is to use the Hessian Binary Web Service Protocol (http://hessian.caucho.com/) which is easily
to integrate with p2j jetty web server (just one jar file added to dependencies), its works like a RMI or RPC solution
having support for serialization but over http/https protocols.
Hessian implementation is released under an open source license (the Apache license).
- The second solution is to use a Jetty WebSocket's solution to do almost the same job.
In this case we have also to design an application protocol (OSI layer 7) used to manage the conversations
between client and server.
Both socket solutions allow local/remote access, are portable over platforms, and are secured due to Web Server secured environment on which they runs.
For simplicity I prefer the first solution Hessian Hessian Binary Web Service Protocol.

Please let me know your opinions regarding this issue.

#90 Updated by Greg Shah over 7 years ago

Before I get into a deep review of these new options, what about my preferred solution as documented in note 74?

A better solution is to implement a temporary P2J session with the server. The idea: create one-time authentication credentials (some kind of random token/key) that can be passed (like our password info) via STDIO to the child process. If we can find a secure way to get it to the P2J client, the P2J client can use this to do a temporary connectDirect() to the server with those credentials, read the key store, initialize the web server and send back the URI. Then it would disconnect and drop into the normal processing. The major advantage of this approach is that it can be extended to the "remote client" scenario, which we will be implementing later. Another major advantage is that we plan to do something similar to allow batch mode P2J clients to be autostarted by the server.

#91 Updated by Marius Gligor over 7 years ago

OK. Where I can find such an example in our p2j project?

#92 Updated by Greg Shah over 7 years ago

Eric has done some work with creating an authentication token for remote database access. Perhaps that can help.

The authentication token/certificate should be 1-time use only. It must have a large enough random hash that it has enough entropy that it cannot be reasonably broken by any kind of brute force attack.

Constantin: I'll repeat the previous question:

do we have anything already made for appserver that can help us here? How do the new appserver "clients" authenticate?

Today we also already have an infrastructure for certificate based authentication. If we can enhance that to make the certificates a one-time use thing (generated on the fly?), then the rest of the solution should be an effort in properly configuring the client and server and then calling the connectDirect().

#93 Updated by Eric Faulhaber over 7 years ago

See com.goldencode.util.RandomWordGenerator and its use in com.goldencode.persist.DatabaseManager.createEmbeddedProperties(Database).

#94 Updated by Constantin Asofiei over 7 years ago

do we have anything already made for appserver that can help us here? How do the new appserver "clients" authenticate?

There was nothing special added for the authentication of appserver agents; but, the configuration of an agent requires it to have a passwordless login (i.e. certificate, trusted or something else).

#95 Updated by Eric Faulhaber over 7 years ago

Eric Faulhaber wrote:

See com.goldencode.util.RandomWordGenerator and its use in com.goldencode.persist.DatabaseManager.createEmbeddedProperties(Database).

Please note the performance-related implementation note in RandomWordGenerator.fillRandomSlots. If you need to generate very long words frequently, the algorithm can be improved by decrementing the len value by 1 each time through the loop and walking the array to the generated index, skipping slots already full (also will need to account for elements already in the array from previous passes when determining initial value of len). Not terribly efficient, but possibly better than calling Math.random many times unnecessarily. Another approach might be to add a level of indirection using an additional array to track used indexes in the first array. Another is to simply advance to the next open slot if the slot at the generated index is full, though we lose some randomness with this approach. I'm sure there are other approaches as well...

Anyway, I didn't need this extra efficiency/complexity for the initial use case, which is called infrequently and only generates words of 128 characters, so I didn't complicate the initial implementation.

#96 Updated by Greg Shah over 7 years ago

You can also look on devsrv01 at the ~/testing/majic/run/batch/ directory to find examples of batch client configurations that can be used with run/client/client.sh when you also use the -b option. Of course, these are each also configured already in the server's directory. The trick in this case is to dynamically create all this configuration in a manner that can only ever be used once.

I do prefer the use of a real certificate over an auth token, as it should enable full use of the rest of our batch process support without any changes to the authentication process.

#97 Updated by Constantin Asofiei over 7 years ago

Eric, about optimizing the RandomWordGenerator.fillRandomSlots - take a look at this approach:
- each possible slot in the buffer (from 0 to buffer.length - 1) is added to a list, L. This is passed to each call of fillRandomSlots.
- when determining which slot is next to be assigned, we generate a random number R between 0 and L.size() - 1. The next slot in the buffer will be determined by the value of element at position R in list L.
- after R is determined, we remove the element at position R from list L. Thus, list L will decrease in size each time a buffer slot is assigned and there will be at most buffer.length random calls (for the buffer slot computation).

#98 Updated by Eric Faulhaber over 7 years ago

This is similar to the additional array approach I noted, but I agree a list (specifically linked list) is a better implementation choice, given the removal idea. Thanks for the details.

The current approach is not a bottleneck at all if it is not being called constantly for very large strings, so we should only complicate the implementation if there is a performance need.

#99 Updated by Greg Shah over 7 years ago

Code Review 1209a

I am fine with the changes.

#100 Updated by Marius Gligor over 7 years ago

Today I started to learn about SecurityManager and SessionManager which are complex technologies.
So far I investigated the SessionManager#connectDirect client side flow and I hove some questions:

Doing debugs on the flow finally I found the TransportSecurity constructor class.
Here on the Java doc I found that on the client side a combination of truststore and keystore is used to create an in memory KeyStore.
Looking on Majic client start-up configurations both a truststore and a keystore are used on almost all cases.

1. Could you explain me in a few words what kind of information is stored in this files?
2. Is any server dependency inside this information data?
3. On the client side processalias and useralias are used as aliases on client truststore and keystore resources.
On the server side inside directory I found 2 containers users and processes.
Is any relation between the directory containers and aliases? What are the differences between users and processes regarding authentication?

4. When no truststore and keystore are used on client side an anonymous (guest) authentication is considered.
Currently this guest account is defined inside server directory users container and is mapped to user "bogus".
As I understood the guest account is not secured and could be invalidate is this true?

5. On SecurityManager#authenticateClientWorker I found that multiple authentication modes are available.

AUTH_MODE_X509
AUTH_MODE_IDPW
AUTH_MODE_X509_IDPW
AUTH_MODE_CUSTOM
Each kind of authentication mode has specific use.
AUTH_MODE_X509 - using c X509 certificate
AUTH_MODE_IDPW - user name and password
AUTH_MODE_CUSTOM - seems to be the case of using a custom plug-in like you told me you already use on Majic.
The authentication mode is established by server is this true or could be requested by client?
Where is specified the authentication mode inside server?

6. The TransportSecurity is a singleton as is described by the javadoc. We cannot not call SessionManager#connectDirect twice using 2 different configurations.
Maybe some changes should be made if we will need to use 2 calls. What do you think about this?

7. It is possible to dynamic generate a KeyStore and a X509 certificate self signed at any moment. It is not possible to design such objects for single use.
The expiration date is the single element that is used to validate / invalidate the certificate.
Using an expiration date closest to the issue date a certificate could be used for a short time but not only once.

8. It's still not clear for me how to implements your requests.
So far my idea is to do the following:

On client side:
a) Generate truststore and keystore dynamically on the client side.
b) Try to connect to the server and create a session via a bootstrap configuration containing the generated truststore and keystore and other specific settings.
The authentication mode could be AUTH_MODE_X509.
c) On success, import server key store, start the embedded server and notify back on server about the URL on which the embedded server listen.
d) Close the connection (session) with the server. This was just a temporary session used only for our purposes.
e) Continue the flow, create a new session with server using the default configuration, etc.
...
On server side:
I don't know yet but I think that as a minimum an account having X509 certificate authentication mode should be defined.
Here I have to do more investigations.

Please help me with any suggestions.

#101 Updated by Marius Gligor over 7 years ago

Inside directory.xml I just found:

<node class="authMode" name="auth-mode">
<node-attribute name="mode" value="4"/>
<node-attribute name="retries" value="0"/>
<node-attribute name="plugin" value="com.goldencode.p2j.security.GuestAccess"/>
</node>
<node class="container" name="auth-plugins">
<node class="container" name="guest_login">
<node class="string" name="classname">
<node-attribute name="value" value="com.goldencode.p2j.security.GuestAccess"/>
</node>
<node class="string" name="description">
<node-attribute name="value" value="Non-Interactive Guest Auto-Login"/>
</node>
<node class="string" name="option">
<node-attribute name="value" value="bogus"/>
</node>
</node>

Something similar I found inside Majic server directory.xml
The question here is:

Only a single auth-mode and a single plugin can be defined per server instance?
If yes we cannot create a temporary session for server key store import because the same plugin is always present on all sessions.

#102 Updated by Greg Shah over 7 years ago

1. Could you explain me in a few words what kind of information is stored in this files?
2. Is any server dependency inside this information data?

For details, please see the "Key-Stores and Trust-Stores" chapter of the "Progress 4GL to Java (P2J) Installation, Configuration and Administration Guide".

3. On the client side processalias and useralias are used as aliases on client truststore and keystore resources.
On the server side inside directory I found 2 containers users and processes.
Is any relation between the directory containers and aliases?

Yes, in an indirect manner. In the directory, account information for user accounts is kept in the users container and process accounts in the processes container. The useralias is how the client code finds the certificate to represent a user account and the processalias is how it finds the cert for a process account.

What are the differences between users and processes regarding authentication?

It is an unfortunate design decision that I should have eliminated early in the project. The bottom line: a process account is essentially one that is not directly associated with a specific end-user, but is instead usually representing some non-interactive process like a:

  • converted 4GL "batch" program
  • remote Java application that is connecting to the P2J server to make remote object calls

The authentication mode is established by server is this true

Yes.

or could be requested by client?

The client must be trying to use one of the modes that are supported by the server. Otherwise this would be a security hole.

Where is specified the authentication mode inside server?

It is configured in the directory (as you have recently found).

6. The TransportSecurity is a singleton as is described by the javadoc. We cannot not call SessionManager#connectDirect twice using 2 different configurations.

As far as I know it is not a singleton. Where do you see this?

7. It is possible to dynamic generate a KeyStore and a X509 certificate self signed at any moment. It is not possible to design such objects for single use.
The expiration date is the single element that is used to validate / invalidate the certificate.
Using an expiration date closest to the issue date a certificate could be used for a short time but not only once.

An idea here would be to remove the certificate once it has been used one time. In other words, if the SecurityManager no longer honors that cert after it has been used once, then it doesn't matter that the expiration time is not over yet.

Please help me with any suggestions.

http://security.stackexchange.com/questions/14000/environment-variable-accessibility-in-linux

On modern systems, it seems that the process' environment is reasonably secure (only accessible by root and by other processes of that same euid). Can we do the following:

  1. On the server, generate a one-time use userid and password that is random enough and big enough to be proof against brute force attacks.
  2. On the server, the security manager would have to be aware of this special account (it would have to be enabled and then disable itself after being used once).
  3. The server would pass this information to the spawn executable.
  4. The spawn executable would insert this into the environment and the spawned child process can read it there.
  5. The client will use this to establish the temporary session.
  6. In the temporary session, the keystore can be read, the embedded web server will be initialized and the URI will be written back.
  7. The client will disconnect the temporary session and continue with the normal ClientCore processing.
  8. The server will redirect the web browser to that URI.

This avoids the complexity of the certificate processing. We could even pass an encoded keystore down to the P2J client through the environment BUT I don't see a way to get the URI back, so I think some kind of temporary session is still needed.

#103 Updated by Greg Shah over 7 years ago

Only a single auth-mode and a single plugin can be defined per server instance?

No, we allow different auth modes as a default, server-specific default, group-default or account level.

#104 Updated by Marius Gligor over 7 years ago

OK Thanks. I will try to do that.
Regarding the singleton instance of TransportSecurity please read the javadoc for SecurityManager#getTransportSecurity at line 6908

#105 Updated by Greg Shah over 7 years ago

Regarding the singleton instance of TransportSecurity please read the javadoc for SecurityManager#getTransportSecurity at line 6908

I see the text now. It is inaccurate. Please fix it as part of this work.

#106 Updated by Marius Gligor over 7 years ago

A more portable solution is to send the random generated user-id and password directly as arguments in command line.
To be more secured we could also encrypt the values for this arguments.

#107 Updated by Greg Shah over 7 years ago

The problem is that on Linux/UNIX, command line args are visible to anyone. Using encryption would require both sides know a particular key in order to decrypt the values. That suggests we would have to hard code the key in our code (which is not secure) or we would have to otherwise provide the key as part of configuration (which is the very problem we're trying to solve in the first place).

#108 Updated by Marius Gligor over 7 years ago

Today I started to implement the direct connection using a strong random subject id and password passed as environment variables.
Using RandomWordGenerator I generated strong values for both subject id and password.
The values are used to create and add two environment variables using ProcessBuilder#environment interface.
The values are converted to hex strings on server side and converted back to the original values on client side.
I added this conversion steps in order to avoid any problems caused by special characters generated by RandomWordGenerator.

Next I tested on both Linux and Windows OS and I found to works fine. On client now I'm able to get the values with no problems.

On the next step I want to implement the server side code and I have some questions:
I found the AdminServerImpl class in admin package which export a lot of administrative operations.
I saw that is used by the admin console applet.

1. I would like to use this helper class to create and delete temporary users. Is this OK?

2. When we create a new user (account) we have to provide an UserDef structure.
I think that I have to create a protected user (with password) and a custom authentication mode.
I have to specify also a custom authentication plug-in.
Should I design and implement a new custom login plug-in for this purpose?

3. In P2J and Majic I found the following entries registered inside directory

P2J (directory.xml):
<node class="authMode" name="auth-mode">
<node-attribute name="mode" value="4"/>
<node-attribute name="retries" value="0"/>
<node-attribute name="plugin" value="com.goldencode.p2j.security.GuestAccess"/>
</node>

Majic (gso-directory.xml):
<node class="authMode" name="auth-mode">
<node-attribute name="mode" value="4"/>
<node-attribute name="retries" value="-1"/>
<node-attribute name="plugin" value="aero.timco.majic.security.Login"/>
</node>

Now if we have to add a new plug-in for the same mode how we can do that?

#109 Updated by Greg Shah over 7 years ago

1. I would like to use this helper class to create and delete temporary users. Is this OK?

Yes, this should be OK.

You will have to meet some requirements for this to work:

  • The thread on which you make these calls must have a proper security context AND that context's account must include sufficient permissions to execute all of the actions required.
  • You will have to honor the control flow that would be followed by a "real" admin console user. In particular, you will have to force a "refresh" of the security cache after adding/removing a user.

It may make sense to create multiple accounts, in advance, one time. The idea is to permanently leave these accounts in the system. Then you can just:

  1. set the password and enable an account from the pool when you are launching a new client
  2. disable that account when the logon has occurred

The only drawback to this approach is that the overall number of simultaneous logons will be limited by the size of this pool. However, in practice, the actual amount of time needed to use this mechanism during the client spawning process will be very small. I suspect that even a relatively small pool will scale pretty well.

a custom authentication mode.
I have to specify also a custom authentication plug-in.
Should I design and implement a new custom login plug-in for this purpose?

I'm not completely sure of this. The permissions of what can be called by this account can be very "locked-down". We can limit this to just the right to call a specific exported remote object interface. Then the client just uses "normal" userid/password authentication mode, obtains the proper remote object, calls that remote object and then disconnects. On the server side, the call to that remote object should be sufficient to notify the server to disable the account and when the logoff occurs, the account must go back into the pool.

There may be some minor changes needed in the security package to enable this, but I don't see the need for a custom auth plugin.

Now if we have to add a new plug-in for the same mode how we can do that?

This specifies a single global DEFAULT auth-mode (and default plugin).

But you can override this at the group account and user account levels (and maybe the server level too, I'm not sure).

#110 Updated by Marius Gligor over 7 years ago

On StandardServer start-up the AdminExports interface is registered:

ifaces = new Class[] { AdminExports.class };
RemoteObject.registerStaticNetworkServer(ifaces, AdminServerImpl.class);

On the AdminLogon after a successful logon a remote reference to this interface is obtained and used via a connectDirect call.
I tried to do the same but so far with no success. I'm using the admin account credentials.
The problem is that so far I cannot create a connection to the server using admin credentials and get the AdminExports remote interface.
Is any default session available to get the interface reference without connectDirect?

Is this the right way that we could follow, using such a connection to get access to the AdminExports interface?

#111 Updated by Greg Shah over 7 years ago

Is this the right way that we could follow, using such a connection to get access to the AdminExports interface?

No.

There is no reason for a session or to be using the remote object interface when the code that is running is in the same JVM process. I think this is as simple as calling AdminServerImpl.addUser(), right? You just need to do it from a P2J server thread that has the proper security context.

In addition, please try my idea of creating a pool of pre-existing accounts. This will greatly reduce the effort for each new session AND it should be much faster too. The creation of this pool can be a 1-time thing (if the pool doesn't exist, it gets created) that is persistent (the accounts aren't ever deleted).

#112 Updated by Marius Gligor over 7 years ago

I tried to create a temporary account when the StandardSever start-up using AdminServerImpl.addUser() but with no success.
I think that StandardSever runs on P2J server thread because I can read from directory on this thread.
The problem is coming from here:

// access the target
WorkArea wa = workArea.get();
if (!wa.targeted || !wa.sa.adminAccess()) {
return false;
}

wa.targeted is false and wa.sa is null.

#113 Updated by Marius Gligor over 7 years ago

I added AdminServerImpl.setTargetLive(true) to solve the previous issue but now It said that I have no admin rights. SystemResource#checkAdminAccess return false.
return systemAccessWorker("admin", SYSTEM_ACCESS);

#114 Updated by Constantin Asofiei over 7 years ago

I think you should bypass AdminServerImpl entirely and just use SecurityAdmin to add a new user.

I added AdminServerImpl.setTargetLive(true) to solve the previous issue but now It said that I have no admin rights.

As this is done using the security context of the server process (i.e. standard for the test directory.xml), you need to make sure that the server process account has the appropriate ACLs; check the ACLs for the "admin" account and "admins" group - I think it could work just by adding "standard" account to the "admins" group.

Greg, about the pool of "generated, pre-existing, accounts": wouldn't adding them to the "live" security cache generation expose them to the Admin Console? I think we will need some separate management for this pool, and not just add them to the global user list.

#115 Updated by Greg Shah over 7 years ago

wouldn't adding them to the "live" security cache generation expose them to the Admin Console

Yes, but that is probably OK. We can create them with verbose descriptions that make it clear these are internal/web accounts. Admins can be trained to leave these alone.

I think we will need some separate management for this pool, and not just add them to the global user list.

I worry that creating "shadow" users would require major changes throughout our security package. I want to keep this as simple as possible. These accounts will normally be disabled anyway, except during the moment we are trying to spawn a client.

#116 Updated by Marius Gligor over 7 years ago

I'm using SecurityAdmin instead AdminServerImpl but unfortunately the problem remains the same.
On the server directory I have for groups:

<node class="container" name="groups">
<node class="group" name="everybody">
<node-attribute name="description" value="all users"/>
</node>
<node class="group" name="admins">
<node-attribute name="description" value="all admins"/>
</node>
</node>

The standard is defined as process account.

<node class="container" name="processes">
<node class="process" name="standard">
<node-attribute name="enabled" value="TRUE"/>
<node-attribute name="description" value="P2J server"/>
<node-attribute name="master" value="TRUE"/>
<node-attribute name="server" value="TRUE"/>
<node-attribute name="alias" value="standard"/>
</node>
</node>

I tried to add the standard account to admin group without success.

&lt;node-attribute name="groups" value="admins"/&gt;

This kind of node is not allowed.

#117 Updated by Marius Gligor over 7 years ago

I fixed the issue by adding the process standard to the subject list in ACL (see bellow)
Now I'm able to create user accounts from StandardServer thread.

<node class="container" name="acl">
<node class="container" name="system">
<node class="container" name="000100">
<node class="resource" name="resource-instance">
<node-attribute name="reference" value="admin"/>
<node-attribute name="reftype" value="TRUE"/>
</node>
<node class="systemRights" name="rights">
<node-attribute name="check" value="true"/>
</node>
<node class="strings" name="subjects">
<node-attribute name="values" value="admins"/>
<node-attribute name="values" value="standard"/>
</node>
</node>

#118 Updated by Marius Gligor over 7 years ago

I started to design and implements the temporary account pool. Finally I was able to create user accounts from StandardServer thread.
I added the "standard" account to the ACL list of subjects in directory.xml for "admin" rights.

- I designed a temporary account pool using a FIFO queue (ConcurrentLinkedQueue) to hold temporary accounts.

- When the pool is created on start-up also a group "temps" is created if the group does not already exists. All temporary accounts created and added to the pool belongs to this group.

- Also the temporary accounts are created if necessary to fill the entire pool. The size of the pool is configured form directory.

- When a temporary account is requested from pool the item form the head of the pool is returned or null if the pool is empty. The item is removed from pool.

- When the item is no longer needed is returned to pool, enqueued at the tail of pool and reused.

This design works fine. However I found a lot of constraints and I want to ask you about a possible solution.

1. The pool is created from the StandardServer thread on start-up when we have enough rights to manipulate user accounts.
Unfortunately after the initial creation it is no longer possible to manipulate user accounts from other threads (jetty).
As a consequence we cannot enable/disable accounts on "the fly". Basically we cannot do any changes to accounts.
Also in my mind was to create new accounts whenever the pool became empty instead to return a null value but this is also not possible.
Do you known any idea on how we could fix that?

2. I saw that the user name is stored always in lower case regarding that the generated string contains upper case characters.
This is just an observation because have no side effects in my implementation.

3. Regarding the password I sow that using the same password for two different users one created form admin console the other
created by me with SecurityAdmin API have different encrypted format!!! Inside the code the decrypted values are OK.
However using the account created from console manager allow me to authenticate remote but using my generated accounts does not.
Do you have any hint for me here? Anyway on the next step I want to do deep investigations on this issue.

#119 Updated by Greg Shah over 7 years ago

1. The pool is created from the StandardServer thread on start-up when we have enough rights to manipulate user accounts.
Unfortunately after the initial creation it is no longer possible to manipulate user accounts from other threads (jetty).
As a consequence we cannot enable/disable accounts on "the fly". Basically we cannot do any changes to accounts.

I think we should create a daemon thread that can handle such tasks. That thread would have to be setup with the proper security context when it is started. In fact, the initial setup of the accounts and so forth can be done there too.

Then it is a matter of establishing the appropriate queueing or IPC mechanism between jetty and this daemon thread. This ensures that the jetty thread doesn't need special permissions.

3. Regarding the password I sow that using the same password for two different users one created form admin console the other created by me with SecurityAdmin API have different encrypted format!!!

This is a surprise to me as well.

#120 Updated by Marius Gligor over 7 years ago

In the mean time I found why the password format are different. AdminConsole does not store the clear password encrypted.
Instead first a SHA hash is created using the clear password bytes using HashPassword.hashPassword tool.
The generated hash is encoded and stored in directory as password. This make the password secured, cannot be decrypted ever!
The problem is that we have to store the password like AdminConsole and send the clear password encrypted as environment variable.
On the client side we have to provide the clear password for authentication when we create a server direct connection session.
Because we cannot decrypt the password we have to keep the encoded password on another parameter in user account. Maybe the owner field would be used.
Or another solution should be found.

#121 Updated by Marius Gligor over 7 years ago

Also the Base64 encryption/decryption algorithm is used. Basically isn't a strong encryption.

#122 Updated by Marius Gligor over 7 years ago

I have an idea. We can use a random string constant like a "salt" and the password should be constructed
from user name also random generated plus the "salt" combined using a specific function.
We have to send remote only the user name and on the client side using the same "salt" and algorithm the password is reconstructed.

#123 Updated by Marius Gligor over 7 years ago

Another solution is to delete all accounts from temps group on each server start-up and create new accounts.
The passwords for the new accounts are kept in memory and sent remote. This approach seems to be more secured.

#124 Updated by Greg Shah over 7 years ago

The problem is that we have to store the password like AdminConsole and send the clear password encrypted as environment variable.
On the client side we have to provide the clear password for authentication when we create a server direct connection session.

This is OK.

These accounts will be disabled except during the moment of use. As long as the account has its password changed to a newly created random text just before it is enabled, then it is OK to send the clear version of this down to the client. Only someone with the rights to run a debugger or otherwise inspect the kernel data structures/proc file system can see this information. It is OK.

Also the Base64 encryption/decryption algorithm is used. Basically isn't a strong encryption.

Indeed. It is obscuring the data only and can be reversed so it isn't encryption at all.

We can use a random string constant like a "salt" and the password should be constructed from user name also random generated plus the "salt" combined using a specific function. We have to send remote only the user name and on the client side using the same "salt" and algorithm the password is reconstructed.

We don't need to do to this level. The real security it adds is not enough because anyone with the salt value can still get the password.

Another solution is to delete all accounts from temps group on each server start-up and create new accounts.
The passwords for the new accounts are kept in memory and sent remote. This approach seems to be more secured.

I like this. We still need to disable/enable accounts only when they need to be used, but leaving this data out of the persistent directory does seem cleaner.

#125 Updated by Marius Gligor over 7 years ago

I implemented the temporary account pool by deleting all temporary accounts and create new ones on server start-up.
Now it's works fine but without enabled/disable account at this moment.
I will try to fix later this issue because now I have another big issue to fix and I need some helps.

When the ClientDriver is started using chui_native or swing_chui_frame a terminal window is open ant the users are working inside this TTY window.
As I understood that on Majic application an authentication plug-in is used and the user should enter his credentials before start working on the p4g converted application.

In our case however the flaw is different. The web login page is used to authenticate the user against OS.
On success the user is redirected to a web page containing a JS TTY emulator.
No other console windows are open for user. Even more the user should be located somewhere remote on another machine.
In this case the authentication for p4g converted application should be done using JS TTY emulator.
Is this true?

Here I have some remarks and questions:

When I tried first to open a server session using a temporary account always the GuestAccess plug-in is used.
Using this plug-in Regarding the user name and password for temporary account the authentication succeed.
The user is always authenticate using bogus account as is specified inside directory.
I tried to set custom plug-ins to temporary accounts when I created temporary users but it does not work.
It seems the auth-mode is global and unique.

<node class="authMode" name="auth-mode">
<node-attribute name="mode" value="4"/>
<node-attribute name="retries" value="0"/>
<node-attribute name="plugin" value="com.goldencode.p2j.security.GuestAccess"/>
</node>

1. How we can fix this issue?

2. If we change this plug-in how the ClientDriver will work with other screen drivers?

3. Using chui_web screen driver the user authentication should be done using the JS TTY window?
If YES should ClientDriver still have an open session?

#126 Updated by Greg Shah over 7 years ago

In this case the authentication for p4g converted application should be done using JS TTY emulator.
Is this true?

Yes. Of course, we haven't gotten that written yet, but so long as the JS driver works like any other driver, then it will naturally allow the client to operate just like the swing or native client.

1. How we can fix this issue?

I think the problem comes from not knowing the userid until after connectDirect() is called. I think this can be solved using some settings in the bootstrap configuration. Please look at SecurityManager.sendAuthType() which is called from SecurityManager.authenticateClientWorker() which is the main client-side authentication processing method.

2. If we change this plug-in how the ClientDriver will work with other screen drivers?

There is no need to change the plugin.

3. Using chui_web screen driver the user authentication should be done using the JS TTY window?
If YES should ClientDriver still have an open session?

Yes. You are writing a "JS driver" right? It is a kind of plugin that provides UI input and output. In the JS driver case, the driver itself is really split into 2 pieces. The "top half" is written in java and runs in the P2J client JVM process. The "bottom half" is written in javascript and runs in the web browser. But this is no different than the other ChUI drivers that we have. None of them know anything about the authentication process. BUT they all get used by the authentication process when an interactive login is needed.

I think you may need to look at the SecurityManager.authenticateLocal() which is the server-side of the interactive login. Note there that in CUSTOM mode we serialize an Authenticator class and send the class' bytes to the client side, where it is loaded and executed as a special piece of UI client code. That code uses the UI services of the driver (whichever one is there) to display a UI to the user and obtain userid/password credentials. Then it packages these credentials and sends them back to the server. The driver has no idea this has anything to do with login. The driver is just acting like its normal UI service.

To see a real example of this, please look at the code in testing/majic/src/aero/timco/majic/security/LoginClient.java. A simpler version can be found in src/com/goldencode/p2j/security/DefaultLoginPanel.java.

There is no problem to solve here that won't be automatically solved by just properly implementing the JS driver.

#127 Updated by Marius Gligor over 7 years ago

OK. I fixed the authentication issue using security:authentication:type=program like AdminLogon do.
Using an interactive type security:authentication:type=user an authentication plug-in is mandatory.
Remains to fix the issue related to enable/disable temporary accounts using a thread having an appropriate security context.

Regarding JS TTY during the last weekends I exercised my JS skills to create a prototype.
The result is here http://www.gmax.ws/tty.html
It's just an exercise but I think that looks good, isn't it?

#128 Updated by Greg Shah over 7 years ago

Code Review 1213a

The changes continue to move in the right direction. Some minor feedback:

1. I prefer to have a flag at the top of ClientCore.start() named "web". You can set this true if the uuid is in the bootstrap config. If it is set true, only then would you call setWebClientConfig() and startUp(). This allows the reader to understand that these are conditional, without having to read "into" those methods. This is similar to how we have the "dbg" flag.

2. Please rename setWebClientConfig() to addWebClientConfig().

3. Please rename startUp() to initializeWebServer().

4. In WebClientSpawner, I think we may need more than 5 seconds of timeout on systems with a heavy load.

5. In WebClientSpawner, I prefer to wait a reasonable amount of time for an account to become available instead of throwing an exception.

#129 Updated by Greg Shah over 7 years ago

The result is here http://www.gmax.ws/tty.html
It's just an exercise but I think that looks good, isn't it?

Very cool!!! Well done.

Yes, I am really excited to see us get to work on that part.

#130 Updated by Marius Gligor over 7 years ago

I fixed the enable/disable temporary account using the AssociatedThread class.
Bellow is a short description of temporary accounts cash implementation.

1. On the server start-up all temporary accounts are deleted (accounts belong to temps group)

2. New temporary accounts are created and enqueued to pool. The created accounts are created disabled.
The number of initial temporary accounts is defined in registry.

3. When an item is requested from pool the status of the related account is changed to enabled.

4. When an item is returned to pool the status of related account is change to be disabled.

5. On heavy loaded systems when the pool potentially became empty a new temporary account is created and returned.
This kind of accounts are created enabled and served immediately to caller.
Later when the items are returned to pool the related accounts will be disabled.
Basically all requests for temporary accounts are honoured.

#131 Updated by Constantin Asofiei over 7 years ago

Marius Gligor wrote:

5. On heavy loaded systems when the pool potentially became empty a new temporary account is created and returned.
This kind of accounts are created enabled and served immediately to caller.
Later when the items are returned to pool the related accounts will be disabled.
Basically all requests for temporary accounts are honoured.

A quick thought: is it possible for the queue size to get out of control? As I understand, this step is before any user authentication is done, thus someone could build a script to send a massive amount of requests, thus OOME-ing the server... I think a static pool size is better; you can limit the amount of time to wait until an account can be retrieved from the pool (this can be made configurable too).

#132 Updated by Marius Gligor over 7 years ago

OK. I started to change the design to have a fixed pool of accounts.

#133 Updated by Marius Gligor over 7 years ago

New design. Fixed size for temporary accounts pool.

#134 Updated by Marius Gligor over 7 years ago

Add code to terminate the thread on server shut down some minor changes.

#135 Updated by Greg Shah over 7 years ago

Code Review 1217a

1. if(web) in ClientCore should be if (web).

2. In TemporaryAccountPool, the import of LogHelper should be using the wildcard.

3. I don't understand the purpose of this code in TemporaryAccountPool.createUser():

         if (user != null && user.enabled == enabled)
         {
            account = new TemporaryAccount(user.subjectId, password);
         }

4. I prefer if you would rename TemporaryAccountPool.createInstance() back to getInstance(). I think it better describes what is being done AND it is more consistent with other singleton classes in P2J (like SecurityManager...).

5. In TemporaryAccountPool.poll(), there may be a logic problem. In the item == null (pool empty) case, it seems that if an account is found before the timeout expires, the account is returned back to the caller without being enabled.

Shouldn't this code:

            // Change user status
            Runnable core = new UpdateAccountTask(item.getSubject(), true);
            TemporaryAccountPool.poolTask.execute(core);

execute no matter how the "item" is found?

6. I would like you to make a change to the build process to bypass the execution of postbuild.sh by default. In other words, for the default build, no postbuild.sh would get run. This is the most common case (where people don't need that to run) and I don't want to make all users force a new sudoers configuration change or a requirement to enter the password. If you can make this something that can be added optionally from the command line, that may be best.

#136 Updated by Greg Shah over 7 years ago

After #2201, you will next be working on #2212. I will add more details/guidance tomorrow. If you run out of work before then, you can start by analyzing the kinds of communication flows that will be needed to support the ChUI driver. Make sure you consider the needs of both directions of flow. Document your findings/thoughts here.

#137 Updated by Constantin Asofiei over 7 years ago

Review for 1217a (beside Greg's notes):
  • in makefile.spawn, you are using tabs for indenting. More, in postbuild.sh, spawn.c and winspawn.c, the indenting is done at 4 chars, not 3.
  • ChuiWebDriver.startupServer is missing javadoc
  • WebHandler - the header is indented one-space to the right...
  • WebHandler.handle/doGet/doPost - check the formatting of the method definition
  • Clientcore.initializeWebServer - you are checking for the driver to be a ChuiWebDriver instance. Isn't this somehow restricting? I think there should be a WebDriver interface, to mark the web drivers.
  • For the new added classes, the class javadoc is minimal. Please improve the javadoc to explain what these classes do and how they should be used (this will be helpful later on, when we will write documentation for all of this).
  • TemporaryAccountPool.createUser - I think the certificate alias for the new user should be null, not hard-coded to shared.
  • in UpdateAccountTask.run, we are calling AdminServerImpl.targetRefresh();. This is a very expensive operation (as it generates a new security cache generation on each call) and I'm not sure how this will behave in high-load moments, when a high number of users are authenticating simultaneously.

#138 Updated by Greg Shah over 7 years ago

in UpdateAccountTask.run, we are calling AdminServerImpl.targetRefresh();. This is a very expensive operation (as it generates a new security cache generation on each call) and I'm not sure how this will behave in high-load moments, when a high number of users are authenticating simultaneously.

This is a very good point. The security generations will grow with no bound in this case AND it can have performance implications.

Please look into ways that we can solve this:

1. dynamically edit all existing security cache(s) in which the temp accounts exist, without using a refresh or creating a new generation
2. make the security and admin code aware of the concept of truly temporary account that is not backed by the directory (automatically goes away at server shutdown) and is kept separate from the generations of security cache, allowing these accounts to be managed as a single pool that is not subject to the refresh requirements

I prefer option 2 if it is reasonable to implement.

#139 Updated by Marius Gligor over 7 years ago

According to my tests without a "refresh" the account changes (enable/disable) are not "visible" and when the client try
to do a remote authentication an error "account is disabled" is throw and the authentication fails.

On the other hand the account pool is mandatory because we need to keep the clear password in memory.
This means that we have always to generate this accounts on server start-up when we know the clear passwords.
The pool is used just to serve the account subject and a clear password used for authentication.
The authentication process it self has nothings to do with our pool.

In my opinion authentication is done relative rarely and isn't used intensively by users.
That's why we assert that a small size pool should works fine.
Anyway I'll try to find a solution as you suggested.

#140 Updated by Greg Shah over 7 years ago

In my opinion authentication is done relative rarely and isn't used intensively by users.

In comparison to the runtime of a session, yes. In other words, each session is commonly expected to be long-lived. But on any given day there will be hundreds or even thousands of logins to a particular server instance. Server instances that are left running for months will have a very large number of security generations that may start to cause problems.

That's why we assert that a small size pool should works fine.

The small pool is OK so long as there are not too many simultaneous logins. Except for companies that have shift operations where a large number of users login within a short interval, a small pool will work OK.

But that doesn't mean that this processing can't have an effect. When the security cache is refreshed, it is a costly activity in the security code. It will affect the performance of all other ongoing security operations, such as ACL checks. This isn't only about login processing.

According to my tests without a "refresh" the account changes (enable/disable) are not "visible" and when the client try to do a remote authentication an error "account is disabled" is throw and the authentication fails.

Yes, this is expected and correct.

There is one other thing, that I should have noticed before. Please change the password every time you re-enable the account. I don't want the passwords to be re-used. The accounts themselves can be reused. But otherwise, someone that breaks a password can just keep trying it in a tight loop while spawning new sessions and eventually they will get in. It is perfectly possible to login using the same account more than 1 time simultaneously. That is by design because our users must be able to have multiple sessions. But it exposes a security hole in the scenario where we re-use passwords.

#141 Updated by Marius Gligor over 7 years ago

I've fixed almost all the issues reported on your code reviews with some remarks:

1. In TemporaryAccountPool, the import of LogHelper should be using the wildcard. This issue cannot be fixed due to conflict with another class
from package com.goldencode.util.LogHelper and I need the class from this package com.goldencode.p2j.util.LogHelper

2. I introduced a command line parameter to execute the postbuild.sh script
The parameter should be use as follow:
a.) ant native
will run build native artifacts without running postbuilt target on makefile.spawn

b.) ant -Dpost.build=yes native
will run build native artifacts and execute also the postbuilt target on makefile.spawn
In other words the parameter "post.build" should be provided in command line and his value should be "yes".
-Dpost.build=yes

3. Checking ChuiScreenDriver issue. In order to be more generic I created a new interface WebScreenDriver that should be implemented by other web screen drivers in the future.

4. Related to AdminServerImpl.targetRefresh(). Whithout the refresh the enable/disable account is not visible and
the remote authentication fails due to "account is disbaled" error.
Greg suggested me some possible fixes. I have to do a research for a possible solution.
So far I found that a list of user accounts is stored inside the SecurityCache.
I think that doing a short work around the SecurityManager and SecurityCache it is possible
to add a method to update (refresh) a user account inside the SecurityCache instead to refresh the entire context.
We have to search on the list of accounts stored inside the SecurityCache instance and update only one account.
Do you think that this scenario could be a solution?

5. For security reasons when an account is (re)enabled the password is also changed, that is, a new random password is generated.
This is also related to issue 4 because is a change of a user account which is synchronized with directory but not updated inside the security cache.

6. I added more descriptions on javadocs especially on the description of classes.

7. I fixed all issues related to use TAB characters instead spaces and space size identation.

#142 Updated by Greg Shah over 7 years ago

Code Review 1218a

Really good improvements. Yes, we are close now.

5. For security reasons when an account is (re)enabled the password is also changed, that is, a new random password is generated.
This is also related to issue 4 because is a change of a user account which is synchronized with directory but not updated inside the security cache.

I understand. All changes to the account are subject to this limit. It is by design.

I think that doing a short work around the SecurityManager and SecurityCache it is possible to add a method to update (refresh) a user account inside the SecurityCache instead to refresh the entire context.
We have to search on the list of accounts stored inside the SecurityCache instance and update only one account.
Do you think that this scenario could be a solution?

Yes, I do think this can work. This is what I was suggesting with my "option 1".

BUT I do think we must update all instances of that account. I believe there may be separate instances of those accounts in every cache generation since the account was created. This would mean that you must edit a list of instances of that account, otherwise when the latest generation is popped, the previous generation may have a different state and cause problems. If I am mis-remembering how we do this, let me know.

#143 Updated by Marius Gligor over 7 years ago

Unfortunately I found that changing the TAB characters in makefile results in a make error related to missing separator.
So I restored the makefiles to have TAB characters.

#144 Updated by Marius Gligor over 7 years ago

I did some small changes inside the security package to allow update of user accounts stored in security cache.

1. I added the method SecurityManager#updateCachedUserAccount which is restricted to be executed only from UpdateAccountTask.run

On running the AssociateThread force a SecurityContext by duplicating the current SecurityContextStack.
I tried to access the SecurityCache from the thread SecurityContext but no accounts were found in cache.
Finally after a lot of tests I found that is enough to search for accounts inside the SecurityCache of the current SecurityManager instance.
I added also the same method to SecurityCache#updateCachedUserAccount which iterate over the cached accounts collection looking for a specific account.
If the specified account is found only the enabled and password attributes are changed in cache.
The SecurityCache class has package-private modifier and could be accessed only from classes inside the package like SecurityManager.

2. Theoretical it's possible to generate the temporary accounts only in memory (pool) and inject the generated list of accounts directly into the SecurityCache list of accounts.
Thought that other threads like AdminConsole could do at one moment a SecurityCache refresh by calling the AdminServerImpl.targetRefresh this could cause problems.
The injected accounts will disappear from SecurityCache and the new SecurityCache instance will hold only accounts which are get from directory.
The same problems could occur also in case we do the account changes only in memory (security cache).
Supposing we change the password and enable attributed for an account into security cache and send the subject id and password to a spawned process.
If a new SecurityCache is requested by another thread before the spawned process is using the password to authenticate, than the enabled and password
for this account will be loaded from directory having other values and later the client (spawned process) authentication will fail.

With the current implementation this side effects does not occur because first the directory is updated 
follow by an cache updated. In case another thread will create a new SecurityCache the directory is in an
consistent state with in memory pool and the temporary accounts are loaded in cache from directory.

#145 Updated by Greg Shah over 7 years ago

Code Review 1219a

Everything looks good. Please get this regression tested (only runtime testing is needed).

I agree that my idea of having truly non-persistent accounts would require much more significant change to the security package. You did the right thing in choosing to implement in the simplest manner.

#146 Updated by Marius Gligor over 7 years ago

I've started to do the regression tests. When I executed this step:

run_regression.sh build && run_regression.sh save-generated -Dsave.candidate=y

the build of project fails due to the missing of PAM libraries when spawn.c is build.
I need this libraries to be installed on devsvr01 but I need root permissions for sudo apt-get install
Another option is to change the makefile.spawn and compile without PAM options.

#147 Updated by Marius Gligor over 7 years ago

In order to install PAM libraries we need to install this package:

sudo apt-get install libpam0g-dev

#148 Updated by Marius Gligor over 7 years ago

Another option is to define a build variable like -Dpam=yes in order to build spawn.c with PAM option.
By default the build will be without PAM options.
This is useful because other members of team should not install the libpam0g-dev on their workstations.

#149 Updated by Greg Shah over 7 years ago

Another option is to define a build variable like -Dpam=yes in order to build spawn.c with PAM option.
By default the build will be without PAM options.

This is a good idea. Please do make this change.

I have also just installed the PAM libraries on devsrv01, so that we can test both with and without PAM support.

#150 Updated by Marius Gligor over 7 years ago

OK. Now the project was successfully build on devsvr01.
Nevertheless I'm going to add the PAM variable for conditional build similar with skip "postbuild" target already fixed.

#151 Updated by Marius Gligor over 7 years ago

I added a conditional build variable ${have.pam} for Linux / Unix native build targets in build.xml script file.
The variable should be provided on the command line in order to build the spawn.c with PAM options.
By default the spawn.c is build without PAM options.
Example:
ant -Dhave.pam=yes native (build with PAM options)
or
ant -Dhave.pam=yes -Dpost.build=yes native (build with PAM options and do the postbuild target)

#152 Updated by Greg Shah over 7 years ago

Code Review 1219b

The changes look good.

#153 Updated by Greg Shah over 7 years ago

-------- Original Message --------
Subject:     regression tests
Date:     Fri, 20 Dec 2013 12:38:35 +0200
From:     Marius Gligor <mag@goldencode.com>
Reply-To:     mag@goldencode.com
To:     Constantin Asofiei <ca@goldencode.com>, Greg Shah <ges@goldencode.com>

Constantin,

The package that I started to test require some changes inside directory.
Do you think that missing this values from directory could cause a failure?

Thanks
Marius

1. The standard subject should be added  to admin ACL in order to create temporary account on start-up:

 <node class="container" name="system">
          <node class="container" name="000100">
            <node class="resource" name="resource-instance">
              <node-attribute name="reference" value="admin"/>
              <node-attribute name="reftype" value="TRUE"/>
            </node>
            <node class="systemRights" name="rights">
              <node-attribute name="check" value="true"/>
            </node>
            <node class="strings" name="subjects">
              <node-attribute name="values" value="admins"/>
              <node-attribute name="values" value="standard"/>
            </node>
          </node>

2. The node chuiWeb should be also added to directory used on client spawn.

   <node class="container" name="server">
      <node class="container" name="default">
        <node class="boolean" name="adminEnabled">
          <node-attribute name="value" value="TRUE"/>
        </node>
        <node class="container" name="chuiWeb">
          <node class="boolean" name="enabled">
            <node-attribute name="value" value="TRUE"/>
          </node>
          <node class="boolean" name="secure">
            <node-attribute name="value" value="TRUE"/>
          </node>
          <node class="string" name="spawner">
            <node-attribute name="value" value="/usr/share/p2j/spawn"/>
          </node>
          <node class="string" name="jvmArgs">
            <node-attribute name="value" value="-Xmx512m -XX:MaxPermSize=64m -Djava.awt.headless=true"/>
          </node>
          <node class="string" name="classpath">
            <node-attribute name="value" value="/usr/share/p2j/lib/p2j.jar"/>
          </node>
          <node class="string" name="libPath">
            <node-attribute name="value" value="/usr/share/p2j/lib"/>
          </node>
          <node class="string" name="workingDir">
            <node-attribute name="value" value="./"/>
          </node>
          <node class="string" name="configFile">
            <node-attribute name="value" value="config.xml"/>
          </node>
          <node class="integer" name="rows">
            <node-attribute name="value" value="24"/>
          </node>
          <node class="integer" name="columns">
            <node-attribute name="value" value="80"/>
          </node>
          <node class="string" name="background">
            <node-attribute name="value" value="0x000000"/>
          </node>
          <node class="string" name="foreground">
            <node-attribute name="value" value="0xFFA500"/>
          </node>
          <node class="string" name="selection">
            <node-attribute name="value" value="0x0000FF"/>
          </node>
          <node class="string" name="fontname">
            <node-attribute name="value" value="monospaced"/>
          </node>
          <node class="integer" name="fontsize">
            <node-attribute name="value" value="12"/>
          </node>
        </node>

In regard to this, please setup our code so that most of this has sensible default values when the backing node is missing from the directory. Perhaps everything except for chuiWeb/enabled can be defaulted.

This will greatly reduce the amount of configuration needed. The only tricky part is where there are platform-specific values (e.g. /usr/...), in such cases we will need different defaults by platform.

#154 Updated by Marius Gligor over 7 years ago

I provided default values for all this parameters (see ClientBuilderOptions).

Now the server starts without errors if directory settings are not in place:
- If node chuiWeb is missing the http://server_address:admin_port/chui returns HTTP 404 page not fond because no handler is registered for /chui target.
- If the server standard subject is not added to admin ACL the pool is empty and no client could be spawned.

The changes are currently in regression tests. At the end of tests if no errors I have to download the Majic build to my workstation add appropriate directory configuration
and do a test to spawn web clients.

#155 Updated by Greg Shah over 7 years ago

If the server standard subject is not added to admin ACL the pool is empty and no client could be spawned.

I'm fine with this, but I would like you to log a diagnostic message that reports that the web chui is enabled but the sufficient rights have not been granted to the standard subject such that the web interface can function properly.

Please upload the new code here. You can keep going with testing.

#156 Updated by Marius Gligor over 7 years ago

Also I could provide platform specific (Linux/Windows) default values and do only runtime tests with Majic build on my workstation without to add chuiWeb node to Majic directory.
The server standard subject MUST be added on tests.

#157 Updated by Greg Shah over 7 years ago

I still prefer you to test on devsrv01 with the chui web active. This will make sure that we have a real chui environment (MAJIC) to test the web chui driver when that is ready. TIMCO will also be very interested in using the chui web driver, so we need to make sure it works well for them.

Please work with Constantin to figure out how to make the changes to the directory template files that are being used for the devsrv01 environment.

Finally, it will be important to make sure that the code can optionally use relative pathing for all the jars, classpath, libpath etc... which will allow the same configuration to work for multple users in our testing environment. So instead of absolute paths /usr/... it would be relative to the user's home dir.

#158 Updated by Marius Gligor over 7 years ago

Classpath, native libraries path and spawner location should be shared by all web clients.
We cannot provide relative paths for this values we have to specify absolute paths.
With relative paths we have to replicate the jar's on all users home directory.

#159 Updated by Constantin Asofiei over 7 years ago

I think the solution for devsrv01 is not about setting relative paths, but computing the full paths dynamically for each user. Inside the directory template, we can set the i.e. /usr/share/p2j/lib/p2j.jar path to something like P2J_JAR and change the majic/run/server/prepare_dir.sh script so that these values are replaced with the proper full path for the current user. Keep in mind that each user on devsrv01 has a ~/testing/majic/p2j folder.

#160 Updated by Greg Shah over 7 years ago

We cannot provide relative paths for this values we have to specify absolute paths.

Most production environments would follow this approach, that is true.

With relative paths we have to replicate the jar's on all users home directory.

This is exactly what we already do in our testing environments. So this is OK and in fact it is desired, because each user may have different changes to test.

#161 Updated by Greg Shah over 7 years ago

Keep in mind that each user on devsrv01 has a ~/testing/majic/p2j folder.

This is why I prefer to use relative paths on devsrv01. It reduces the changes to be made in the directory template and it should still work.

#162 Updated by Constantin Asofiei over 7 years ago

Actually, I think we should be able to determine the library path and the classpath dynamically. See the code in AppServerLauncher.builder - there I determine the jar which stores the ClientDriver class and this will be added to the classpath; also, the assumption is that the library path is the parent folder of this jar.

What remains is the spawner folder - can you remind me what this is used for?

#163 Updated by Greg Shah over 7 years ago

I think we should be able to determine the library path and the classpath dynamically.

This will work as a default. But I do want to make sure that we can specify a completely independent installation. The idea: the server and clients may not even be on the same machine and/or the clients may not be allowed access to the directory structure in which the server runs.

#164 Updated by Marius Gligor over 7 years ago

We can detect the location of p2j.jar as you already mention and I use this method in the previous snapshots but I think this should be used
as a default value in case no other value is specified inside directory.
In other words we have to find a set of default values which are relative to user and override this values in directory if we need absolute or other values.
Please let me know if this is what you are looking for and if we can use as default values the following:

- p2j.jar location for classpath native libraries directory and spawner location.
- user (current) directory for default working directory.

They are also many other issues that should be solved related to spawner build.
In Linux the postbuild target is mandatory and require sudo permissions.
Are all users (developers) on devsvr01 permissions tu use sudo if they want to do tests?

#165 Updated by Marius Gligor over 7 years ago

Also keep in mind that the p2j.jar location detection is done on the server instance
and is absolute to the user on which the server is running.
For other users this method of location detection does not work.

#166 Updated by Greg Shah over 7 years ago

Here are some details on our approach in regard to javascript frameworks.

1. Cross-browser compatibility is still a problem. Primarily I refer here to issues caused by browsers implementing the same features in incompatible ways (e.g. the IE event model versus everyone else).
2. Javascript standards move slowly and many problems that have yet to be standardized are already solved cleanly by existing frameworks.
3. Uneven implementation of existing javascript standards across browsers.
4. The event-driven (asynchronous) + single-threaded nature of javascript can lead to a great deal of boilerplate code being needed for things that in other languages would be shorter and simpler. Often javascript frameworks provide mechanisms to reduce this.
5. There are common javascript problems that can be solved with helper functions and other utility features which enhance productivity and simplify code.

I think it is reasonable to use a javascript framework to reduce or eliminate the above issues.

I have taken a close look at jQuery and Dojo. I've taken a more cursory look at Prototype, GWT and a few others. Based on what I have seen, Dojo seems to be the best option for "web applications". It can also be used for simpler web pages, wherever javascript is used. But it seems to really shine when one is developing an entire web app in javascript. In other words, when the user is not moving from page to page, but instead there is some technique (e.g. AJAX, websockets...) that is used to communicate to the back end server while the javascript drives a dynamic interface on a single page load.

The open source licensing, breadth of features and depth of features of Dojo are all very good. It is also moving forward at a fast pace. As a technology it is vibrant and healthy.

The plan is to include the necessary Dojo files from our own local installation instead of from a content network. Where we can do something simply in javascript that isn't enhanced or made more simple by using Dojo, then we should use our own javascript. But we don't need to be worried about trying Dojo functionality if it makes sense.

#167 Updated by Greg Shah over 7 years ago

Also keep in mind that the p2j.jar location detection is done on the server instance and is absolute to the user on which the server is running.

This will be fine for our use on devsrv01. It may also be good for other simple test environments. Production environments can configure more explicit choices, so that is OK.

Are all users (developers) on devsvr01 permissions tu use sudo if they want to do tests?

This hasn't been done yet. Unfortunately, I have to step out for a couple of hours.

Constantin: can you make the sudoers file changes? See the postbuild.sh for details.

#168 Updated by Constantin Asofiei over 7 years ago

Greg Shah wrote:

Constantin: can you make the sudoers file changes? See the postbuild.sh for details.

Yes, I can make the change. Just confirm that this line needs to be added (i.e. for user mag):

mag ALL=(ALL) NOPASSWD: /home/mag/testing/majic/p2j/src/native/postbuild.sh

#169 Updated by Marius Gligor over 7 years ago

Here are the latest changes that I made. Monday I want to do runtime tests first on my workstation and then on devsvr01.
I have to download the Majic build on my station add the later p2j build from my workspace and change the server directory.
If the running tests works on my station than I have to upload the p2j build and changed directory on devsrv01 and do runtime tests for user mag.
Constatin please let me know if is this OK. Are you available on Monday to help me if necessary?

#170 Updated by Constantin Asofiei over 7 years ago

Marius Gligor wrote:

Monday I want to do runtime tests first on my workstation and then on devsvr01.

Regression testing can be ran only on devsrv01; what is possible on your machine is to run a local MAJIC server. For this, you need to download the MAJIC jars and other setup (without converted and generated sources) to your machine. If MAJIC is starting OK and clients can login, then you can move to regression testing on devsrv01.

Constatin please let me know if is this OK. Are you available on Monday to help me if necessary?

Yes, I'll be available on Monday.

BTW, you need sudo rights too, beside the line at note 168, right? I'll make the sudo-related changes on Monday (if Greg doesn't get a chance to make them).

#171 Updated by Marius Gligor over 7 years ago

I did something similar with AdminConsole if you remember.
After the local tests you suggest me to repeat the regression tests?
What I want to do is to start the Majic server on devsrv01 and try to access https://localhost:7443/chui from my web browser.
But I'm afraid will not work 100% due to the redirection on a random port which require a ssh tunnel.

#172 Updated by Constantin Asofiei over 7 years ago

Marius Gligor wrote:

After the local tests you suggest me to repeat the regression tests?

Yes

What I want to do is to start the Majic server on devsrv01 and try to access https://localhost:7443/chui from my web browser.
But I'm afraid will not work 100% due to the redirection on a random port which require a ssh tunnel.

For this reason is best to start the Majic server on your machine (that's why we allow remote connections to each user's dedicated postgres cluster on devsrv01). Have you done this before?

#173 Updated by Marius Gligor over 7 years ago

Yes I did that when I tested Admin Console on my local station. So I have to do tests on my local station and after that redo regression tests with the latest changes.

#174 Updated by Marius Gligor over 7 years ago

Related to JS frameworks. I have some experience using jQuery but no experience using other frameworks.

#175 Updated by Marius Gligor over 7 years ago

On Saturday and today I did a lot of tests on a local copy of Majic project.
Finally I succeeded to configure the directory nodes for Majic with properly values.
Using these settings the web client spawn process works properly.
The steps and the changes that I did are described on the attached file.
I attached also the latest changes that I did (minor changes) as a result of my tests and the merged changes on StandardServer.java committed today by Ovidiu.

#176 Updated by Constantin Asofiei over 7 years ago

Marius: I've made the sudo-related changes for your account on devsrv01. After regression testing passes, make sure the /etc/sudoers.d/sudoers-ext contains the settings for all other users.

#177 Updated by Greg Shah over 7 years ago

Go ahead with regression testing on devsrv01 without making changes to enable the web chui. That will allow us to get this code checked in sooner.

Meanwhile, Constantin and I will make the devsrv01 changes needed to enable the web chui in the near future. In regard to the sudo approach, I have been thinking about it. The current plan needs to be changed because it opens a security hole.

The core issue is the path to the postbuild.sh and to the target spawn executible are both under user control. That means that the script can be edited to do anything at all. Likewise, the spawn source code is under the control of the user leading to the same problem. Although we could go with an approach that puts the specific commands (chown, chmod including their specific arguments so that the commands are very limited in scope) into sudoers intead of the script, that only protects from the modification of postbuild.sh.

Just as our customers would do, I think we will manually install a prebuilt spawn to a common location. Then we will not need sudo rights for each user. The downside is that changes can't be automatically applied.

#178 Updated by Marius Gligor over 7 years ago

Constantin: I think that is no need to do that. This issue should be tested only on a local copy of Majic not on devsvr01.
The currently regressions tests scripts does not test the spawn of web client.
Greg: Should I repeat the regressions tests on devsvr01 using the current changes?

#179 Updated by Constantin Asofiei over 7 years ago

Marius Gligor wrote:

Constantin: I think that is no need to do that. This issue should be tested only on a local copy of Majic not on devsvr01.

OK, I've backed out the sudo changes. But, the problem is: you need to change the "native" task so that sudo rights are not required; with your current build, you need sudo rights on each machine were P2J needs to be built (and the native task is executed by default - maybe make a separate task to build the spawner?).

#180 Updated by Marius Gligor over 7 years ago

Actually the build does not require sudo permissions.
You have to use some command line arguments to execute the postbuild.sh

-Dhave.pam=yes // used to compile with PAM options
-Dpost.build=yes // to execute the postbuild target which in turn run postbuild.sh script

The normal ant task
ant native
will build the native artifacts without ask for sudo permissions.

The following build (manual)
ant -Dhave.pam=yes -Dpost.build=yes native
will ask for sudo password and you shoul have PAM libraries installed on machine.

#181 Updated by Constantin Asofiei over 7 years ago

Marius Gligor wrote:

Actually the build does not require sudo permissions. You have to use some command line arguments to execute the postbuild.sh

Ahh, I missed that, is OK then.

#182 Updated by Greg Shah over 7 years ago

Code Review 1223a

I am fine with the changes.

Please merge up to the latest bzr revision (10426) and then run runtime regression testing on devsrv01. Do not test the web chui on devsrv01, just get it past regression testing so that we can get it checked in.

#183 Updated by Greg Shah over 7 years ago

We have to design the protocol used by websockets.
We have two options: JSON or binary payloads. As I remember you said that you prefer a binary protocol.

Yes, for network efficiency it is important this is binary.

Are you already designed such a protocol?

No.

#184 Updated by Greg Shah over 7 years ago

Additional Details on the Protocol

1. The protocol will need to be full-duplex. It must be possible to have both sides sending/recving independently.

One big reason for this is that even when the normal client processing is blocked (because the flow of control for the app is executing on the "server" side), the end-user can still execute a CTRL-C which should cause an interruption of the server. It is important that you review the CTRL-C processing which we implement today. It is a bit complex because we have split a single threaded process into 2 processes that are connected via a socket for interprocess communications (IPC). We use the "conversation" thread on both sides of the socket to force this to behave like a single-threaded environment even though we are inherently multi-threaded. In other words, we make all the primary control flow paths operate synchronously. But since the CTRL-C is an asynchronous event, it must interrupt the flow of control wherever it is executing (locally in the P2J client or remotely on the P2J server). This is where it is tricky. Please review the following files to understand this better:

com/goldencode/p2j/net/Control.java
com/goldencode/p2j/net/Conversation.java
com/goldencode/p2j/net/Dispatcher.java (search on RoutingKey.INTERRUPT_SESSION)
com/goldencode/p2j/net/InterruptHandler.java
com/goldencode/p2j/util/InterruptedExceptionHandler
com/goldencode/p2j/ui/chui/ThinClient.java (see watchCtrlC())
com/goldencode/p2j/ui/client/TypeAhead.java (search on Key.VK_CTRL_C)
com/goldencode/p2j/ui/client/Watcher.java

I hope that you can simply deliver the CTRL-C event to the P2J client side and let the rest of the infrastructure execute as is coded.

2. It is unknown if we need a full remote procedure call (RPC) implementation. I would like to avoid it if possible, BUT I don't want to make a poor solution just to avoid such an implementation. Our standard P2J protocol (see the net package for the Protocol, Queue, Dispatcher and Message classes) is message based and we implement both synchronous (RPC) and asynchronous (events) using the same transport.

3. As mentioned above, the message encoding must be very efficient and we already know we want to go binary. There are a couple of existing open source options that can mix JSON with a binary transport (UBJSON and messagepack).

4. On the javascript side, we may want (or even need) to use Shared Workers (background threads) and MessagePorts/MessageChannel to implement the CTRL-C notification. Or maybe that is not necessary.

Start by focusing on the requirements for the protocol (based on what services the java driver will need and what services the javascript side will need). Document your findings here. You should also investigate the binary JSON options. If those are not useful/appropriate, we an write our own.

#185 Updated by Greg Shah over 7 years ago

Something else to consider with the protocol is that the GUI driver/javascript will have more significant requirements (and a different driver interface) than can be seen right now with the ChUI driver.

Mouse events (movement, clicks...), fonts, images, drag and drop and a whole different level of drawing will be needed at a minimum.

#186 Updated by Marius Gligor over 7 years ago

Here are the latest changes on my local repository after an update from remote repository.
Only two files was merged on update StandardServer.java and ClientCore.java

#187 Updated by Greg Shah over 7 years ago

Code Review 0107a

I am fine with the changes. When it passes regression testing, please check it in and distribute it.

#188 Updated by Marius Gligor over 7 years ago

Ref. #2201 implement the web client process spawning and server redirect.
Passed the regression tests.
Committed revision 10429.

added src/com/goldencode/p2j/main/ClientBuilder.java
added src/com/goldencode/p2j/main/ClientBuilderOptions.java
added src/com/goldencode/p2j/main/ClientBuilderParameters.java
modified src/com/goldencode/p2j/main/ClientCore.java
added src/com/goldencode/p2j/main/ServerKeyStore.java
added src/com/goldencode/p2j/main/Spawner.java
added src/com/goldencode/p2j/main/SpawnerImpl.java
added src/com/goldencode/p2j/main/SpawnerListener.java
modified src/com/goldencode/p2j/main/StandardServer.java
added src/com/goldencode/p2j/main/TemporaryAccount.java
added src/com/goldencode/p2j/main/TemporaryAccountPool.java
added src/com/goldencode/p2j/main/TemporaryAccountWorker.java
added src/com/goldencode/p2j/main/UpdateAccountTask.java
added src/com/goldencode/p2j/main/WebClientSpawner.java
modified src/com/goldencode/p2j/main/WebHandler.java
modified src/com/goldencode/p2j/security/SecurityCache.java
modified src/com/goldencode/p2j/security/SecurityManager.java
modified src/com/goldencode/p2j/ui/client/chui/driver/swing/ChuiSimulator.java
modified src/com/goldencode/p2j/ui/client/chui/driver/web/ChuiWebDriver.java
added src/com/goldencode/p2j/ui/client/chui/driver/web/WebScreenDriver.java
added src/com/goldencode/p2j/ui/client/chui/driver/web/chui_web.html
modified src/com/goldencode/p2j/ui/client/swing/SwingHelper.java
modified src/com/goldencode/p2j/web/GenericWebServer.java
modified src/native/makefile
added src/native/makefile.spawn
added src/native/postbuild.sh
added src/native/spawn.c
added src/native/winspawn.c
modified build.xml

#189 Updated by Marius Gligor over 7 years ago

1. WebSockets are full-duplex communication channels over TCP. So we have to implements a communication protocol on server side in Java and on the client side using JS.
Maybe a short research on how to use GZIP with embedded jetty could be useful.
Do you think that a JSON and GZIP could be an alternative to binary compressed JSON?

2. At this stage I have a general overview over the implementation issues.
On the client side the protocol should looks like:

- The input events from keyboard and mouse are send over the network using the WebSockets and binary protocols toward chui web driver.
- The commands from chui driver on server side are send over the network toward the client page and rendered on the screen.
- The java script code on client side should manage a video memory array to render the output commands on a canvas, a screen cursor, the keyboard
and the mouse events as input commands send to server.

On the server side:
- The output commands from chui screen driver are send to client where are rendered on the screen or on the speaker (beep command)
- The inputs coming from client consisting in keystrokes and mouse events should be enqueued on the server side and dequeued on screen driver requests.
Regarding the keystrokes and CTRL-C handle in particular, according to my researches could be easily implemented to use the existing flaw with no changes in code.
The ScreenDriver define a readKey() method used to get keystrokes from a type-ahead buffer using a ChuiSimulator instance.
Our ChuiWebDriver use an instance of ChuiWebSimulator. ChuiWebSimulator extends ChuiSimulator and override the readKey() method.
So far we have a simple implementation for readKey() method which should be improved if necessary.

3. Dojo toolkit is distributed as a zip file containing a lot of Dojo modules (short java script files)
The latest release is 1.9.2 which is around 13 Mb in size. (the compressed format) A new release 2.0 is expected earlier this year.
The best way to handle the Dojo modules inside p2j project is to put the zip file on the classpath (add to the list of dependencies inside the manifest file)
and access modules as classpath resources.
In order to do that I have to design a separate jetty Handler class which handle all Dojo targets and add this handler to the embedded server handlers collection.
I think that is better to use the Dojo toolkit wherever is possible in our java script code for example in WebSockets communication implementation, DOM manipulation etc.

4. I think also that we have to use canvas HTML5 API which in turn restricts the use of application only on web browsers HTML5 compliant.

#190 Updated by Greg Shah over 7 years ago

Maybe a short research on how to use GZIP with embedded jetty could be useful.
Do you think that a JSON and GZIP could be an alternative to binary compressed JSON?

I am willing to consider this. However, I am generally concerned with any approach that encodes more data than necessary and uses compression to "fix" the issue. Ultimately, the network traffic is reduced but by trading CPU cycles.

We should also consider just implementing our own non-JSON binary message passing protocol where we handle the encoding/decoding ourselves.

2. At this stage I have a general overview over the implementation issues.

Yes, this all sounds like you are going in the right direction.

Dojo toolkit is distributed as a zip file containing a lot of Dojo modules (short java script files)
The latest release is 1.9.2 which is around 13 Mb in size. (the compressed format) A new release 2.0 is expected earlier this year.

We should not need the full distribution, especially for the ChUI support.

Dojo has 2 things that may help us:

http://dojotoolkit.org/documentation/tutorials/1.9/build/ - a build toolset to create a custom dojo distribution (that can be much smaller)
http://dojotoolkit.org/documentation/tutorials/1.9/modules/ - asynchronous module definition support to make code modular in a standard way

The best way to handle the Dojo modules inside p2j project is to put the zip file on the classpath (add to the list of dependencies inside the manifest file) and access modules as classpath resources.

I can see this may be needed. But I wonder if we can't customize/minimize our distribution using the Dojo build tools and get a simpler result. Please review and respond.

I think that is better to use the Dojo toolkit wherever is possible in our java script code for example in WebSockets communication implementation, DOM manipulation etc.

If we can write simple non-Dojo javascript to do a job, I prefer that option except for the following cases:

1. The javascript must be browser-specific and/or would have compatibility issues and Dojo has a clean solution.
2. The Dojo approach provides a much cleaner result because it has functionality that would have to be otherwise duplicated ourselves.

This means that the Dojo code may indeed be used often. But I don't want to default to writing Dojo code. Here is why:

1. Dojo changes rapidly and the more dependent we are upon it, the more effort there is to keep up.
2. High levels of dependency on Dojo will make it harder or more costly to get rid of it/replace it later, if ever the time comes where that is desired or necessary.
3. More code and complexity will be added to the project than may be strictly needed. With more code and complexity comes more bugs and a harder to maintain/understand code base.

I want to find the right balance. So the first time you plan to use Dojo for some particular feature, please consider the above and document (here) why it makes sense to use it, based on the criteria above.

4. I think also that we have to use canvas HTML5 API which in turn restricts the use of application only on web browsers HTML5 compliant.

Indeed, this is certainly true. It is not a problem.

#191 Updated by Marius Gligor over 7 years ago

1. I think that is better to start by designing our custom binary protocol. This means that we have to define the structure
of binary packages for each communication between web server and browser JS code.

2. The Dojo toolkit SDK allow us to do the following:

- To build a custom toolkit minimized and optimized containing only the used dependencies.
The result is a zip file containing only an subset of Dojo modules. The size of the zip file and the modules
could be reduced which is useful for us. I did a custom build today and it works fine.
However because we are using an embedded web server all the resources (web pages, css files, js files, image files, etc.)
should be delivered to browsers using jetty Handlers which are designed as Java classes.
So far I started to design such a Handler for Dojo toolkit using a full build Dojo toolkit minimized (13 Mb) as a zip file resource.
The requested JS modules are get from the zip file on request and serialized as HTTP response back to requester.
Finally I have to substitute this Dojo toolkit with a custom build containing only the used dependencies.
- To integrate the user applications with Dojo toolkit. Because we decided to avoid Dojo dependencies as much as possible 
this option is not in our interest.

3. For CHUI application we have to manage the following resources using WebSockets and JS code:

- Screen and cursor rendering. This could be done using HTML5 Canvas API without using any Dojo toolkit modules.
- Keyboard and mouse events. Here the Dojo toolkit could help us because this part depend on browser implementation so my suggestion is to use it.
- WebSocket communication. Here the Dojo toolkit added JS code (Long-Polling) to emulate WebSockets communications on web browsers (or servers)
which doesn't support the standard WebSockets API. Because our jetty server support standard WebSockets API and
the API is simple I think that we could design our custom JS standard module. (no other JS frameworks like Dojo).
Also I think that we will use only WebSockets API no other protocols supported by Dojo toolkit.

3. For a GUI application Dojo toolkit modules could help us but for the moment I did no investigations on this area.

#192 Updated by Greg Shah over 7 years ago

1. I think that is better to start by designing our custom binary protocol.

OK, good.

2. The Dojo toolkit SDK allow us to do the following:

OK.

Screen and cursor rendering. This could be done using HTML5 Canvas API without using any Dojo toolkit modules.

Yes.

Keyboard and mouse events. Here the Dojo toolkit could help us because this part depend on browser implementation so my suggestion is to use it.

Agreed, use Dojo here.

Because our jetty server support standard WebSockets API and the API is simple I think that we could design our custom JS standard module. (no other JS frameworks like Dojo). Also I think that we will use only WebSockets API no other protocols supported by Dojo toolkit.

Just like with HTML5 canvas, we are making it a requirement that only browsers that support websockets will be supported in our ajax driver. So it is OK to code our own approach directly using the standard websockets API instead of Dojo.

3. For a GUI application Dojo toolkit modules could help us but for the moment I did no investigations on this area.

This is a question we will answer later.

#193 Updated by Marius Gligor over 7 years ago

I designed minimal JS modules for Screen, Screen cursor, Message Beep and WebSocket communication management.
The attached archive does not contains the Dojo toolkit distribution (dojo-release-1.9.2.zip).
On the next step I'm going to design and implements the protocol to manage "Screen" and "Screen cursor" vis WebSockets.

#194 Updated by Greg Shah over 7 years ago

Code Review 0110a

I am fine with the general approach and how the code is progressing.

I only have 3 comments:

1. Although we don't have coding standards for Javascript, please consider that all Java coding standards are basically the same standards we should use for Javascript.

2. Please think about breaking the JS code into smaller modules. I'm not sure if we are at that point of needing it yet with tty.js, but I can see that we probably will need it before the chui JS client is done.

3. The HTML formatting should follow most of the same Java standards concepts. I know this may sound weird. But I think many of the standards make sense in general. For example, your HTML usually has no indents, but I would expect us to use indents of 3 spaces for each nested level like:

<!DOCTYPE html>
<html>
   <head>
      <script language="javascript" type="text/javascript" src="tty.js">
      </script>
      <script>
         window.onload=function()
         {
            tty.initSound('beep');
            tty.initScreen('screen', 25, 80, 'monospace', 20);
            tty.initWebSocket('wss://' + window.location.host + '${context}/ajax');
         };
      </script>
   </head>
   <body>
   </body>
</html>

#195 Updated by Marius Gligor over 7 years ago

I did a lot of improvements inside the JS code:
- I'm using JS sub modules stored on separate script files.
- Screen cursor improvements (could be sold/line, blinking/unblinking, visible/not visible)
- Use place holders in html page to provide initial values.

#196 Updated by Marius Gligor over 7 years ago

The following classes are used to emulate a terminal:

ChuiWebDriver extends (ChuiScreenDriver implements ScreenDriver)
ChuiPrimitives extends DriverPrimitives
ChuiWebSimulator extends (ChuiSimulator extends JComponent implements Printable)

The following methods are used by the terminal driver:

ChuiScreenDriver: ================
O beep() -> sim.beep()
O clear() -> sim.clear();
I readKey() -> sim.readKey();
O setCursorStatus(boolean on) -> sim.setCursorStatus(on);

ChuiPrimitives: ===============
O append(String str, Color color) -> sim.append(data);
O cursorAt(int x, int y) -> sim.setCursor(x, y);
O cursorStay(int x, int y) -> sim.setCursor(x, y);
I getX() -> sim.getCursorX();
I getY() -> sim.getCursorY();
I screenHeight() -> sim.getRows();
I screenWidth() -> sim.getColumns();
O sync() -> sim.triggerRepaint(true);
O updateScreen(ScreenData screen) -> sim.replace(screen.asArray(), screen.getCursorX(), screen.getCursorY());

Translation of driver calls into simulator calls:

ChuiScreenDriver,ChuiPrimitives ChuiSimulator I/O Remarks
(ChuiWebDriver) (ChuiWebSimulator) ===============================================================================================
beep() beep() O
clear() clear() O
readKey() readKey() I
setCursorStatus(on) setCursorStatus(on) O
append(str, color) append(data) O
cursorAt(x, y) setCursor(x, y) 0
cursorStay(x, y) setCursor(x, y) 0
getX() getCursorX() I (update by mouse event)
getY() getCursorY() I (update by mouse event)
screenHeight getRows() I (from configuration)
screenWidth getColumns() I (from configuration)
sync() triggerRepaint(true) O
updateScreen(screen) replace(arry, x, y) O

A possible binary protocol might looks like:

(O)utput Protocol Packet ===============================================================================================
beep() 0x80
clear() 0x81
setCursorStatus(on) 0x82 // cursor on
0x83 // cursor off
setCursor(x, y) 0x83 x y // new x, y cursor position
setCursor(x, y) 0x83 x y // new x, y cursor position
triggerRepaint(full) 0x84 size cell cell cell ... cell // synchronize screen memory
// cell = [x, y, colour, character]
// should be efficient implemented
// the cell list should contains only the changed cells.

(I)nput Protocol Packet ===============================================================================================
KeyEvent 0x00 meta key // meta = CTRL, ALT, SHIFT etc.
// key = key code
MouseEvent 0x01 meta x y // meta = click, contextmenu, dblclick
// x, y = mouse cursor position (row and col not pixels)

#197 Updated by Marius Gligor over 7 years ago

For a better view.

#198 Updated by Greg Shah about 7 years ago

It looks good (both the code in 0113a and the proposed protocol).

#199 Updated by Marius Gligor about 7 years ago

Today I started to implements the WebSockets protocol for screen management (server -> browser JS)
Please look at the attached picture in order to see the results (Ask.p)

Some observations:

The Cell class in the current CHUI screen implementation has two members, a character value and a Color object.
According to my researches the Color is not used to render the character, in other words the current implementation is for a monochrome terminal.
Please let me know if my assertion is true or not.
Finally I decided to use a JSON format for payload because is more easily to manage in JS.
Binary payloads require more processing in JS which it's a very slow process.
Only the operation to synchronize the screen is subject to be optimized.
The rest of operations are very simple and the JSON payloads are very small.
So far due to the existing implementation in p2j all cells in the screen are send as a text message.
On the client side only the changed values are rendered in other words the rendering process is optimized because this is a slow process in JS.
Regarding the network performances for an Ethernet MTU of 1500 bytes in size only 2 packets are used to send the entire screen over the network.
According to my tests this implementation seems to be very fast on my workstation.
Nevertheless we can think to further improvements but only after we have also the input protocol in place.
Also if we need colour support for cells inside converted P4G applications we have to do some changes.
For more details please look at ChuiWebSimulator class.

#200 Updated by Greg Shah about 7 years ago

The code changes in 0114a look good. There is some missing javadoc, but otherwise things seem pretty much on target.

According to my researches the Color is not used to render the character, in other words the current implementation is for a monochrome terminal.
Please let me know if my assertion is true or not.

It is true for today. But Hynek is working on proper COLOR support in #2037 and so this is about to change (in the next 2 weeks).

Finally I decided to use a JSON format for payload because is more easily to manage in JS.
Binary payloads require more processing in JS which it's a very slow process.

I know that other implementations have found it possible to improve performance using binary transfers. Have you reviewed the contents of UBJSON (http://http://ubjson.org/)?

We can go text for now, but we may need to shift this approach to binary in the future.

#201 Updated by Marius Gligor about 7 years ago

I've started to implements the protocol for keyboard management.
I have good news! So far It's works fine see the attached picture.
However invoking the primes.p results in server crash because my converted application is obsolete.
Since the time of my conversion many other changes were done on the project.
So I tried to do a new conversion but unfortunately without success.
Is the testcases project updated?
Should I do an update for testcases project from the remote repository?

#202 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

So I tried to do a new conversion but unfortunately without success.

What command do you use for conversion? When converting directly under the uast/ folder, is best to pass an explicit file list to the ConversionDriver.

Should I do an update for testcases project from the remote repository?

Yes, you should update it (there are changes in some scripts, like server.sh, not just tests).

#203 Updated by Marius Gligor about 7 years ago

According to conversion handbook:

export P2J=/mag/p2j/to/p2j
export CLASSPATH=$P2J/build/lib/p2j.jar

java com.goldencode.p2j.convert.ConversionDriver -d2 f2+m0+cb ask.p primes.p

#204 Updated by Constantin Asofiei about 7 years ago

And what error do you get?

#205 Updated by Marius Gligor about 7 years ago

Marius Gligor wrote:

According to conversion handbook:

export P2J=/home/mag/p2j
export CLASSPATH=$P2J/build/lib/p2j.jar

java com.goldencode.p2j.convert.ConversionDriver -d2 f2+m0+cb ask.p primes.p

#206 Updated by Marius Gligor about 7 years ago

I attached the build log.

#207 Updated by Constantin Asofiei about 7 years ago

Make sure you are using the latest revision (and P2J is built). From the log, you have an outdated rules/annotations/record_scoping_prep.rules file.

#208 Updated by Marius Gligor about 7 years ago

My p2j project is updated and the build all is OK, no errors. However the conversion does not work.
The rules/annotations/record_scoping_prep.rules file size is 24.326 bytes and the date is 6 December 2013 13:58

#209 Updated by Constantin Asofiei about 7 years ago

Then I think there is some other p2j/ installation on the path; and I can't be of much help with that. Why I say this: from the conversion log, the failure is at this expression:

buf.registerBuffer(uniqueName, bufName, schemaName, javaName, getNoteString("dbname"), (level == -1), upPath("TRIGGER_PROCEDURE/KW_OLD") or upPath("STATEMENT/KW_ON/KW_OLD/"))

but in the record_scoping_prep line 406 (file history H019 - this is what we use to check the file version) there is this expression:
               buf.registerBuffer(uniqueName,
                                  bufName,
                                  getNoteString("force_dmo_alias"),
                                  schemaName,
                                  javaName,
                                  getNoteString("dbname"),
                                  (level == -1),
                                  upPath("TRIGGER_PROCEDURE/KW_OLD") or
                                     upPath("STATEMENT/KW_ON/KW_OLD/"),
                                  isNote("is_dynamic_tables") and
                                     getNoteBoolean("is_dynamic_tables"))

See how the getNoteString("force_dmo_alias") and isNote("is_dynamic_tables") and getNoteBoolean("is_dynamic_tables") arguments are missing from the logged expression.

Search your drive for the record_scoping_prep.rules files and see which P2J has the incorrect file.

#210 Updated by Marius Gligor about 7 years ago

Thanks I fixed the problem.
I just copied the rules directory from p2j on testcases/uast/p2j/rules and the conversion was done.

#211 Updated by Greg Shah about 7 years ago

Please note that you should always have a symbolic link in your project home (testcases/uast/) to the p2j/ directory that you are using.

#212 Updated by Greg Shah about 7 years ago

I've started to implements the protocol for keyboard management.
I have good news! So far It's works fine see the attached picture.

That is AWESOME! I am very excited to see this come together. Great work!

#213 Updated by Marius Gligor about 7 years ago

I did a new conversion for Ask.p and Primes.p
Unfortunately when I want to start the primes.p on my emulator the server and the client crash.
The same results I obtained when I'm using the swing client (see the attached files)!
Seems to be a common problem but I'm not sure if is the result of my recently changes.
Could you help me please with some hints?

#214 Updated by Constantin Asofiei about 7 years ago

Take the testcases/simple/server/server.sh from the repository. The problem is caused by the fact that the server is missing this JVM argument:

-Djava.system.class.loader=com.goldencode.p2j.classloader.MultiClassLoader

#215 Updated by Marius Gligor about 7 years ago

Now seems to be a serialization problem.

WARNING: {Reader} failure in reading loop
java.io.InvalidClassException: com.goldencode.p2j.util.NumberType; local class incompatible: stream classdesc serialVersionUID = -505777196529990035, local class serialVersionUID = 4677557220506711006
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at com.goldencode.p2j.ui.client.WidgetEntry.readExternal(WidgetEntry.java:116)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1704)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1342)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at com.goldencode.p2j.ui.ScreenBuffer.readExternal(ScreenBuffer.java:929)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1704)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1342)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at com.goldencode.p2j.net.Message.readExternal(Message.java:367)
at com.goldencode.p2j.net.SyncMessage.readExternal(SyncMessage.java:102)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1835)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1794)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at com.goldencode.p2j.net.Protocol$Reader.byteArrayToObj(Protocol.java:436)
at com.goldencode.p2j.net.Protocol$Reader.run(Protocol.java:347)
at java.lang.Thread.run(Thread.java:724)

com.goldencode.p2j.net.SilentUnwindException: Connection ended abnormally
at com.goldencode.p2j.net.Queue.transactImpl(Queue.java:1140)
at com.goldencode.p2j.net.Queue.transact(Queue.java:583)
at com.goldencode.p2j.net.BaseSession.transact(BaseSession.java:179)
at com.goldencode.p2j.net.HighLevelObject.transact(HighLevelObject.java:163)
at com.goldencode.p2j.net.RemoteObject$RemoteAccess.invokeCore(RemoteObject.java:1416)
at com.goldencode.p2j.net.InvocationStub.invoke(InvocationStub.java:97)
at com.sun.proxy.$Proxy4.standardEntry(Unknown Source)
at com.goldencode.p2j.main.ClientCore.start(ClientCore.java:260)
at com.goldencode.p2j.main.ClientCore.start(ClientCore.java:80)
at com.goldencode.p2j.main.ClientDriver.start(ClientDriver.java:223)
at com.goldencode.p2j.main.CommonDriver.process(CommonDriver.java:423)
at com.goldencode.p2j.main.ClientDriver.process(ClientDriver.java:1)
at com.goldencode.p2j.main.ClientDriver.main(ClientDriver.java:241)
Caused by: com.goldencode.p2j.net.ApplicationRequestedStop: Queue stop is requested by application
at com.goldencode.p2j.net.Conversation.block(Conversation.java:304)
at com.goldencode.p2j.net.Conversation.waitMessage(Conversation.java:257)
at com.goldencode.p2j.net.Queue.transactImpl(Queue.java:1126)
... 12 more

#216 Updated by Constantin Asofiei about 7 years ago

The problem is you have two distinct P2J builds - one for the server, and one for the client. And the serialVersionUID in the two builds (For a certain class) are not the same. You should always have a single P2J build and symlink to it as necessary, as the P2J server and the P2J client must use the same compiled sources.

#217 Updated by Marius Gligor about 7 years ago

OK Thanks. Now the SWING driver works fine.
I have configurations on my IDE to start the server and the client which helps on development.
I have to find a solution to change also my IDE run configurations.

#218 Updated by Constantin Asofiei about 7 years ago

The swing chui client can be started with no problem from Eclipse (and I guess from other IDE too); just set it up like this:
  • working dir: testcases/simple/client
  • main class: com.goldencode.p2j.main.ClientDriver
  • program args: client.xml; its content is (change the ports accordingly):
    <?xml version="1.0"?>
    <node type="client">
       <net>
          <server secure="false" />
          <server host="localhost" />
          <server secure_port="3334" />
          <server insecure_port="3434" />
       </net>
       <client>
          <driver type="swing_chui_frame"/>
          <chui rows="24"/>
          <chui columns="80"/>
          <chui background="0x000000"/>
          <chui foreground="0xFFA500"/>
          <chui selection="0x0000FF"/>
          <chui fontname="monospaced"/>
          <chui fontsize="12"/>
       </client>
    </node>
    
  • JVM args: -Dfile.encoding="UTF-8" -XX:+HeapDumpOnOutOfMemoryError -Xdebug -Xnoagent -Djava.compiler=NONE -Djava.library.path=../../../build/lib/ (library path can be changed as needed)

#219 Updated by Marius Gligor about 7 years ago

OK. Now it works from both command line and from IDE.
I attached some well known screens this time taken from my web browser :-)

#220 Updated by Greg Shah about 7 years ago

Very cool stuff. Please use testcases/uast/keys.p to test out your key processing. Compare the results to running the Swing client and the NCURSES client.

#221 Updated by Greg Shah about 7 years ago

By the way, our current ChUI client (swing and ncurses) does not support mouse events. You don't need to add support for mouse to the AJAX client at this time.

#222 Updated by Marius Gligor about 7 years ago

Today I started to implements the keyboard events protocol.
So far I tested:

-printable characters OK
-CTRL-C OK
-beep sound OK
-mouse events removed (not used)

However we have to do more tests as you already recommended.
On the next step I'm going to do extended tests also with other web browsers.

#223 Updated by Constantin Asofiei about 7 years ago

Marius: what happens when the browser is closed? Does the P2J client process get shut down properly?

#224 Updated by Marius Gligor about 7 years ago

This feature isn't yet implemented but will be. We have a notification on web sockets when the connection is closed on browser.
My intention is to implements here the code to terminate the client process.
Also other changes I have to made which are currently in my short list.
For example on CTRL-C the ClientCore#start loop is executed again and we should avoid to reinitialize the embedded server.
I did some temporary changes inside ClientCore class which allow me to run the client directly.
Also temporary the server embedded port is fixed to 5555 instead a random value.
That's why the ClientCore in not yet in my attached archive.
Finally I have to tests the entire flow by eliminating all temporary changes.

#225 Updated by Greg Shah about 7 years ago

Code Review 0115a

I am generally fine with the changes. Minor feedback:

1. Some imports need to be shifted to *.

2. Please put a comment in tty.js that the socket module must be initialized before the keyboard module.

3. You can put the mouse code back into screen.js. I don't think it hurts anything to have it.

#226 Updated by Marius Gligor about 7 years ago

1. Today I converted the keys.p application and I implemented the keyboard codes conversion for p4g.
I observed that the ESC-KEY is equivalent to ALT-KEY.
On Linux it works fine but I have to do some changes to work also in Windows OS.
On Windows OS the CTRL and ALT key are repeatead by keeping the keys pressed on Linux not.

2. On Linux I tested using Firefox, Chromium and Chrome browsers:
- Firefox works perfect.
- Chromium works fine but the font base line in canvas context is calculated wrong and the characters are rendered in a wrong position! (see chromium.png)
- On Chrome doesn't work and I got an error:
WebSocket connection to 'wss://192.168.1.15:5555/ajax' failed: Could not decode a text frame as UTF-8.

3. On Windows I tested using Firefox, Chrome, Safari, IE and Opera browsers:
- Firefox works perfect with some minor problems on rendering on old versions works perfect.
- On Chrome does not work and I got an error like on Linux:
WebSocket connection to 'wss://192.168.1.15:5555/ajax' failed: Could not decode a text frame as UTF-8.
- Safary like on Chrome since is based on the same engine.
- IE 9 does not support web socket only from IE 10. I have no IE 10 on my workstation.
- On Opera works like on Chromium Linux.

4. In conclusion so far only Mozilla Firefox on Linux works perfect.
Unfortunately we have to do more works in order to support other browsers.

5. Another issue is related to fonts. In Web Browsers the fonts are rendered different than in Swing.
For example I'm using the monospace font which looks good but the Swing client does not recognize the font as monospaced!
Using monospaced font results in an ugly screen so in JS I changed the monospaced with monospace font (see monospaced.png).

6. When the user press CTRL-C on ClientCore#start a loop is executed. The current session is closed and another session is open.
This works fine with guest account but doesn't work with our temporary accounts. Possible solutions for this issue might be:
- Re initialize the application without to close the session.
- Use the initial credentials to reopen the session. In this case we have to keep the passwords for temporary accounts unchanged in cache.

#227 Updated by Marius Gligor about 7 years ago

On Chrome seems to be a bug when secured connection (WSS) with self signed certificate is used.
https://code.google.com/p/chromium/issues/detail?id=119793

#228 Updated by Greg Shah about 7 years ago

Code Review 0116a

Generally, I'm OK with the code. My only concern: the use of Runtime.getRuntime().halt(statusCode) seems a bit "heavy handed". What is the reason it is needed?

#229 Updated by Greg Shah about 7 years ago

Using monospaced font results in an ugly screen so in JS I changed the monospaced with monospace font (see monospaced.png).

It is fine to have different defaults for the JS and Swing clients.

In regard to the cross-browser compatibility issues, please limit the effort you put in on that right now. For the near term, we are only going to support very recent versions of:

Firefox
IE
Safari
Chrome

Opera, mobile browsers and older browsers will be out of scope for now.

As to the versions of the above which we will support, we will pick those based on our requirements. So long as we limit our requirements to pretty common features (html5 canvas, html5 audio tag, websockets...), then we will be able to pick reasonable versions that already exist. But it may be needed to use very recent versions of these browsers (e.g. IE 10).

Our customers won't be rolling this out til late this year or next year at the earliest. I don't want to spend time on support for browsers that won't even be heavily installed at that time. Our customer has already agreed to this.

#230 Updated by Marius Gligor about 7 years ago

1. I found another way to terminate the client process on connection close. Just simple put an END-ERROR key (code 304) on the keyboard queue.
Doing so I force the ClientCore.start loop to exit and finally to terminate the client process.

2. I found two pieces of code SwingKeyboardReader.java and Key.java which helps me to implements the p4g keyboard mapper in java script.
Today I redesigned the JS keyboard code accordingly. On the next step I have to do more tests and compare the results by running keys.p
So far according to my tests seems to be OK on almost all cases.

3. I did also more tests on Windows OS:

- I tested with Firefox on Windows 8.1 and it works almost perfect. Very small problems on rendering.
- I tested with IE 11 on Windows 8.1 and it works fine. Just some small rendering problems.
- I tested with Chrome 31 and it works! The same observations like on IE 11.
- After updated to Chrome 32 (the latest release) it doesn't work ;-(
I updated the jetty library to the latest stable release (jetty-9.1.1.v20140108) and I tried again with Chrome 32
Unfortunately the same results, doesn't work. Finally I reported the issue to Google Chrome team.

#231 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

1. I found another way to terminate the client process on connection close. Just simple put an END-ERROR key (code 304) on the keyboard queue.
Doing so I force the ClientCore.start loop to exit and finally to terminate the client process.

I don't think this will work. The application logic can catch the raised ERROR condition and choose to ignore it. The fact that the client ends in your case is just a coincidence specific to that program. To catch the ERROR condition, use something like:

do on error undo, next:
  /* main logic goes here */
end.

LE: the code needs to ignore, leave.

#232 Updated by Greg Shah about 7 years ago

I checked in the TerminalMenu class and we have this code for the Exit menu item:

      if (container.canExit())
      {
         exit.addActionListener(new ActionListener()
         {
            public void actionPerformed(ActionEvent evt) 
            {
               sim.quit();
               System.exit(0);
            }
         });
      }

Go ahead and use System.exit() to cause the process to halt.

#233 Updated by Greg Shah about 7 years ago

Code Review 0117a

The changes look fine except for the END-ERROR approach already discussed above.

The use of System.exit() should result in the session termination anyway, so I don't think you need the session object to be there.

Obviously, there is still some work to go with getting the restart behavior right on a CTRL-C, but it is shaping up nicely.

Good stuff!

#234 Updated by Marius Gligor about 7 years ago

System.exit(code) works fine.

#235 Updated by Greg Shah about 7 years ago

Code Review 0116b

The code looks good. Does ChuiWebSimulator really need to call super.quit()?

#236 Updated by Marius Gligor about 7 years ago

1. I did more investigations regarding the Google Chrome web browser.
Both Google Chrome and jetty teams works currently to implements WebSockets extensions.
Google Chrome is ahead and the latest release is not compatible with the current jetty implementation.
Here is a snapshot representing the handshake between browser and server.

Request URL:wss://192.168.1.15:5555/ajax
Request Method:GET
Status Code:101 Switching Protocols
Request Headers CAUTION: Provisional headers are shown.
Cache-Control:no-cache
Connection:Upgrade
Host:192.168.1.15:5555
Origin:https://192.168.1.15:5555
Pragma:no-cache
Sec-WebSocket-Extensions:permessage-deflate; client_max_window_bits, x-webkit-deflate-frame
Sec-WebSocket-Key:J23OCuuwjQ4i/evCnDnNog==
Sec-WebSocket-Version:13
Upgrade:websocket
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36

Response Headersview source
Connection:Upgrade
Sec-WebSocket-Accept:S/YgcUNrTi05FKJdi+l0aO6PkJo=
Sec-WebSocket-Extensions:permessage-deflate, x-webkit-deflate-frame
Upgrade:WebSocket

As you can see Google Chrome already implemented the client_max_window_bits WebSocket extension but jetty not.
So we have to wait until all WebSockets extensions will be implemented by jetyy and than upgrade our library.

2. I did some small changes inside the java script screen module. I fixed all issues related to canvas rendering.
Fortunately the old Chrome 31 release remains on my Windows installation and can be used for tests.
Now the applications looks good on all major web browser Chrome, Firefox, IE, Opera.
I cannot do tests using Safari on Windows because the version installed on my laptop does not work like the new Google release.

3. Currently I'm working to implements the keyboard interface in JS.
In JS we have 3 events related to keystrokes which are fired in the following order:

keydown
keypress
keyup

The keypress event is fired only for keys mapped to printable characters not for modifier keys (CTRL, ALT, SHIFT).
This event provide the printable character code for the key pressed.
Also the event object does not contains any information related to CTRL ALT and SHIFT keys
because if keys like CTRL, ALT or SHIFT are pressed the keypress event is not fired at all.

The keydown event is fired on all keys but unfortunately we receive a key scan code not the mapped printable character code.
Even more not all the key scan codes are the same across the web browsers. (see http://www.javascripter.net/faq/keycodes.htm)
The mapping between scan codes and character codes depends on the keyboard layout (US, EN, RO etc.)
So I think that we have to design and implements also a key mapper for a US keyboard layout inside the JS code.
Please let me know your opinion related to this issue.

#237 Updated by Greg Shah about 7 years ago

Please document the differences between Java/Swing key processing and the Javascript equivalent. It was my understanding that the two models were pretty close. And I thought that the event.shiftKey, event.ctrlKey, event.altKey and event.metaKey properties were defined on the keypress event object. Is that incorrect?

If those modifier properties are there, then it seems we could leverage our SwingKeyboardReader code. Perhaps we can simulate the Java events using the JS input and let the SwingKeyboardReader code handle the processing on the Java side.

#238 Updated by Marius Gligor about 7 years ago

1. Today I did the appropriate changes inside the JS code in order to manage the keyboard events.
So far all my tests reveals the same results using keys.p on Swing emulator and JS code emulator.
Please take a look over my attached code.

2. In JS we have all modifiers keys inside the event object but on the keydown event not in keypress event.
In keydown event we have only the key scan code not the printed character code.
They are no ways to convert the scan code to character code even using JS tools like Dojo, jQuery etc.
On my JS code I made a conversion from scan code to character code only for printable keys inside the keydown event.
Doing so I'm able now cto ompute the key codes for combinations like CTRL and/or ALT.
The key map is done for a US keyboard layout.
Unfortunately JS is much more restricted and limited than Java.

The keypress event is used to process the printable characters which are available on the event object.
I'm not sure because I have only US keyboards on my computers, but processing the special characters on keydown
and printable characters on keypress event could solve also the keyboard layout issue.

3. In my opinion trying to leverage the use of SwingKeyboardReader require more work effort and I'm not sure if it finally works.
The keyboard events in Java and JS are different as I already described in paragraph 2.

#239 Updated by Marius Gligor about 7 years ago

  • File mag_upd20140117b.zip added

Here is an improved version for keyboard management. Seems to be closed to the final solution.
Nevertheless I will do more tests.

#240 Updated by Greg Shah about 7 years ago

I have not gotten to look at your latest code yet.

However, I think you have come to the wrong conclusion in regard to the keypress event. You can definitely get the state of the modifier keys. Please try out this code:

<html>
   <head>
      <title>keypress Event Tester</title>

      <script type="text/javascript">
         keyHandler = function(event)
         {
            text = "keypress event:<br>";

            for (attr in event)
            {
               text = text + attr + " = " + event[attr] + "<br>" 
            }

            element.innerHTML = text;
         }

         var element;

         if (window.addEventListener)
         {
            window.addEventListener("load", function()
            {
               document.addEventListener("keypress", keyHandler, false);
               element = document.getElementById("something");
            }
            , false);
         }
      </script>

   </head>
   <body>
      <div id="something">Type keys!</div>
   </body>
</html>

You can see that there are Boolean properties for each of the modifier keys. As far as I can tell, we should be able to use keypressed for the printable keys (eliminating the layout issues) and use keydown for non-printable stuff, which is just like we do in Java.

Perhaps the Dojo event processing is obscuring this? I deliberately just used basic JS for this example. I have only tested it in Firefox, but it works great there.

#241 Updated by Greg Shah about 7 years ago

By the way, you re-uploaded mag_upd20140117b.zip when I think you intended to upload mag_upd20140121b.zip.

#242 Updated by Marius Gligor about 7 years ago

  • File deleted (mag_upd20140117b.zip)

#243 Updated by Marius Gligor about 7 years ago

#244 Updated by Marius Gligor about 7 years ago

I Tested your code and here are my conclusions:

1. The code does not work at all on IE (Windows) web browser.
The event keypress is fired only for keys mapped to printable characters.
For example if CTRL or ALT key is pressed the event keypress isn't fired.

2. The code does works partial on Chrome (Linux and Windows).
- If CTRL key is pressed works only if event.defaultPrevented = true. For example it's works for CTRL-Q but does not work for CTRL-A, CTRL-B, CTRL-S etc.
- If ALT key is pressed the event is not fired.

3. The code does not work on Safari (Windows).
The event keypress is fired only for keys mapped to printable characters.
Foe example if we press CTRL-Q the browser is closed because on CTRL-Q is mapped to exit command.

4. On Firefox on the other hand it's works indeed but with a small change.
In the original code when you press CTRL-S for example the keystroke also open the browser "Save AS" dialog.
In order to prevent this behaviour we have to avoid the browser action execution associated with the event by adding the following line: event.preventDefault()
With this changes in place the code works as we expected.

<!DOCTYPE html>
<html>
<head>
<title>keypress Event Tester</title>

&lt;script type="text/javascript"&gt;
keyHandler = function(event) {
event.preventDefault();
text = "keypress event:&lt;br&gt;";
for (attr in event)
{
text = text + attr + " = " + event[attr] + "&lt;br&gt;"
}
element.innerHTML = text;
}
var element;
if (window.addEventListener)
{
window.addEventListener("load", function() {
document.addEventListener("keypress", keyHandler, true);
element = document.getElementById("something");
}
, false);
}
&lt;/script&gt;
&lt;/head&gt;
&lt;body&gt;
&lt;div id="something"&gt;Type keys!&lt;/div&gt;
&lt;/body&gt;
&lt;/html&gt;

5. The event object structure is different on each web browsers. It contains common fields as well as custom fields specific to each web browser.
Another important difference between JS and Java: CTRL key in not processed in JS.
We have only the key code or char code for the key and a flag which store the CTRL key status (true - pressed, false - not pressed)

6. In conclusion the Firefox behaviour on keypress event seems to be a really useful solution for our purposes.
Unfortunately cannot be used for a cross web browsers solution.

#245 Updated by Marius Gligor about 7 years ago

I did some small changes and improvements.
Also I tested a version using native JS instead Dojo. The results are the same.

Doing tests I found some interesting behaviours running on Linux (Kubuntu):
Some keystrokes like ALT+F4, ALT+F2 are processed by OS and are not available to applications!
I obtained the same behaviour running the ChuiSwing client and your JS example code:
-On ALT+F4 the application windows is closed.
-On ALT+F2 the KDE search window is open.
Doing tests in Windows OS this not happens.

My current JS implementation works exactly like ChuiSwing simulator when I executed key.p application.
So far I found no differences but nevertheless might be since testing all possible combinations took a while.

#246 Updated by Greg Shah about 7 years ago

OK, these are good findings.

Please review the DOM level 3 events specification: http://www.w3.org/TR/DOM-Level-3-Events/#event-type-input

I wonder if the keydown can be used for most processing but then for the printable chars the corresponding input event can be used.

The issue here is simply that we will constantly be having to add support for new keyboard layouts if we cannot find a better solution. I assume the customer uses a UK English layout. We have other potential customers around the world that will certainly have other layouts in use. Now if the time to see if we can find a more elegant solution. Obviously, I prefer this to be completely in JS.

But certainly we do know that the Java Swing code has already solved this issue. I don't just mean our code in SwingKeyboardReader. I mean that the Java platform solves this problem. Is there a way to leverage this? For example, if we detect the locale/layout in JS and then match that with the same locale in Java, we can force it into a Swing component using component.getInputContext().selectInputMethod(locale). From there we can potentially use that component to process the low level events that we create OR we can dynamically create a map (of low-level to high-level transformations) which can be sent down to the JS code.

#247 Updated by Marius Gligor about 7 years ago

1. The DOM level 3 events specifications are implemented total or partial on all major web browser but sligtly different.
Using your JS code which is really a valuable tool I did a lot of tests on different web browsers for keydown and keypress events (see the attached files)
The conclusion is that we cannot get the character code of a key using the keydown event.
We have to combine the keydown event and key press event as follow:

- On keydown we have to handle only the control keys, that is, keys having non printable characters mapped.
This works fine even for diferent layouts. I tested a QWERTZ DE (German) layout keyboard and a US QWERTY layout keyboard.
Pressing CTRL+Y and CTRL+Z for example works fine even thinking that the position on keyboard differ on US and DE layouts.

- On keypress we have to handle only keys having printable characters mapped.
In this case the charCode of event object contains the Unicode value for the character which is mapped to key in format UTF-16.
Example: 8364 (U+20AC) which correspond to EURO sign. The question here is: Does p4g process Unicode UTF-16 characters?

2. A keyboard could have many layouts associated with locales. Even a US key has more than one layout: US standard and US International layout (see the attached picture)
The layouts except the US Standard extends the keyboard capabilities by adding more characters on some keys (3 or 4)
In this case the ALT key from the right is used as a control key which in combination with another key generate the character code.
This special key is called ALTCHAR key. Example ALT-CHAR+E on a DE layout will get 8364 (the EURO sign)
Even more the ALTCHAR key could be combined using the SHIFT key like ALTCHAR+SHIFT+E and allow to define up to 4 characters / key.
Using the DOM level 3 events specifications according to my researches the the ALT key from the right works as follow:

- On the US Standard layout only for a keydown event the event object has the following properties:
altKey = true
ctrlKey = false
location = DOM_KEY_LOCATION_RIGHT (2)
keyCode = ALT (18)
- On all other layouts which extends the keyboard capabilities:
altKey = true
ctrlKey = true
location = DOM_KEY_LOCATION_RIGHT (2)
keyCode = ALT (18)
In conclusion the location and keyCode define the ALT key from the right (we have 2 ALT keys on a keyboard, one on the left and one on the right)
The ctrlKey specify the cope of the key. If ctrlKey=false means normal if ctrlKey=true menas that the key is used as ALTCHAR.

As a result of my researches I implemented and tested the keyboard layout handle in JS.
Now the code works fine using different layouts like US International, English UK, German DE, Romanian RO, etc.
I did all the tests on Windows OS only using IE, Chrome, Firefox, Safari and Opera web browsers.

The only problem that I found is in Safari which does not set the location field on event object.
The last release 5.1.7 for Windows Safari is dated May 9 2012 and think that we could consider that Safari for Windows is obsolete.
The same observation for Opera, 2012 is the year of the latest release!

#248 Updated by Greg Shah about 7 years ago

Code Review 0123a

This is really, really good! I'm glad that you were able to work out an approach that is much more generic. Well done.

My only question: Should mapUSkeys() be renamed to something else, since this is no longer a US-only solution?

Answers to your questions:

The question here is: Does p4g process Unicode UTF-16 characters?

It can. But more importantly, when we are operating in Java we DO process using Unicode. So it is perfectly valid to have the JS code deal entirely in Unicode.

Where we output something that can have different codepages, we have to write code to handle that (to convert the Unicode data into the codepage-specific output). Likewise, where we read input that can come in multiple codepages, we must have code that is aware (so that it can translate from the input codepage into Unicode). If everything is already in Unicode, no translation is needed.

The only problem that I found is in Safari which does not set the location field on event object.
The last release 5.1.7 for Windows Safari is dated May 9 2012 and think that we could consider that Safari for Windows is obsolete.
The same observation for Opera, 2012 is the year of the latest release!

Old releases are not a problem. We aren't planning to support Opera at all right now and for Safari, we can choose a much newer version that does work.

#249 Updated by Marius Gligor about 7 years ago

I did some small improvements and bug fixing. Basically I eliminated the dojo/keys module dependencies.
With this changes in place it's very easy to eliminate all the Dojo dependencies.
I found no differences using Dojo (a) or native (b) code, so I think that at this stage we could use the jS native code version for CHUI.
Please let me known your opinion regarding this issue.

#250 Updated by Greg Shah about 7 years ago

Code Review 0124a

All the changes look good. I like the more modular approach.

Code Review 0124b

I prefer the non-Dojo version. My only question: does the document.addEventListener() work properly on IE (at least older versions didn't have it). If it is working properly there, then let's go with the non-Dojo version for now. Good stuff.

#251 Updated by Greg Shah about 7 years ago

Can you please post a list of what is left to do on #2212?

#252 Updated by Marius Gligor about 7 years ago

1. We have to solve the CTRL-C issue to initialize the converted application without a new server authentication if possible.
2. I have to do some tests with IE and will be back to you.

#253 Updated by Marius Gligor about 7 years ago

It's works in IE but I have to fix a small issue. Event preventDefault() does not work properly with IE.
I tested both versions Dojo/Native and the problem is the same.
I'm going to fix this issue.

#254 Updated by Marius Gligor about 7 years ago

Some small changes to fix ESC key in IE.

I did more tests on Windows using IE 11 and Chrome 31.
Unfortunately on both web browser does not work properly ;-(
The short-cuts keys are always processed by the web browser.

For example CTRL+N in Chrome always open a new Window, CTRL+T a new tab page.
On IE ALT+H goes to Help option on menu bar, ALT+F4 close the application.

Firefox seems to be the only web browser which works properly (as we need).
I will try to find a solution if exists for IE and Chrome but seems to be a big challenge.

#255 Updated by Greg Shah about 7 years ago

To disable default behavior you may have to do 2 things:

1. Prevent the default behavior. In browsers that support addEventListener() there is the event.preventDefault() which does this. IE in versions prior to IE9 used event.returnValue = false to cause the same result.

2. Stop the event from "bubbling" or propagation. The idea is that there may be other event handlers registered on the current DOM element or on one of the parent DOM elements. Unless you take some action, these other event handlers will be executed. In browsers that support addEventListener() there is the event.stopPropagation() which stops invocation of any event handlers registered on the PARENT DOM elements. Prior to IE9, IE had event.cancelBubble = true to achieve the same thing.

But the new DOM Level 3 Events supports a event.stopImmediatePropagation() which is even better because it stops invocation of additional handlers that are registered at the current DOM element (not just those registered at the parent DOM element).
Have you tried event.stopImmediatePropagation()?

One other idea: instead of registering for key events at the document level, I wonder if registering at the canvas may be better for cancellation purposes. On the other hand, it may create conditions where some events are not delivered if the keyboard focus is somehow placed outside of the canvas (e.g. with a mouse click).

#256 Updated by Greg Shah about 7 years ago

Code Review 0124c

My only thought here is that we probably need to always preventDefault() and stopImmediatePropagation() for every event, since these events are meant to be consumed by us. For now, you only do this when you detect the modifier keys.

#257 Updated by Marius Gligor about 7 years ago

Starting from IE 9 preventDefault() is supported. We are interested on IE 10 and IE 11 because IE 9 does not support WebSockets.
In my code I'm using both preventDefault() and stopImmediatePropagation() but with no effect.
Anyway during this weekend I will try to find a solution.
One idea is to use a pop-up window for our JS simulator.
I saw that when a pop-up window is created we can eliminate menu bars and toll bars.
Do you believe that could be a solution?

#258 Updated by Greg Shah about 7 years ago

I saw that when a pop-up window is created we can eliminate menu bars and toll bars.
Do you believe that could be a solution?

Possibly.

I have been considering this approach in general, as it has some other advantages. But I am worried about the disadvantages that may come along with this. Popup blocking configurations in the browser can interfere and there may be other issues. Please analyze the pros/cons of this approach and document them here.

#259 Updated by Marius Gligor about 7 years ago

Looking for a solution to fix the cross browser keystrokes handle issues I found:

Supported Key Combinations:
The actual possible key combinations vary greatly between browsers, as most browsers have a number of built-in short-cut keys, which can not be used in web applications.
For example, Mozilla Firefox allows binding almost any key combination, while Opera does not even allow binding Alt short-cuts.
Other browsers are generally in between these two.
Also, the operating system can reserve some key combinations and some computer manufacturers define their own system key combinations.
If the browser handles the short-cut key automatically and does not pass the event on there is really not much we can do to fix it.

Indeed IE and Chrome define a set of reserved keys which cannot be override. We have no way to prevent their actions!

1. IE - Most of the reserved keys are ALT+key combinations but also CTRL+key combinations exists.
The actions assigned to the reserved keys are always executed and the keystrokes are sent also to JS code with few exceptions (ALT+F4 close the browser).
Here: http://connect.microsoft.com/IE/feedback/details/778745/default-browser-behavior-for-alt-key-shorcuts-can-be-overridden-by-the-accesskey-attribute-but-not-by-scripts
the Microsoft IE team advice us that we cannot override the browser default due to a design constraint.

2. Google Chrome define also a set of reserved keys (short-cuts) that cannot be override.
Even more some combination like CTRL+N, CTRL+T never reach the page JS code in other words we never receive events for this keystrokes.
Goolge Chrome could be lunched in a kiosk mode. On this mode we have control over Fn keys and ALT+Fn keys.
ALT+F4 never close the window but CTRL+N, CTRL+T, etc. cannot be ever controled.
The worst case ever CTRL+W close the Chrome browser on both modes.

3. Mozilla Firefox is the only web browser which allow us to control the keyboard keystrokes almost 100%.
So far I found no keys combinations which could not be processed in JS and prevent default behaviour.
In Windows using Firefox 26 (the latest release) works perfect with our JS simulator.
In Kubuntu Firefox 26 works the same except for some key combinations which are reserved by the OS
(ALT+F2 open the search window, ALT+F3 open window menu, ALT+F4 close the application main window).
Fortunately we have the ESCAPE+Fn key combination as an alternative.

4. My idea was to use a modal pop-up window hoping that we have full control over all keyboard events.
Unfortunately the assertion is not true and the actions assigned to reserved keys cannot be prevented.

5. In conclusion only the Firefox (26) seems to offer us the entire control over keyboard events.
The question is: Will Firefox offer this capabilities also in the future?
It's no reason to spend more time searching for a solution to use IE and Chrome because we have no solution!
Is a huge frustration to see these constraints that can not be fixed in any way!

#260 Updated by Marius Gligor about 7 years ago

The keyboard listeners are bind on canvas instead document.
The canvas should have the focus in order to receive keyboard events. On start the canvas is the focus owner.
It's works basically like previous version.
The problems with IE and Chrome reserved keys remains the same.

#261 Updated by Greg Shah about 7 years ago

Code Review 0125a

I don't want to remove all the server-side Dojo support that you already provided. Put that work back in so that when the time comes for us to use Dojo features, we will be able to easily do so.

About the unavailable keys, don't worry. It was on my list as a potential issue that might not be resolved. We will document this set of limitations and customer's can choose browsers knowing the limitations they will have to deal with.

With enough effort, we could solve this problem. But it would require a non-trivial bit of native code for each platform we would support (and probably some native code for each browser too). For example, most platforms provide an architected hook interface to intercept messages before they are delivered to the application's specific queue. On Windows:

http://msdn.microsoft.com/en-us/library/windows/desktop/ms632589%28v=vs.85%29.aspx

Since most (nearly all) GUI implementations are based on a message-passing design, this trick is something that is commonly available. But it is NASTY and FRAGILE. And in the end it is not our responsibility to make up for the design decisions of the browser developers.

At this time, just make sure you document the list of browser-specific key handling findings so that we don't have to do more research later.

#262 Updated by Greg Shah about 7 years ago

Code Review 0125b

I like this version better from the perspective of how you made the code better separated.

But since the canvas approach doesn't solve any real problems for us AND it does add a new focus problem, I think that it is best to go back to registering the handlers on the document.

#263 Updated by Marius Gligor about 7 years ago

  • File mag_upd20140125c.zip added

Done.

#264 Updated by Greg Shah about 7 years ago

Code Review 0125c

I like it. The only thing I don't understand is why the tty.keymap.js is missing.

#265 Updated by Marius Gligor about 7 years ago

  • File deleted (mag_upd20140125c.zip)

#266 Updated by Marius Gligor about 7 years ago

Sorry, was my fault.

#267 Updated by Greg Shah about 7 years ago

It is good, thanks.

#268 Updated by Marius Gligor about 7 years ago

1. Today I did a code review over the JS code. The changes are attached.

2. Also I started to investigate the CTRL+C issue. The behaviour is done by the stop_disposition condition value get from registry:

<node class="integer" name="stop_disposition">
<node-attribute name="value" value="1"/>
</node>

Three values are possible:

0 - retry without logging off
1 - force log-off and then re-logon (default)
2 - force log-off

Values 0 and 2 works fine without any changes.

For value 1 however we have to do some changes. I think that we have two options.

A. Keep the temporary account unchanged and reuse the same credentials on re-logon. I tested and this works fine.

Advantages:
- Easy to implements.
- Very fast. On systems having big directory file the temporary accounts are changed only at server start-up.
No changes are made after start-up so no time is consumed to update the directory file whenever a temporary account is changed.
Disadvantages:
- Less secured. The temporary accounts are changed per server session, that is, only at server start-up.

B. Another solution is to use the existing RemoteObject to get a new account and do a re-logon. This scenario should be implemented.

In other words the first authentication is done using the environment variables values as currently is already implemented.
Than we have a valid session and a valid RemoteObject interface.
We could add a new method to this interface in order to provide a temporary account from server pool remotely.
On the client we can get a new account before the actual session is closed and use this accounts to do a re-logon.
Advantages:
- More secured.
Disadvantages:
- Require more changes to add this feature.

Is any time-out value set for RemotObjects sessions?

#269 Updated by Greg Shah about 7 years ago

B. Another solution is to use the existing RemoteObject to get a new account and do a re-logon. This scenario should be implemented.

How about a slight variation on this? We already return a boolean from the standardEntry() method. Let's just return back an object that contains both the boolean value and any re-logon/authentication info needed to re-establish the next session. That way, it minimizes changes.

Is any time-out value set for RemotObjects sessions?

The "original" temporary session was disconnected, so it no longer will work. But at the time standardEntry() returns, there is still a valid session and the data we would obtain via environment var can be just returned back using that existing session.

#270 Updated by Greg Shah about 7 years ago

Code Review 0127a

I am fine with the code. It looks very good.

My only request is for tty.keyboard.js, tty.keymap.js, tty.screen.js and tty.socket.js. Please put back (or add if not already there) blank lines that separate "member" variables. For example:

   /* exports object */
   var my = {};    
   /* track escape status */
   var escape = false;                     
   /* track alt char status */
   var altChar = false;

should be:

   /* exports object */
   var my = {};    

   /* track escape status */
   var escape = false;                     

   /* track alt char status */
   var altChar = false;

#271 Updated by Marius Gligor about 7 years ago

  • File mag_upd20140128a.zip added

Here is the implementation for CTRL-C issue and some remarks.

1. A Result object is used instead a boolean value.
2. The temporary account credentials are stored into Result object only in case of re-logon and a web client.
3. In case of a re-logon the temporary account is locked until the client will call again the main entry when the account is unlocked, that is, returned back to pool.
Also as a protection whenever a client call the main entry point the locked accounts list is checked
for so called "old accounts", accounts that are enabled from a specified amount of time (more than 10 seconds)
and not returned back to pool. This accounts are also unlocked. Please see my javadoc remarks.
4. WebSockets also define a time out value. If no information is sent over the web sockets for an amount of time the connection is closed.
This is an annotation value please see the ChuiWebSimulator class annotations.
I had set a value of 900000 which means 15 minutes.

#272 Updated by Greg Shah about 7 years ago

Code Review 0128a

1. In ChuiWebSimulator, I would like to shift to an idle timeout (maxIdleTime) that is set programmatically instead of via an annotation. Using the 15 minute as a default is fine. Please add a mechanism to override the default using configuration in the directory.

2. The Result class is missing from the update.

3. In StandardServer, I am not convinced that the use of a ConcurrentHashMap for lockedAccounts is sufficient to make this code thread safe. There is too much editing of the contents from multiple locations for this to be safe. Please use synchronization to make this safe.

4. The TemporaryAccount now has state that is edited in multiple places. That class needs to be properly synchronized.

5. The MainEntry javadoc should specify that the uuid parameter is optional (and thus it may be safely passed as null) unless this is a web client.

#273 Updated by Marius Gligor about 7 years ago

  • File deleted (mag_upd20140128a.zip)

#274 Updated by Marius Gligor about 7 years ago

I added the missing Result class.
The ConcurrentHashMap is fully thread safe by design and is internally synchronized.
The Java concurrent package is special designed to avoid the external synchronization.

#275 Updated by Greg Shah about 7 years ago

The ConcurrentHashMap is fully thread safe by design and is internally synchronized.
The Java concurrent package is special designed to avoid the external synchronization.

Yes, I'm aware of this. My point is that it is only sufficient for some use cases. Where you are potentially adding/removing/editing contents from multiuple locations on multiple threads, it is not always safe. In this case, the things that make me nervous:

1. lockAccount() and unlockAccounts() are not atomic.
2. There are multiple remove points for the map.
3. The uuid is provided by the client, if there is ever a scenario where the same uuid is accidentially (or maliciously) reused within a short interval of time, the results would probably break the logic since it would be possible to replace a TemporaryAccount instance that is already there and which is not closed. If that were to happen, that temporary account would never be disabled and it wouldn't timeout. It could be used over and over to access the server.

I think this is a case for proper synchronization.

#276 Updated by Marius Gligor about 7 years ago

  • File mag_upd20140128b.zip added

1. Done. ChuiWebSimulator.onConnect allow us to set the session idle time out which in turn override the default value.

2. Done.

3. Done. I understood what you said about this issue and is true. The lock and unlock account static methods are now synchronized and by consequence atomic.

4. Sorry, but I don't find any place in this class to use synchronization.
The open and close methods are synchronized in TemporaryAccountPool class.
Other places?? Could you help me please with a short hint?

5. Done.

#277 Updated by Marius Gligor about 7 years ago

  • File deleted (mag_upd20140128b.zip)

#278 Updated by Marius Gligor about 7 years ago

  • File deleted (mag_upd20140128b.zip)

#279 Updated by Marius Gligor about 7 years ago

Sorry, I changed the date of last changes on the ClientBuilderOptions class header.

#280 Updated by Greg Shah about 7 years ago

Sorry, but I don't find any place in this class to use synchronization.
The open and close methods are synchronized in TemporaryAccountPool class.
Other places?? Could you help me please with a short hint?

TemporaryAccount instances are created on one thread (in TemporaryAccountPool), updated on other threads (in UpdateAccountTask) and read on other threads (from StandardServer)...

This means it is unsafe to share the 3 members of the class (subject, password and timeout) without synchronization. Make sure that any access to those members are done from synchronized methods. It doesn't solve all access problems, but it ensures that the data being read/written is done atomically.

#281 Updated by Marius Gligor about 7 years ago

I did the changes. Normally no concurrency should occur on this class but just for safety is OK.

I have two other questions please:

1. We are not use Dojo at this stage. Should I provide the Dojo jar dependencies on lib folder or we will add this jar later?

2. I have a new build for jetty 9 the latest available. Should we update the jetty jar file or we have to wait until the final release.
NOTE: When I did the new build for jetty the name of the jar remains the same but is a little bit more big in size.

#282 Updated by Greg Shah about 7 years ago

Code Review 0128c

The code looks good.

but just for safety is OK.

Exactly.

1. We are not use Dojo at this stage. Should I provide the Dojo jar dependencies on lib folder or we will add this jar later?

Go ahead and add this now so that it is easy to use later.

I have a new build for jetty 9 the latest available. Should we update the jetty jar file or we have to wait until the final release.

Update it now.

#283 Updated by Marius Gligor about 7 years ago

Here is the package that I tested today. The package contains the new jetty 9.1 build.
Due to it size the Dojo zip file has been uploaded at the following location:

~/shared/projects/p2j/dojo/dojo-release-1.9.2.zip

The CTRL-C part of the regression tests FAILS and I have to retry this part.
The main part of the regression tests passed.

#284 Updated by Greg Shah about 7 years ago

Code Review 0129a

There are no concerns. Please note that I don't plan to check in the Dojo build, into bzr. I assume that it can be present or not without any errors (unless we try to use it and it isn't there). Is that correct?

Please continue getting this tested.

#285 Updated by Greg Shah about 7 years ago

For #2238: the following are some potential issues that we need to consider/research. Document your thoughts and findings here. If they are in fact real issues, then work to resolve them.

  • What happens when the user forces the browser to exit (without exiting the converted app first)? We would want to make sure that we get a notification when this happens, so that we can tell the server-side to exit.
  • Can we disable the browser's context menu?
  • How do we handle the use of the browser's FWD and BACK buttons (and history in general)? Can we disable those?

#286 Updated by Marius Gligor about 7 years ago

The Dojo dependencies zip file is always in the CLASSPATH because is in the list of dependencies inside the manifest file.
The build of p2j is not affected if the presence of Dojo zip file in lib directory.
Also at runtime as long as Dojo dependencies are not use no problems occur.

#287 Updated by Marius Gligor about 7 years ago

When the page having the JS code running is loaded a WebSocket connection is open.
If the user close the browser the WebSocket connection is closed and the server is notified.
On close notification the client exit and the process is terminated.
Also once the WebSocket connection is open we have an idle time out defined.
On time out also the client force an exit and the process is terminated.
However if from vary reasons the WebSocket is not open (the page is not loaded, WebSocket are not supported, JS disabled, others) the process could remains on running state for an indefinite period of time.
Maybe for safety it's a good idea to design and implements something like a "watchdog" timer used to
force an exit and terminate the client if no WebSocket connection is open during a specified amount of time.
What do you think about this?

#288 Updated by Greg Shah about 7 years ago

Maybe for safety it's a good idea to design and implements something like a "watchdog" timer used to force an exit and terminate the client if no WebSocket connection is open during a specified amount of time.

It is an excellent idea. Implement this, but don't include it in the current update. I want to commit the 0129a update as it is (assuming it passes testing).

#289 Updated by Greg Shah about 7 years ago

The Dojo dependencies zip file is always in the CLASSPATH because is in the list of dependencies inside the manifest file.

You mean it is implicitly in the CLASSPATH because of the manifest file? But there are no negative effects for the actual file being missing (unless we change our code to use Dojo).

#290 Updated by Marius Gligor about 7 years ago

To avoid any confusions maybe is better to eliminate the Dojo zip from CLASSPATH (from manifest file).
We could add this later when we effectively use the Dojo dependencies.

#291 Updated by Greg Shah about 7 years ago

To avoid any confusions maybe is better to eliminate the Dojo zip from CLASSPATH (from manifest file).

No, it is OK to leave it. I just wanted to confirm that my understanding was correct.

#292 Updated by Marius Gligor about 7 years ago

After three executions of CTRL-C part regression tests I obtained the same results (see attachments).
The main part of the regression tests passed.
The server.log file contains some strange messages related to persistence layer.
Example:

[01/30/2014 06:21:23 EST] (com.goldencode.p2j.persist.trigger.DatabaseTriggerManager:INFO) DatabaseTriggerManager registered with TransactionManager.
[01/30/2014 06:21:27 EST] (com.goldencode.p2j.persist.lock.InMemoryLockManager:WARNING) [00000016:0000005F:syman] --> local/majic/primary: cleaning up 1 leaked record lock(s) for exiting context ({temp_sum:T=SHARE [syman:00000016

#293 Updated by Marius Gligor about 7 years ago

Regarding browser session and JS code.
The web browser are designed to navigate over the Internet and for security reasons have a lot of restrictions.
Most of them are related to JS code which directly interact with browser environment.

1. The browser context menu can be controlled from JS on all major web browsers IE, Firefox and Chrome:

// disable browser context menu
document.oncontextmenu = function () {
return false;
};
Using this piece of JS code the context menu will be completed disabled.

2. The navigation buttons on the other hand cannot be controlled from JS to be hidden or disabled.
However it is possible to control the navigation from JS in many ways:

a). By using the following JS code on the page which is the target of the back navigation:
// prevent back navigation
function noBack() {
window.history.forward();
}
noBack();
window.onload = noBack;
window.onpageshow = function(evt) {
if (evt.persisted)
noBack();
}
window.onunload = function() {
void (0);
}
With this JS code in place when the user press the back button the back operation is executed
but the JS code forward the request back to the page. (from B page back to A page and than from A page forward to B page)
b). A better solution is to open our TTY JS on a new page (pop-up window).
This require an intermediate html page something like a "splash" window from which the pop-up window is created and open.
Here we have also 2 options:
- Use JS and automatically create the pop-up window when the "splash" page is loaded.
In this case the browser pop-up blocker blocks the pop-up window.
The user should confirm that the pop-up window could be open.
- In order to avoid the pop-up blocker action we can put a link inside the "splash" window and when the user click the link
open the pop-up window. In this case the pop-up blocker is inactive, no action is take.
The problem on both cases is that we cannot close the "splash" window after the pop-up window is open. It's works
on Chrome but IE ask for a confirmation and Firefox do nothings. Nevertheless we can keep both windows opened and hide the link on the "splash" window.
On the pop-up window we can disable or hide a lot of objects like menu, context menu, scroll bars, etc.
The only object that cannot be eliminated from the pop-up window is the address bar.
I have also to do some tests in order to find out if it's possible to use short-cuts keys in pop-up windows to navigate back.
It is possible also to combine pop-up windows with JS code to avoid back navigation.

Currently whenever the user try to navigate back or reload (refresh) the page the web socket connection is closed and the client process is terminated.
The user should go to the authentication page and restart a new session.
A good idea is not to terminate the process when the user refresh the page and close the socket connection but instead we should use the "watchdog" timer.

On web sockets close - start (arm) the watchdog timer.
On web socket open - stop the watchdog timer.

In other words the exit statement should be executed only when the converted application is ended (F4 for example)
or by the watchdog timer if the web socket connection is not open after a specified amount of time.

#294 Updated by Greg Shah about 7 years ago

Please check in and distribute the 0129a update. Make sure to post the bzr revision number here.

#295 Updated by Marius Gligor about 7 years ago

The changes inside the mag_upd20140129a.zip archive has been committed to bzr revision 10457

#296 Updated by Constantin Asofiei about 7 years ago

Marius, I think there is a problem with the temporary accounts. If I understood properly, the process of authentication works like this:
  1. user accesses the https://localhost:<port>/chui, where the username/password for an existing account on the machine is needed
  2. if these credentials are valid on the machine, it will run the command to spawn a new P2J Client.
  3. the P2J Client will use the credentials read from the P2J_SUBJECT and P2J_PASSWORD environment variables (this will point to the temporary account) to authenticate. After control gets back to P2J Client, it will compute the redirect URI and return it back to the server, to redirect the user to that page.
Where are the problems:
  1. in case there is a custom authentication hook, this will be bypassed.
  2. after step 3, the Reader, Writer and Conversation threads remain on the context for the temporary account. This doesn't look right to me. Even when using the testcases project, the Conversation thread is still on the context for the temporary account. The correct account should have been the bogus account, specified at the configuration for the com.goldencode.p2j.security.GuestAccess auth plugin.

What we are missing is the implementation for this part from note 74 from Greg (can't tell from the comments if this is still WIP or not):

A better solution is to implement a temporary P2J session with the server. The idea: create one-time authentication credentials (some kind of random token/key) that can be passed (like our password info) via STDIO to the child process. If we can find a secure way to get it to the P2J client, the P2J client can use this to do a temporary connectDirect() to the server with those credentials, read the key store, initialize the web server and send back the URI. Then it would disconnect and drop into the normal processing. The major advantage of this approach is that it can be extended to the "remote client" scenario, which we will be implementing later. Another major advantage is that we plan to do something similar to allow batch mode P2J clients to be autostarted by the server.

The change is in ClientCore.start - this should connect to the server twice:
  1. using the temporary credentials, to start the web server. This needs to use a copy of the original BootstrapConfig instance, to not lose any explicit user/passwd passed via access:subject:id and access:password:user configurations (or other configuration).
  2. if web server was started, connect again. This time, fallback using the normal processing: use auth plugin as necessary, let the user input credentials, etc.

Finally, when you get a chance, please document all the rights needed for the server account (i.e. standard account for the test server or gso for MAJIC). Currently, I had to remove the NetResource, AdminResource and DireectorResource security plugins to be able to populate the temporary account pool. The default security plugins should be:

       <node class="strings" name="resource-plugins">
         <node-attribute name="values" value="com.goldencode.p2j.security.SystemResource"/>
          <node-attribute name="values" value="com.goldencode.p2j.security.AdminResource"/>
          <node-attribute name="values" value="com.goldencode.p2j.net.NetResource"/>
          <node-attribute name="values" value="com.goldencode.p2j.directory.DirectoryResource"/>
      </node>

but the testcases project has only the SystmeResource plugin - this is why adding the server account to the system/admin ACL worked for you.

#297 Updated by Marius Gligor about 7 years ago

OK. I have to think to this issue.

At this moment what I understood is as follow:

1. Use the temporary account to connect to the server, import the server key, start server and redirect user.
2. Close the temporary account session.
3. Create a new session this time using the credentials provided by the user. In other words the ThinClient
should be initialized using this session and the MainEntry should be executed also using this session.

What it's not clear for me. In case of a custom authentication plug-in where the user should enter the credentials?
The plug-in is like a p4g program which provide a login form on the terminal, in this case in the web page?

#298 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

At this moment what I understood is as follow:
1. Use the temporary account to connect to the server, import the server key, start server and redirect user.
2. Close the temporary account session.
3. Create a new session this time using the credentials provided by the user. In other words the ThinClient
should be initialized using this session and the MainEntry should be executed also using this session.

You understood correct.

What it's not clear for me. In case of a custom authentication plug-in where the user should enter the credentials?
The plug-in is like a p4g program which provide a login form on the terminal, in this case in the web page?

Yes, in the web page; in MAJIC case, the first thing the user will see will be the MAJIC login screen. If the system credentials are the same as the P2J credentials (when user/pass is used), we could reuse that, but this would require changes in the authentication code (and the custom auth plugin probably).

#299 Updated by Marius Gligor about 7 years ago

OK. In this case I have to do some changes on the ClientCore and potentially on other places.
I think that using the temporary accounts implementation is OK to authenticate for server key import and user redirection.
I have to use a copy of the original bootstrap configuration file for this purpose.
On CTRL-C in this case we don't need a temporary account credentials, I think that the Majic login plug-in should occur on the client web page.
Is this correct? If yes we have to revert also the changes form StandardServer.

#300 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

I think that using the temporary accounts implementation is OK to authenticate for server key import and user redirection.

Correct, this is their purpose.

I have to use a copy of the original bootstrap configuration file for this purpose.

Correct.

On CTRL-C in this case we don't need a temporary account credentials, I think that the Majic login plug-in should occur on the client web page.

Correct, the custom login plug-in is invoked and must be shown on the client web page.

Is this correct?

CTRL-C in MAJIC doesn't unwind all the way back to the login screen - is caught and handled in the business logic, to keep the user in the main menu. In normal cases, when CTRL-C is not caught, it will need to act as the client is restarted - so yes, this will unwind all the way to the while (running) loop in ClientCore.start.

If yes we have to revert also the changes form StandardServer.

If the changes are no longer needed, they should be removed.

#301 Updated by Marius Gligor about 7 years ago

OK Constatin thanks. Tomorrow I will try to fix this issue.
Are you did tests with the web client? How it's works?

#302 Updated by Greg Shah about 7 years ago

1. The browser context menu can be controlled from JS on all major web browsers IE, Firefox and Chrome:

Go ahead and implement this.

2. The navigation buttons on the other hand cannot be controlled from JS to be hidden or disabled.

For now, I want to avoid the problems that come with a popup window.

We could put an implementation of 2a (JS approach) in ui/client/chui/driver/web/chui_web.html. This is the previous page in our case, right?

Another approach is to edit/control the browser's history. If the previous entry in the history is the current page, this may be a solution to the back/forward problem. Please see:

https://developer.mozilla.org/en-US/docs/Web/Guide/API/DOM/Manipulating_the_browser_history

I have also to do some tests in order to find out if it's possible to use short-cuts keys in pop-up windows to navigate back.

Please do let me know the results of your tests here.

Currently whenever the user try to navigate back or reload (refresh) the page the web socket connection is closed and the client process is terminated.
The user should go to the authentication page and restart a new session.
A good idea is not to terminate the process when the user refresh the page and close the socket connection but instead we should use the "watchdog" timer.

Hmmm. This is a good find on the refresh issue. I had not thought of that.

Some questions:

1. Does the use of the 2a JS "solution" still cause the back to sever the web socket connection?
2. Can the browser history manipulation help here?
3. Can we get a notification on refresh? If so, can we cancel it?
4. Can we make the websocket survive by using a separate worker thread (and separate window object) as documented in note 184 point number 4?

#303 Updated by Marius Gligor about 7 years ago

I fixed the issue related to server key store import. It was a misunderstanding related to implementation.
For a web client first a temporary session is established using a temporary account.
Inside the main loop of ClientCore.start user credentials are used.
As a result I reverted the changes from MainEntry, StandardServer and TemporaryAccount classes.
Also the Result class is no longer used and has been deleted.
This design gracefully simplify the ClientCore.start code.
Basically the code looks (main loop of ClientCore.start) like before introducing web clients.
Also I added some JS code inside the web pages which prevents the back navigation and disable the browser context menu.
Nevertheless we have to do more works on the JS front-end depending on the final solution we will choose.

#304 Updated by Constantin Asofiei about 7 years ago

Marius, 0131a.zip solves the login problem (I'll let Greg do a full review), but there are some other issues in MAJIC (I'm using firefox 26.0 for ubuntu canonical 1):
  1. in the main MAJIC menu, when moving around, the selected option is "highlighted", via the put screen color message statement. This does not work in web mode.
  2. the cursor is not blinking in MAJIC - don't know if this is a real issue or not.
  3. I don't think we should allow the input cursor position to be changed at mouse click. IIRC, the position of the cursor is changed during chui drawing and it should be out of user reach.
  4. there are artefacts remaining after the screen is cleared (is like some portions of the chars are left behind...); see for the "breadcrumbs".
  5. you should to redirect the standard error and output streams for the spawned process to a file somewhere - else, it will be difficult to debug through configuration or other kind of problems, on the client side.
  6. not sure why, but process launching does not work while in web mode. Use a code like this to test (a.txt is a file with a single line of text):
    def var ch as char.
    input thru value("cat a.txt") no-echo.
    import unformatted ch.
    input close.
    message ch.
    
  7. in firefox, when redirecting to the custom url, I always get a This Connection is Untrusted screen first (to accept the certificate).

#305 Updated by Marius Gligor about 7 years ago

1. Put screen color message statement isn't implemented on the web client. I have to check how to do that. Could you please attach a picture for this issue?
2. The cursor could be "solid" or "line" blinking and not blinking, visible or hidden according to the cursor settings (see the JSON used for initialization inside index.html).
3. Greg told me to keep the mouse events but it is not a problem to cut off.
4. Rendering is another issue that is browser and platform dependent. In this case we have to do some small changes in the drawing code to fix the problem.
5. I think that we can use the client log file. Maybe a good idea is to spawn the client and activate the log at this time.
6. I have no idea about this issue, maybe you can offer me more details like where is the file located, how is loaded etc.
I'm working only for 3 moths on this complex system and so far I have no enough knowledges about all aspects regarding the system capabilities.
7. This message cannot be eliminated as long as we use self signed certificates.

#306 Updated by Marius Gligor about 7 years ago

A better way to prevent back navigation by editing history.

#307 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

1. Put screen color message statement isn't implemented on the web client. I have to check how to do that. Could you please attach a picture for this issue?

See the attached .
You can convert the menu.p program from the testcases project and run it with the ask.p.

2. The cursor could be "solid" or "line" blinking and not blinking, visible or hidden according to the cursor settings (see the JSON used for initialization inside index.html).

You can run menu.p (via the pro -p menu.p command) on lindev01 and see how it works. In 4GL, the cursor is solid and not blinking.

3. Greg told me to keep the mouse events but it is not a problem to cut off.

OK, I understand now. I was worried that the drawing code will read the cursor position after/during drawing, and it will read this modified position (thus breaking drawing). But I don't think this is the case.

4. Rendering is another issue that is browser and platform dependent. In this case we have to do some small changes in the drawing code to fix the problem.

OK

5. I think that we can use the client log file. Maybe a good idea is to spawn the client and activate the log at this time.

Maybe change spawn.c to accept a file name where to redirect the stdout and stderr? Otherwise, another option is to redirect them via ProcessBuilder.redirectError and redirectOutput.

6. I have no idea about this issue, maybe you can offer me more details like where is the file located, how is loaded etc.

Run the io/process_streamt_test.p program via ask.p - this will do a basic process test. This test will create a file named some_output_file.txt with this content:

txt
--------
This is

7. This message cannot be eliminated as long as we use self signed certificates.

OK, I understand.

Greg, some notes:
- the untrusted connection issue will be annoying for "live" usage; for testing is OK, but having to go through this screen each time during login in a production environment, I don't think is OK.
- about the web socket timeout: IIRC, in MAJIC there are some end of month reports which take a lot of time (much more than 15-30 minutes). The timeout is OK in a "away from keyboard" scenario, but it will be problematic in cases of long-running server-side tasks or reports.

#308 Updated by Marius Gligor about 7 years ago

1. The redirection to a log file is not possible from ProcessBuilder because we redirect the stderr for spawner not for web client process.
Form spawner is also difficult and we have to found solutions for Linux and Windows.
However I found a better way which works like magic!
- I did the redirection from ClientDriver.main function the entry point of the client process. If the process is a web client
(if found the following string client:driver:type=chui_web in args list) than I constructed a file name like client_$user_$time.log
and I redirected the stderr to this file. The log file is created into client working directory.

2. The socket timeout is configurable. Nevertheless we have to improve this part (see watchdog proposals)

#309 Updated by Marius Gligor about 7 years ago

I added the ClientDriver log file redirection implementation.

#310 Updated by Marius Gligor about 7 years ago

1. The user Refresh action could not be detected. Is the same like closing the browser.
When the user reload the page the web socket is closed and the client process exit.
In order to avoid this behaviour I designed a watchdog timer thread.

When the process (chui simulator) is started also the watchdog timer thread is started.
When a connection is made over web sockets the watchdog is killed.
When the web socket connection is closed the watchdog timer thread is restarted.
In respect with this the whatchdog looks like an idle task.
The whatchdog timer thread is responsible to terminate the process when timeout expire.
Currently I'm working to implement the whatchdog timer.
Because now the whatchdog timer is responsible with process termination I think that the timeout stored
on registry could be used by whatchdog and the socket timeout is no longer a critical issue.
With the whatchdog in place the user could connect and reconnect to server at any moment.
Now the Refresh (reloading the page) will work properly.

2. On a short TODO list I have to fix also:

- send also text colour to web chui simulator.
- fix the drawing issue reported by Constantin.
- other issues reported during tests.

#311 Updated by Constantin Asofiei about 7 years ago

Marius, about 0131c.zip:
  • please make an update without the JS changes in the .html files, and get it conversion test it. I need this update, as others can start these kind of clients using the spawner: appserver agents and batch processes. Also, I'm refactoring some of the spawner-related infrastructure, to make it unaware of who starts the P2J client.
  • you removed the synchronization from TemporaryAccount - this is still needed.

#312 Updated by Marius Gligor about 7 years ago

Here are the package that I used today on the regression tests.
Passed the regression tests.

#313 Updated by Constantin Asofiei about 7 years ago

The changes look good to me. If Greg has no objections, you can release them.

#314 Updated by Greg Shah about 7 years ago

In regard to 0203a, I have some minor concerns with the ClientDriver changes, but please go ahead and commit/distribute the update anyway.

In your next update, address the following in ClientDriver:

1. Setting the driver can be done from both the bootstrap config file as well as from the command line. I realize that we currently drive this from the spawner and it will (for now) always come from the command line. But this creates a hidden dependency that may "bite us" later. I prefer to process off the bootstrap config after the command line overrides are merged in. Hopefully that is not too late for the redirection to be useful.

2. We should provide a mechanism to override the log file name (and optionally the pathing too) rather than only allow a fixed log name. Perhaps we should calculate the logfile on the server and set it into the command line args/bootstrap config. That would allow us to read overrides from the directory if present.

#315 Updated by Marius Gligor about 7 years ago

During the weekend I tried to do some tests on Windows OS and I found two important issues:

1. The File.renameTo(File target) does not work properly in Windows. If target file already exists the method fails.
Tried to rename the directory.xml.tmp to directory.xml fails because directory.xml already exists.
Seems to be a known issue since I found a TODO remarks inside the class.
I fixed this issue inside the class XmlRemapper using the JDK 7 Files.move call.
Now it's works fine on both Linux and Windows OS.

2. I cannot build the native artefacts due to the new libffi dependencies. I tried to fix this issue but unfortunately with no success.
I cloned the libffi project from git hub repository and I tried to build the libffi.
The build seems to be OK but I cannot use the resulted lib file for linking in Windows OS.
I'm not sure if building under Linux for x86 could be used also in Windows.
I didn't spent much time with this issue.
Is anybody on the team that could help us with this issue?
The problem is for Windows OS only.

Today I started to fix the terminal colour issue. Seems to be not a colour issue but a font colour attributes issue.
So far I did some changes inside the code to implements text colour attributes and the results are good (see the attached picture).
I adjusted also some rendering parameters in order to fix the drawing issue. I have to do some tests with my new changes.
Unfortunately another issue occur, this time at spawner level.
Bellow are Constantin remarks:

The problem is because a process can not be launched (it needs to run "nice mail -e 2> /dev/null ; echo $?" using 4GL's INPUT THROUGH statement).
You can ignore this error while adding support for the highlight feature.
But please investigate why, for a P2J client started using the spawner, processes can not be started.
See the pseudoTerminalLaunch function from src/naive/process_linux.c. Process launching fails at this test:

if (!isatty(STDIN_FILENO))
{
return -2;
}

I'll try to investigate also this new issue.

#316 Updated by Greg Shah about 7 years ago

In regard to the libffi issue, please see note 74 in #1634.

#317 Updated by Greg Shah about 7 years ago

Code Review 0203b

I am fine with this change.

#318 Updated by Greg Shah about 7 years ago

- the untrusted connection issue will be annoying for "live" usage; for testing is OK, but having to go through this screen each time during login in a production environment, I don't think is OK.

Self-signed certificates must be allowed and would be the default. But any customer can do the following to eliminate the problem:

1. Implement a certificate that is signed by a root authority that is already recognized by the browsers in use. This is commonly done for HTTPS servers and it will work for our server too. It is just a matter of properly importing the cert into our directory.

2. Add the root authority used for self-signing to the browser's configuration. This is possible if all the browsers are deployed under the control of that customer.

Anyway, do you see something else we need to implement to make this better? As far as I know, there is nothing else to do from our perspective. Our current approach just reads the server's keystore over a RemoteObject connection and re-uses that on the embedded web server in the client. Basically, any certificate configured to secure the P2J server will be used automatically by all the web clients.

#319 Updated by Marius Gligor about 7 years ago

Here is the implementation of text attributes (color)
I found the following attributes defined (see ColorAttribute).

NORMAL(0x00), 
BLINK(0x01),
BOLD(0x02),
UNDERLINE(0x04),
REVERSE(0x08);

I implemented all this attributes except BLINK.
Theoretical is possible to implement the BLINK in JS but I'm afraid the performances will be affected and the feature not usable.
For performances reasons I used a binary protocol to send attributes down to client.

I did also some small changes on the drawing code.
Unfortunately this part depends on browser and platform on which the JS code runs.
According to my tests it works fine on Firefox/Linux having a font size between 16 and 20.

I designed and implemented also a Watchdog Timer to guard the web client life.
I moved to JS code used to avoid back navigation and access the browser menu context inside tty.js file.

#320 Updated by Greg Shah about 7 years ago

Code Review 0204a

Overall, this is very good.

1. In ChuiWebSimulator, please synchonize any access or assignment to the watchdog data member since there is no guarantee that it won't be set and accessed from different threads.

2. In the WatchdogTimer.run(), please check the running flag after exiting the sleep() since it is possible that the user could have connected and a kill() was issued, dropping the thread out of the sleep early. But if the timer had elapsed, the system will exit, even though the user may think they have come back in. In other words, let the user connection take precedence over the timer.

3. As Constantin noted above, it is a common case for these clients to be left running for a very long time (e.g. during lunch). We may need to provide an option to completely disable the watchdog timer. Constantin, what do you think?

4. Our NCURSES implementation has recently be updated to support real colors (see ColorPalette, Color and ConsolePrimitives classes). Our Swing and Web ChUI clients need to be similarly enhanced.

#321 Updated by Constantin Asofiei about 7 years ago

Greg Shah wrote:

3. As Constantin noted above, it is a common case for these clients to be left running for a very long time (e.g. during lunch). We may need to provide an option to completely disable the watchdog timer. Constantin, what do you think?

If I understand correctly, this watchdog timer and web socket timeout is needed to be able to close the P2J client when the user closes the browser, right? If this is the case, why don't we add a "heartbeat" in JS code? As long as this heartbeat is received by the web server, we can assume that the user is still using the application (and the watchdog can can listen only for this heartbeat).

#322 Updated by Marius Gligor about 7 years ago

Indeed. The watchdog timer has been designed to terminate the process when the user close the browser.
Even more when the client use the reload button to reload the page the web socket is closed and than reopen after page is reloaded.
During the page reload the application should remain active. The watchdog guardian help us to do that istead to terminate the application like on the previous implementation.
If the page is not reloaded due to an unexpected reason the guardian is active and the process will finally terminate after an amount of time.

#323 Updated by Marius Gligor about 7 years ago

Here are the latest changes. I fixed also the issue related to isatty() when using spawner.
It was a bug on my spawner code. Now it works properly.

#324 Updated by Greg Shah about 7 years ago

Code Review 0205a

1. The WAIT_TO_DIE javadoc is inaccurate.

2. Please rename the sartWatchdog() to startWatchdog().

3. What happens when someone executes something in the converted app that causes the app to exit (e.g. execute a QUIT statement)? What does the user see in this case? I guess it should probably redirect back to the original login web page.

#325 Updated by Marius Gligor about 7 years ago

  • File mag_upd20140206a.zip added

1. The WAIT_TO_DIE javadoc is inaccurate.
Done.

2. Please rename the sartWatchdog() to startWatchdog().
Done.

3. What happens when someone executes something in the converted app that causes the app to exit (e.g. execute a QUIT statement)?
What does the user see in this case? I guess it should probably redirect back to the original login web page.

Good questions? I designed and implemented a flow to redirect the user to the login page in case of converted application exit.
a) On the server side:
I added a new command line parameter web:referrer:url which store the referrer http header value, that is,
the URL address at which the client should be redirected. This command line argument is used by the client builder when a new client is spawn.
b) On the client side:
On the main loop of ClientCore.start when the converted application exit (running = false) for web clients I added a method quit() on screen driver
which further will notify the JS code running on the web browser via ChuiWebSimulator.quit and a new web sockets command.
This command provide the referrer address to the remote JS code. When the command arrive to the JS code in browser the client is redirected to the referrer address.
As a protection I added also a Cache-Control field into HTTP headers when the page containing the JS code is loaded (index.html).
This instruct the browser to not cache or store the page.
When the user try to go back to this page after the redirection is done the browser will inform the client about the missing page.
The client must follow the normal way and do another authentication in order to start a new session.

#326 Updated by Greg Shah about 7 years ago

Were the changes to tty.screen.js and tty.socket.js deliberately removed from 0206a?

#327 Updated by Marius Gligor about 7 years ago

  • File deleted (mag_upd20140206a.zip)

#328 Updated by Marius Gligor about 7 years ago

Sorry, it was my mistake. I have a short script to build the archive and I had a small problem inside the script.
Please take the new archive.

#329 Updated by Greg Shah about 7 years ago

Code Review 0206a

It looks really good. Other than adding real color support, is there anything else left to do?

The real color support can be done later.

#330 Updated by Marius Gligor about 7 years ago

From my point of view I think no.

#331 Updated by Greg Shah about 7 years ago

OK. Get this regression tested.

#332 Updated by Greg Shah about 7 years ago

Hynek recently added proper color support for our NCURSES/WIN32 Console based ChUI client (the ConsoleDriver which is backed by the libp2j.so/p2j.dll).

Please extend this same level of support to both the Swing ChUI and AJAX ChUI clients.

#333 Updated by Constantin Asofiei about 7 years ago

Marius, a note about 0129a.zip: the archive has a tty.keymap.js file which is not in bzr rev 10457. Please check 0129a.zip against 10457 revision, to be sure nothing was lost. You can include any missing files with the 0206a.zip update.

#334 Updated by Marius Gligor about 7 years ago

I sow now that is missing. It wasn't add to bzr. I will include this file also on the next commit.
Thanks.

#335 Updated by Marius Gligor about 7 years ago

  • File regression_tests_reports_01.zip added
  • File regression_tests_reports_02.zip added

Here are the results of my regression tests.
For the first time the CTRL-C test passed.
The main test part has only one failure but is a negative one.

#336 Updated by Greg Shah about 7 years ago

The tc_job_002 is expected to fail at that point, so this testing passes.

Please commit the update and distribute it.

#337 Updated by Marius Gligor about 7 years ago

May I delete the attached files containing the test results?

#338 Updated by Greg Shah about 7 years ago

Yes. You just need to post your assessment of the regression testing into Redmine, not the results themselves. At this point, I think you have enough experience with the testing process to assess things yourself.

#339 Updated by Marius Gligor about 7 years ago

  • File deleted (regression_tests_reports_01.zip)

#340 Updated by Marius Gligor about 7 years ago

  • File deleted (regression_tests_reports_02.zip)

#341 Updated by Marius Gligor about 7 years ago

Update [mag_upd20140206a.zip]
Passed regression tests, committed to bzr revision 10459.

#342 Updated by Marius Gligor about 7 years ago

Today I started to work on issue #2240 regarding color implementation on swing and web chui clients.
I studied the implementation for NCURSES and I reread also the p4g manual regarding colors in character terminals.
What I understood so far is that the color field in Color class holds a pair value foreground/background color.
The integer values for FG and BG colors are indexes into a color palette structure.
Currently the ColorPalette has only 8 entries. The p4g states the we could have max 127 colors (entries).

1. It is possible in the future to have more entries inside ColorPalette?
It is important to know in order to optimize the transfer of colors wia web sockets.
Up to 16 entries we could use 4 bits/color and 1 byte for FG/BG pairs.
If we have more than 16 entries we should use 1 byte/color and 2 bytes for FG/BG pairs.

2. The files color-map.dtd and color-map.xml are the substitute of protermcap file?

3. Where is the color-map.xml? (see the attached client log)
Without this file the native client seems to not work with colors.

4. The color-map.dtd define also 8 colors for FG and BG.
The same question regarding the number of colors that could be defined.

#343 Updated by Greg Shah about 7 years ago

First, I should have previously mentioned that details about the most recent color work can be found in #1801.

1. It is possible in the future to have more entries inside ColorPalette?

This enum does define the well-known terminal colors. So we won't be directly adding colors in the ColorPalette enum, but custom colors will be possible in the terminal.

2. The files color-map.dtd and color-map.xml are the substitute of protermcap file?

The color-map.xml is an optional file that is read from the converted application's jar file and which defines custom colors and color name/index mappings that can be referenced from the converted code.

The DTD is just to define/validate the proper structure of the color-map.xml.

3. Where is the color-map.xml? (see the attached client log)

See #1801 for an example. This is application specific, so by default there won't be one in P2J.

The same question regarding the number of colors that could be defined.

We have to assume that we must provide support for the maximum number of colors that are supported by the 4GL.

But the details of a given color definition are different from the color index that will be passed. In particular, we can split the implementation in 2 parts:

1. Custom color definitions get registered in the driver. This effectively would tell the driver how to render a new color AND what color index is associated with it. This would be done one time, when the client is initialized.

2. For a given character on the screen we just need to know the index data of the color being used. This is the data actually sent at drawing time for each character. This is the data to optimize.

Please review #1801.

#344 Updated by Constantin Asofiei about 7 years ago

Marius, I'm merging your 0206a.zip update with mine and mainPage and indexPage handlers in ChuiWebDriver.startupServer don't look right: shouldn't mainPage target /, instead of /index.html? Now they look like they target both the same page...

#345 Updated by Hynek Cihlar about 7 years ago

Hynek recently added proper color support for our NCURSES/WIN32 Console based ChUI client (the ConsoleDriver which is backed by the libp2j.so/p2j.dll).

Note that color support in WIN32 part of the native terminal driver is not implemented yet.

#346 Updated by Hynek Cihlar about 7 years ago

I am not sure how the custom colors will be defined for the Swing and Web chuis, but at the moment color-map.xml may be too much terminal-oriented. For example, for bright-red color in xterm terminal, you have to combine the (simple) Red color and the Blink attribute.

#347 Updated by Marius Gligor about 7 years ago

We have 2 jetty handlers for two targets ./ and ./index.html which load the same page:

1. When the user is first redirected at https://server:port/ the target is /. which load the page index.html
On browser address bar the URL is https://server:port/index.html after redirection.

2. Once redirected if the user do a reload (refresh) of page having the address https://server:port/index.html
the target is now /index.html which in turn will load the same page index.html

#348 Updated by Greg Shah about 7 years ago

Please add some javadoc to describe these important points.

#349 Updated by Marius Gligor about 7 years ago

Today I did a lot of investigations related to COLOR implementation on CHUI Swing and WEB clients.
First I attached some important paragraphs extracted from the P4G manuals (see color.txt).
According to the P4G manuals:

1. COLOR phrase is mentained for compatibility with the old versions.
The COLOR phrase was superseded by DCOLOR and PFCOLOR on character interfaces and FGCOLOR and BGCOLOR for graphical interfaces.
Actually in P2J only DCOLOR, PFCOLOR and partial old COLOR (NORMAL | INPUT | MESSAGES, protermcap-attribute (color paires like BLACK/WHITE), dos-hex-attribute are supported.

2. According to the P4G manuals the DEFAULT_COLOR_TABLE has 16 colors (0...15) which are reserved, that is, SHOULD not be changed.
However the user could define custom colors starting with index 16. No limits are specified but 256 entries in the table might be a good limit.

3. In character interface FGCOLOR and BGCOLOR are undefined. The DCOLOR and PFCOLOR are indexes in a COLOR PAIR table.
The entries in COLOR_PAIR table holds foreground/background color pairs.
The colors MUST be defined, that is, should be defined inside the COLOR TABLE (DEFAULT_COLOR_TABLE + custom colors).
The first 5 (0 to 4) entries in the COLOR_PAIR table are reserved. The user could define custom pairs values starting with index 5 to 127.
In total the COLOR_PAIR table could holds maximum 127 entries.

4. The actual implementation for chui native driver introduces some limitations.
- It define only 8 default colors.
- Custom colors aren't supported.

5. Some Java code is common for all terminal implementations and I think that we have to find a solution
to have the same code on all drivers. For example the ColorSpec and Color classes.

In conclusion, in my opinion before to implements the screen rendering we have to design and implements the P4G color mapping parts according to P4G specifications.
Than we could easily convert the P4G color indexes into rendering colors (Swing and Web colors) and implements also the rendering part.
In order to do that we have to decide where to store the CUSTOM COLORS defined by user and the PROTERMCAP file to holds the custom COLOR_PAIR custom entries.
Could we leverage the existing color_map.xml file to be used for our purposes?

#350 Updated by Marius Gligor about 7 years ago

Based on the current COLOR implementation for ChUI native driver I implemented the COLOR for ChUI Swing driver too. (see attached picture)

1. First I converted the chui_colors.p from testcases/uast.
Because I got an error when chui_colors.p was converted I changed the first line:
define var ed as character view-as editor size 20 by 2 label "Editor" pfcolor 3.
to:
define var ed as character view-as editor size 20 by 2 label "Editor".

Seems that pfcolor 3 cannot be converted or not supported.

2. I used the color-map.xml file posted by Hynek and I implemented the COLOR for Swing driver.
First I defined colors for terminal type "vt100" because the vt100 is defined as monochrome (BLACK/WHITE) inside color-map.xml.
By the way from where it is possible to set the terminal type?
Also I improved the Swing drawing using the Double Buffered option for JComponent.
The current changed are attached together with some picture snapshots.

3. Now I'm working to implement the COLOR for ChUI Web driver.

#351 Updated by Hynek Cihlar about 7 years ago

Seems that pfcolor 3 cannot be converted or not supported.

Looks like PFCOLOR is parsed ok and a valid AST is built. Probably a bug during emitting the Java code. I'll create an issue for this. Also I committed a fixed chui_colors.p so it compiles fine.

By the way from where it is possible to set the terminal type?

Terminal type is retrieved from the actual screen driver (see ScreenDriver.terminalType). For the linux driver for example, the terminal type is ultimately fetched from the TERM env variable.

Progress also contains the TERMINAL statement for changing the terminal type. However I am not sure whether this is currently supported and I didn't scope this in when implementing #2037.

#352 Updated by Greg Shah about 7 years ago

2. According to the P4G manuals the DEFAULT_COLOR_TABLE has 16 colors (0...15) which are reserved, that is, SHOULD not be changed. However the user could define custom colors starting with index 16. No limits are specified but 256 entries in the table might be a good limit.

We have to assume that some customers will actually modify these first 16 entries, so don't exclude that case.

in my opinion before to implements the screen rendering we have to design and implements the P4G color mapping parts according to P4G specifications.

Agreed. I prefer to maximize common code and only have differences at the driver level (if possible).

In order to do that we have to decide where to store the CUSTOM COLORS defined by user and the PROTERMCAP file to holds the custom COLOR_PAIR custom entries.
Could we leverage the existing color_map.xml file to be used for our purposes?

The color-map.xml is intended to be the P2J replacement for the PROTERMCAP. It will also store the custom colors defined on Windows (which the 4GL stores in the registry).

We will have a single, simplified approach and only use this color-map.xml file.

#353 Updated by Greg Shah about 7 years ago

Code Review 0211a

1. In ColorPalette, let's use RGB values instead of the AWT Color instances. This is more universal and can be easily translated to both Swing and Web clients.

2. In the ChuiSimulator, I would like to avoid the color/font processing and drawing for each character. Please consider ways to batch up use of the same font (including the same foreground/background color) for multiple characters.

#354 Updated by Marius Gligor about 7 years ago

1. AWT Color instances are build using now RGB values in ColorPalette.

2. The code used to set the Font color attributes was moved to the right place ColorMapper class.
The ChuiSimulator code has been reverted to it original optimized code. I added only the double buffered option.

3. Implemented COLOR in ChUI Web client. The color palette is send down to JS client only when web sockets connection is open.
Apart the changes to implements COLOR I did also a lot of major changes inside JS code (tty.screen.js) in order to manage the screen cursor.
In the current implementation we have only 8 colors defined so for performances reasons the color pairs FG/BG are assembled and stored
in only 1 byte, MSB 4 bits for FG and LSB 4 bits for BG color numbers. This encode scheme allow mapping of up to 16 values in the color palette.
For character interfaces a limit of 16 colors is a reasonable value.
If you consider to have more colors let say 1 byte for FG and another byte for BG which allows to map up to 256 colors please let me know.

#355 Updated by Greg Shah about 7 years ago

Code Review 0212a

1. Please note that our maximum line length is 98 characters. The docs haven't been updated. You don't need to break down the javadoc lines to fit within 78 chars.

2. Please rename ChuiWebSimulator.builPaletteMessage() to ChuiWebSimulator.buildPaletteMessage().

3. ChuiWebSimulator.clear(), ChuiWebSimulator.setCursor() and ChuiWebSimulator.setCursorStatus() are missing javadoc.

4. You mention that the color palette is sent to the client at websocket open. But actually, it is sent down every time we refresh the screen (i.e. sync()). That seems too often. I don't think Progress has a feature to dynamically modify the color palette. If I'm correct, it only needs to be sent once.

so for performances reasons the color pairs FG/BG are assembled and stored
in only 1 byte, MSB 4 bits for FG and LSB 4 bits for BG color numbers. This encode scheme allow mapping of up to 16 values in the color palette.
For character interfaces a limit of 16 colors is a reasonable value.

This is fine for now. We can change it later if requirements present themselves.

#356 Updated by Marius Gligor about 7 years ago

1. Regarding sync() method. This is a private method called from the onConnect(session) method, when a new session is open.
The onConnect() is called when the main web page is loaded by web browsers.
This could happened on two cases:

- When the user is first redirected to the main page URL. In this case is MANDATORY to call sync()
in order the synchronize the virtual tty with JS tty implementation.
- A second scenario is when the user force the page to be reloaded by pressing the browser Refresh button.
Unfortunately we cannot disable this button from JS. The action of this button is equivalent with the first scenario and the sync() call is also MANDATORY.
Whenever the main page is reloaded as I described the JS code embedded on the page is reinitialized.
That's why we MUST send down color palette because after the page is reloaded the value for this JS variable became "undefined".
Basically the color palette is loaded only on this two cases. During the normal operation no color palette is send down to JS.

2. Today I designed and implemented a cached mechanism for color and attributes messges.
The last send color or attribute messages are cached and always compared with the current message.
If no changes are found on the current message the message is not send.
This is a significant improvement of performances especial on monochrome applications where the
color message is basically unchanged during the execution. The color message is send only once
except the case of a refresh like I described on the previous paragraph.

3. I added log messages on WatchdogTimer and I did some tests. According to my tests the watchdog
timer works properly (see attached log file). I tested on two cases:
- when the browser is closed.
- when the tab page of the browser is closed but the browser remains active.

#357 Updated by Greg Shah about 7 years ago

Code Review 0213a

I am fine with the code, except for 1 change. I should have looked more carefully at the sync() method. The problem is that we have a well known sync() method that is part of the driver (actually, the OutputPrimitives instance contained in the driver). That method has a very specific meaning/purpose and when I read the name sync(), I made an assumption that it was the same method. Please change the name of the method to something more descriptive of its purpose so that in the future people won't make the same silly mistake I made.

Unless you have any other remaining changes on this, you can go ahead with regression testing the updated version.

#358 Updated by Marius Gligor about 7 years ago

OK. I have an idea to cache also the text messages. I'm going to implements this feature.

#359 Updated by Marius Gligor about 7 years ago

Renamed sync() method to initRemoteTerminal()
I implemented also the screen characters message caching.

#360 Updated by Greg Shah about 7 years ago

Code Review 0213b

The changes look good. Please get it runtime regression tested.

#361 Updated by Constantin Asofiei about 7 years ago

Marius, I've managed to get a NPE in TemporaryAccountPoolWorker:

Exception in thread "Thread-46" java.lang.NullPointerException
        at com.goldencode.p2j.main.TemporaryAccountWorker.run(TemporaryAccountWorker.java:119)
        at com.goldencode.p2j.security.AssociatedThread.run(AssociatedThread.java:127)

Can't duplicate yet... this was on devsrv01, not in debug mode.

#362 Updated by Marius Gligor about 7 years ago

  • File netbeans-7.4.desktop added

It's a bug. In TemporaryAccountWorker line 119 I omitted to check the callerLatch for null value.
The correct code should looks like:

if (callerLatch != null)
{
   callerLatch.countDown();
}

This could happen when the TemporaryAccountWorker thread is terminated and the execute() method was never call.
The callerLatch remains at initial value which is null.
Looking inside the code this happen when the server is start-up and later shut-down without using any temporary account from pool, no process was spawned during the server session.
Please fix this issue.

#363 Updated by Marius Gligor about 7 years ago

TemporaryAccountWorker thread is used to enable/disable temporary accounts.
Since this operation require a specific security context I used an AssociatedThread instance created earlier.
The access to TemporaryAccountWorker is serialized because we have a single instance.
TemporaryAccountWorker like AssociatedThread accept a Runnable object as task to be executed.
I used two CountDownLatch instances to synchronize between execute() and run() methods.

#364 Updated by Marius Gligor about 7 years ago

I just manage to reproduce the bug. I started than I stopped the server without any other action and the bug occurs.

[02/14/2014 13:40:43 EET] (SecurityManager:FINER) {00000000:00000001:standard} Search done: handle 65, decision true, cache false
Exception in thread "Thread-6" java.lang.NullPointerException
    at com.goldencode.p2j.main.TemporaryAccountWorker.run(TemporaryAccountWorker.java:119)
    at com.goldencode.p2j.security.AssociatedThread.run(AssociatedThread.java:127)

#365 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

I just manage to reproduce the bug. I started than I stopped the server without any other action and the bug occurs.

[...]

OK, I'll include the fix with my #2124 update.

#366 Updated by Marius Gligor about 7 years ago

Unfortunately I found another NPE bug which occur if the spawn or spawn.exe is missing.
In this case the ProcessBuilder throw an exception at line 101 and the shell variable remains null.

[02/14/2014 13:52:55 EET] (WebHandler.spawn():SEVERE) {qtp1523495872-25} Cannot spawn the process.
java.io.IOException: Cannot run program "/home/mag/p2j/build/lib/spawn": error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)

Later at line 125 a NPE is throw.

02/14/2014 13:52:55 EET] (org.eclipse.jetty.server.HttpChannel:WARNING) /chui
java.lang.NullPointerException
        at com.goldencode.p2j.main.WebClientSpawner.spawn(WebClientSpawner.java:125)

The bug must be fixed in line 123 as follow:

   if (remoteUri == null && shell != null)
   {
      shell.destroy();
   }

#367 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

The bug must be fixed in line 123 as follow:

OK, I'll include the fix with my #2124 update, too.

#368 Updated by Marius Gligor about 7 years ago

I saw that you already did some changes inside WebClientSpawner and the code was moved to ClientSpawner.
However the bug remains an could occur in ClientSpawner line 154.

#369 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

I saw that you already did some changes inside WebClientSpawner and the code was moved to ClientSpawner.
However the bug remains an could occur in ClientSpawner line 154.

Yes, don't worry, the fixes will be in the proper files.

#370 Updated by Constantin Asofiei about 7 years ago

I found another problem in ClientCore this time. There are cases when the P2J Client terminates via System.exit. Thus, the browser will not be redirected back to the login page.
I've tried adding this code before the loop in ClientCore.start (the correct place now would be in WebClientSpawner$TemporaryClient.doWork):

      if (driver instanceof ChuiWebDriver)
      {
         final BootstrapConfig cfg = config;
         final ChuiWebDriver webDriver = (ChuiWebDriver) driver;

         Runtime.getRuntime().addShutdownHook(new Thread()
         {
            @Override
            public void run()
            {
               // remote application is terminated.
               // notify the web client by sending a referrer address.
               String url = cfg.getString("web", "referrer", "url", null);      
               webDriver.quit(url);
            }
         });
      }

but this ends up in a deadlock on SecurityManager singleton, in the catch block of the ChuiWebSimulator.quit method, which catches this exception:
java.io.IOException: Connection output is closed
   at org.eclipse.jetty.websocket.common.io.IOState.assertOutputOpen(IOState.java:101)
   at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendFrame(WebSocketRemoteEndpoint.java:167)
   at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:88)
   at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:331)
   at com.goldencode.p2j.ui.client.chui.driver.web.ChuiWebSimulator.sendTextMessage(ChuiWebSimulator.java:266)
   at com.goldencode.p2j.ui.client.chui.driver.web.ChuiWebSimulator.quit(ChuiWebSimulator.java:466)
   at com.goldencode.p2j.ui.client.chui.driver.web.ChuiWebDriver.quit(ChuiWebDriver.java:223)
   at com.goldencode.p2j.main.ClientCore$1.run(ClientCore.java:147)

This tells me we socket is already closed, when this hook is called...

On a side note:
  • the monospaced font is not the same as Monospaced font reported by TerminalOptions.DEF_FONTNAME - I think the names are case-sensitive...
  • Color.getRGB sets bits 24-31 to the alpha value, and are set to FF. For now, readTerminalOptions discards bits 24-31, so only bits 0-23 will be used.
  • I'm running MAJIC on devsrv01 with my #2124 update, I can access the login page, but when I get redirected to the (static) new location, the browser disconnects me. At first I thought I had conflicting certificates, but that is not the case: the client's log says that Jetty is getting started, but for some reason, I can't access the port.

#371 Updated by Marius Gligor about 7 years ago

1. Regarding fonts: During development I realized that fonts are rendered slightly different on web browsers and Java Swing.
The "monospaced" or "Monospaced" is the same but results in an ugly rendering on web client.
I fixed this issue by replacing the "monospaced" with "monospace" in tty.screen.js line 377 because using this font works fine.
"monospace" is not recognized as a monospaced font by SwingHelper.isMonospaced that's why has been fixed on JS code.

I tested also other fixed fonts (http://www.webupd8.org/2010/07/7-of-best-ubuntu-terminal-fixed-width.html)
but the test SwingHelper.isMonospaced fails at the following test:

if (!font.canDisplay('\u2500'))
return false;

which means that this font cannot be used to draw graphical characters (borders). I tested and is true in Swing does not draw borders.
Surprisingly on web browser it works fine! Also as I know that if the browser does not support a specific font he will use another.
Maybe a good idea is to use another parameter to specify the font family for web clients.
The web client does not require fixed fonts because the screen (canvas) is organized as a grid and characters are draw on the canvas.
All that we need is to find the largest width character of the font.

2. Regarding the RGB colors the alpha blending byte is used to define transparency level. A value of 0xFF means opaque.
I thinks that on ChUI interfaces the alpha blending is not used.

#372 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

I fixed this issue by replacing the "monospaced" with "monospace" in tty.screen.js line 377 because using this font works fine.

Is this replacement done case-insensitive? If not, make it so, because it will allow Monospaced to become monospace in the browser.

2. Regarding the RGB colors the alpha blending byte is used to define transparency level. A value of 0xFF means opaque.
I thinks that on ChUI interfaces the alpha blending is not used.

You might be right, but I can't say for sure. My problem was that the alpha byte was being set to 0xFF which made the web client unusable. That's why I'm dropping the alpha byte (for now) in the last #2124 update.

#373 Updated by Marius Gligor about 7 years ago

Here are the changes which passed today regression tests.

#374 Updated by Marius Gligor about 7 years ago

  • File deleted (netbeans-7.4.desktop)

#375 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

Here are the changes which passed today regression tests.

You can go ahead and release it.

#376 Updated by Marius Gligor about 7 years ago

I builded a new jetty release. The good news are that web sockets are working now in Google Chrome 32.

#377 Updated by Greg Shah about 7 years ago

I'm OK with this update. Please manually test the following:

1. The admin console on the MAJIC server.
2. Normal (swing client) login to MAJIC and usage of some menus.

I think that if those things are working then there is no need to run standard regression testing since the web server is not in the testing paths.

#378 Updated by Constantin Asofiei about 7 years ago

Marius, there are problems when terminating the P2J server via the "shutdown" command (i.e. -k argument). After at least one web client has accessed the P2J server's jetty server, the normal shutdown doesn't work anymore. There are at least these problems:
  • the TemporaryAccountWorker.run loop ignores InterruptedException's and doesn't terminate.
  • the StandardServer.registerDefaultServices:958 (where the web server is started) must implement the terminate method, so that it will terminate the jetty server, when P2J server shuts down.

I think the same goes for the P2J Client - when the P2J session ends and the queue is stopped (i.e. the server goes down), the embedded jetty needs to be explicitly terminated, so the client can terminate, too.

#379 Updated by Constantin Asofiei about 7 years ago

LE: check if the WatchdogTimer.run:81 while (true) loop needs to terminate if it was interrupted.

#380 Updated by Marius Gligor about 7 years ago

Marius Gligor wrote:

Here are the changes which passed today regression tests.

Passed regression tests. Committed to bzr revision 10464.

#381 Updated by Marius Gligor about 7 years ago

Constantin Asofiei wrote:

LE: check if the WatchdogTimer.run:81 while (true) loop needs to terminate if it was interrupted.

OK I fixed. Inside the loop first check for time elapsed than wait.

#382 Updated by Marius Gligor about 7 years ago

I did an update form the remote repository and I tried a full build "ant all".
The spawn build fails due to:

/usr/bin/ld: cannot find -ljvm

[exec] make -f /home/mag/p2j/src/native/makefile.spawn
[exec] make[1]: Entering directory `/home/mag/p2j/build/native'
[exec] rm -fv ../../build/native/spawn
[exec] gcc -g -Wall -O2 -s -DNDEBUG -I/usr/lib/jvm/java-7-openjdk-amd64/jre/../include -I/usr/lib/jvm/java-7-openjdk-amd64/jre/../include/linux -I/home/mag/p2j/src/native /home/mag/p2j/src/native/spawn.c -o ../../build/native/spawn -lm -lcrypt -ljvm
[exec] /usr/bin/ld: cannot find -ljvm
[exec] collect2: error: ld returned 1 exit status
[exec] make[1]: * [build] Error 1
[exec] make: *
[spawn] Error 2
[exec] make[1]: Leaving directory `/home/mag/p2j/build/native'

#383 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

I did an update form the remote repository and I tried a full build "ant all".

From my followup email:

Followup: to compile the native code, you need to bring the libjvm.so into the global path. For devsrv01, this was used:

sudo ln -s /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so /usr/lib/libjvm.so

Use "locate libjvm.so" to find the correct location for your system.

This should solve your problem.

#384 Updated by Marius Gligor about 7 years ago

The build of native artefacts now works but the server start-up fails probably due to major changes in design and directory structure.
What can I do in order to start the simple server?
Without the server running I have no options to tests my changes.

#385 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

The build of native artefacts now works but the server start-up fails probably due to major changes in design and directory structure.
What can I do in order to start the simple server?

See note 38 from #2124 - parts 3, 4, 5.

#386 Updated by Marius Gligor about 7 years ago

I configured the directory.xml and now it works fine.
Just a minor observation.
On note 38 from #2124 - parts 4 for full configuration the "host" attribute should be of type "string" instead "integer".

<node class="integer" name="host">
<node-attribute name="value" value="localhost" />
</node>

correct is:

<node class="string" name="host">
<node-attribute name="value" value="localhost" />
</node>

#387 Updated by Marius Gligor about 7 years ago

Here are the changes that I did today:

1. Redirect web client when client exit via System.exit()
We have two scenario for redirection.

a.) When the remote application is about to end.
b.) When the running client exit via System.exit().
To fix both cases I did the following changes:
- The referrer address is always embedded on the web page inside configuration JSON structure.
a.) When the remote application is about to end the JS client is notified like on the previous
implementation but the referrer in no longer send because is already embedded inside the page.
b.) When the running client exit via System.exit(). The detection of this event on web browser is not possible.
On the web browser page all that we can detect is a web socket close event.
This event could occur in the following cases:
- when the remote server close the connection due to System.exit() call or other reasons.
- when the user press the browser "Refresh" button and the page is reloaded.
In this case a message is show to client in order to inform about the connection closed.
If in an amount of time (5 seconds) the web socket connection is not open or the page is
not loaded the client is redirected to the login page. (see tty.socket.js line 135)

2. I did also a lot of changes in order to fix some bugs or to do improvements.

#388 Updated by Greg Shah about 7 years ago

Code Review 0217a

I am fine with the changes. As far as I can tell, this resolves both the server shutdown issue (note 378) as well as the ClientCore/System.exit() issue. Correct?

Constantin: please review.

Fix anything noted in Constantin's review and upload the final update (if any). Then please get it runtime regression tested.

#389 Updated by Greg Shah about 7 years ago

In regard to the web client performance DOWN frame issue:

I suspect that the issues we are seeing are probably due to the extra trip (from the Java client to the JS client) being added to every DISPLAY. This is done synchronously, right? In other words, a DISPLAY call to the Java client won't return to the P2J server until after the results are flushed down to the JS client, right?

If so, it may be possible to resolve this by making the sync() asynchronous. In other words, sync() in the Java code can push the buffer into the driver and return. Then a separate thread would actually push the screen down to the JS client. This intermediate buffer must be very carefully synchronized such that the pusher thread can only ever see a complete update to the buffer and we never try to push a partial/corrupt change.

Constantin: can you think of any 4GL behavior that this would break?

I will be interested in your findings on the interactive input/editing slowness. I wonder if there isn't a similar solution there too.

#390 Updated by Marius Gligor about 7 years ago

The main difference between web client and swing client is the network layer. Here we have to find solutions to improve performances.
One solution could be the asynchronous transfer mode but we have to investigate also other issues.
The ChuiWebSimulator is based on ChuiSimulator used by swing client and unfortunately is not designed and optimized for web clients.
For example when a keystroke is send to the application the entire screen is sent down to JS client (all cells) in order to update the screen.
I'm sure that here we could implements a better solution to send only one character and the cell location instead the entire screen.

#391 Updated by Constantin Asofiei about 7 years ago

About 0217a: my feeling is that the TemporaryAccountPool and TemporaryAccountWorker changes are prone to deadlocks. You made the TemporaryAccountWorker.execute and terminate synchronized, but what if we are on callerLatch.await(); line 82 on execute and we need to call terminate? As we have a lock on the TemporaryAccountWorker instance when execute was called, that lock was not released by callerLatch.await();, so it is in place; thus, terminate can not be called until the lock on TemporaryAccountWorker instance is released.

Also, combining synchronized blocks with latches is confusing. Have you considered using only synchronized blocks and basic wait/notify, to pick up the task? This would imply posting the task to a queue instead of an instance field. Look at how scheduler.JobProcessor works, although there is a bug I just noticed in there - wait should start only if the queue is empty; the while loop should have been like this:

      while (true)
      {
         synchronized (pendingJobs)
         {
            if (pendingJobs.isEmpty())
            {
               try
               {
                  pendingJobs.wait();
               }
               catch (InterruptedException e)
               {
                  // someone interrupted me, exit
                  break;
               }
            }  

            core = (pendingJobs.isEmpty()) ? null : pendingJobs.removeFirst();
         }

         if (core != null)
         {
            // execute the task.
         }
      }

#392 Updated by Constantin Asofiei about 7 years ago

Ah, and if you need for the caller to wait for the task to be finished, you can always use task.wait() at the caller and task.notify() at the TemporaryAccountWorker. actually, this will not work if the thread executes the task before you call wait, so sending a latch alongside with the task will be required (or do not wait if the task is already terminated...).

#393 Updated by Marius Gligor about 7 years ago

The deadlocks does not occur. The callerLatch is always counted down on the finally block when the current task execution ended. Than the execute method will release the lock (synchronized). In other words either execute() or terminate() has the lock at one moment.
The call to execute() method is always synchronized on TemporaryAccountPool so I eliminated the synchronized from TemporaryAccountWorker and I added
synchronized for call to terminate() also in TemporaryAccountPool.
The old method for synchronization using wait/notify should be designed carefully due to "spurious" wake-ups the should occur especially on Linux OS.

#394 Updated by Marius Gligor about 7 years ago

Here are my latest changes.

#395 Updated by Marius Gligor about 7 years ago

The TemporaryAccountWorker is a single instance as is declared as static. It was designed to execute one single task at a moment.
When no tasks are executed the TemporaryAccountWorker should wait. Obviously the task execution is serialized.
When a task is executed the caller MUST wait until task execution end.

#396 Updated by Greg Shah about 7 years ago

Code Review 0217b

I am fine with the changes.

Constantin: based on these changes and Marius' comments, do you have any other feedback?

#397 Updated by Constantin Asofiei about 7 years ago

Greg Shah wrote:

Constantin: based on these changes and Marius' comments, do you have any other feedback?

The code makes sense now.

Marius: make sure to test that, in MAJIC:
1. ./server.sh -k -i<instance> shuts down the server, in these two cases:
- server just started, login with terminal client, shut down server
- server just started, login with web client, shut down the server - the web client should redirect to login page, too.

2. having an active P2J web client, the P2J server abends (i.e. terminate the P2J server process manually with the kill command) - the P2J web client needs to be terminated and the browser should redirect.

#398 Updated by Marius Gligor about 7 years ago

I tested the redirection by killing P2J web client manually and its works on my workstation.
Nevertheless I will test also with Majic. I already downloaded the Majic from my testing environment on devsrv01.
I have to do the appropriate configurations and than try to improve performances and fix all other issues.
Should I started the regression tests using my latest changes or is better to wait until the Majic tests and improvements will be done?

#399 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

Should I started the regression tests using my latest changes or is better to wait until the Majic tests and improvements will be done?

You need to include the scenarios I've noted above as part of the regression testing for 0217b (test them using MAJIC). The improvements will be in another update, we need to fix the "-k", as it is broken now.

#400 Updated by Greg Shah about 7 years ago

You are a go for regression testing on 0217b.

#401 Updated by Constantin Asofiei about 7 years ago

Greg Shah wrote:

Constantin: can you think of any 4GL behavior that this would break?

In terms of 4GL behaviour, I don't see any problems at this time. But I wonder if we shouldn't push to the JS side just the portion(s) of the screen being changed... to not redraw the entire screen each time something changed. For a 24x80 terminal the drawing already seems slow, but with a larger terminal or when we will move into GUI, we will experience problems.

Also, about the interactive input: we need to keep in mind that a user key-press is not processed on the JS side; the key first needs to reach the remote P2J Client side (and even the P2J Server side, in case of triggers) and only after that may or may not end up on screen. So, I'm not sure how much we can improve this, as the JS screen will "see" the key only when it picks up the updated screen from P2J Client side. At this time, my feeling is that a major part of the slowdown is caused by the fact that we redraw the entire screen, no matter how small the affected portion is. What you suggest will do some improvement, as the P2J side will not have to wait for the JS side to finish drawing, but the JS side will still be slow on drawing - and it might have a hard time to keep up with all the updated screens, coming from the P2J side.

#402 Updated by Constantin Asofiei about 7 years ago

Marius: another part which is not working in Web (or Swing) mode is the UNIX. statement. This suspends the P2J client and opens a shell. See the shell_commands.p and unix-cmd.p programs; adjust shell_commands.p (or write another test) to add a plain UNIX. statement, and see how this behaves in terminal and web mode. See #2244 for the MAJIC scenario about how to launch the shell.

#403 Updated by Marius Gligor about 7 years ago

1. I started to find a solution to improve the web ChUI client. I have to improve both network traffic and JS rendering part.
Because JS code is very slow I have and idea to send cells data down to JS code in a binary format and per/line not the entire screen.
In other words whenever something is changed on a screen line the line is send down to JS to be updated.
On the JS only the changed positions from the line should be redraw. After this flaw is implemented and tested the next step is to send
data to JS asynchronous using a separate thread to push messages.

2. Regarding UNIX statement, at this moment I have no idea how to solve. We will see after I will finished the improvements.
In my opinion this should be fixed on the P2J client code.
But what happen when the client are using the ChUI web client and he is on another machine?

3. I restarted today the CTRL-C regression tests once again. Why it not passes?

#404 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

2. Regarding UNIX statement, at this moment I have no idea how to solve. We will see after I will finished the improvements.
In my opinion this should be fixed on the P2J client code.

This will be a tricky one, because UNIX suspends the P2J Client and currently we don't have a way of capturing the native process input/output. We might need changes in the process launching code.

But what happen when the client are using the ChUI web client and he is on another machine?

The shell, other process launching and even file reading/writing is always relative to the P2J Client, not the P2J Server. So if the P2J Client is on another machine, it must open a shell on that machine, not the server's.

3. I restarted today the CTRL-C regression tests once again. Why it not passes?

I know CTRL-C becomes more and more annoying, but we need to deal with it as it is, for now. The failures in your last email look like false-negatives; for some reason, at some point the connection to the remote server can't be established, and I don't know if is related to machine load or some race condition already existing in the P2J persistence code which establishes the remote DB connection (CTRL-C uses some very quick connect/disconnect scenarios, so this might not even be a real issue after all). Greg: at some point we should add some logging in the MAJIC program which establishes the remote connection, and see if the ERROR-STATUS reports any errors; we should make CTRL-C stable, as at some point it might hurt us if some bug hides behind the failures.

#405 Updated by Greg Shah about 7 years ago

But I wonder if we shouldn't push to the JS side just the portion(s) of the screen being changed

Yes, we certainly need to do this.

In other words whenever something is changed on a screen line the line is send down to JS to be updated.

We should think about this problem in rectangles or changed regions, not on a line by line basis. It is very common for us to update just a portion of the screen. This portion will often be broken over multiple lines but only a specific rectangular region of the screen will be updated: there will be characters to the left and right of that region that are not changed. The the example below, the O characters are the region of the screen being changed.

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXOOOOOOOXXXXXX
XXXXXXXXXXXXXXXXXOOOOOOOXXXXXX
XXXXXXXXXXXXXXXXXOOOOOOOXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

I don't want to have a line-by-line oriented approach because that will entail too many trips to the client. When a region changes, only that region should be sent down.

Likewise, I don't want to send the unchanged portions of the lines down to the JS client. Let's just send the changed portions and we can include enough data to tell the JS portion what the X and Y offsets are for the region (and the length of each row). With this, it is a simple effort to just draw the sent cells in the right place. There will be no need to track the diffs in JS, just at the P2J client side.

the next step is to send data to JS asynchronous using a separate thread to push messages.

OK.

2. Regarding UNIX statement, at this moment I have no idea how to solve. We will see after I will finished the improvements.
In my opinion this should be fixed on the P2J client code.

This will be a tricky one, because UNIX suspends the P2J Client and currently we don't have a way of capturing the native process input/output. We might need changes in the process launching code.

Indeed. I had thought of this interactive child process problem a few times during the early development stages but I neglected to write down that thought and now you have found my mistake.

The use of INPUT THROUGH, INPUT-OUTPUT THROUGH and OUTPUT THROUGH should be fine. These are all ways to execute a child process with redirected terminal.

The problem is in the use of the interactive terminal. Progress allows the 4GL to "shell out" to the child process and the child process controls the terminal. The user can then interact directly with the child process until they enter input that causes it to exit. Then control naturally returns to the 4GL code and the application's processing continues. There is no way for the user to switch back and forth from the child process to the 4GL. The child process can only be launched, used and then terminated.

There are 2 problems that I see here:

1. Since the user is actually at the JS client (and not on the system on which the child process is being executed), we must find some way to feed the JS user's input to the child process AND capture the child process' output and display it on the JS client. We CANNOT use terminal redirection because such a mode changes the terminal to disable features that are needed by many interactive programs. In other words, these child processes may be programs that do not operate on a stream basis, but which truly manage the terminal processing on an entire screen/keyboard basis. The good news is that we already manage this launching process using native code and that is where we will have to implement our solution. The bad news is that we will have to figure out how we to insert ourselves into the terminal driver chain to allow us to intercept output and insert input. But we do have a program to potentially use as a starting point: screen. This program does pretty much exactly what we want to do. Please inspect its source code to see what techniques it uses and determine if we can use these same techniques to solve our problem.

2. How do we suspend the P2J client while still continuing to communicate with the JS client. At least this problem is fully under our control, because it is just about the P2J client design. I expect those asynchronous I/O threads will be very important to this solution.

But what happen when the client are using the ChUI web client and he is on another machine?

The shell, other process launching and even file reading/writing is always relative to the P2J Client, not the P2J Server. So if the P2J Client is on another machine, it must open a shell on that machine, not the server's.

Luckily, this is already exactly how we do this. Nothing more is needed to implement this, since we already have a separate P2J client process and that process can already use the libp2j.so/p2j.dll to launch a child process. This works today, even for the web client.

When we provide "remote launching" support such that the P2J clients can be launched on a separate machine from the P2J server, then the process launching will automatically be done on that other system, by virtue of the fact that the P2J client already manages the child process launching. Nothing extra is needed for this to work.

#406 Updated by Marius Gligor about 7 years ago

Passed regression tests. Committed to bzr revision 10471.

I created an Majic environment on my work station based on the converted project from devsrv01 testing environment.
Now I'm able to start both the ChUI native and ChUI web clients on my workstation.

My conclusions running Majic:

1. The server shut-down works properly. Using ./server.sh -k i10 for example results in a silently shut-down of the server.
I tested the same command to shut-down the two server instances on my devsrv01 testing environment and it works also properly.

2. Killing the spawned client "by hand" results in a client redirection to login page.

3. Terminating or auto terminating the Majic application results also in a client redirection to the login page.

4. Admin console works properly with the new jetty libraries.

5. Web sockets works on both Firefox and Chrome (Chromium) web browsers maybe a little bit more faster on Google Chrome due to it faster JS engine.

6. Definitely we have to find solutions to improve the performances on ChUI web client.

#407 Updated by Marius Gligor about 7 years ago

Here is the first release containing my changes in order to improve the ChUI web client performances.
I did major changes in both Java and JS code in order to improve performances.

1. For web socket protocol I switched to a binary format for payloads except for color palette message which remains in text JSON format.
The network packages are now optimized in term of size.

2. The messages are send asynchronously. First the messages are stored on a FIFO queue. From this queue the messages are extracted by a thread and pushed down to JS client.

3. The screen buffer is scanned line by line and only the cells which has changes on character, attribute or color are send to JS client.

4. I did also changes in JS code to be fastest as much as possible.

5. Based on some Internet articles regarding HTML5 canvas improvements I implemented a double buffered mode of drawing.
The screen is draw in an invisible canvas and finally the main canvas (visible) is updated.

6. The rendering problems are not yet fixed. I have to do a lot of tests here using different fonts.

7. Doing tests on Majic the results are comparable with the native client except for Majic login plug-in which is loaded in 2-5 seconds.
According to my tests the problem isn't from the JS code or from network traffic.
How I can use the Swing chui driver to do tests on Majic to see if we have the same problems?
I saw that the applications are rendered different on native on Swing (Web) chui drivers. It seems that native driver has more than 24 lines.
I obtained the fastest results using Google Chrome web browser. Today Google had issued a new release Chrome 33 which fortunately works fine.
The native is obviously fastest and we cannot obtains the same results using the web driver.

#408 Updated by Constantin Asofiei about 7 years ago

Marius, the changes are good and is a step forward into performance improvement. See bellow for my review:

ChuiWebSimulator
  • add javadoc for all the MSG_ constants.
  • document this protocol in the class javadoc (we might want to add it to our books, too).
  • triggerRepaint:
    - add a lazy initialization of the ByteArrayOutpuStream line var, so that we don't build a new instance for lines which weren't change.
    - can you think of a (fast) way of merging contiguos cells with the same attributes (text/color) into a single packet? I'm thinking about something like this: rectangle(x, y, widht, height) has these (text, color, etc) attributes. This way, a screen cleanup (when we need to set all cells to the same value) can be easily sent to the remote side, and the remote side can do a single color-fill for the specified rectangle, instead of drawing each cell. I think this would improve a lot when we need to draw borders or just empty content.
  • please enhance the terminal size to i.e. full-screen and check the JS performance. Use a test like this, with some heavy drawing:
    def var i as int.
    def var n as int.
    /* replace "24" with your actual number of rows */
    n = 24 - 7.
    
    form i with n down frame f1.
    
    do i = 1 to n * 10000: /* adjust this to something smaller */
       if i mod n = 0
       then up n - 1 with frame f1.
       else down 1 with frame f1.
    
       display i with frame f1.
    
       hide all no-pause.
    end.
    
PushMessagesWorker
  • there should be a space between while and condition in run() method, as in while (running).
  • even if messages is a ConcurrentLinkedDeque, you should enclose this code in a synchronized block, so is atomic:
                synchronized (messages)
                {
                   if (messages.isEmpty())
                   {
                      // no messages on the queue. wait.
                      messages.wait();
                   }
                }
    

    Otherwise, we might lose notifications between the isEmpty and the lock acquisition.

Finally, each time you change something in the communication protocol (as with this update, when you added a new thread), please don't forget to double check the P2J server termination scenarios you mention in note 406.

How I can use the Swing chui driver to do tests on Majic to see if we have the same problems?

To test the swing client, add this client node to your client.xml and use client.sh to start it, as usually:

<node type="client">
   <client>
      <driver type="swing_chui_frame"/>
      <chui rows="24"/>
      <chui columns="80"/>
      <chui background="0x000000"/>
      <chui foreground="0xFFA500"/>
      <chui selection="0x0000FF"/>
      <chui fontname="monospaced"/>
      <chui fontsize="12"/>
   </client>
</node>

I saw that the applications are rendered different on native on Swing (Web) chui drivers. It seems that native driver has more than 24 lines.

You need to manually set your terminal to 24 rows by 80 columns, as in native mode, MAJIC (as 4GL does) uses the entire available screen.

except for Majic login plug-in which is loaded in 2-5 seconds.

I'll take a look at this tomorrow.

#409 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

except for Majic login plug-in which is loaded in 2-5 seconds.

I think I know where the problem is: the web socket is not open until we accept the certificate and the page is loaded. In the mean time (while there is no web socket), the authenication plugin finished drawing the screen, but there is no web socket open - so it doesn't have where to send the screen (the pushWorker is null). This is OK, the screen will be fully drawn when the web socket is connected. The delay comes from the Jetty/web socket side/JS: it takes a while to bring up the connection, but once it does, the screen is drawn immediately.

BTW, I just noticed that you if a line has changed cells, you send them to the JS immediately: can you test if is faster to send all the changed cells at once, for the entire screen? And not just for a certain line.

#410 Updated by Greg Shah about 7 years ago

I just noticed that you if a line has changed cells, you send them to the JS immediately: can you test if is faster to send all the changed cells at once, for the entire screen? And not just for a certain line.

Yes, exactly. We already know that extra trips to the client will reduce performance. It is better to do more work on the Java side to batch up a larger update and send it with one round trip.

This was my intention when I noted:

I don't want to have a line-by-line oriented approach because that will entail too many trips to the client. When a region changes, only that region should be sent down.

#411 Updated by Marius Gligor about 7 years ago

Rectangles detection on screen entropies doesn't help us too much. Let me explain:

Having a pattern like:

XXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXX LINE 1 XXXXXXXXXXX
XXXXXXXXX LINE 2 XXXXXXXXXXX
XXXXXXXXX LINE 3 XXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXX

When we draw the text on the canvas we have to draw line by line in other words we have 3 text draws operations.
The only advantages is that we have a single rectangle clear instead three.
Another example:

XXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXX1XXXXXXXXXXXXXXXXXX
XXXXXXXXX2XXXXXXXXXXXXXXXXXX
XXXXXXXXX3XXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXX

In this case we have also to draw text line by line but a single cell at a time.
On the other hand an algorithm to detect rectangles is not so easy to implements and require a lot of computations.

However a simple approach which could improve the draw is to group all cells in a line, something like an rectangle but on an single line.
This is very easy to implements require few computations and finally we have also 3 text draws and 3 rectangle clear instead 1.
I just implemented such an algorithm and now I have to do tests.

#412 Updated by Greg Shah about 7 years ago

The 4GL approach to all of this processing is executed in a single process. When we designed P2J, we deliberately chose to implement a thin presentation engine approach to the client. But this meant that what was inherently a single system/single process was now a 2 process approach and for many customers it is also split across 2 systems. We found there were many cases where the performance of our approach was not comparable to the original 4GL. When we worked to improve performance, the majority of the improvements came from eliminating trips down to the client. This often came from sending more data down in one trip rather than splitting the same data over multiple trips. We found there was a massive improvement in this case. It was worth spending more time to gather up/process the data on the server so that it could be batched.

Now that we have split this particular driver over 3 processes and potentially up to 3 separate systems, I am willing to bet that this same approach is still very important.

Please do spend the time to calculate the actual rectangle that is changed. I'm confident that the extra processing time will be worth reducing the size of each change set in the majority of cases.

It is OK if the canvas drawing code still needs to draw line by line, that is independent of how many trips we take over the network from the P2J client to JS. It is the trips we are trying to eliminate, not the looping in the JS drawing code.

#413 Updated by Marius Gligor about 7 years ago

Did you mean to find all rectangles or only the biggest one?

#414 Updated by Greg Shah about 7 years ago

Start with the rectangle specified by the union of the changes. If we still need more optimizations past that, we can look at sending multiple smaller rectangles down in a second phase.

#415 Updated by Marius Gligor about 7 years ago

Please take a look over this changes. All the changes on the screen are sent on a single message over the network.
Each message is composed from packages. Each package contains the adjacent changed cells that should be draw
on the screen grouped by attribute and color. Each packet will be draw on a single shut.
According to my tests is faster even my DB connection is slow.
I have to think to a better solution if exists as you already suggested.
Also the rendering issue is still not fixed.

#416 Updated by Marius Gligor about 7 years ago

Cleaning JS code after so many changes. I will try also to simplify even more the JS code.
I think that is possible to not keep any status in JS just drawing.

#417 Updated by Greg Shah about 7 years ago

Code Review 0221b

The changes look good. I look forward to seeing the next changes (batching multiple rows in a single message and fixing the rendering issue).

#418 Updated by Constantin Asofiei about 7 years ago

Marius: we are running blind when we are trying to determine if there were errors when execvp launches the command. All we get is a status code, but no actual stacktraces/messages/etc, if i.e. java can't launch the P2J client because some other P2J client uses the same debug port or for some other reason (faulty claspath, incorrect syntax for the command, etc). I know you've added some code in ClientDriver to redirect stderr for web clients (btw, I think we need to redirect stderr in case of P2J processes, too, not just web clients), but that code is used only for a successful command... if the command fails, its output and errors are lost.

On a side note: with #1787, we will be determining if a P2J client has its standard output or input redirected. This is useful for appservers, as this is how the agents work: all statements generating output are sending it to standard output (bypassing the 4GL infrastructure), which is redirected to a file. So: can we redirect the standard output for the child process, in spawn? Or better said, if we directly invoke spawn with output redirected (i.e. spawn ... > file.log), will this redirection be picked up by the child process?

#419 Updated by Marius Gligor about 7 years ago

The syslog is used by the audit function in spawn.c to log errors during the process spawning.
When the spawner starts first a child process is created.
The child process became a session leader and is detached from his parent.
The parent end execution here and exit.
Than the session leader process create than another child process.
The session leader will wait until child exit to avoid creation of orphan processes.
A pseudo terminal (PTY) is assigned to the child process and STDIN STDOUT and STDERR are redirected to PTY.
The STDIN MUST be redirected to PTY otherwise the test isatty() in process_linux.c line 91 fails.
This was a bug which was fixed recently if you remember.
I'm not sure but you could try to redirect STDOUT for the child process by putting a comment in spawn.c line 261.
As long as STDOUT is inherited from initial process to session leader and than to further to child process the redirection should works.

#420 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

The syslog is used by the audit function in spawn.c to log errors during the process spawning.

Does this mean that errors during command invocation by execvp should end up in syslog? If so, please do some tests with a faulty classpath (so that the main class can't be found), and post here how the syslog messages should look.

#421 Updated by Marius Gligor about 7 years ago

On syslog only error messages related to system call's are stored.

syslog(LOG_ERR, "Error:%d method:%s (%m)\n", code, method);

What you describe indeed is not in the system log, the error message is lost.
The redirection of the child process STDOUT could be a solution.

#422 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

The redirection of the child process STDOUT could be a solution.

I don't think we can redirect STDOUT by default, because this can cause conflicts in the 4GL behaviour. When writing from a 4GL batch program/appserver, behaviour is different with STDOUT redirected and with STDOUT not redirected; and we want to have full control of cases when we redirect STDOUT and when we don't.

#423 Updated by Constantin Asofiei about 7 years ago

PS: if the command has error messages, wouldn't these end up to STDERR? So, what about inheriting the STDERR from the P2J server; as ClientDriver.setWebClientLog redirects STDERR to file once the P2J Client is started, the STDERR will end up to the file, right? And P2J Server will see only the command-related errors, not from P2J Client.

#424 Updated by Marius Gligor about 7 years ago

A possible solution is to put in a command line parameter the name of the file we want to redirect the output.
We can open this file inside the spawner and redirect STDERR or STDOUT to this file.

#425 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

We can open this file inside the spawner and redirect STDERR or STDOUT to this file.

This is good when we explicitly need to redirect STDOUT to a file (you can go ahead and start adding support for this), but I don't think is needed for STDERR. Try this in ClientBuilder.start:222:

pb.redirectError(ProcessBuilder.Redirect.INHERIT);

This should make the native code inherit the STDERR from the P2J Server. Let me know if it works.

#426 Updated by Marius Gligor about 7 years ago

It works. Nice! The messages from the client are stored now into the server log.

#427 Updated by Marius Gligor about 7 years ago

1. No longer dirty screens. I found what cause the canvas to becomes dirty and I fixed this issue.

When a text is drawn on the canvas the text characters are converted into graphical glyphs which are rendered on a paint area.
Sometimes the browser draw some pixels outside the paint area and the canvas becomes dirty.
Drawing of the letter R is one such case that I found.
The solution is to protect the outside area whenever we draw the text on the canvas by defining a clip rectangle as a paint area.
All paints are done only inside the clip rectangle and the outside area remains unchanged.
The HTML5 canvas API has support to define and use clipping regions.

At this stage the font metrics is poor implemented on HTML5 canvas API. It allows to measure a text width only.
I found a method to measure also the height of a font.
The text rendering differs on web browsers. I created a test page (font.html)
Please open this page on Firefox and Chrome and observe differences.
According to my tests in Widows OS the character height is the same in Firefox, Chrome and IE.
However the space at the top of the character is 1 in Firefox and 4 in Chrome and IE.
As a result the cell height on the canvas is 3 pixels smaller in Firefox and the screen aspect ratio is different in Firefox versus Chrome and IE.
On Linux OS the top is 0 on Firefox and 4 in Chrome. The character height is the same.
By measuring the font height I eliminated the browser dependencies on this issue.

2. Looking to improve data transfer from client to JS code I designed an improvement of the current implementation.
Before starting to implements I would like to describe what is already done and what I would like to do next.

- drawing on canvas are expensive operations.
- the number of canvas text drawings must be reduced to a minimum value.
- drawing texts on the canvas is possible only one line per each draw operation.
- each draw operations are done in 2 steps. First the paint area is filled with the background color (fillStyle, fillRect) then the text is drawing with the foreground color (fillStyle, fillText).
- we have to send down to JS only the changed screen cells that have to be redraw.

In order to minimize the number of drawings the adjacent screen cells in a line having the same attributes and color are merged in a single text string having the same attributes, stored in a package and appended to a message. Finally the message is send down to JS code. The message contains packages from all screen lines and each package is optimized for drawing. On the JS the packages are extracted from the message and draw one by one on the canvas. This is already implemented and looks like Swing screen driver implementation. (ChuiSimulator#paintComponent).

A step forward is to detect rectangles containing screen cells having the same attributes and color by keeping in mind that we need to have a minimum number of drawings.
Here we have to be carefully. Let see some cases:

a). Patterns like:

 XXXXXXXXXXX
   XXXXX
XXXXXXXXXX

could be packaged as follow:

XXXXXXXXXXX  XXXXX XXXXXXXXXX (3 rectangles with height=1)

which require only 3 drawings (3 fillRect and 3 free fillText) or

XX  XXXXX XXXX
    XXXXX
XXX XXXXX XX

which require 7 drawing (7 fillRect and 7 fillText) operations and some extra computation to split lines.
On this case the optimized mode is to send 3 packages for 3 drawings.

b) Patterns like:

XXXXXXXXXXXX
XXXXXXXXXXXX
XXXXXXXXXXXX

or

X
X
X
X
X

are the best candidates to be send as rectangles. When such a rectangle is drawing on the canvas
the clipping rectangle is bigger and we have a single fillRect operation and 3 fillText operations.

This part (detection of rectangles) is not yet implemented and I would like to implements on the next step.
Please let me know if you have any other ideas regarding this step.

#428 Updated by Marius Gligor about 7 years ago

Overall best performances has been obtained using Google Chrome which really has a very fast JS engine.

#429 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

This part (detection of rectangles) is not yet implemented and I would like to implements on the next step.

Please postpone this one a little and first build a separate update with following:
- the STDERR fix for spawn.
- add support in spawn.c to redirect the STDOUT to a file; the file name will be received from the arguments.

This is priority work needed by the #1787 and #2228 tasks.

#430 Updated by Constantin Asofiei about 7 years ago

PS: add support for a clientConfig/outputToFile node, with the name of the log file. This flag will be read by ProcessBuilderOptions but not by WebClientBuilderOptions. ProcessClientBuilder will pick up this flag and send it to the spawn tool, via an argument; make sure both syntaxes support it: spawn 1 for password auth and spawn 0 for P2J secure connection auth.

Think of a way to include additional info in the file name (i.e. process ID, P2J user, appserver name). Do you think append mode (>>) can be easily added? If yes, we can use a clientConfig/appendToFile node, to mark the append mode.

#431 Updated by Marius Gligor about 7 years ago

OK. A first question please. This change remains or was only for test purposes.
pb.redirectError(ProcessBuilder.Redirect.INHERIT);

#432 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

OK. A first question please. This change remains or was only for test purposes.
pb.redirectError(ProcessBuilder.Redirect.INHERIT);

This change needs to remain; otherwise it will be almost impossible to determine why a P2J client couldn't start, when there are problems with the command.

#433 Updated by Marius Gligor about 7 years ago

I understood the STDERR is redirected foe both modes 0 and 1 so I kept the code.
The difference between > and >> is reflected in the open file mode.
On > the file is always created on >> we will try to use an existing file.
This could be implemented by having an aditional command line parameter which instruct the spawner how to open the file.
How could I do tests using the mode 0 )with no password)?

#434 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

How could I do tests using the mode 0 )with no password)?

Mode 0 is used for starting batch processes or appserver clients. Here are the steps to configure your directory for a batch process:
  1. configure a process account for your directory; there are two steps:
    - in security/accounts/processes
              <node class="process" name="p2j_proc">
                <node-attribute name="enabled" value="TRUE"/>
                <node-attribute name="alias" value="p2j_proc_alias"/>
                <node-attribute name="description" value="a P2J batch process"/>
                <node-attribute name="server" value="FALSE"/>
                <node-attribute name="master" value="FALSE"/>
              </node>
    

    - in server/default/runtime, to set the p2j-entry, use this node:
              <node class="container" name="p2j_proc">
                <node class="string" name="p2j-entry">
                  <node-attribute name="value" value="com.goldencode.testcases.SomeProgram"/>
                </node>
                <node class="integer" name="stop_disposition">
                  <node-attribute name="value" value="2"/>
                </node>
              </node>
    

    Is important to replace SomeProgram with a converted program which IS NOT USER-INTERACTIVE. Use a program with something like this:
    def stream rpt.
    
    output stream rpt to some-file.txt.
    message "some-test".
    output stream rpt close.
    

    Currently, you can't use DISPLAY or MESSAGE to write. Instead, you can edit the converted program by hand and add some System.out.println and System.err.println statements, to check where the output will go. BTW, while you are at it, please fix ClientDriver to redirect STDERR for batch processes, too; currently, this is redirected only for web clients.
  2. Use the -c option of the ServerDriver to re-configure your directory's certificates and private keys. At some point, it will ask you if you want to save the server certificates in a file; enter srv-certs.store as the file name. This is the only important step; leave all the private keys in the directory, so it will be easier to run.
  3. install the files needed by spawner's "no-password authentication" mode. In this mode, it needs to connect to the P2J server and extract the command. Beside the spawn tool, you need in the same folder: the p2j.jar and srv-certs.store files, both with 0550 modes. If everything is setup correctly, spawn 0 will output the syntax. BTW, I think the puts calls in spawn.c will go to STDOUT, not STDERR: please change these so that they will write to STDERR.
  4. start the server normally.
  5. to start a batch process, you can use the -b p2j_proc arguments for the ServerDriver. You can create a separate server.sh file which hard-codes the -b p2j_proc and use this to start the batch process. If successful, this will:
    a. connect to the target P2J server and instruct it to start a batch process
    b. the target P2J server will call spawn, which will:
    - connect back to the P2J server using a secure connection and the temporary credentials (see NativeSecureConnection)
    - if connection was possible, use RemoteSpawner.getCommand (implemented by ClientBuilder.getCommand) to retrieve the real command
    - pass this command back to the native side of spawn, which will execute it. The P2J client for the batch process will use the same temporary credentials to retrieve the trust-store and the process key-store from the P2J server. These will be used to authenticate the batch process.

#435 Updated by Greg Shah about 7 years ago

Code Review 0224a

Your use of the direct pixel access in canvas for the calculation of font metrics is very clever. I love it!

The changes in tty.screen.js are good. Based on the described changes, I did expect some code for the Java side too, but the only file that was different in the update was tty.screen.js. Are there some files missing?

#436 Updated by Marius Gligor about 7 years ago

No files are missing. I did changes only on JS code.
On the next step I have to implements the rectangle detections on Java code. This work require also some changes on JS code.
On my previous post I described the next step that I have to do and I specified that is not yet implemented.
My expectations are to obtain better results after implementation of rectangles detection.

#437 Updated by Greg Shah about 7 years ago

OK, I understand.

#438 Updated by Constantin Asofiei about 7 years ago

Marius: can you tell me the changes needed in spawn.c, to redirect the child process to a file? I don't need the full support, I just want to hard-code the output to a file and test some of the #1787 changes made by Eugenie.

#439 Updated by Marius Gligor about 7 years ago

You have to delete the lines when the STDOUT and STDERR are redirected to PTY.
The open the file on which you want to redirect the STDOUT or STDERR and redirect to file descriptor.
I already implemented the redirection and now I have to configure my environment and do some tests.
So far I tested "by hand".
Please take a look over my current implementation.

#440 Updated by Constantin Asofiei about 7 years ago

About 0225a.zip:
  • ClientBuilderParameters
    - can you think of a way to pass the PID in the file name, too? Maybe %pid% can be replaced by spawn.c?
    - all process launching is done from a thread with the server's context, so SecurityManager.getUserId will always return the server's id. But, the process name is known to the ProcessClientSpawner. Add a ProcessBuilderParameters subclass of ClientBuilderParameters which saves the process name passed to ProcessClientSpawner. Add a ClientBuilderParameters.getUser() method, which by default returns SecurityManager.getUserId() and will be overridden by subclasses to return the correct subject ID, for each case: i.e. web clients the system username and P2J processes the process name.
    - use SecurityManager.getAppServerForProcess to determine if this has an appserver or not; and replace %appserver% accordingly.
    - if outputToFile is not set, we do not want to redirect.
    - also, I think is best for each sub-class to prepare the filename on its own. So, the ClientBuilderParameters.getOutputToFile logic will treat only the %userid% and %timestamp% cases and subclasses can resolve the other cases (i.e. %appserver%).
  • ClientDriver
    - rename setWebClientLog to setClientLog
    - I think is enough to check only for the server:spawner:uuid config: we can assume that all clients started by the P2J server receive this config.
  • ProcessBuilderOptions
    - the javadoc for outputToFile is incorrect
    - you need to default to null if the outputToFile node is missing - we don't want to redirect always.
    - you are duplicating logic from ClientBuilderParameters.getOutputToFile. All logic should be in ClientBuilderParameters.getOutputToFile and related subclasses.
  • spawn.c
    - why do you redirect STDERR to a distinct file? I thought the pb.redirectError(ProcessBuilder.Redirect.INHERIT); is enough... as we want the STDERR from spawn.c to end up in the server's log, right?
    - I'm not sure I understand why you are using spawn:stdout:file to pass the log file name... the spawn tool should receive only one additional argument in spawn 0 syntax, which is the name of the file to which STDOUT needs to be redirected and this argument needs to be optional. In spawn 1 case, we don't want to redirect the STDOUT.

#441 Updated by Marius Gligor about 7 years ago

It was a miss understanding in requested design. Let clarify.
What I understood is that outputToFile is a file to redirect STDERR on case 0.

You said that this is not true and the correct is:

The redirection of STDERR is the same on both cases and is done on ProcessBuilder.
The outputToFile is a file to redirect STDOUT on case 0.
The redirection of STDOUT is optional and could missing.
No redirection of STDOUT on case 1 so spawn:stdout:file should be eliminated.

We have 3 process ID's. If we need to put the id of the child process on the file name we have to do the
redirection on the child process before execvp.

Please let me know if this is the correct design.

#442 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

You said that this is not true and the correct is:

The redirection of STDERR is the same on both cases and is done on ProcessBuilder.
The outputToFile is a file to redirect STDOUT on case 0.
The redirection of STDOUT is optional and could missing.
No redirection of STDOUT on case 1 so spawn:stdout:file should be eliminated.

All are correct.

We have 3 process ID's. If we need to put the id of the child process on the file name we have to do the redirection on the child process before execvp.

I have these processes on devsrv01, after I connect with a web client:

ca        3934  0.0  0.0  41548  1528 pts/28   Ss+  05:32   0:00 /home/ca/testing/spawner/spawn 1 ca /home/ca/testing/majic/run/client java -Dlbin=/home/ca/testing/majic/lbin -DHOME=/home/ca -Djava.library.path=/home/ca/testing/majic/p2j/build/lib/ -classpath /home/ca/testing/majic/p2j/build/lib/p2j.jar com.goldencode.p2j.main.ClientDriver server:spawner:uuid=3e2699fc-467b-45bd-b448-017659318890 net:server:host=localhost net:connection:secure=true net:server:secure_port=3373 client:driver:type=chui_web client:chui:rows=24 client:chui:columns=80 client:chui:background=0x000000 client:chui:foreground=0xffa500 client:chui:selection=0x0000ff client:chui:fontname=Monospaced client:chui:fontsize=12 client:web:socketTimeout=-1 client:web:watchdogTimeout=-1 client:web:port=10473 client:web:host=localhost web:referrer:url=https://localhost:7473/chui
ca        3935  109  0.1 21863464 98760 pts/28 Sl   05:32   0:18 java -Dlbin=/home/ca/testing/majic/lbin -DHOME=/home/ca -Djava.library.path=/home/ca/testing/majic/p2j/build/lib/ -classpath /home/ca/testing/majic/p2j/build/lib/p2j.jar com.goldencode.p2j.main.ClientDriver server:spawner:uuid=3e2699fc-467b-45bd-b448-017659318890 net:server:host=localhost net:connection:secure=true net:server:secure_port=3373 client:driver:type=chui_web client:chui:rows=24 client:chui:columns=80 client:chui:background=0x000000 client:chui:foreground=0xffa500 client:chui:selection=0x0000ff client:chui:fontname=Monospaced client:chui:fontsize=12 client:web:socketTimeout=-1 client:web:watchdogTimeout=-1 client:web:port=10473 client:web:host=localhost web:referrer:url=https://localhost:7473/chui

I guess the spawn here is the parent process, right?

#443 Updated by Marius Gligor about 7 years ago

Indeed the spawn is the session leader process the parent of the child process, our client.
The 2 remaining processes are the session leader and his child, our client.
The parent process of the session leader (the initial process created by Java ProcessBuilder) is no longer running.
After the session leader process is created it exit.

#444 Updated by Constantin Asofiei about 7 years ago

Marius, some more issues/notes:
  1. adding System.err.println calls directly in the java program will write to the server log (which is OK, as we are on the server side). I forgot about this when I mentioned it in note 434.
  2. I've hard-coded the redirection after switching to the working dir, just before execvp (as you mentioned) and Eugenie's update on #1787 seems to work OK. But I'll keep testing it a little more.

#445 Updated by Eugenie Lyzenko about 7 years ago

Marius, Constantin,

I need some clarification for output redirection.

Do we need to check STDERR in addition to STDOUT to find out if the output is redirected? Is it possible to have STDERR redirected but STDOUT - not or STDOUT redirected and STDERR - not, these cases must be considered as redirected? Another word we are considering the output is redirected when at least one of the standard output streams is redirected, correct?

#446 Updated by Constantin Asofiei about 7 years ago

Eugenie Lyzenko wrote:

Do we need to check STDERR in addition to STDOUT to find out if the output is redirected?

No. I don't think 4GL even allows access to the STDERR (from 4GL code). Running a program in batch mode using pro -b -p run_batch.p 2> a.txt behaves the same way as there was no redirection. Only STDOUT redirection looks like is picked up.

#447 Updated by Marius Gligor about 7 years ago

No. it's not necessary to check STDERR redirection before redirect the STDOUT and vice versa.

All 4 cases are possible:

STDOUT and STDERR not redirected.
STDOUT and STDERR both redirected.
STDOUT redirected and STDERR not redirected.
STDOUT not redirected and STDERR redirected.

#448 Updated by Eugenie Lyzenko about 7 years ago

Running a program in batch mode using pro -b -p run_batch.p 2> a.txt behaves the same way as there was no redirection.

Thanks, this is exactly what I wanted to know. Because if this was not a true I had to add STDERR checking into native call that detect redirection.

#449 Updated by Constantin Asofiei about 7 years ago

Marius, something else to think about and add to your web client issue list: once we are redirected, there is nothing stopping us from copy-pasting the URL on another browser/tab/machine. This is also a security issue, because if someone guesses the port, it will be able to see that port's active web client without having to authenticate; I've just tested this on two machines:
- on machine A I've logged in and started the web client
- on machine B I've copy-pasted the URL and I can see the web client. After this, the terminal on machine A seems to freeze (as all refreshes are sent to machine B); but if keys are pressed - they will end up to machine B's screen.

We need to secure the redirected URL, so that only the authorized browser session can use it.

#450 Updated by Marius Gligor about 7 years ago

I'm trying to start a batch process. I did all configurations but when I started the batch client I got:

Error connecting to server
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

#451 Updated by Greg Shah about 7 years ago

Once a web socket is open, we shouldn't allow any other connections. Likewise, only the browser that initially is redirected should be allowed to connect in the first place. Perhaps we need to pass a token (a different one than we use between the server and P2J client) in the redirection URL?

#452 Updated by Marius Gligor about 7 years ago

I have to found a solution to set-up the embedded server on the client to accept only one connection.

#453 Updated by Constantin Asofiei about 7 years ago

Greg Shah wrote:

Perhaps we need to pass a token (a different one than we use between the server and P2J client) in the redirection URL?

Using a token is OK, but this one I think it needs to be generated by the JS side when the user enters the credentials and pushed to the server via POST. Both sides will save this toke and after redirect, each time the web socket sends or receives a request (on both P2J client and browser side), it must receive a token and reject any requests with an unmatched token.

This way, the P2J client can receive requests only from the authorized browser and viceversa.

#454 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

I'm trying to start a batch process. I did all configurations but when I started the batch client I got:

Error connecting to server
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Can you send me the directory.xml and srv-certs.store files you used?

#455 Updated by Constantin Asofiei about 7 years ago

I forgot to mention: remove all the security/keystore and access/password nodes from server.xml and client.xml.

#456 Updated by Constantin Asofiei about 7 years ago

See the 0225b.zip update from #2124: I forgot to change the code in ServerDriver.connect, for the case when the private keys are read from the directory.

Beside this, you need to add the security:truststore:alias=standard in the server.xml, so that the server connection is validated.

#457 Updated by Marius Gligor about 7 years ago

The certificate problem is now fixed. However I have another problem.
When I started the batch client an error message "Invalid user name." I found inside the server log file. This message comes from spawner.

#458 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

The certificate problem is now fixed. However I have another problem.
When I started the batch client an error message "Invalid user name." I found inside the server log file. This message comes from spawner.

Add a server/default/runtime/p2j_proc/clientConfig node with this content:

<node class="string" name="systemUser">
   <node-attribute name="value" value="mag"/>
</node>

#459 Updated by Marius Gligor about 7 years ago

Now the redirections are works but I still have an error. The batch process cannot be started.
Tomorrow I have to do more tests to find out why this happen.
The Batch.java is the result of batch.p conversion. Is the node "p2j_proc" properly configured?

<node class="container" name="p2j_proc">
  <node class="string" name="p2j-entry">
    <node-attribute name="value" value="com.goldencode.testcases.Batch"/>
  </node>
  <node class="integer" name="stop_disposition">
    <node-attribute name="value" value="2"/>
  </node>
  <node class="container" name="clientConfig">
    <node class="string" name="systemUser">
      <node-attribute name="value" value="mag"/>
    </node>
  </node>
</node>

For outputToFile I used a node like this:

<node class="string" name="outputToFile">
  <node-attribute name="value" value="client_%pid%_%appserver%_%userid%_%timestamp%.log"/>
</node>

The palce holders are replaced except appserver because I have no application server.
The pid is replaced inside the spawner code.

I attached also my changes.

#460 Updated by Constantin Asofiei about 7 years ago

Marius, the entry point for the process in directory.xml needs to end with .execute (i.e. the method which we need to run). This should get you past.

About 0225b.zip: your changes look good; see my comments bellow:
  • ClientBuilderParameters
    - you've replaced all "at" characters with "%"
  • ProcessBuilderOptions
    - line 115: you have a typo in "place holder" and some extra spaces on line 112
  • ProcessBuiderParameters
    - change the c'tor to accept a ProcessBuilderOptions instead of ClientBuilderOptions.
    - please centralize the logic of replacing the placeholders - move the %appserver% from ProcesSBuilderOptions c'tor to getOutputToFile.

#461 Updated by Marius Gligor about 7 years ago

The outputToFile parameter is an optional command line parameter. In the future it is possible to add more optional parameters.
In order to parse multiple optional parameters I think that a best idea is to send a pair of command line parameters like:
-O <outputToFile>
or a single argument like:
-O:<outputToFile>
What do you think about this?

#462 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

The outputToFile parameter is an optional command line parameter. In the future it is possible to add more optional parameters.
In order to parse multiple optional parameters I think that a best idea is to send a pair of command line parameters like:
-O <outputToFile>
or a single argument like:
-O:<outputToFile>
What do you think about this?

I like the idea, go ahead with it.

#463 Updated by Marius Gligor about 7 years ago

Here are the latest changes and the results of my tests with and without STDOUT redirected to a file.
The STDERR is always inherited from server.
I tested both 0 and 1 options for spawner and found to work properly.

#464 Updated by Constantin Asofiei about 7 years ago

Marius, the update is OK. Please put the ProcessBuilderOptions.getProcess together with the other public methods. You can go ahead with regression testing.

#465 Updated by Constantin Asofiei about 7 years ago

Marius, remember the stress drawing test on note 408? Well, I've ran it with your 0224a.zip update and it broke the browser. Why is that: although PushMessagesWorker sends the messages in sync, the browser receives (and interprets) them async. What I did to conclude this:
  • I've placed a breakpoint in the browser, tty.screen.js:375 - the drawText function
  • I've placed a breakpoint in PushMessagesWorker.run:114 and stepped over to send the message to the browser.
  • the browser was in breakpoint at drawText. Removed the breakpoint in PushMessagesWorker, let the thread run.
  • the browser started drawing other text, although I was still blocked in drawText for the very first message.

I know the program on note 408 is not some normal usage, but this might hurt us anyway. I'm not sure, but at some point there was some mentioning that the P2J client was waiting for the browser side to finish, before returning the control back to the P2J server side: did you change the sockets on the JS side to be async in the mean time?

#466 Updated by Greg Shah about 7 years ago

did you change the sockets on the JS side to be async in the mean time?

JS is single threaded (unless worker threads are used; in our case they aren't) BUT to eliminate the natural problems that occur due to the single threaded nature, it is highly asynchronous. Many APIs, including websockets are designed as a set of asynchronous callbacks. There is nothing we can do to set or not set this for websockets.

If I understand the problem, we just need to make sure to always apply the screen changes IN ORDER, right? So we just need some sequencing information passed with each message and then honor that sequencing info.

#467 Updated by Constantin Asofiei about 7 years ago

Greg Shah wrote:

If I understand the problem, we just need to make sure to always apply the screen changes IN ORDER, right? So we just need some sequencing information passed with each message and then honor that sequencing info.

Yes, this would solve the drawing order. We can use a priority queue on JS side to sort the requests in real-time, but we will need to ensure that JS waits for the next message (in sequence) to arrive, before pulling the next request from the queue. Also, if the JS is not processing the requests fast enough (i.e. consider my drawing stress test) and the queue fills up with lots of pending requests, can this destabilize the browser?

I wonder if is not better to synchronize the PushMessagesWorker with the JS side: wait for the JS to inform that it finished processing the request before sending the next one.

#468 Updated by Marius Gligor about 7 years ago

What you ask me to do I'm afraid that is impossible in JS.
Unlike Java the JS is a very simple programming language having a lot of limitations like:
- It's not OO oriented. The OO is simulated by a construction name closure.
- JS is a functional language and accept other functions as parameters for functions.
- JS is not multithread. We cannot create threads and synchronize the execution of threads.
- The JS engine is under the browser control and as developer we have no access to the core engine.
- JS is very slow and performances are accordingly. Google Chrome has the faster JS engine and it use a JIT compiler to obtain a better speed.

Please read:
http://www.sitepoint.com/multi-threading-javascript/

#469 Updated by Constantin Asofiei about 7 years ago

If we don't sequence the drawings, what I'm worried about is that the screen can end up in the wrong state: i.e. if message M1 needs to draw on the same area as message M2 and M1 is drawn after (or at the same time) as M2, then the screen will not be in the correct state. As I understand, the web socket onmessage is an event executed when the socket receives something, and this event is pushed to the main JS thread, to be executed, right? And this is why the browser keeps executing other incoming messages, even if I had a breakpoint for the first message: this suspends the execution of this JS code, and not the entire JS thread.

Anyway, what about the other approach I mentioned: is there any reason for the PushMessagesWorker not wait until is notified by the JS that it finished drawing?

#470 Updated by Greg Shah about 7 years ago

is there any reason for the PushMessagesWorker not wait until is notified by the JS that it finished drawing?

I don't think this is needed. It should be pretty easy to only apply the messages in order.

If a message comes in with an out-of order sequence number, then it can be added to a sparse array using the sequence number as the subscript.

When a message comes in that has the next expected sequence number, then this will be drawn AND we can continue drawing any already received messages that have contiguous following sequence numbers. This is easy to detect by subscripting the array. Of course, each drawn message is cleared from the array and as soon as the next expected sequence number is not found in the array, the drawing stops.

#471 Updated by Marius Gligor about 7 years ago

WebSockets is built on TCP. TCP guarantees delivery and ordering of packets. In addition, unlike TCP, WebSockets is message based which means that WebSocket messages are received as an entire message (TCP is streaming and 'messages' may get fragmented from the listener's perspective)

#472 Updated by Greg Shah about 7 years ago

TCP has nothing to do with it. I agree that the entire message will be received before it is delivered and if that message takes more than 1 TCP frame, those frames will be put together into the full message before delivery.

The problem is the asynchronous nature of JS. Just because the messages are sent (and presumably received via TCP) in a known order doesn't mean that the websockets callback function is called in the same order on the JS side. I'm guessing that the websockets messages are being delivered to our code out of order when there is network stress.

If that is not the cause of the problem, then please do debug into the stress test and find the real problem.

#473 Updated by Hynek Cihlar about 7 years ago

Also, if the JS is not processing the requests fast enough (i.e. consider my drawing stress test) and the >queue fills up with lots of pending requests, can this destabilize the browser?

If a message is not lost and JS will not process the incoming messages fast enough, your network buffers will eventually fill up which will cause TCP flow control to slow or stop the far end from writing more data to the socket.

#474 Updated by Marius Gligor about 7 years ago

Sorry I think that is my fault. I found a serious bug inside my code which is the cause of describe bevaviour.
By mistake the message for the entire screen is send on each line ;-( The send message remains inside the loop instead outside.
I did some tests and I added a sequence to the draw message.
Good new is that the messages are delivered in right order.
I have to prepare a package having my changes I will send to you.

#475 Updated by Marius Gligor about 7 years ago

Here are the changes containing the fixed bug. You can use JS console to see the order of the packages.
This part will be changed on rectangle implementation.
Obviously the drawing performances are also improved.
Please do some tests using this changes and let me know your results.

#476 Updated by Marius Gligor about 7 years ago

The package was send just for testing purposes. Please do not code review.

#477 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

Please do some tests using this changes and let me know your results.

I'm using Firefox 26.0 - for Ubuntu canonical 1.0. Although the update is an improvement, the drawings are still done async and the browser stops responding at some point (as I guess it can't keep up with the drawings). Have you ran the stress test? How does it behave for you?

You can use JS console to see the order of the packages.

OK, they seem to be in order, until I place a breakpoint in tty.socket.js:66, just at the start of the onmessage callback: messages still be received and processed, even if this one is paused. After I resume it, from the JS console:

08:50:21.354 "message: 10505" tty.socket.js:82
08:50:22.326 "message: 10506" tty.socket.js:82
08:50:22.330 "message: 10507" tty.socket.js:82
08:50:32.320 "message: 10329" tty.socket.js:82

-------

OK, I've just installed latest chrome and did some tests: the behaviour is a lot different, in a good way; the stress test is even faster than swing. The drawings are in sync and a breakpoint in onmessage suspends the entire JS thread, not just that message. So I guess what I saw in firefox is a difference in JS implementation between browsers.

#478 Updated by Marius Gligor about 7 years ago

Implements rectangle detection on screen cells that should be update on JS code.
With a few extra computation in Java rectangle detection is a step forward on improvements.
This time I tested carefully.

#479 Updated by Marius Gligor about 7 years ago

Implements STDOUT and STDERR redirection for spawner Ref. #2124
Passed regression tests. Committed to revision 10483

#480 Updated by Marius Gligor about 7 years ago

Here are my last cahanges after Majic tests on devsrv01.

- I added character \u2502 (|) to measure the height of the screen cell. The result is an improvement especially when Firefox is used since we provide more space between lines.
- I did some small changes to protect the canvas to became dirty when draw text lines.
- I added specific javadoc comments in all JS scripts.
- I added javadoc on ChuiWebSimulator.triggerRepaint method like follow:

When data is prepared to send down to browser JS code to be draw the data should be
optimized for drawing on canvas.
<p>
The video screen memory is composed form cells stored in a 2 dimensional matrix,
The size of the matrix is rows x columns where rows are the number of rows in the screen
and columns are the number of columns. For example 24 x 80 for a screen having 24 rows
and 80 columns. Each cell is a structure which holds cell attributes, color and character.
The status of video screen memory is cached and preserved the last screen status.
Whenever a new screen should be send down the new screen is compared with the cached screen
and only the changed cells are send down. Also the cached status is updated to reflect the
latest changes. The cell is considered changed when at least one of the cell properties
attribute, color or character had changed.
<p>
Drawing texts on HTML5 canvas using JS code is done on two steps:
<ul>
<li> First the background rectangle containing the text is filled with the background color.
The largest the filled rectangle is the drawing is optimal.

<li> The second step is to fill the rectangle with the containing text using the foreground
color. On HTML5 canvas this can be done only on a line basis by filling with text
rectangles having the height equals to one character. An optimized draw on this case
require a text length as much as possible long.
</ul>
<p>
When a network message is constructed this is done on two steps by keeping in mind the
performances constraints described above.
<ul>
<li> The new screen that have to be send is scanned line by line column by column to detect
changed cells. Adjacent changed cells on a line having the same color and attributes are
stored together on a Chunk structure. The chunks are stored on a linked map structure having
the cell Color (attribute and color) structure as a key and a linked list of chunks as value.
Both the linked map and the linked list preserve the inserting order.

<li> On the next step the message is build and contains all chunks from a screen detected on
the first step. For each entry on the map the list of chunks is used to build rectangle
structures. The chunks are extracted from the list one by one. If multiple chunks could be
combined on larger rectangles the chunks are combined to form a largest rectangle. The width
of a rectangle is the number of characters from a chunk and the height are the number of chunks
combined. For example a rectangle of 25x5 holds 5 lines of 25 text characters each.
All characters have the same color and text attributes. The condition for two chunks to form a
rectangle is to have the same length, to start at the same position within the line and are from
adjacent lines. Chunks that cannot be combined as large rectangles are stored inside the message
as simple rectangles, rectangles having height equals to one.
</ul>

#481 Updated by Marius Gligor about 7 years ago

Just a minor change. Use \u2501 (-) character to measure the width of a cell.
Character \u2501 should be the largest. However I observed that sometimes some dirty pixels still remains on the screen.
I observed that drawing character & on a 22 font size cause such kind of effects despite our clipping rectangles definition.
Nevertheless I have to do more tests and investigations.

#482 Updated by Constantin Asofiei about 7 years ago

Marius, the changes in 0228b.zip look good to me. Greg: please take a look too.

#483 Updated by Marius Gligor about 7 years ago

1. During this weekend I did a lot of tests on Linux and Windows OS. Defining clip rectangles when drawing text lines does not protect 100% against dirty screens.
Depending on the text pattern which is draw dirty screens could be seen.
In order to have 100% clean screens we have to protect the screen each time we draw a character in other words to draw the text line character by character.
This approach does not affect the drawing performances in a visible way, so the application remains fast and the screen remains always clean regarding the text pattern we draw on the canvas. Also for OS compatibility I used the \u2500 and \u2503 characters to measure the screen cell dimensions.

2. I started to fix also the security issue related to multiple access. The security issue is implemented at two levels.

a) On successful login before client redirection a cookie containing an authorization token is added to the response object.
A SHA message digest is generated from the random UUID and used as an authorization token.
On the client embedded server when the browser try to load the main page should present the cookie containing the authorization token.
If the cookie is missing an 404 error is returned.
This will protect the application when multiple accesses are requested from different instances of web browser/browsers.

b) The number of simultaneous web sockets connections is set to 1 by a simple logic implemented in ChuiWebSimulator using an integer count.
This protect the application against possible "attacks" comings from entities who stolen the authentication token.
Also this provide protection when the application is accessed within the same browser from other tab pages.

Although this works fine from 3-4 hours I'm struggle to find a solution for the following scenario:
The user is redirected and the Majic login plug-in is presented waiting to enter credentials.
If at this point we reload the page using the browser "Reload" button the screen is clear but the Majic plug-in never occur again!!!
After entering credentials and Majic conversion application is running using the browser "Reload" button works properly, the page is reloaded.

#484 Updated by Constantin Asofiei about 7 years ago

Marius Gligor wrote:

A SHA message digest is generated from the random UUID and used as an authorization token.

Is best to use some new random UUID, and not reuse the UUID from the P2J client.

b) The number of simultaneous web sockets connections is set to 1 by a simple logic implemented in ChuiWebSimulator using an integer count.

I don't think this is secure enough: after an onClose, until onConnect is called again, the connection is open for anyone to access! And if someone else steals the connection, with your current changes the authorized user can no longer get it back, until the unauthorized one disconnects. We need to authenticate each incoming web socket connection, not just limit the number of connections.

If at this point we reload the page using the browser "Reload" button the screen is clear but the Majic plug-in never occur again!!!

The login client is stuck in a WAIT-FOR loop, which doesn't refresh the screen anymore. If the browser is refreshed, there are no more screen draw events in PushMessagesWorker, so there is nothing to draw. In these cases, you need to determine that a browser page refresh is in progress and send the entire screen down to the browser.

After entering credentials and Majic conversion application is running using the browser "Reload" button works properly, the page is reloaded.

Main menu in MAJIC refreshes the screen every 5 (I think) seconds, to update an upper-right clock.

#485 Updated by Greg Shah about 7 years ago

Code Review 0303a

1. About the authorization approach:

I agree, we cannot reuse the spawn UUID. As I noted in note 453:

Perhaps we need to pass a token (a different one than we use between the server and P2J client) in the redirection URL?

Using a token is OK, but this one I think it needs to be generated by the JS side when the user enters the credentials and pushed to the server via POST. Both sides will save this toke and after redirect, each time the web socket sends or receives a request (on both P2J client and browser side), it must receive a token and reject any requests with an unmatched token.

Some important points:

  • Inserting the token in a URL is not the safest approach since this can be stored in the browser history which is not the best place for sensitive data.
  • If the JS side generates a unique token and passes it during the POST at the time of the LOGIN (chui_web.html), the question is: what is the best way to get access to it from the page to which we are redirected? The data cannot be embedded in that page, because then the first browser to load that page would work (but that isn't necessarily the same browser for the user that did the OS-level login, which is a massive security problem).
  • We can't use the "Web Storage" (the localStorage and sessionStorage) API because the login document (chui_web.html) will have a different origin than the index.html that will be returned from the client Jetty instance.
  • The use of cookies is likewise not possible because of the same origin issue. In addition, we want to avoid using cookies because they may be limited/blocked in many browsers. If we do have a need in the future, we can use the Web Storage API, which is well supported in recent browsers.
  • The reason I want to have the JS side generate this and POST it is because then the JS side will have the unique token and only that user's browser instance will be able to authenticate with the associated spawned jetty client.
  • There is a technique called "cross-document messaging" which is well supported in current browsers. You can invoke Window.postMessage() and communicate between 2 documents of different origin. Please look at this as a potential solution. My only concern is whether this will work in a redirect case since the login page (chui_web.html) will be replaced by the JS web client page (index.html) and the pages may not simultaneously exist to allow such a communication.

2. I'm still thinking about the PushMessagesWorker changes. Can you describe the reasons for the changes? I think the approach is probably OK.

3. Chunk.java has some methods that are missing javadoc (hashCode() and equals()).

#486 Updated by Marius Gligor about 7 years ago

1. Today I did many tests and investigations regarding the "Refresh" button on web browser so far unfortunately without success.
All that I found is that after the button is pressed the web socket connection is closed and the code code is 1001 0n Firerfox and 1005 on Chrome.
What's strange is that happen only when the p2j Majic plug-in refresh. After Majic login everything works fine.
I remember that some times ago I tested this issue and I found to works!

2. You suggest me to find another solution without using cookies. Here is a possible solution the we can try to implements on JS code. We have two web servers M (main) and C (client).

- The user access the main server M and get the login page as usual.
- Using a JS AJAX call send a POST message (submit the login form) containing the user name, password and a random generated token down to server M. We could use DOJO or jQuery to make AJAX calls.
- On successful login the target URL on server C (redirection url) is returned into the AJAX response. On error the error message is returned and displayed on the main page.
Also the authorization tag is exported to server C when the server key store is exported.
- The client is no longer redirected. The main page on server M contains also an iframe on which the target page from server C will be loaded.
Also is possible to open a new window (pop-up) to load the page from server C.
I found this useful article regarding cross domain messages [[http://blog.teamtreehouse.com/cross-domain-messaging-with-postmessage]]
Than using a cross domain communication postmessage the generated token is transferred to JS web sockets code from the main page.
After the token is received the web socket connection is open. On web socket open the token is send down to server C.
On server C if the token match the exported tag the web sockets session is used if not the session is closed.
If the page from server C is loaded directly the web sockets connection remains closed and will not affect the running code.

The problem regarding the browser refresh button remains as well. Please let me know your opinion.

#487 Updated by Greg Shah about 7 years ago

What's strange is that happen only when the p2j Majic plug-in refresh. After Majic login everything works fine.
I remember that some times ago I tested this issue and I found to works!

Did you previously test from the MAJIC login screen? The screen created by the authentication plugin is not a "full" client and there is no corresponding server-side converted code that is running at the time this is executed. This means that the "normal" environment is not yet fully functional and in fact, the user's P2J "session" is not yet fully created since this is the authentication step for the session setup.

Please review the code in aero.timco.majic.security.LoginClient.java (you can see it in your devsrv01 ~/testing/majic/src/ directory).

I think you will have to debug into this code to see how it is reacting to the refresh.

Please also try CTRL-C on this screen. This may also be instructive, however I presume that the issue is really about how the session management works when it is disconnected during authentication.

Here is a possible solution the we can try to implements on JS code. We have two web servers M (main) and C (client).

I like it. Using an iframe is a good solution (and actually, I like it better than the redirect solution). This should also make it easier to "go back" to the login page on any disconnect.

Of course, the login portion should be hidden and the iframe must display as the "content" of the page, once connected.

The problem regarding the browser refresh button remains as well. Please let me know your opinion.

Yes, I would expect this to be present in either approach.

#488 Updated by Greg Shah about 7 years ago

Also is possible to open a new window (pop-up) to load the page from server C.

Please avoid pop-ups. Just make the iframe visible and position it to be the primary content for the screen. The login form should be hidden.

#489 Updated by Marius Gligor about 7 years ago

1. I found and fixed the issue related to Majic login plug-in.
When Majic plug-in run within ThinClient using system logger cause application blocking.
For example on the following snippet from ChuiWebSimulator when the call LOG.logp is executed the application remains blocked forever!!!
As a result no other threads can acquire the lock on keyBuffer. More generally seems that calling static methods cause the same effect.
Writing simple messages to STDERR works. When I tried to build a message using a String.format this remains also blocked!!!
So I eliminated all log messages from all affected classes ChuiWebSimulator, PushMessagesWorker and WatchdogTimer.
For example with log messages in place the WatchdogTimer remains blocked and never force the client exit!
As a result now the Majic plug-in is loaded instantly on first load as well as on page refresh.
The WatchdogTimer and PushMessagesWorker are working properly.
I attached a version using a random generated token exported via server key store and used to set the cookie value.
The cookie is valid for a single client session only.
This is a complete solution using redirection and cookies for authorization.

@OnWebSocketClose
public void onClose(int statusCode, String reason)
{
  synchronized (keyBuffer)
  {
      if (--connections == 0)
      {
    // Stop push worker thread
    stopPushWorker();
    // Start watchdog timer
    startWatchdog(watchdogTimeout);

    // here remains blocked            
    LOG.logp(Level.INFO,
          "ChuiWebSimulator.onClose",
          "",
          LogHelper.generate("Connection closed status=%d reason=%s", statusCode, reason));
      }
  }
}

2. Regarding the JS solution I identified two major issues:

a) Supposing that client is authenticated and the application is loaded and running into the iframe.
When the user reload the page using browser button or the keyboard (F5) the application is gone away forever.
The main page is reloaded from main server M and the user should start a new session.
It is not possible to stop this user action on web browsers using JS.

b) The main server M became heavy loaded if all users are connected to this server.

#490 Updated by Greg Shah about 7 years ago

Code Review 0305a

1. In HtmlResourceHandler, in what case will isAuthorized() find that authorizationToken is null?

      // if authorization token is null don't check
      if (authorizationToken == null)
      {
         return true;
      }

This is an unsafe default behavior. The default behavior must always be to refuse access if things are not configured properly.

2. ChuiWebSimulator, PushMessagesWorker and WatchdogTimer classes need comments to explain that the logging should not be used and why this the case. The comments need to be in the places that can be affected.

3. In ChuiWebSimulator, you have added an improper usage of volatile.

   /** Keyboard input queue */
   private volatile Deque<Integer> keyBuffer = new ConcurrentLinkedDeque<>();

The construction of keyBuffer is only done during this line. That means that there is no case where keyBuffer will refer to an instance that can change at any time. The use of volatile here suggests that this is being made safer in some manner, but in fact it makes no difference since the reference itself is never going to change. Since the volatile property does not extend to the contents of the Deque and only affects the keyBuffer reference itself, this is confusing/dangerous. It may lead future readers to think it is safer than it is.

4. The ChuiWebSimulator.connections member has lost its javadoc.

5. WebHandler:

If the instanceof protection is really needed, then you will never get to that code because you will have already raised a ClassCastException:

         Object store = (ServerKeyStore) webSpawner.getServerData();   // will raise ClassCastException

         if (store instanceof ServerKeyStore)  // if you get here you already know that store is a ServerKeyStore
         {
            // Use UUID to build a SHA message digest as token. 
            Cookie cookie = new Cookie(WebHandler.AUTHORIZATION_TOKEN, 
                                       ((ServerKeyStore)store).getAuthorizationToken());
            // add cookie
            response.addCookie(cookie);            
         }

6. Please confirm my understanding of how this works:

a. the M server generates an authorization token, stores it in the specific ServerKeyStore instance for the new client
b. the M server spawns the new client, which reads the authorization token (which is unique to that specific client instance) and as part of the initialization of server C, it configures server C to expect that token
c. when server C is ready, the M server inserts this token as a cookie in the redirect response and redirects the original browser session to server C
d. the original browser will read the redirect response and this will generate a GET request to the server C
e. the GET request will have the cookie set EVEN THOUGH the request is going to a different origin (server C) than that in which the cookie was generated/set (server M)
f. the request seen by server C has the cookie set (if and only if the request was the redirected GET from the original browser) and the authorization passes

The part I am really unclear about is step (e). It seems to me that the cross-origin nature of the cookie should break this step.

Please explain exactly how the redirect is working such that the cross-origin cookie is OK.

7. I'm also unclear on how long the cookie will remain set for the resulting page loaded from server C. In other words, what is the scope of this cookie (when does it go away)?

#491 Updated by Greg Shah about 7 years ago

a) Supposing that client is authenticated and the application is loaded and running into the iframe.
When the user reload the page using browser button or the keyboard (F5) the application is gone away forever.
The main page is reloaded from main server M and the user should start a new session.
It is not possible to stop this user action on web browsers using JS.

I think it would be possible using Web Storage. We could easily store the authorization token and redirect url there which leads the page to immediately reload the iframe instead of making the form visible.

b) The main server M became heavy loaded if all users are connected to this server.

This one is definitely more concerning to me. I still wonder if we couldn't find a way to replace the current window's URL while setting the authorization token it that context. Once that is done, we don't have to be in an iframe environment anymore.

Anyway, if you have worked out all issues with the redirect/cookie implementation, we will go with that.

#492 Updated by Marius Gligor about 7 years ago

1. Done. HtmlResourceHandler the base class of ChuiWebPageHandler is used to load other pages like the admin page which does not require authorization.
I changed the implementation and I moved the authorization code only on ChuiWebPageHandler.

2. Logging use should be totally avoided anywhere inside this classes ChuiWebSimulator, PushMessagesWorker and WatchdogTimer.
So I added a comment to the class javadoc.

3. Done.
volatile keyword tell that the variable is accessed by multiple threads and has two effects:
- In Java read/write operations are atomic for reference variables and for most primitive variables except long and double.
Using volatile the read/write operations are atomic for all variables.
- The volatile also prevent variables to be cached by threads.

see http://docs.oracle.com/javase/tutorial/essential/concurrency/atomic.html

4. Done

5. Done.

6. This is almost exactly with a few remarks:

a. the M server generates an authorization token, stores it in the specific ServerKeyStore instance for the new client
b. the M server spawns the new client, which reads the authorization token (which is unique to that specific client instance) and as part of the initialization of server C, it configures server C to expect that token
c. when server C is ready, the M server inserts this token as a cookie in the redirect response and redirects the original browser session to server C
d. the original browser will read the redirect response and this will generate a GET request to the server C
e. the GET request will have the cookie set EVEN THOUGH the request is going to a different origin (server C) than that in which the cookie was generated/set (server M)

On the first redirection the origin is the server M but when the user reload the page by pressing refresh button the page is reloaded and the origin became C.
Unfortunately this differ on web browsers. Chrome keep the referrer to server M on all requests Firefox keep the referrer header to server C.
On both cases the authorization token should be presented by the web browser. Cookies was invented exactly to do that. The cookies are always presented into HTTP headers regarding referrer.
A possible improvement here might be:
- Accept the authorization key if the cookie token match and change dynamically the token and the cookie value.
Whenever the page is loaded and cookie token match the token on server C the authorization token is changed again.

f. the request seen by server C has the cookie set (if and only if the request was the redirected GET from the original browser) and the authorization passes

The part I am really unclear about is step (e). It seems to me that the cross-origin nature of the cookie should break this step.

Please explain exactly how the redirect is working such that the cross-origin cookie is OK.

7. I'm also unclear on how long the cookie will remain set for the resulting page loaded from server C. In other words, what is the scope of this cookie (when does it go away)?

The life of the cookie is the browser session. When the web browser is closed the cookie is destroyed.
On the embedded server the cookie remains until server shutdown, client exit.

8. In a production environment the JS files could be minified into a single file something like tty_min.js and on index.html use this file instead the separate JS files.
This make the code difficult to read and minimize the amount of data to be downloaded when the page is loaded.

#493 Updated by Greg Shah about 7 years ago

Code Review 0305b

1. In ChuiWebPageHandler, please use * for imports.

2. Please remove the reference to "Majic" in the javadoc for classes ChuiWebSimulator, PushMessagesWorker and WatchdogTimer. We must not have customer-specific information in our documentation for P2J. The plugin you are referencing is a generic feature that can be called an "authentication plugin that is used during session establishment with the P2J server".

Otherwise I am fine with the code. Is there anything left to do?

#494 Updated by Greg Shah about 7 years ago

If your testing is not showing any noticeable performance differences between Majic in the browser and Majic in the terminal, then you can stop your performance improvement work now.

8. In a production environment the JS files could be minified into a single file something like tty_min.js and on index.html use this file instead the separate JS files.
This make the code difficult to read and minimize the amount of data to be downloaded when the page is loaded.

Yes, we will keep obfuscation in mind for the future. I think we will leave this as is for now, especially since we will have many more changes to this area for GUI support. Getting GUI running is the top priority.

#495 Updated by Marius Gligor about 7 years ago

Here are the changes after your last code review.
In the mean time I prepared the testing environment by doing a complete build of Majic project on devsrv01 having the latest changes.
Using the jar files of this Majic converted release I did the final tests on my workstation and I found no problems.
I think that we could go forward and start the regression tests.

#496 Updated by Greg Shah about 7 years ago

I'm fine with the code in 0306a. Go ahead with runtime regression testing.

This should close off #2244, right?

Please work #2243 next and then #2247.

#497 Updated by Marius Gligor about 7 years ago

OK. I started the regression tests on devsrv01.
Regarding #2243 if I understood better we could encounter a problem.
I'm afraid that in Windows OS it's not possible to create a process on another account using only the user name.
We must provide both user name and password. Without credentials we can create processes only for the account on which the server (spawner) is running.

#498 Updated by Greg Shah about 7 years ago

We must provide both user name and password. Without credentials we can create processes only for the account on which the server (spawner) is running.

Nasty. After 3 decades it is sad to see that there is such poor support for non-interactive use cases. I guess we will have to store the OS password for such accounts in our directory. It should be optional (not needed on UNIX or Linux) and it should be stored with the account information.

Constantin: what do you think?

#499 Updated by Greg Shah about 7 years ago

Constantin: How does the 4GL handle this for appserver launching? Does it always assume everything runs in the same context (as the parent)?

#500 Updated by Constantin Asofiei about 7 years ago

Greg Shah wrote:

I guess we will have to store the OS password for such accounts in our directory.

This can work, but ProcessClientBuilder needs to be changed so that it will access spawn.c in "password-mode", if the account has a system password.

It should be optional (not needed on UNIX or Linux) and it should be stored with the account information.

I agree it should be optional, but I don't think is needed to be kept at the account. The system user is read from the i.e. runtime/default/runtime/<account-id>/clientConfig/systemUser path (for both user and process accounts). So, is best to keep it with the system user, in clientConfig/systemPassword node.

Constantin: How does the 4GL handle this for appserver launching? Does it always assume everything runs in the same context (as the parent)?

The appserver launching in 4GL can be setup to launch agents either in server's account or in a specific account (by setting the system username and password). This is done at the Broker. Our current model is OK, as the agents can be started in a specific account.

#501 Updated by Marius Gligor about 7 years ago

ChUI web client improvements ref. #2244
Passed regression tests. Committed revision 10487.

#502 Updated by Marius Gligor about 7 years ago

Here are the winspawn.c changes that I did in order to integrate the code from Linux spawn.c version.

1. In order to build spawn.exe you should do:

- Set JRE_HOME environment variable to a JRE installation. Example: JRE_HOME=F:\jdk1.7.0_21\jre
- Copy the file jvm.lib from JDK installation folder to MinGW lib folder. Example:
copy /B F:\jdk1.7.0_21\lib\jvm.lib C:\mingw64\x86_64-w64-mingw32\lib\ /B
- In order to build spawn.exe you could do: ant native

- When the spawn.exe runs jvm.dll file is loaded. Windows should search for jvm.dll in order to load the library.
The simple way to set the jvm.dll location is to add to PATH variable the path to jvm.dll file.
Example: If the file is located at F:\jdk1.7.0_21\jre\bin\server\ we have to add to PATH variable the file location. PATH=F:\jdk1.7.0_21\jre\bin\server;…

2. Changes from Linux/Unix version to Windows. I found a lot of differences between Linux and Windows described bellow.

- Unlike Linux on Windows we have to provide also a password when we spawn a process on another account.
I added a new node clientConfig/systemPassword to store the user password. The password is added to the list of arguments
only for agents (mode = 0) which is send down via a secured channel (RemoteObject). For Linux/Unix this argument is totally ignored.
On Windows if the password is present the agent will be spawn on the specified account (user name / password)
If password is missing the user name argument which is always present must be the user name of the system account on which
the server (spawn.exe) runs. In this case the client is spawn on the server account. If user name does not match the current user name
the spawn is refused. Because sending null or empty passwords are not enabled an empty password is send as and empty quoted string "".

3. Redirection of STDOUT has some major differences that we have to discuss.

- On Windows we cannot execute part of the client process from spawn code like in Linux. We have to redirect the STDOUT before to client spawn. 
At the spawn moment the PID of spawned client is unknown because will be assigned later by Windows, so I used the spawn PID to replace the pid. on SRDOUT file name.
- In order to redirect STDOUT we have to create the file and pass the file handle to replace the STDOUT.
At the moment of the file creation the client is not running, we are still on the spawn process.
If the client is spawned on a specified account other than server account, spawn process must have enough rights to create the redirection file inside
the working directory of the specified account.

- The ChUI batch driver extend ChUI console driver. ChUI console driver on Windows is a JNI implementation.
The JNI implementation on terminal_win.c use STDIN, STDOUT system files of the client process to manage the terminal screen.
Those handlers must be console handles.
If the STDIN or STDOUT are not console handlers due to a redirection to an ordinary file or pipe the terminl_win.c throw an exception. The ChUI batch client won't work.
I fixed this issue by doing some changes inside terminal_win.c which allow the code to always use console handlers. Now the STDOUT redirection is possible.
(see [[http://msdn.microsoft.com/en-us/library/windows/desktop/ms682075(v=vs.85).aspx]])
However unlike on Linux I saw that at the end of the batch execution a screen having the following prompt
"Procedure complete. Press space bar to continue." is shown and the batch process wait for a keyboard event.
In other words in order to terminate the batch process the user should press a key.
Also Even if the STDOUT is redirected I'm not sure if the batch messages are written to this file or on console.
For example the message "some-text" is written on the redirected file on Linux but on the screen in Windows and not inside the redirected file.
Using System.out statement from ClientDriver is written on the redirected file which prove that the redirection is in place.

#503 Updated by Marius Gligor about 7 years ago

Regarding to STDOUT redirection in terminal_win.c it seems that we have to do more changes here.
On init console we have to find if STDOUT is redirected. If STDOUT is redirected we have to keep both two handles for console and for redirected file.
Whenever a character is written on screen we have to write also the same character to redirected file or just only to redirected file.
Maybe other informations could be useful like checking if is a batch client or not.

#504 Updated by Marius Gligor about 7 years ago

On console initialization STDIN and STDOUT are checked for redirection.
If STDOUT is redirected the redirected handle is kept and the characters are written to redirected handle instead console handle.
However looking inside the P2J project the batch mode seems to be not yet implemented on Windows OS.
A batch process should not interact with user in any way. For example waiting for a key to end a batch process is not a good idea.

#505 Updated by Greg Shah about 7 years ago

It is my understanding that batch most is mostly implemented for Windows. I think an exception is the support for message box.

Eugenie?

#506 Updated by Eugenie Lyzenko about 7 years ago

It is my understanding that batch most is mostly implemented for Windows. I think an exception is the support for message box.

Eugenie?

The batch mode in Windows support is on the same level as for Linux. It is mandatory to set up the batch mode in client.sh/cmd:

swing="client:driver:type=chui_batch client:driver:background=true" 

Without this settings the batch mode will not work. And BTW the ThinClient has API call to check if it is running in batch mode and even if the batch is in background.

#507 Updated by Marius Gligor about 7 years ago

1. I'm using a command like: com.goldencode.p2j.main.ServerDriver -b p2j_proc
This is not a batch mode?

2. The redirection of STDOUT won't work without my changes. I'm not sure that my changes for redirection are enough or should be more changes.

#508 Updated by Marius Gligor about 7 years ago

Some time ago I worked to implements the STDOUT redirection in spawn.c for Linux/Unix OS (see note 434).
Now I'm working to do the same in winspawn.c for Widows OS.
The only problem is that redirection of STDOUT does not work on Windows OS. It works on Linux/Unix after the changes on spawn.c
I provided a possible solution at low level by doing some changes in terminal_win.c but I'm not sure this is OK.
This is the only problem from me regarding this issue.

#509 Updated by Eugenie Lyzenko about 7 years ago

1. I'm using a command like: com.goldencode.p2j.main.ServerDriver -b p2j_proc
This is not a batch mode?

Until the directory.xml contains client-mode-batch == true this is not a batch mode from clients point of view.

#510 Updated by Greg Shah about 7 years ago

There is no batch mode for the ServerDriver because there is no concept of a real server in the 4GL. All 4GL sessions are clients (appserver agents, webspeed agents, batch clients and interactive clients). Because there is no concept of a server (let alone a batch-mode server), we have no concept of this either.

When we start appserver agents or batch mode clients from the server, these clients are automatically started in batch mode. Batch mode means that the terminal input and output is largely nullified.

The appserver agents are also started in "background" mode, which means that the output (stdout) is redirected. This can be optionally added for batch clients.

#511 Updated by Marius Gligor about 7 years ago

OK. I understood. Is about appserver agents running in batch mode. The appserver agents implementation is to spawn clients in batch mode.
If the STDOUT is redirected appserver agents runs in "background mode".

At the time when I did changes in spawn.c I didn't do any other changes in screen drivers Java or native C code and the redirection for STDOUT works on Linux.
I need the same behaviour for Windows OS as well but the redirection of STDOUT is not yet implemented in screen drivers code for Windows OS.
Maybe I should check the existing code to see how the "background" mode is implemented on Linux OS and try to provide implementation for Windows OS.

#512 Updated by Greg Shah about 7 years ago

the redirection of STDOUT is not yet implemented in screen drivers code for Windows OS.
Maybe I should check the existing code to see how the "background" mode is implemented on Linux OS and try to provide implementation for Windows OS

I guess so.

I must be mistaken, because I thought that it was your STDOUT redirection for spawn that properly created the background mode for appserver clients.

#513 Updated by Marius Gligor about 7 years ago

The STDOUT redirection is implemented for Windows OS at high level but not at low level.
So my changes in terminal_win.c should remains because is the low level STDOUT redirection except the writing on the redirected handle.
I have to eliminate the writing on redirection handle inside terminal_win.c because this is already done at high level.
Normally should work without any other changes but I sow that in Windows does not work. Maybe is a configuration problem or something else.
I'm working to do some debugs to see what's happen when the appserver agent is spawn in Windows OS.

#514 Updated by Marius Gligor about 7 years ago

I found the problem. I fixed my bug and now it works properly. What was missing is the last argument client:driver:background=true.
On Windows when the client is spawn we have to provide a single command line string not an array of strings like on Unix.
On my method which build the command line the last argument was ignored due to a small bug.
Now the behaviour is the same on Windows and Linux / Unix.

#515 Updated by Greg Shah about 7 years ago

Code Review 0311b

The code changes make sense.

My only questions are in regard to the terminal_win.c change:

1. In what circumstances does this code get triggered? (why is it needed now when it wasn't before)

2. Does this have any implications when using the interactive client? (manually launching ClientDriver with the "Console" ChUI driver)

#516 Updated by Marius Gligor about 7 years ago

1. The code inside terminal_win.c is the native implementation of chui_native screen driver for Windows OS using console API.
GetStdHandle(STD_OUTPUT_HANDLE) returns process STDOUT handle. Console API works only if this handle is for a character device.
If the process SDTOUT is redirected to a file or a pipe the chui_native won't work.
Before fixing my bug regarding background agents the chui_console was spawned instead a background agent and because the STDOUT was redirected to a file the spawned client did not work. However after my bug has been fixed I realized that batch clients and background agents does not use the terminal_win.c code and the redirection works without my changes.
My changes make possible to always run chui_native regarding if STDOUT or STDIN is redirected or not.
Since the redirection of STDOUT or STDIN for chui_native clients make no sense I think that is better to keep the original terminal_win.c code which prohibits redirection.

#517 Updated by Greg Shah about 7 years ago

Since the redirection of STDOUT or STDIN for chui_native clients make no sense I think that is better to keep the original terminal_win.c code which prohibits redirection.

Agreed.

#518 Updated by Marius Gligor about 7 years ago

Today I designed and tested a simple algorithm in C code to encode/decode the password on Windows OS.
I think that is better to keep the password in an encoded form in directory instead a clear-text.
What do you think about this?

#519 Updated by Greg Shah about 7 years ago

Normally, I would absolutely want to do this.

Can the encoding be easily reversed? It seems like it must be reversible in order to decode it for Windows. So this can help obscure the plain text, but anyone with access will still be able to recover it, right?

We already have a great deal of sensitive data in the directory, such as our private encryption keys. This file must be tightly protected such that only system admins have access.

We need to balance the value of obscuring the password with the cost of requiring the admin to specially encode it before setting up the configuration. If this was truly encrypted and could not be recovered, then I would say we would definitely do it.

#520 Updated by Greg Shah about 7 years ago

Code Review 0312a

The code looks fine. What is left to do with this task?

#521 Updated by Marius Gligor about 7 years ago

Here is my implementation for password encryption.
For Windows OS we have to keep the systemPassword in clear-text if the appserver agents runs on an specific account.
I think that is better to keep the password in an encoded format rather then clear-text.
We can do that in Java code or in C code inside winspawner.c
Because Windows OS require clear-text password we have to encrypt and decrypt the password using the same key, a symmetrical algorithm.
The problem is where the key for encryption should be store.
Using a Java helper class to encode/decode the password is not the good solution because Java classes could be easily decompiled and the key is no longer secret.
So I implemented a simple encrypt/decrypt algorithm in C code inside the winspawn.c
The reverse engineering of winspawn.c is more difficult than a Java decompiler. Also the spawn is a private tool.

The algorithm use 3 hard coded keys to encrypt/decrypt the password.
I added a new option in command line used to find the encrypted value for a specified password.

spawn 9 &lt;password&gt;

This option will display the encoded value for password on STDOUT or on a file if redirected.

spawn 9 &lt;password&gt; 1>password.txt

Just copy and paste the encoded value and put on directory as a password value.
Let me know your opinion.

#522 Updated by Greg Shah about 7 years ago

I like the idea.

Some issues:

  1. We can't use encryption keys that are hard coded in our source code. We provide our source code to our customers and in the future we may even open source it. This means that the encryption keys are not very secret. Instead we would have to use keys provided by the admin. These keys could be in a file that is stored with spawn and which has permissions that disallow anyone but the admin from reading it.
  2. Please provide a plain-text mode such that if the admin doesn't care to use this feature, we can still implement the system.

#523 Updated by Marius Gligor about 7 years ago

The encryption use is now optional. By default no encryption is used just clear-text.
In order to use encryption a configuration file spawn.ini should be provided on the same folder where spawn.exe is located.
Inside the configuration file on the Encryption section the use of encryption could be enabled/disabled
Inside this file the encryptions keys are provided as well.
Whenever spawn is running this file is loaded. If file exists defined options are used.
If no such file is found no encryption is used just clear-text.

#524 Updated by Greg Shah about 7 years ago

Code Review 0312c

I'm fine with the code.

Is there anything left to do for #2243?

#525 Updated by Marius Gligor about 7 years ago

No. I have no other changes to do.

#526 Updated by Greg Shah about 7 years ago

OK, then go ahead and get this runtime regression tested.

#527 Updated by Marius Gligor about 7 years ago

0312c.zip Integrate spawn.c changes into winspawn.c ref. #2243
Passed regression tests. Committed revision 10492.

#528 Updated by Marius Gligor about 7 years ago

1. The interactive client on Linux/Unix could be implemented using a pseudo terminal.
A pseudo terminal (PTY) is assigned to interactive process during creation.
The parent process will use master channel of PTY to communicate with child process connected to slave channel.
On the parent side we have to emulate a physical terminal like VT 100. The terminal emulator is the key
part of this system and should provide a full implementation of VT 100 specifications. The emulator
provide also rendering services for SWING or JS virtual terminals. Key strokes from keyboard are pushed downstream to child
process via master/slave channel. The outputs of the child process are pulled upstream via slave/master channel
processed by terminal emulator and finally rendered on the screen by SWING or JS virtual terminals.
The scheme bellow depict the communication chains. Also I posted some useful links related to this topic.

http://rachid.koucha.free.fr/tech_corner/pty_pdip.html
http://man7.org/linux/man-pages/man7/pty.7.html
http://man7.org/linux/man-pages/man1/screen.1.html

|--------------|  Key stroke   |---------------|         
|              |-------------->|               |
| JS render    |               |               |           Pseudo terminal PTY          |-----------------------|
|              |<--------------|    VT 100     |   OUT   |---------------------|   IN   |                       |
|--------------|  Screen cells |    xterm      |-------->|          |          |------->|    child process      |       
                               |               |    IN   |  master  |  slave   |  OUT   |     (os-command)      |
|--------------|  Key stroke   |   emulator    |<--------|          |          |<-------|                       |
|              |-------------->|               |         |---------------------|        |                       |
| SWING render |               |               |                                        |-----------------------|
|              |<--------------|               |         
|--------------|  Screen cells |---------------|

<----------------------- P2J client (parent) -----------------------><------- interactive (child) process ------>
<-------------- Java/JS code ------------------><---- JNI C code --->

2. VT 100 terminal emulator is the core part of interactive process implementation.
Besides processing the specific commands for a VT 100 terminal the emulator should manage a lot of screen buffers.
Imagine an interactive process on which the user open the nano or vi text editor in order to edit a file having hundreds of text lines.
At one moment only one screen buffer is displayed on screen inside a scrollable viewport.

|---------------------------------------|
| Screen buffers                        |
|                                       |
|                                       |
|                                       |
|                                       |
|                                       |
|---------------------------------------|
| Scrollable viewort                    |
|                                       |
|                                       |
|---------------------------------------|
|                                       |
|                                       |
|                                       |
|                                       |
|                                       |
|                                       |
|                                       |
|---------------------------------------|

The VT 100 terminal emulator should be a Java implementation. Searching on Internet I found two implementations in two different projects.
Bellow are the links to those projects.

http://www.jcraft.com/jcterm/
https://code.google.com/p/zterm/

3. On Windows OS no pseudo terminals are available. However using anonymous or named pipes something similar might be implemented.
Instead to use a PTY pair channels we have to redirect the interactive process standard files to a set of pipes which allow to communicate between
the parent and the child process.
The result is the input and output streams capture of the child process.
This kind of input/output capture of a process is widely used in IDE's. For example Eclipse IDE use java.exe command to compile
sources inside IDE. The input and output of java.exe are captured and displayed inside the IDE console view.
It's not usual to have text editors like vi or nano running on a console window.
However a nano version for Windows (http://www.nano-editor.org/download.php) is available.
For this kind of applications we couldn't capture the input and output streams.
On my last worked issue I just found that such kind of applications MUST use console handles and not pipe or file handles.

#529 Updated by Greg Shah about 7 years ago

A couple of quick points:

The VT 100 terminal emulator should be a Java implementation. Searching on Internet I found two implementations in two different projects.

For your information, we already use JTA (and JSch) for this purpose in our Harness project (the regression testing automation toolset). JTA can be found here:

http://www.javatelnet.org/space/start

The problem with it is that it is GPL, so we can't use it in P2J. Zterm is also GPL so we can't use that.

I am interested in how JCTerm will work. From a license perspective, it is fine because it is LGPL (and the JSch dependency is BSD).

On Windows OS no pseudo terminals are available. However using anonymous or named pipes something similar might be implemented.

Before analyzing the Windows solution, please do some tests to see what kind of ChUI support the 4GL provides on Windows. Interactive child process support may be different on Windows. In other words: does the child process really take over the console on Windows OR does it launch a separate "windows" with a new console OR do something else?

#530 Updated by Marius Gligor about 7 years ago

None of the existing Java VT 100 emulators can be used as they are.
We have to create a derivative work if license allow or just use as examples.
Maybe a good idea is to create our own VT 100 emulator. I have VT 100 specifications.

#531 Updated by Greg Shah about 7 years ago

None of the existing Java VT 100 emulators can be used as they are.

Please provide more details.

  • What are the problems?
  • What would we have to do to make them usable?

Maybe a good idea is to create our own VT 100 emulator. I have VT 100 specifications.

I am open to this idea. But first I need to understand the issues with the existing options.

If we were to go down this path, what would you estimate as the effort?

#532 Updated by Marius Gligor about 7 years ago

The attached examples are form projects which implements SSH standalone clients.
Those projects use the same interprocess communication via a PTY and need a terminal emulator.
We cannot use any of those products as jar dependencies because we need only a VT 100 emulator which should be integrated in our P2J design for interactive client.

Our interactive client need only a terminal emulator not an entire standalone SSH client.
The VT 100 implementation in JCTerm has an ugly code inside and VT 100 emulation is incomplete as is described on the product web site
http://www.jcraft.com/jcterm

We have to design our specific interfaces for P2J interactive client.
The VT 100 emulator could be designed by getting some code from JCTerm project and adapt to our needs.
Also we could provide the missing implementation from VT100 specifications.
This could take 3-4 days work efforts.

#533 Updated by Greg Shah about 7 years ago

The VT 100 emulator could be designed by getting some code from JCTerm project and adapt to our needs.
Also we could provide the missing implementation from VT100 specifications.

I don't want to mix code into our project from other projects, especially since it is licensed differently. Even with an LGPL license, any code that we merge together with that code would have to be handled differently (and open sourced). I don't have a problem contributing our code back to a larger project and/or open sourcing a VT100 terminal implementation of our own. But since we need such a tight coupling between the terminal code and our client, we can't mix that code together.

Without taking code from another project, what is the effort estimate?

In addition, I have these other questions:

1. Are there any native solutions that can be used as a library by our JNI code to handle this terminal emulation for us? Does Konsole (and the other native Linux solutions like Gnome Terminal) not have a library for terminal emulation?

2. What does screen do to handle this? Does screen implement terminal emulation?

#534 Updated by Marius Gligor about 7 years ago

1. I don't know I have to check if such libraries exists.

2. Screen is writen in C and YES he provide also a VT 100 terminal emulation but the code is designed to be integrated into the screen project which is a complex one. I attached the ansi.c file from screen project which
contains the VT 100 emulation just to see how it looks.

3. A Java design is the best solution in my opinion. Finaly we have to design also renderers objects for this emulator for both Swing and Web Sockets clients. Than we have to integrate this parts with the existing native code which launch the interactive client. So far such a native code exists only for chui console client. The design of a VT 100 emulator from scratch could take 4 to 7 days but I cannot estimate the entire work effort for interactive client implementation.

#535 Updated by Marius Gligor about 7 years ago

I didn't found any native library that could be used in our P2J interactive client.
Each application which need a terminal emulator has designed their own solution.

#536 Updated by Greg Shah about 7 years ago

OK, go ahead and write our own vt100 emulation layer, from scratch. Of course, it is OK to look at the other projects to see how to do things, but just don't copy any code.

Keep it as simple and clean as possible. Also: keep it as separate as possible from the rest of the client code. Design an interface that allows its usage without deeply coding linkages into its internals.

#537 Updated by Marius Gligor about 7 years ago

Here is the RC (release candidate) of VT-100 emulator. It is an implementation of DEC VT 100 video terminal specifications.
The Swing renderer part looks like P2J ChuiSimulator implementation.
Starting from this design we have to integrate this standalone interactive client into the P2J Swing and Web clients.

How to do tests:
In order to see how it works you should compile both Java and native artefacts. The native code is loaded from libp2j.so
After P2J build you have to start Konsole.java as a Java application. You should get a Swing frame and a $ sh shell prompt.
Than you could type any command like bash, ssh, mc, nano, vi etc exactly like on Ubuntu Konsole application.

If you want to compare how commands runs on Ubuntu console and on VT-100 emulator you should do:

1. Open a console window on Ubuntu (KUbuntu)
2. Set terminal type to VT100: export TERM=vt100
3. Start the Java emulator test application Konsole.java as a Java application. Don't forget to specify the native lib location using the JVM argument -Djava.library.path=
4. Now you could execute and compare the same commands on both Ubuntu Konsole and on VT100 emulator.

You may now compare the execution for different commands on Linux console and on our VT-100 emulator.

#538 Updated by Marius Gligor about 7 years ago

I fixed a minor bug when using colors.
We have to add 1 to FG and BG because according to the current P2J implementation color 0 is not used.
Colors indexes are from 1...8 not from 0...7

#539 Updated by Marius Gligor about 7 years ago

Add code to synchronize with child process exit.

#540 Updated by Greg Shah about 7 years ago

I'm struggle to understand the current implementation on interactive process for native ChUI client.

We don't need to integrate it for the native client because the support is already inherently there. By definition, the native client is directly connected with a terminal emulator to the Linux/UNIX system. Our native code just has to "get out of the way" ("suspend") while the child process takes over control of the terminal. When the process ends, we "resume" and use the terminal ourselves.

Only the Swing and AJAX clients need this extra facility.

#541 Updated by Marius Gligor about 7 years ago

I implemented the interactive process for Swing ChUI client.
When a Linux OS command is executed from the converted application the shell command is executed on a modal dialog using the VT-100 emulator.
In order to resume application the user must close the dialog by executing an exit command or just closing the dialog.

#542 Updated by Greg Shah about 7 years ago

Please document how interactive child process support works on Windows (in the 4GL). When you are running a ChUI 4GL session, does the same feature work there?

#543 Updated by Greg Shah about 7 years ago

Code Review 0326a - Part 1

I am splitting this into multiple parts because I don't have time to finish the entire code review in one sitting.

1. For any new JNI interface, we implement it with the following standard approach:

  • there is a single file for the JNI API itself, which is cross-platform (e.g. filesys.c); this file hides all the JNI nastiness AND it must be written with no platform-specific code
  • this common file is what is added to the standard list of files for javah (in build.xml) and in the SOURCES= line (in makefile)
  • all exported entry points in the library must be in common code
  • if the common functionality is dependent upon platform-specific implementation code, then we design an internal-use API that will be called from the common code; this API is defined in a .h file (e.g. filesys.h)
  • then we implement a platform-specific version of this API for each platform (today we only support Linux/UNIX and Windows)
  • these platform-specific files are stored in files that are denoted by their target system (e.g. filesys_linux.c for the Linux/UNIX code and filesys_win.c for Windows)
  • platform-specific code does not ever get added to the SOURCES= line in the makefile
  • platform-specific code does not use any of the JNI features (calling up to Java instance or static methods, reading/writing arrays in the Java heap, reading/writing data members of Java classes...); the API is designed only using datatypes that are available on a cross-platform basis
  • architecture and word-size differences are hidden using makefile defines and #ifdef conditional preprocessing
  • Windows code must be compilable with MinGW and can use both WIN32 API calls and C runtime functions that are available in MSVCRT
  • any common code should only use C runtime functions that are available in both the glibc and MSVCRT

2. Are process IDs always only 32-bit?

3. In the native method shell(), why are you calling GetIntField() before setting those fields? The value is not being used so this seems like an unnecessary round trip through the JNI layer (which can be costly).

#544 Updated by Greg Shah about 7 years ago

Code Review 0326a - Part 2

I have finished going through all the code. It is really impressive.

4. It seems that there may be some missing parts of the VT100 emulator support. For example, start/end alternate character set support. I don't have any idea how important the missing pieces are. Please describe the limitations of the emulator.

5. The primary thing I'd like you to try is to eliminate the separate Swing window (VT100SwingRenderer/VT100SwingKeyboard). There are 2 problems with the current approach:

  • It only works for the Swing driver and not for the AJAX ChUI driver.
  • It pops up a dialog which is a different behavior than the user's normally see. In the native client, everything is in the same emulator window since the entire app is executing remotely and everything is being pushed down to the emulator.

With the AJAX client we will want to fix this issue anyway. The idea is to map the rendering/key processing to the driver's standard interface instead of having separate code for this. I realize that the current interface may need changes to enable this. That is OK, but please do minimize any changes.

Much of the VT100SwingRenderer looks just like the code in the ChuiSimulator, so I am hopeful that the basic implementation is not far off, even though it may be tricky to figure out.

The key point here is to shift into a special mode where we use the ChUI driver code for rendering and key processing, while suspending normal 4GL processing. We already have the suspend/resume hooks implemented to the driver level (see ProcessDaemon.launch()) to enable this aspect of the solution. This was going to be needed for supporting the AJAX ChUI driver anyway, so it seems that so long as we make the same changes to the driver API for the Swing client, that it will work there too.

6. Please move the terminal package to com/goldencode/ instead of com/goldencode/p2j/. This code seems to be generic (not P2J-specific) AND we will probably want to use this code in the Harness in the future (replacing our use of JTA).

#545 Updated by Marius Gligor about 7 years ago

This archive contains my later changes for interactive process implementation.
It is not the final release is only an intermediate work and is posted just to accompany my remarks.
However it works as was initially designed on a separate window.

1. I implemented almost all DEC VT-100 specifications except those related to terminal hardware.
For example start/end alternate character set support selects the character set from an EPROM inside terminal.
Also some commands are used by service personnel to calibrate the VT-100 terminal.
This kind of commands cannot be implemented and are not used by applications.
For command line applications like sh, bash, ssh VT-100 terminal features are not used,this kind of applications are running on row mode only.
Other applications like mc, nano, vi are using special VT-100 features because they are working on screen mode.
According to my tests all this applications works properly.
The VT-100 is the most simple terminal that can be emulated. The limitations of VT-100 are 24 rows, 80 columns, 8 colors etc.

2. I implemented the interactive process according to P4G OS-COMMAND options SILENT and NO-WAIT.
Using terminal.suspend() on Swing screen driver never returns so I used another way to implements WAIT (see ProcessDaemon.lunch).

3. I implemented also the interactive process for Windows OS as well. Unlike on Linux the Windows shell cmd.exe INPUT/OUTPUT cannot
be captured because cmd.exe use internal console handles which cannot be controlled from outside.
However simple commands like dir, copy works fine. Also I found a small Windows application named choice which display
a prompt and wait for an user input. This kind of commands are working in interactive mode.
Another issue on Windows is related to pipes. Read and write operations on pipes are blocked operations.
Here I have to do a small workaround inside the native code to cancel pipe read operation when interactive process exit.

4. Next starting from the already designed and implemented code I have to do other changes to integrate the interactive process
with the existing Swing and Web client code as you suggested on the previous code review.

#546 Updated by Marius Gligor about 7 years ago

The interactive process implementation fo0r ChUI Swing and ChUI Web (AJAX) clients with minimum changes on the existing code.

#547 Updated by Marius Gligor almost 7 years ago

1. Use ansi STARTUPINFO on shell_win.c
2. Merge changes on build.xml.

#548 Updated by Greg Shah almost 7 years ago

Code Review 0417a

1. Java_com_goldencode_terminal_NativePty_shell() in shell.c has some possible resource leaks:

  • The pointers created with strdup() are never freed.
  • The terminal type string pointer that is obtained using GetStringUTFChars is not released if the code returns on line 73. This is most likely an OOM situation, but if the JVM recovers, there will be a remaining reference on the heap that can never be garbage collected.

2. Is the shell_win.c implementation compatible with how the 4GL supports this feature? In other words, is this really how Progress does it too? Are there any limitations or open issues?

3. The 8192 command line limit is one that is enforced by the Windows CMD.EXE shell itself. But CreateProcess() has a 32K command line limit. We use CreateProcess so the limit could be higher. Does the 4GL limit the command line to 8192? If so, I wonder if it is using CMD.EXE in some way.

4. Have you tested what happens (on both Linux and Windows) when P2J is in wait_pid() and the child process is killed?

5. Have you tested what happens (on both Linux and Windows) when P2J is in wait_pid() and the P2J process is killed?

6. I don't think you are properly duplicating the logic of the ProcessDaemon.launch() else block on line 191. The following seem to be missing:

  • An ERROR condition should be raised if the child process can't be launched.
  • Avoids zombie processes when wait false by still calling LaunchManager.pseudoTerminalWait(pid) via an asynchronous thread. This is needed on UNIX/Linux to ensure that the child process' resources get reaped when it dies. Your code does seem to unconditionally call wait_pid (MasterPty.await() called from MasterPty.Runnable.run()) but I worry about the implications of always doing this on a separate thread. What exactly is the original client thread doing while the child process is executing?
  • On UNIX/Linux in silent false mode the ThinClient.suspendedPause() must be called when the child process exits. The current approach in MasterPty.Runnable.run() does not duplicate this sufficiently. The ThinClient code has some important features like proper condition processing, which cannot (and should not) be handled in the terminal package.
  • ConditionExceptions should be propagated to the caller, when generated. Note that they can be generated in ThinClient.suspendedPause() by user input.
  • Other exceptions are silently eaten/ignored. You may already be handling this OK, but it is hard to see exactly how this may or may not work because of a lack of exception handling in key places like MasterPty.Runnable.run().

7. I would like to eliminate our Swing dependencies in com.goldencode.terminal. For example, I don't see why we need the Swing parts of VT100SwingRenderer. It seems to me that the ChuiSimulator should already be able to handle all the paintComponent() needs by just calling getScreenData() just like the web client does. This just makes the solution more complex, it duplicates code and pulls in Swing dependencies for the terminal package that seem to be out of place. The terminal package should be purely about launching the child process, attaching to it and providing an API for reading/writing keys and screens.

8. In line with number 7 above, the code in com.goldencode.terminal is still too tightly coupled to the com.goldencode.p2j.ui packages. First of all, there should be no imports of com.goldencode.p2j (or its sub-packages like .ui) from com.goldencode.terminal. If there are some data structures like Cell that more properly belong in the terminal package, then we can move them. But if there are just integration points for call backs, then such APIs must be abstracted and provided as an interface that can be implemented in the com.goldencode.p2j.ui packages and "client" code in those packages can register instances of themselves for call backs. The idea is to make the terminal package completely standalone. The only exception at this time is the dependency on libp2j. That can be left in place. But otherwise, the package should stand on its own.

9. There is missing javadoc in VT100Emulator.run().

#549 Updated by Marius Gligor almost 7 years ago

I have no idea how Progress 4GL runs on Windows OS.
So far I have access only on the customer server but the Progress 4GL is running on Linux using a ChUI interface not on Windows.

1. How could I have access to a Progress 4GL for Windows environment?

2. When I designed the interactive process for Windows OS I tried to implements something like on Linux OS.
Looking at current implementation for native client on Windows OS the interactive process seems to run on a separate window.
However for our web client the interactive process should always use the browser for input/output (keyboard/screen) on both Windows or Linux OS.
How we can solve this issue? Please let me know if you have any idea?

3. The interactive process always execute an OS shell command: "sh -c" on Linux and "cmd.exe /c" on Windows. (see ProcessDaemon.prepareCommandLine)

#550 Updated by Greg Shah almost 7 years ago

I have no idea how Progress 4GL runs on Windows OS.
So far I have access only on the customer server but the Progress 4GL is running on Linux using a ChUI interface not on Windows.

Uh oh. Don't you have the getting_started_with_p2j_development.html document (should have been provided on Sept 29, 2013)? That document describes the details regarding access to the customer's Windows 4GL Development Box (WINDEV01). You have full access to this system to run 4GL code. I have been assuming that all your Windows testing of 4GL behavior has been done on that system.

Now I am a bit worried about the Windows interactive child process code. It may not be compatible with the real 4GL behavior. Please connect to that system and do some tests to confirm how things work.

Looking at current implementation for native client on Windows OS the interactive process seems to run on a separate window.

Please check the real Windows 4GL behavior and document it here.

#551 Updated by Marius Gligor almost 7 years ago

Sorry I have this file on my machine but I cannot find any instructions regarding the customer's Windows 4GL Development Box.

#552 Updated by Greg Shah almost 7 years ago

Please see the Test and Development Systems section.

#553 Updated by Marius Gligor almost 7 years ago

Here are the changes for Swing and AJAX interactive process implementation as a result of your first code review.

1. The com.goldencode.terminal package is now more abstract. No longer contains dependencies from other P2J packages.
2. When the child process is killed "by hand" the behaviour is like a process exit. (see attached picture for an OS-COMMAND statement with wait and no silent options)
3. When P2J process is killed by hand or normal terminated its child processes are also terminated.

#554 Updated by Greg Shah almost 7 years ago

Code Review 0424a

This is a very good revision. It is almost there.

1. It looks to me like the majority of the VT100SwingRenderer is generic. I'd like you to consider if most of this class can be made into an abstract base class that exists in com.goldencode.terminal. Then you can implement the minimum P2J-specific portion in com.goldencode.p2j.ui.client.chui.driver.swing. The key idea here is that much of this logic really does belong in the terminal package (because other users of that package would have to duplicate the logic). We can leave a smaller abstract set of functions that are specific to the use case.

2. The onTerminalExit() call on Windows platforms is not supposed to display the message or pause in non-silent mode. See ProcessDaemon:

            // if not silent we display a default message and pause indefinitely
            if (!silent && !PlatformHelper.isUnderWindowsFamily())
            {
               lastkey = terminal.suspendedPause();
            }

This is something EVL added after testing on Windows.

#555 Updated by Marius Gligor almost 7 years ago

A. I created a generic abstract class for VT100 renderer.

B. Regarding Windows OS and AJAX interactive client:

1. In Windows OS the P2J native implementation of interactive client spawn the OS command on a separate console widow. The executed command looks like: cmd.exe /c <command>
On this case is no need to call ThinClient.suspendedPause() because the user can see the results of the command execution on the interactive process window.

2. For Swing clients we use the same window for both P2J client and interactive client. This is default behaviour on Linux/Unix OS.
Using the same window for P2J client and interactive client is better to use ThinClient.suspendedPause() which allow the user to inspect the results after command execution before switch back to P2J screen.

3. For AJAX clients which are in fact remote clients we cannot spawn the interactive process on a separate window. We have to capture the input/output of interactive process and send over the network to the remote JS client running on the web browser. However we have some restrictions here. Supposing that P2J client (embedded server) is running on machine A and the user browser in on machine B if the user spawn a command like konsole on Linux or notepad.exe on Windows the processes are spawned on machine A and the user cannot interact with the spawned processes! On AJAX clients only commands that runs on the interactive window like ls, dir, vi, nano, mc, ssh etc. could be used.

4. Another important limitation in Windows OS is related to capture the input/output of a process. This is possible an almost all processes except some processes which use console I/O.
For example cmd.exe cannot be executed as an interactive client because we cannot control the STDIN for this process.
In other words according to paragraph #1 spawning something like: "cmd.exe /c" or "cmd.exe /c cmd.exe" will not work.
However spawning something like "cmd.exe /c dir *.tmp" works fine.

#556 Updated by Marius Gligor almost 7 years ago

I did more researches on Windows OS regarding the console standard files redirection.
I have good news. Finally I succeeded to make cmd.exe interactive.
I found 2 issues on Windows:

- We should send \r\n at the end of the line instead \n like on Linux
- In windows we have receive no echo when we send a character to STDIN

Now I'm working to fix also this issue for Windows OS interactive process.

#557 Updated by Greg Shah almost 7 years ago

Code Review 0425a

1. I really like the new VT100AbstractRenderer. The only change needed is to put the private methods at the bottom of the class (below the protected ones) and move all public methods to the top.

2. The VT100SwingRenderer is nice and clean but it needs the protected methods moved to the bottom.

#558 Updated by Greg Shah almost 7 years ago

1. In Windows OS the P2J native implementation of interactive client spawn the OS command on a separate console widow. The executed command looks like: cmd.exe /c <command>
On this case is no need to call ThinClient.suspendedPause() because the user can see the results of the command execution on the interactive process window.

Whether this is OK or not depends upon how the 4GL works on Windows. I hope that the recently sent customer document provides enough information to access the Windows desktop via ICA. From there you can run the 4GL just like a Windows user would. Please note that there is both a GUI and ChUI (console) version of the 4GL on Windows. For this purpose, you must run the ChUI version. We must launch child processes just like that version. If it is in a separate window, then we may be fine. If it is in the same console, then I think we will need changes.

Using the same window for P2J client and interactive client is better to use ThinClient.suspendedPause() which allow the user to inspect the results after command execution before switch back to P2J screen.

Yes, the message and pause are required to get the compatible implementation.

Using the same window for P2J client and interactive client is better to use ThinClient.suspendedPause() which allow the user to inspect the results after command execution before switch back to P2J screen.

In Linux, Konsole is an X application (GUI) and on Windows notepad.exe is a WIN32 GUI app. We are only supporting ChUI/console apps as interactive child processes.

On AJAX clients only commands that runs on the interactive window like ls, dir, vi, nano, mc, ssh etc. could be used.

Yes, this is understood.

#559 Updated by Marius Gligor almost 7 years ago

A.) Today I succeeded to connect to windev01 via remote desktop connection.
I did some tests related to implementation for OS-COMMAND in Windows OS.
Here are my conclusions:

1. OS-COMMAND is always executed in Progress 4 GL for Windows OS by opening a separate console windows applications running the Windows shell cmd.exe

2. With default options SILENT=false NO-WAIT=false the cmd.exe is executed on interactive mode: cmd.exe /K <command>

3. On all other cases the cmd.exe is executed on non interactive mode: cmd.exe /C <command>

4. No message "Press space bar to continue." is displayed on all cases. This behaviour is available only on Linux.

B.) I changed my code according to this behaviour for Windows OS.

1. Swing clients the OS-COMMAND is always executed on a separate native console window in interactive / non interactive mode depending on options.

2. For web (AJAX) clients I tried to execute the cmd.exe inside the JS canvas like on Linux using native code by redirecting the STDIN, STDOUT and STDERR to anonymous pipes.
Unfortunately the cmd.exe on Windows cannot be executed properly on interactive mode because cmd.exe doesn't use STDIN for input.
When the STDIN is redirected to a pipe cmd.exe execute the command from pipe only after we write a line feed code (0x0a) on the pipe which is done when the ENTER key is pressed.
No echo characters are send during the write of other keystrokes on the pipe.
After the ENTER key is pressed the STDIN pipe is read and the command line is displayed on the screen and executed. If /Q (echo off) is used no echo is displayed.

I found this article http://support.microsoft.com/kb/q190351 from Microsoft which state:

Note Some console based applications do not use the standard handles for their input/output (IO) operations.
The Win32 API does not support redirection of these processes.

I tried many other ways to inject keystrokes inside cmd.exe console but with no success. I searched on the net for a possible solution but I didn't found any hint.
In conclusion at this moment the OS-COMMAND for AJAX clients in Windows works properly only for commands which use STDIN as input.
For applications using Console I/O like cmd.exe it works fine only in non interactive mode.

#560 Updated by Greg Shah almost 7 years ago

Code Review 0428a

1. In VT100SwingKeyboard and ChuiScreenDriver, please only query PlatformHelper.isUnderWindowsFamily() once (at construction) and save that value. It won't change during the lifetime of the instance.

2. In ChuiScreenDriver, is there a reason to explicitly import PlatformHelper?

3. command_line() in shell_win.c can overflow the cmd buffer (size MAX_COMMAND_LINE or 8192) if the contents of argv are big enough. This should be protected from overflow.

#561 Updated by Greg Shah almost 7 years ago

In conclusion at this moment the OS-COMMAND for AJAX clients in Windows works properly only for commands which use STDIN as input.
For applications using Console I/O like cmd.exe it works fine only in non interactive mode.

Is there any reason to continue searching?

If not, then make sure to document this limitation (in an obvious way) in the implementation classes. This should be in the javadoc AND in code comments where it makes sense (e.g. in VT100SwingKeyboard.handle() the code comments should explain WHY the keystrokes are "sent directly" and what "sent directly" really means).

#562 Updated by Marius Gligor almost 7 years ago

1. I found no way to make Windows shell cmd.exe to run properly in interactive mode for AJAX clients. It's works properly only in non interactive mode.
Only processes using STDIN, STDOUT and STDERR as input/output streams are working properly in interactive/ non interactive mode for AJAX clients in Windows.

2. Swing clients in Windows OS are working properly using a native shell window like Progress 4 GL for Windows.

#563 Updated by Marius Gligor almost 7 years ago

Just added some javadoc explanations.

#564 Updated by Greg Shah almost 7 years ago

Code Review 0429b

Everything is fine, except for this item from the previous code review:

1. In VT100SwingKeyboard and ChuiScreenDriver, please only query PlatformHelper.isUnderWindowsFamily() once (at construction) and save that value. It won't change during the lifetime of the instance.

#565 Updated by Marius Gligor almost 7 years ago

Fixed.

#566 Updated by Greg Shah almost 7 years ago

Code Review 0429c

Everything looks good. Please get it runtime regression tested.

#567 Updated by Marius Gligor almost 7 years ago

0429c - Interactive child process support ref. #2247
Passed regression tests. Committed revision 10517.

#568 Updated by Constantin Asofiei almost 7 years ago

Marius, I think there is a problem in ChuiScreenDriver.osCommand:460:

      if (isOsDefault && isWindowsOS)
      {
         // If defaults options and Windows OS use interactive mode.
         cmdList[1] = "/k";
      }

You are assuming that the received cmdList will always be of length 2... but this is not the case, a plain DOS statement will fail on Windows with an ArrayIndexOutOfBounds exception. The following fixes this:
      if (isOsDefault && isWindowsOS)
      {
         // If defaults options and Windows OS use interactive mode.
         cmdList = new String[] { cmdList[0], "/k" };
      }

I'll include this in an update with some other misc changes, if the fix looks OK to you.

#569 Updated by Constantin Asofiei almost 7 years ago

PS: something else I've forgot to mention: on Windows, using a program with only the DOS command, P2J waits for a key to be pressed before launching the terminal - which is not the case in 4GL.

#570 Updated by Marius Gligor almost 7 years ago

The cmdList is constructed on ProcessDaemon.prepareCommandLine(String[])
Indeed now I saw that if cmdList is empty the array is only 1 size.
You fix should be not enough because the cmdList could have a size greater than 2.
My idea was to transform a command like this:

cmd.exe /c &lt;other args&gt;

into

cmd.exe /k &lt;other args&gt;

So a fix should looks like:

      if (isOsDefault && isWindowsOS)
      {
         // If defaults options and Windows OS use interactive mode.
         if (cmdList != null && cmdList.length > 1 && "/c".equalsIgnoreCase(cmdList[1]))
         {
            cmdList[1] = "/k";
         }
      }

#571 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

So a fix should looks like:

OK, is more complex than I expected, so please fix it.

#572 Updated by Marius Gligor almost 7 years ago

Constantin Asofiei wrote:

PS: something else I've forgot to mention: on Windows, using a program with only the DOS command, P2J waits for a key to be pressed before launching the terminal - which is not the case in 4GL.

Yes I know about this. Seems to come from ThinClient.suspend() implementation not from my code.
The same behaviour is under Linux or Windows using native client which was implemented before.

#573 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

Constantin Asofiei wrote:

PS: something else I've forgot to mention: on Windows, using a program with only the DOS command, P2J waits for a key to be pressed before launching the terminal - which is not the case in 4GL.

Yes I know about this. Seems to come from ThinClient.suspend() implementation not from my code.
The same behaviour is under Linux or Windows using native client which was implemented before.

OK, I'm not sure if this is a regression of how 4GL 9.1c behaves.

Something else I've noticed: are the Backspace/Delete keys supposed to work in the VT100 implementation?

#574 Updated by Marius Gligor almost 7 years ago

Constantin Asofiei wrote:

Marius Gligor wrote:

Constantin Asofiei wrote:

PS: something else I've forgot to mention: on Windows, using a program with only the DOS command, P2J waits for a key to be pressed before launching the terminal - which is not the case in 4GL.

Yes I know about this. Seems to come from ThinClient.suspend() implementation not from my code.
The same behaviour is under Linux or Windows using native client which was implemented before.

OK, I'm not sure if this is a regression of how 4GL 9.1c behaves.

Something else I've noticed: are the Backspace/Delete keys supposed to work in the VT100 implementation?

Yes. I just tested on Linux using Swing client and it's works.
On the web client seems to be a small problem. I have to fix this on JS code.

For web clients on Windows OS is a limitation:

The Windows OS shell cmd.exe doesn't work properly in interactive mode.
Windows shell cmd.exe does not use STDIN as input nor STDOUT and STDERR as outputs.
Commands using STDIN, STDOUT and STDERR are working properly.

#575 Updated by Marius Gligor almost 7 years ago

This are some bugs fixes for interactive client:

- On Windows OS if OS-COMMAND has default values the command contains only one argument "cmd.exe"
Is no longer needed to look for /c and change to /k because only cmd.exe in the same as cmd.exe /k
This behaviour was already implemented in ProcessDaemon.prepareCommandLine(String[]).

- I fixed the DELETE key code for JS clients to send down the correct code.

- I fixed other issues related to screen cursor on JS and the ENTER key for Windows OS AJAX clients

#576 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

This are some bugs fixes for interactive client:

For some reason, when launching a process using the Swing client (on my Ubuntu, configured as standard dev workstation) the Backspace and Delete keys do not work...

#577 Updated by Marius Gligor almost 7 years ago

Have you tried a sh or a /bin/bash command?

#578 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

Have you tried a sh or a /bin/bash command?

Yes, they are working. This is how it looks when Backspace and Delete are pressed (in this order):

Also: is VT100 supposed to wrap output from i.e. echo $PATH?

#579 Updated by Marius Gligor almost 7 years ago

The wrap ON and OFF could be changed by keyboard sequences.
In class VT100AbstractRenderer the autoWrap was set to false by default.
I changed the default value to true and now it's works.

Regarding BACKSPACE and DELETE keys:

I tried the sh command on a Konsole application:
- The BACKSPACE works but DELETE not returns the same sequences. I have to do more investigations here.

When I executed a /bin/bash command both BACKSPACE and DELETE are working properly.

#580 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

Regarding BACKSPACE and DELETE keys:

I tried the sh command on a Konsole application:
- The BACKSPACE works but DELETE not returns the same sequences. I have to do more investigations here.

I think you are correct, only BACKSPACE is problematic.

When I executed a /bin/bash command both BACKSPACE and DELETE are working properly.

Yes, this works fine for me too.

#581 Updated by Marius Gligor almost 7 years ago

I searched on the Internet regarding BACKSPACE key:

1.The ^H is the normal response for VT100 terminal on response to a BACKSPACE key.

2. However I found that this could be changed by executing the following command:

$ stty erase ^H

Please try to execute this command from Swing interactive client at sh prompt.
Maybe a better solution is to send this command from our code prior to use the interactive client.

#582 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

Please try to execute this command from Swing interactive client at sh prompt.

Yes, now it works.

Maybe a better solution is to send this command from our code prior to use the interactive client.

I'm not sure if this is the right way. Can this be related to how the TERMINFO DB is configured?

#583 Updated by Marius Gligor almost 7 years ago

I did more tests on the customer's Windows server and I found that executing a command like:

OS-COMMAND "dir".

which means default values the dir command is executed and the shell window remain active.
On P2J a command like this is transformed in:
cmd.exe /c dir
but must be
cmd.exe /k dir

So I reverted the changes in ChuiScreenDriver and I did the appropriate changes.

#584 Updated by Greg Shah almost 7 years ago

Marius Gligor wrote:

Constantin Asofiei wrote:

PS: something else I've forgot to mention: on Windows, using a program with only the DOS command, P2J waits for a key to be pressed before launching the terminal - which is not the case in 4GL.

Yes I know about this. Seems to come from ThinClient.suspend() implementation not from my code.
The same behaviour is under Linux or Windows using native client which was implemented before.

Eugenie: I think this is in your Windows console support code. What are your thoughts?

#585 Updated by Greg Shah almost 7 years ago

Constantin Asofiei wrote:

Marius Gligor wrote:

Please try to execute this command from Swing interactive client at sh prompt.

Yes, now it works.

Maybe a better solution is to send this command from our code prior to use the interactive client.

I'm not sure if this is the right way. Can this be related to how the TERMINFO DB is configured?

Previously, we have had to make fixes to TERMINFO to get VT220, VT320 and xterm to work properly. This is documented in our books. Marius, please check if we need to make similar changes for VT100.

#586 Updated by Greg Shah almost 7 years ago

Code Review 0507c

I am fine with the changes so far. One note: the VT100AbstractRenderer has the variable named autoWarp instead of autoWrap. Although autoWarp is sounds cooler, I guess we should name it more descriptively. :)

Let's figure out these last two issues (Windows console pausing and BACKSPACE) and then we will get this in testing.

#587 Updated by Marius Gligor almost 7 years ago

I'm working to change the TERMINFO for vt100 as follow:

sudo cd /usr/share/terminfo/v/
infocmp vt100 > vt100.tmp

The content of vt100.tmp is:

#       Reconstructed via infocmp from file: /usr/share/terminfo/v/vt100
vt100|vt100-am|dec vt100 (w/advanced video),
        am, mc5i, msgr, xenl, xon,
        cols#80, it#8, lines#24, vt#3,
        acsc=``aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~,
        bel=^G, blink=\E[5m$<2>, bold=\E[1m$<2>,
        clear=\E[H\E[J$<50>, cr=^M, csr=\E[%i%p1%d;%p2%dr,
        cub=\E[%p1%dD, cub1=^H, cud=\E[%p1%dB, cud1=^J,
        cuf=\E[%p1%dC, cuf1=\E[C$<2>,
        cup=\E[%i%p1%d;%p2%dH$<5>, cuu=\E[%p1%dA,
        cuu1=\E[A$<2>, ed=\E[J$<50>, el=\E[K$<3>, el1=\E[1K$<3>,
        enacs=\E(B\E)0, home=\E[H, ht=^I, hts=\EH, ind=^J, ka1=\EOq,
        ka3=\EOs, kb2=\EOr, kbs=^H, kc1=\EOp, kc3=\EOn, kcub1=\EOD,
        kcud1=\EOB, kcuf1=\EOC, kcuu1=\EOA, kent=\EOM, kf0=\EOy,
        kf1=\EOP, kf10=\EOx, kf2=\EOQ, kf3=\EOR, kf4=\EOS, kf5=\EOt,
        kf6=\EOu, kf7=\EOv, kf8=\EOl, kf9=\EOw, lf1=pf1, lf2=pf2,
        lf3=pf3, lf4=pf4, mc0=\E[0i, mc4=\E[4i, mc5=\E[5i, rc=\E8,
        rev=\E[7m$<2>, ri=\EM$<5>, rmacs=^O, rmam=\E[?7l,
        rmkx=\E[?1l\E>, rmso=\E[m$<2>, rmul=\E[m$<2>,
        rs2=\E>\E[?3l\E[?4l\E[?5l\E[?7h\E[?8h, sc=\E7,
        sgr=\E[0%?%p1%p6%|%t;1%;%?%p2%t;4%;%?%p1%p3%|%t;7%;%?%p4%t;5%;m%?%p9%t\016%e\017%;$<2>,
        sgr0=\E[m\017$<2>, smacs=^N, smam=\E[?7h, smkx=\E[?1h\E=,
        smso=\E[7m$<2>, smul=\E[4m$<2>, tbc=\E[3g,

I changed the kbs=^H trying the following values one by one kbs=\177, kbs=\b, kbs=\011 and kbs=\EOD (kcub1)
Looking inside my emulator the more appropriate values are \011 and \EOD if this represents the sequences which are send as a response to BACKSPACE key.
Also I did the same changes for cubl and I recompiled the database

tic vt100.tmp

I tested using infocmp vt100 > vt100.tmp and the changes was made.
However no changes on console always the same ^H on BACKSPACE.
Do you have any idea how to fix that?

#588 Updated by Greg Shah almost 7 years ago

I'm not sure I have too much guidance here. The only thing I see is that I think the ASCII BS is 0x08 or octal \010. Isn't octal \011 the 0x09 or TAB key?

There are a couple of documents that are potentially useful:

http://www.ibb.net/~anne/keyboard.html
http://www.ibb.net/~anne/keyboard/keyboardprev.html (this is the one that more closely maps to what we want, see the terminfo section)

#589 Updated by Eugenie Lyzenko almost 7 years ago

Greg Shah wrote:

Marius Gligor wrote:

Constantin Asofiei wrote:

PS: something else I've forgot to mention: on Windows, using a program with only the DOS command, P2J waits for a key to be pressed before launching the terminal - which is not the case in 4GL.

Yes I know about this. Seems to come from ThinClient.suspend() implementation not from my code.
The same behaviour is under Linux or Windows using native client which was implemented before.

Eugenie: I think this is in your Windows console support code. What are your thoughts?

What I did special regarding to Windows console implementation and pause is the eliminating additional pause for Windows P2J but AFTER command is executing, not BEFORE. And this is not in platform dependent code but in ProcessDaemon.launch(). The terminal suspend() was not changed/adjusted.

#590 Updated by Eugenie Lyzenko almost 7 years ago

I changed the kbs=^H trying the following values one by one kbs=\177, kbs=\b, kbs=\011 and kbs=\EOD (kcub1)

Try ^? and may be "\e[3~".

#591 Updated by Marius Gligor almost 7 years ago

Changing TERMINFO seems to not work on all cases see 5.12 on this document
http://www.tldp.org/HOWTO/Keyboard-and-Console-HOWTO-5.html
However on 5.13 he claim to offer a complete solution.

#592 Updated by Greg Shah almost 7 years ago

Does this solution documented in section 5.13 solve the problem with sh?

#593 Updated by Marius Gligor almost 7 years ago

Greg Shah wrote:

Does this solution documented in section 5.13 solve the problem with sh?

I didn't try but I found a more clever solution. I fixed the issue directly on the VT100 emulator
by changes the code for BACKSPACE to 127 (0x7f).
Basically is the same idea to use code 127 for BACKSPACE but implemented directly into the software emulator
without any other changes on system.

#594 Updated by Greg Shah almost 7 years ago

Perfect! Yes, that is clever.

#595 Updated by Marius Gligor almost 7 years ago

1. Fixed the BACKSPACE key on VT100 emulator (VT100AbstractKeyboard).
2. Fixed waiting issue in TypeAhead.suspend()

#596 Updated by Marius Gligor almost 7 years ago

Add also the fix for BGCOLOR and FGCOLOR in BaseEntity to use int values on setters.

#597 Updated by Constantin Asofiei almost 7 years ago

Greg/Marius, something to keep in mind about shell launch on Windows: when using a web client and the clients launches a shell (via DOS), then the shell (cmd.exe) will not be visible/accessible from the web client. I'll add this as a limitation to the docs.

#598 Updated by Marius Gligor almost 7 years ago

Constantin Asofiei wrote:

Greg/Marius, something to keep in mind about shell launch on Windows: when using a web client and the clients launches a shell (via DOS), then the shell (cmd.exe) will not be visible/accessible from the web client. I'll add this as a limitation to the docs.

Yes this is a more general issue using interactive processes on web clients see my remarks on note #555

#599 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

Yes this is a more general issue using interactive processes on web clients see my remarks on note #555

Marius, thanks.

Something else about these notes in note 490:

f. the request seen by server C has the cookie set (if and only if the request was the redirected GET from the original browser) and the authorization passes

The part I am really unclear about is step (e). It seems to me that the cross-origin nature of the cookie should break this step.

You are correct, the cross-origin nature of the cookie breaks this step; if the initial login page is on localhost and the redirect is on i.e. 192.168.1.2, then the redirect will not see the cookie and will not authorize the client. I guess this is OK for now (as the client and server are on the same machine), but we will need to enhance this, once the P2J Server will be allowed to start P2J Clients on other machines.

#600 Updated by Greg Shah almost 7 years ago

Code Review 0508b

My only concern with this update is the change to TypeAhead. Have you confirmed that all ScreenDriver instances will return -1 when readKey() is interrupted? This is a very sensitive area of the code.

#601 Updated by Marius Gligor almost 7 years ago

I tried to do as much as minimum changes in the code due to it's complexity in terms of thread synchronization.
The code inside TypeAhead isn't my work but what I understood is the follow:

- When the keyboard reader thread is suspended the suspended flag notify the thread about the request and goes to wait.
- The KeyReader thread when check the suspended flag and found a true value execute a notify and than goes to wait.
- On the initial implementation seems that both thread goes to wait and a deadlock occur.
- Fortunately a keystroke seems to wakeup the KeyReader which in term notify the caller thread which leave the wait state.

According to my tests the keyReader thread is blocked because it uses a LinkedBlockingQueue.take when get the keystroke from key buffer.
Please look in TypeAhead line 454 int key = OutputManager.instance().readKey();
LinkedBlockingQueue.take is a blocked operation. When is interrupted -1 is retuned.
The code in suspend() is safe because after interrupting the key reader we wait to be notified when reader goes to wait.
When the suspend() exit key reader is suspended

I did tests and I found to works fine with all screen drivers on both Windows and Linux.

#602 Updated by Marius Gligor almost 7 years ago

Another very important aspect is the code in line 443 which force recheck for suspension in order to avoid a spurious wake-up.
Even if we interrupt the thread from suspend() the key reader recheck the suspension and goes back to sleep and notify us about this.

#603 Updated by Greg Shah almost 7 years ago

OK, go ahead with runtime regression testing for this update.

#604 Updated by Marius Gligor almost 7 years ago

0508b - Passed regression tests. Committed revision 10525.

#605 Updated by Marius Gligor almost 7 years ago

Here is a first solution for issue #2306 implements copy/paste for web client

1. The system clipboard cannot be accessed in JS for security reasons. Only IE on Windows allow to access the system clipboard.
As a consequence I designed a simple text clipboard in JS code.

2. For selection menu I used Dojo toolkit the latest release 1.10.0 (I tested also the previous 1.9.2 release)
I designed a JS pop-up context menu having 3 options Select, Copy and Paste.

3. For selection drawing I'm using a layered canvases technique. Basically we have two canvases the main canvas and a transparent canvas.
The two canvases are overlapped. When the selection is draw the transparent canvas is used and It is no need to redraw (refresh) the main canvas.
Also we have two types for graphical selection. A rectangle (stroke) like on the Swing P2J or a fill semi-transparent rectangle.
The selection type is configurable (stroke parameter) and by default is like on the Swing P2J.

4. Doing a lot of tests I observed that often when I reloaded the main html page or even on a first load on Linux the Dojo toolkit throws some JS errors.
This is a strange behaviour and on the next step I have to find a solution to fix this issue.

#606 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

2. For selection menu I used Dojo toolkit the latest release 1.10.0 (I tested also the previous 1.9.2 release)

Is there supposed to be a dojo-release-1.10.0.zip file in the P2J project? Because I can't find it...

#607 Updated by Marius Gligor almost 7 years ago

1. I found and I fixed the issue related to Dojo modules loading. The problem was a race condition in DojoToolkitHandler jetty handler.
The method deliverDojoModule MUST be synchronized. I did new tests and now it works properly.

2. The Dojo toolkit 1.9.2 is available from: ~/shared/projects/p2j/dojo/ location
Today I copied also the latest Dojo 1.10.0 toolkit on the same location.
The manifest was changed some times ago to put on CLASSPATH the Dojo toolkit 1.9.2 and now I changed to 1.10.0

#608 Updated by Greg Shah almost 7 years ago

I haven't gotten the chance to do a code review yet. But I think we need to be using the new HTML5 clipboard API (as long as it can do the job):

http://caniuse.com/clipboard
http://www.w3.org/TR/clipboard-apis/ (new API that is hopefully functional enough to do the job and which is mostly supported in recent browsers)
http://brooknovak.wordpress.com/2009/07/28/accessing-the-system-clipboard-with-javascript/ (nice summary of the "old way" to work around clipboard restrictions)

#609 Updated by Marius Gligor almost 7 years ago

- The HTML 5 Clipboard API is currently only a draft and the specifications aren't implemented yet by any web browser.

- I found some examples for paste from system clipboard but these examples allow only to register a listener for paste command triggered from browser context menu or CTRL-V keyboard.
The implementation is different in Firefox and Chrome. I found no examples for copy to system clipboard.

- Another solution which I've not yet tested is to use a JQuery plug-in based on a zeroclipboard JS library from GitHub.
The solution is based on the injection of a hidden Adobe flush plug-in and use this object to copy and paste to system clipboard.
The solution have a lot of serious limitations and is subject to security threats (see https://github.com/zeroclipboard/zeroclipboard)
Also the keyboard strokes (CTRL-C and CTRL-V) cannot be used on our case to trigger copy and paste commands.

- The only suitable solution available today for our case is the IE implementation which allow direct access to system clipboard to read and write.

#610 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

- I found some examples for paste from system clipboard but these examples allow only to register a listener for paste command triggered from browser context menu or CTRL-V keyboard.

If think we should allow the user to paste text (via CTRL-V); can/should this be done by reading the clipboard content and sending it as a key, char-by-char? Can you check how the console client behaves if i.e. CTRL-V/Shift-Insert is pressed?

- Another solution which I've not yet tested is to use a JQuery plug-in based on a zeroclipboard JS library from GitHub.

I don't think the user will have much use of the copy-paste feature, if pasting the text selection from the ChUI (or GUI) client outside of the browser window is not allowed. More, copy-paste will be much more expected when the GUI ajax client will be in use (just think of all the fill-in's and text editors to which the user can select text and copy/paste it as necessary).

#611 Updated by Marius Gligor almost 7 years ago

  • File paste_example.html added

I tested CTRL-V/Shift-Insert strokes on both P4GL and P2J.
Shift-Insert switch the insert mode on P4GL editor.
CTRL-V seems to have no effects and could be used for paste operations.

In the mean time I created a short example (see paste_example.html) that could be used to paste from system clipboard.
I designed the JS code to work on all major browsers. I tested with IE, Firefox, Chrome and Opera and on all it works!

The limitation of this solution is that it works only from keyboard by pressing the CTRL-V key.
We could try to integrate the code from this example in our web client JS code.

#612 Updated by Marius Gligor almost 7 years ago

  • File deleted (paste_example.html)

#613 Updated by Marius Gligor almost 7 years ago

#614 Updated by Greg Shah almost 7 years ago

Code Review 0703a

Overall the code looks good.

1. The sendResponse() and detectContentType() in DojoToolkitHandler seem generic enough that they would be useful in the parent class.

2. Constantin is correct: we can't implement clipboard support just within our own implementation. The key requirement is for integration with the system clipboard. Your paste example looks good. I am fine with the limitation that SHIFT-INS does not work. CTRL-V and the browser's paste menu item is enough, though if possible I would like to have it linked to our Paste popup menu too.

3. I assume you can make a similar example for copy (as you did for paste). For copy, the tricky part is that we 4GL use CTRL-C for generating the STOP condition. However, I think we can let CTRL-C be used a copy in the case (and only in that case) where the user has made an explicit selection. When this copy occurs, we clear the selection and CTRL-C will be given to the "terminal" again. I also expect the browser's copy menu should work fine. Again, if possible please link this to our popup menu.

#615 Updated by Marius Gligor almost 7 years ago

1. Unfortunately we cannot link CTRL-V to our pop-up menu.
For security reasons the event listener for paste is trigerred only when the CTRL-V is pressed from browser, taht is, only under browser control.
We cannot force a paste operation from JS code using

document.execCommand('paste', ...)
. This works today only on IE which fully support JS clipboard API.
All we could do is to copy from system clipboard to our application clipboard when CTRL-V is pressed and in the same time to send down to server the pasted text.

2. The CTRL-C is even more restrictive. When CTRL-C is pressed the selected text from the current html element having the focus is copied into system clipboard.
The html element must be an editable element (input) and he must be the focus owner at the moment when CTRL-C is pressed.
Our selection on canvas is not a selection in terms of system clipboard and we cannot specify which data should be copied to clipboard.
Again the

document.execCommand('copy', ...)
solution is available only on IE.

#616 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

2. The CTRL-C is even more restrictive. When CTRL-C is pressed the selected text from the current html element having the focus is copied into system clipboard.

Is it possible to use an editable input/textarea element this way: after the user selects the text, the selected text is set as the content for the input/textarea, via js (and everything is selected); this element is set as focus, and if the user presses CTRL-C, should pick the text up. Not sure what happens if the element is hidden.

#617 Updated by Marius Gligor almost 7 years ago

I have to try. It is the best way to find out if it works or not.

#618 Updated by Constantin Asofiei almost 7 years ago

Another solution would be this: for copying, the user selects the text, then right-clicks and chooses from the menu the Copy option; once this option is chosen, a dialog (maybe modal?) is displayed to the user, with the selected text (which is already highlighted/selected). This dialog will inform the user to press CTRL-C to copy it. If CTRL-C is pressed directly (after the selection is completed and before Copy option is chosen), is it possible to add an event which does what the Copy menu does (i.e. the intermediate dialog)?

For pasting, I think your approach to "copy from system clipboard to our application clipboard when CTRL-V is pressed and in the same time to send down to server the pasted text." should be OK - no intermediate dialog is needed.

#619 Updated by Marius Gligor almost 7 years ago

Here is a solution which works with CTRL-C and CTRL-V by hidding the input element using CSS.
If the input is of type="hidden" id doesn't work because he cannot gain the focus.
When the user press CTRL-C we have to copy the data to the input element set focus and select text.

The soulution with dialogs to copy and paste is the simple way to implements copy/paste and can be linked with our pop-up menu.

function copyToClipboard(text) {
  window.prompt("Copy to clipboard: Ctrl+C, Enter", text);
}

function pasteFromClipboard() {
  return window.prompt("Paste from clipboard: Ctrl+V, Enter", "");
}

but this solution is not so elegant because the user select copy from context menu and after that he should press also CTRL-C.

#620 Updated by Marius Gligor almost 7 years ago

Here is a simple soultion using dialogs.

#621 Updated by Marius Gligor almost 7 years ago

1. The sendResponse() and detectContentType() in DojoToolkitHandler seem generic enough that they would be useful in the parent class.

The parrent class of DojoToolkitHandler is AbstractHandler which is part of the jetty library.
We cannot move sendResponse() and detectContentType() to AbstractHandler.

#622 Updated by Greg Shah almost 7 years ago

I don't want to use the dialog approach unless absolutely needed. At a minimum, the paste can be done properly (without dialogs), right?

In regard to the copy, please investigate if we can make a non-hidden focused input element that is not visible (using sizing/positioning, opacity or the visible attribute). That is much cleaner than the dialog approach.

Finally, if we can drive all of this using CTRL-C and CTRL-V, then it seems there is no reason for a popup menu. The selection can be driven purely off mouse events, right?

#623 Updated by Marius Gligor almost 7 years ago

1. The CTRL-C example attached on #619 is exactly what you are looking for. The input is made not visible using CSS style.

2. Yes we could do the selection using only the mouse events and use CTRL-C and CTRL-V for copy and paste.
In this case the Dojo toolkit is no longer needed. However the changes DojoToolkitHandler should be kept because has been tested.

3. May I do changes to use only the mouse events for selection and CTRL_C/CTRL-V for copy/paste?

#624 Updated by Greg Shah almost 7 years ago

1. The sendResponse() and detectContentType() in DojoToolkitHandler seem generic enough that they would be useful in the parent class.

The parrent class of DojoToolkitHandler is AbstractHandler which is part of the jetty library.
We cannot move sendResponse() and detectContentType() to AbstractHandler.

Right. But we can create a new base class (extends AbstractHandler) that has those features and which can be used from DojoToolkitHandler and other subclasses.

1. The CTRL-C example attached on #619 is exactly what you are looking for. The input is made not visible using CSS style.

If you can make both CTRL-C and CTRL-V work for accessing the system clipboard, then that is the best solution.

Is there any problem with event handling during the time when the input element has focus?

Also, when there is no current selection, the CTRL-C needs to still be forwarded to the server for use by the 4GL. Only when there is a selection, should CTRL-C be honored as a copy.

In this case the Dojo toolkit is no longer needed. However the changes DojoToolkitHandler should be kept because has been tested.

Agreed.

3. May I do changes to use only the mouse events for selection and CTRL_C/CTRL-V for copy/paste?

Yes. You should also allow the user to cancel the current selection using ESC. This would only work while a selection was active. Once the selection is cleared, ESC would be forwarded to the 4GL key processing again.

#625 Updated by Marius Gligor almost 7 years ago

I will try to do the appropriate changes to implements the copy/paste with CTRL-C and CTRL-V, ESC to clear selection and mouse events to draw selection.
Regarding DojoToolkitHandler I have to create another abstract class let say AbstractHandlerCommons which extends AbstractHandler and move the generic methods as you suggested.

#626 Updated by Marius Gligor almost 7 years ago

Here is the implementation of copy/paste using:

- Mouse drag to select.
- ESC key to clear selection.
- CTRL-C key to copy only when have a selection.
- CTRL-V key to paste always.

Tested with Chrome, Firefox, IE and Opera.

#627 Updated by Greg Shah almost 7 years ago

Code Review 0709a

The code looks very good.

1. In onkeyup(), shouldn't we still call preventDefault() and stopImmediatePropagation() when the event is not CTRL-C or CTRL-V?

2. CTRL-C still gets passed to the server when there is no selection. My only concern is about how we handle CTRL-V. If I understand correctly, CTRL-V is never passed to the server. This means this is another key combination that is excluded from 4GL app compatibility. That may be OK if we can't think of any way around this, but I wanted to discuss and understand it.

#628 Updated by Marius Gligor almost 7 years ago

1. The onkeyup() is the last event fired on key events. If we prevent default action and stop propagation we could be sure that no other actions are performed by browser.

2. Regarding CTRL-V unfortunately we have no other conditions to use like on CTRL-C.
Supposing that CRTL-V is used by a P4GL application this could be a problem if we send CTRL-V to application and in the same time we perform a copy operation.
Probabbly the user expect something else to happen on CTRL-V.

#629 Updated by Greg Shah almost 7 years ago

The onkeyup() is the last event fired on key events. If we prevent default action and stop propagation we could be sure that no other actions are performed by browser.

Yes, let's do this for all keystrokes that are not CTRL-C or CTRL-V.

Regarding CTRL-V unfortunately we have no other conditions to use like on CTRL-C.

The only thing I can think of is when the text read from the clipboard is the empty string. What do you think?

Supposing that CRTL-V is used by a P4GL application this could be a problem if we send CTRL-V to application and in the same time we perform a copy operation.
Probabbly the user expect something else to happen on CTRL-V.

Other than the empty string case, we will just have to live with this restriction.

#630 Updated by Marius Gligor almost 7 years ago

Yes this might be a solution. At the moment the CTRL-V is pressed we cannot determine if clipboard is empty but we could find later.
In the paste events handler if clipboard is not empty we could send the CTRL-V. I have to check also if paste event is fired when the clipboard is empty on all browsers.

#631 Updated by Constantin Asofiei almost 7 years ago

Do you think is OK to have some browser toggle box to set/change how CTRL-V is used? This toggle box could be defaulted to some chuiWeb setting; its set/unset modes would determine what happens when the user presses CTRL-V.

#632 Updated by Marius Gligor almost 7 years ago

Here are the changes which I made to implements CTRL-V rule.

#633 Updated by Marius Gligor almost 7 years ago

By the way one issue that should be also discussed. When a paste operation is made the text is not discarded from clipboard.
So in order to send CTRL-V to P4GL application the clipboard MUST be empty. The user must clear the clipboard from outside id he need to use CTRL-V in P4G application.
We could clear the clipboard from JS when CTRL-V is handled but this is not a standard mode to work with clipboard.

#634 Updated by Greg Shah almost 7 years ago

Constantin Asofiei wrote:

Do you think is OK to have some browser toggle box to set/change how CTRL-V is used? This toggle box could be defaulted to some chuiWeb setting; its set/unset modes would determine what happens when the user presses CTRL-V.

I think this is an excellent idea. We can default its state from server-side (directory) configuration, if present. If the directory does not have any configuration, then for now we can leave clipboard support defaulted to ON.

To keep the interface clean, let's add the Dojo popup menu back with just this single checkbox item: "Enable Clipboard".

#635 Updated by Greg Shah almost 7 years ago

We could clear the clipboard from JS when CTRL-V is handled but this is not a standard mode to work with clipboard.

Good point.

Since it is not the standard way users expect to work with the clipboard, we won't clear the clipboard at this time.

#636 Updated by Marius Gligor almost 7 years ago

"Enable Clipboard" means also CTRL-C? Basically means the entire clipboard?

#637 Updated by Greg Shah almost 7 years ago

Code Review 0711a

It looks good. Please implement the "Enable Clipboard" feature and then this task will be done.

#638 Updated by Greg Shah almost 7 years ago

Marius Gligor wrote:

"Enable Clipboard" means also CTRL-C? Basically means the entire clipboard?

Yes, exactly. When the "Enable Clipboard" is OFF, the client should work exactly as before. That means the mouse selection processing should be disabled too and all event processing should be restored (allowed to flow to the server).

#639 Updated by Marius Gligor almost 7 years ago

Another question please. If in directory the clipboard is set to OFF (disabled) should we hide also the pop-up menu?

#640 Updated by Constantin Asofiei almost 7 years ago

Marius Gligor wrote:

Another question please. If in directory the clipboard is set to OFF (disabled) should we hide also the pop-up menu?

The directory setting just tells the web client how the clipboard is by default (ON or OFF). Regardless of what the directory setting is, the user needs to be allowed to disable/enable it as needed.

#641 Updated by Marius Gligor almost 7 years ago

Enable/Disable clipboard. Add node "clipboard" to the "container" node "chuiWeb":

  <node class="container" name="chuiWeb">
    ....
    <node class="boolean" name="clipboard">
       <node-attribute name="value" value="FALSE"/>
    </node>     

If missing the default value is TRUE clipboard enabled.
The user may enable/disable the clipboard behaviour using a context pop-up menu created using Dojo toolkit having a single checked item "Enable Clipboard".

#642 Updated by Greg Shah almost 7 years ago

Code Review 0714a

The update looks really good.

I realize that the escape key selection processing in the onkeydown() function (tty.keyboard.js) is safe right now because tty.clipboard.haveSelection() is protected. But I would like the selection clearing to be wrapped in if (tty.clipboard.enabled) anyway for 2 reasons:

  1. Future changes to tty.clipboard.haveSelection() won't cause regressions/surprises.
  2. Future readers/maintainers of tty.keyboard.js will be able to see immediately that it is safe instead of having to look at tty.clipboard.haveSelection() first.

Other than that I think this is good to go. I don't think this needs regression testing. Please make the above change, post it here and then you can check in/distribute the update.

#643 Updated by Constantin Asofiei almost 7 years ago

Greg/Marius: is there a specific reason why the dojo zip is not included in the p2j bzr repo? More, how will the web client act if dojo is not in the classpath (and the JS relies on dojo)?

#644 Updated by Marius Gligor almost 7 years ago

Dojo toolkit is big in size (~13 Mb). So far the dojo toolkit was not used in P2J project. However it was added to the classpath some time ago (see the manifest file)
I updated the classpath inside the manifest file to use the latest Dojo release (1.10). I will commit the Dojo zip at the end of this task and I will inform the team that Dojo zip
is missing in the distributed archive. They could update the dojo zip either from bzr or from ~/shared/projects/p2j/dojo/

#645 Updated by Greg Shah almost 7 years ago

OK, good plan.

#646 Updated by Marius Gligor almost 7 years ago

Here are the requested changes.

#647 Updated by Greg Shah almost 7 years ago

It looks good. Please note that we are backing out 10568, but that is not yet complete. Until that happens, please hold off on your check-in and distribution.

#648 Updated by Marius Gligor almost 7 years ago

0714b.zip committed revision 10570.

#649 Updated by Constantin Asofiei over 6 years ago

Fixes a starvation problem while reading keys via the web client driver. Will runtime test it.

#650 Updated by Constantin Asofiei over 6 years ago

Constantin Asofiei wrote:

Fixes a starvation problem while reading keys via the web client driver. Will runtime test it.

No issues related to this during testing, released to rev 10678./

#651 Updated by Greg Shah almost 6 years ago

The following are the next development steps to implement the GUI web client:

  1. Much of the web-specific code for the ChUI client is not specific to ChUI. But the initial implementation was not designed to maximize reuse. Instead, this common code was embedded inside the ChUI-specific web client. This code needs to be refactored to move into common/generic web support classes so that the GUI web client can share that code.
  2. The ChUI web client was implemented primarily by extending the ChUI Swing client. Some ChUI functionality was pushed into base classes and then reused via inheritance. One problem is that those base classes themselves inherit from Swing components (e.g. JComponent) and the implementation is very Swing-dependent. This makes it very hard to understand which code is actually needed for the web client. This Swing-dependency needs to be eliminated from the ChUI web client. This means moving away from the ChuiSimulator approach.
  3. Rework the client side javascript code to maximize the amount of generic/shared code. This also means implementing multiple locations for loading of javascript code on the server and the ability to specialize or differentially load javascript that is specific to a given UI-modality (ChUI vs GUI).
  4. There is no common parent hierarchy for the 5 existing drivers: native NCURSES ChUI, batch driver ChUI, Swing ChUI, Web ChUI, Swing GUI. We added drivers over time, organically. When adding them, we took some short cuts and never tried to get a clean and understandable parent hierarchy. Instead, there is a great deal of duplication, inconsistencies and it is harder to understand the drivers than necessary. By eliminating the Swing dependent parent classes in the remaining 2 Swing clients, a single, clean hierarchy can be implemented.
  5. Direct usage of AWT (and some Swing) features in the GUI code must be limited to inside the driver itself. Currently, there is quite a bit of such usage from GUI code that is expected to be common (e.g. GuiDriver, GuiPrimitives, PaintStructure, *GuiImpl widget classes). Access to Font, Image, Stroke, Color and other similar AWT features needs to be abstracted behind the driver interface.
  6. There is code in the Swing GUI driver that is generic enough that it should be shared for the web GUI as well. This means that there is some refactoring to move this common code into generic GUI support classes.
  7. Using the massive refactoring in the prior steps, a server-side web GUI driver must be built. Initially, the one area that can be left undone is the extension of the websockets protocol to the GUI driver primitives.
  8. Extend the websockets protocol to support the full set of GUI driver primitives.
  9. Implement the client side GUI-specific driver portions.

There will be more testing and refinement from there. For example, we will need to make sure that we test the implementation for compatibility with the Swing client AND for cross-browser compatibility. But the above steps will get us to a good foundation.

I have finished a first cut at steps 1 - 4 and am currently working on step 5. This is a massive refactoring update and I have not gotten everything currently edited into a working state yet. When it is at least mostly working I will upload an update for review. I intend to get through step 7 and then make sure that everything gets tested and checked in. From there Anton and I will make the final push for steps 8 and 9.

#652 Updated by Greg Shah almost 6 years ago

Research Item

What is the best approach to provide web GUI client support for images as provided in the 4GL?

The 4GL GUI is implemented using Windows and the native WIN32 API. From 4GL syntax, it provides the ability to use images on GUI buttons and it also provides an image widget which allows a specific image to be placed directly in a frame. In addition to this, there are some ways that a 4GL GUI application can implicitly use images. For example, the min/max/restore buttons in each window, the system menu icon (top left corner of the window) and the iconified (minimized) representation of a window as an icon. These are not directly controlled from the 4GL, but we must support the implicit use cases as well as the explicit use cases.

The current design for a GUI driver has a single method defined (drawImage()) to support images. All other loading, sizing, positioning, transformations and other processing is done in common code. But to do this there are multiple dependencies on non-generic classes:

  • java.awt.Image
  • java.awt.image.*
  • javax.imageio.*
  • javax.imageio.stream.*
  • some image types access via javax.imageio require the lib/imageio-*.jar files to be present at runtime

More details can be seen in #2446 (where most image work was done, including both buttons and image widgets) and #2252 (master GUI support task - note 681 is especially useful).

Some use of these is going to be required, even in the generic code, because we must be able to read in image formats that are native to Windows (or at least commonly used in Windows) like .bmp or .ico.

I plan to abstract the core image processing to eliminate direct use of the java.awt.Image class in the generic code. But the plan is not complete and it will depend to some degree on how we intend to push this support out to the browser side.

Some Windows-native image types will need to be translated into forms that can be used in the browser. That translation should certainly happen in Java code.

The biggest question in my mind is how exactly will we take the images from the server and deliver them to the client such that they are loaded as javascript Image objects. I see the following as options:

  • The most common way to do this is a hidden <img> tag and setting the src attribute to a server-side URLs. Our current approach is that the client is a single page web application, loading data/content via URL will require additional infrastructure and some care to handle all security concerns. Besides the additional security issues, this would also represent a deviation from our simple bidirectional websockets approach, meaning that it makes the overall approach harder to understand because it has more "moving parts". I do NOT prefer this approach.
  • We can send the image data as a byte[] using the web socket. This is simple, clean and has not additional security implications. How then would we turn this array of bytes into something usable (optimally to an in-memory Image javascript object)?
    • The ImageData support in canvas could be used (getImageData()/putImageData()) to access the raw pixels. Doing this may have positioning issues and we would also have to handle any alpha support manually. We could also use canvas.toDataURL() to get a data: URI from it, which can be turned into an Image object easily.
    • We could generate a data: URI manually (see https://developer.mozilla.org/en-US/docs/Web/HTTP/data_URIs). This would either require us to implement base-64 encoding in javascript (will atob() work?) or send the bytes down already encoded (not a good idea because this expands the data sent).

One concern I have about data: URI support is whether there are size limits to the images that can be loaded in this manner.

Anton: please do some further analysis and report your findings in this task. Ask any questions here as needed.

#653 Updated by Anton Breaur almost 6 years ago

We can send the image data as a byte[] using the web socket. This is simple, clean and has not additional security implications. How then would we turn this array of bytes into something usable (optimally to an in-memory Image javascript object)?

Agree, this looks like a better approach because of the existing websocket communication. Once we have the image as a byte array in javascript, we can easily convert into base64 using the window.btoa(byteData) function.
From here, creating an Image object means instantiating one and then setting the src attribute as such:
image.src = "data:image/png;base64," + base64
Then, having a canvas context the image can be drawn like this: ctx.drawImage(image, 0, 0)

The ImageData support in canvas could be used (getImageData()/putImageData()) to access the raw pixels. Doing this may have positioning issues and we would also have to handle any alpha support manually. We could also use canvas.toDataURL() to get a data: URI from it, which can be turned into an Image object easily.

We can create an Image object as described earlier, without involving a canvas context, other than the one to actually paint the image.
Not sure why we need to access raw pixels, but we can display the Image cropped or scaled as desired. Also, the alpha will probably need to be handled manually; the canvas context alpha can be updated and restored as such:
ctx.save();
ctx.globalAlpha = 0.4;
ctx.drawImage(img, x, y);
ctx.restore();

We could generate a data: URI manually (see https://developer.mozilla.org/en-US/docs/Web/HTTP/data_URIs). This would either require us to implement base-64 encoding in javascript (will atob() work?) or send the bytes down already encoded (not a good idea because this expands the data sent).

Manually generating data URI is the approach I am in favour of using. The base64 encoding in javascript works as described earlier, using btoa() function. Agree, the base64 encoded image is 1/3 larger than the original, so it is better to send byte data and encode in javascript.

One concern I have about data: URI support is whether there are size limits to the images that can be loaded in this manner.

I cannot find any other limit than an IE8 limit of 32kb for the URI, which means less than 24kb for the original image, but I suppose IE8 is out of our scope.

#654 Updated by Greg Shah almost 6 years ago

Following up on my note 651, the attached update is the first (partially working) pass at steps 1 - 7.

I have also attached a compare.sh helper to make code reviews easier (unzip the update and then run ./compare.sh <p2j_dir_with_unchanged_bzr_10848> <dir_where_update_was_unzipped>).

The following is a summary from bzr of the changes:

removed:
  src/com/goldencode/p2j/ui/client/chui/driver/ChuiScreenDriver.java
  src/com/goldencode/p2j/ui/client/driver/swing/GenericScreenDriver.java
  src/com/goldencode/p2j/ui/client/gui/driver/GuiScreenDriver.java
  src/com/goldencode/p2j/ui/client/swing/
added:
  src/com/goldencode/p2j/ui/client/FontDetails.java
  src/com/goldencode/p2j/ui/client/FontMetricsHelper.java
  src/com/goldencode/p2j/ui/client/FontStyle.java
  src/com/goldencode/p2j/ui/client/chui/driver/AbstractChildProcess.java
  src/com/goldencode/p2j/ui/client/chui/driver/AbstractChuiDriver.java
  src/com/goldencode/p2j/ui/client/chui/driver/EmulatedTerminalState.java
  src/com/goldencode/p2j/ui/client/chui/driver/VT100EmulatorChildProcess.java
  src/com/goldencode/p2j/ui/client/chui/driver/VT100ProgressKeyboard.java
  src/com/goldencode/p2j/ui/client/chui/driver/console/NativeChildProcess.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/p2j.clipboard_helpers.js
  src/com/goldencode/p2j/ui/client/driver/AbstractDriver.java
  src/com/goldencode/p2j/ui/client/driver/web/
  src/com/goldencode/p2j/ui/client/driver/web/ClientProtocolHooks.java
  src/com/goldencode/p2j/ui/client/driver/web/EmbeddedWebServerImpl.java
  src/com/goldencode/p2j/ui/client/driver/web/WebClientMessageTypes.java
  src/com/goldencode/p2j/ui/client/driver/web/WebClientProtocol.java
  src/com/goldencode/p2j/ui/client/driver/web/WebPageHandler.java
  src/com/goldencode/p2j/ui/client/driver/web/res/
  src/com/goldencode/p2j/ui/client/gui/driver/AbstractGuiDriver.java
  src/com/goldencode/p2j/ui/client/gui/driver/CursorType.java
  src/com/goldencode/p2j/ui/client/gui/driver/EmulatedWindowState.java
  src/com/goldencode/p2j/ui/client/gui/driver/ImageWrapper.java
  src/com/goldencode/p2j/ui/client/gui/driver/LineStroke.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/SwingFontMetrics.java
  src/com/goldencode/p2j/ui/client/gui/driver/web/
  src/com/goldencode/p2j/ui/client/gui/driver/web/GuiWebDriver.java
  src/com/goldencode/p2j/ui/client/gui/driver/web/GuiWebEmulatedWindow.java
  src/com/goldencode/p2j/ui/client/gui/driver/web/GuiWebPageHandler.java
  src/com/goldencode/p2j/ui/client/gui/driver/web/GuiWebSocket.java
  src/com/goldencode/p2j/ui/client/gui/driver/web/res/
  src/com/goldencode/p2j/ui/client/gui/driver/web/res/p2j.clipboard_helpers.js
  src/com/goldencode/p2j/ui/client/gui/driver/web/res/p2j.screen.js
  src/com/goldencode/p2j/util/ChildProcessFactory.java
  src/com/goldencode/p2j/util/InteractiveChildProcess.java
renamed:
  src/com/goldencode/p2j/ui/client/AbstractClient.java => src/com/goldencode/p2j/ui/client/driver/swing/AbstractClient.java
  src/com/goldencode/p2j/ui/client/chui/driver/swing/TerminalOptions.java => src/com/goldencode/p2j/ui/client/chui/driver/TerminalOptions.java
  src/com/goldencode/p2j/ui/client/chui/driver/swing/VT100SwingRenderer.java => src/com/goldencode/p2j/ui/client/chui/driver/VT100RendererImpl.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/AbstractHandlerCommons.java => src/com/goldencode/p2j/web/AbstractHandlerCommons.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/DojoToolkitHandler.java => src/com/goldencode/p2j/web/DojoToolkitHandler.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/PushMessagesWorker.java => src/com/goldencode/p2j/ui/client/driver/web/PushMessagesWorker.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/WatchdogTimer.java => src/com/goldencode/p2j/ui/client/driver/web/WatchdogTimer.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/WebScreenDriver.java => src/com/goldencode/p2j/ui/client/driver/web/EmbeddedWebServer.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/chui_web.html => src/com/goldencode/p2j/main/web_client.html
  src/com/goldencode/p2j/ui/client/chui/driver/web/index.html => src/com/goldencode/p2j/ui/client/driver/web/index.html
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/beep.mp3 => src/com/goldencode/p2j/ui/client/driver/web/res/beep.mp3
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/beep.ogg => src/com/goldencode/p2j/ui/client/driver/web/res/beep.ogg
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/beep.wav => src/com/goldencode/p2j/ui/client/driver/web/res/beep.wav
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/tty.clipboard.js => src/com/goldencode/p2j/ui/client/driver/web/res/p2j.clipboard.js
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/tty.js => src/com/goldencode/p2j/ui/client/driver/web/res/p2j.js
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/tty.keyboard.js => src/com/goldencode/p2j/ui/client/driver/web/res/p2j.keyboard.js
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/tty.keymap.js => src/com/goldencode/p2j/ui/client/driver/web/res/p2j.keymap.js
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/tty.screen.js => src/com/goldencode/p2j/ui/client/chui/driver/web/res/p2j.screen.js
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/tty.socket.js => src/com/goldencode/p2j/ui/client/driver/web/res/p2j.socket.js
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/tty.sound.js => src/com/goldencode/p2j/ui/client/driver/web/res/p2j.sound.js
  src/com/goldencode/p2j/ui/client/driver/GenericWebSocketCreator.java => src/com/goldencode/p2j/ui/client/driver/web/GenericWebSocketCreator.java
  src/com/goldencode/p2j/ui/client/driver/swing/AbstractSimulator.java => src/com/goldencode/p2j/ui/client/driver/swing/ContentPane.java
  src/com/goldencode/p2j/ui/client/gui/driver/GuiOptions.java => src/com/goldencode/p2j/ui/client/gui/driver/GuiConstants.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/EditableWidget.java => src/com/goldencode/p2j/ui/client/gui/driver/EditableWidget.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/GuiClient.java => src/com/goldencode/p2j/ui/client/gui/driver/swing/SwingGuiClient.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/GuiSimulator.java => src/com/goldencode/p2j/ui/client/gui/driver/swing/SwingEmulatedWindow.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/MouseHandler.java => src/com/goldencode/p2j/ui/client/gui/driver/MouseHandler.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/MouseMoveable.java => src/com/goldencode/p2j/ui/client/gui/driver/MouseMoveable.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/MousePopupable.java => src/com/goldencode/p2j/ui/client/gui/driver/MousePopupable.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/MouseResizeable.java => src/com/goldencode/p2j/ui/client/gui/driver/MouseResizeable.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/MouseTooltipWorker.java => src/com/goldencode/p2j/ui/client/gui/driver/MouseTooltipWorker.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/MouseWidgetAction.java => src/com/goldencode/p2j/ui/client/gui/driver/MouseWidgetAction.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/PaintPrimitives.java => src/com/goldencode/p2j/ui/client/gui/driver/PaintPrimitives.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/PaintStructure.java => src/com/goldencode/p2j/ui/client/gui/driver/PaintStructure.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/SwingGuiPrimitives.java => src/com/goldencode/p2j/ui/client/gui/driver/GuiPrimitivesImpl.java
  src/com/goldencode/p2j/ui/client/swing/ColorMapper.java => src/com/goldencode/p2j/ui/client/driver/swing/ColorMapper.java
  src/com/goldencode/p2j/ui/client/swing/ContainerOps.java => src/com/goldencode/p2j/ui/client/driver/swing/ContainerOps.java
  src/com/goldencode/p2j/ui/client/swing/ImageXfer.java => src/com/goldencode/p2j/ui/client/driver/swing/ImageXfer.java
  src/com/goldencode/p2j/ui/client/swing/LinuxKeyboardReader.java => src/com/goldencode/p2j/ui/client/driver/swing/LinuxKeyboardReader.java
  src/com/goldencode/p2j/ui/client/swing/PopupTrigger.java => src/com/goldencode/p2j/ui/client/driver/swing/PopupTrigger.java
  src/com/goldencode/p2j/ui/client/swing/SwingHelper.java => src/com/goldencode/p2j/ui/client/driver/swing/SwingHelper.java
  src/com/goldencode/p2j/ui/client/swing/SwingKeyboardReader.java => src/com/goldencode/p2j/ui/client/driver/swing/SwingKeyboardReader.java
  src/com/goldencode/p2j/ui/client/swing/WinKeyboardReader.java => src/com/goldencode/p2j/ui/client/driver/swing/WinKeyboardReader.java
modified:
  build.xml
  src/com/goldencode/p2j/main/Broker.java
  src/com/goldencode/p2j/main/ProcessClientSpawner.java
  src/com/goldencode/p2j/main/StandardServer.java
  src/com/goldencode/p2j/main/WebClientBuilder.java
  src/com/goldencode/p2j/main/WebClientBuilderOptions.java
  src/com/goldencode/p2j/main/WebClientBuilderParameters.java
  src/com/goldencode/p2j/main/WebClientSpawner.java
  src/com/goldencode/p2j/main/WebHandler.java
  src/com/goldencode/p2j/ui/ColorPalette.java
  src/com/goldencode/p2j/ui/ColorRgb.java
  src/com/goldencode/p2j/ui/ColorTable.java
  src/com/goldencode/p2j/ui/FrameWidget.java
  src/com/goldencode/p2j/ui/Keyboard.java
  src/com/goldencode/p2j/ui/chui/ThinClient.java
  src/com/goldencode/p2j/ui/client/Cell.java
  src/com/goldencode/p2j/ui/client/ColorManager.java
  src/com/goldencode/p2j/ui/client/FontManager.java
  src/com/goldencode/p2j/ui/client/OutputManager.java
  src/com/goldencode/p2j/ui/client/UiUtils.java
  src/com/goldencode/p2j/ui/client/chui/driver/ChuiPrimitives.java
  src/com/goldencode/p2j/ui/client/chui/driver/batch/BatchDriver.java
  src/com/goldencode/p2j/ui/client/chui/driver/batch/BatchHelper.java
  src/com/goldencode/p2j/ui/client/chui/driver/batch/BatchPrimitives.java
  src/com/goldencode/p2j/ui/client/chui/driver/console/ConsoleDriver.java
  src/com/goldencode/p2j/ui/client/chui/driver/console/ConsolePrimitives.java
  src/com/goldencode/p2j/ui/client/chui/driver/swing/ChuiClient.java
  src/com/goldencode/p2j/ui/client/chui/driver/swing/ChuiSimulator.java
  src/com/goldencode/p2j/ui/client/chui/driver/swing/SettingsPanel.java
  src/com/goldencode/p2j/ui/client/chui/driver/swing/SwingChuiDriver.java
  src/com/goldencode/p2j/ui/client/chui/driver/swing/TerminalMenu.java
  src/com/goldencode/p2j/ui/client/chui/driver/swing/VT100SwingKeyboard.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/ChuiWebDriver.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/ChuiWebPageHandler.java
  src/com/goldencode/p2j/ui/client/chui/driver/web/ChuiWebSimulator.java
  src/com/goldencode/p2j/ui/client/driver/ScreenDriver.java
  src/com/goldencode/p2j/ui/client/event/EventManager.java
  src/com/goldencode/p2j/ui/client/event/MouseEvt.java
  src/com/goldencode/p2j/ui/client/gui/ButtonGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/CaptionButton.java
  src/com/goldencode/p2j/ui/client/gui/FillInGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/FrameGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/GuiColorResolver.java
  src/com/goldencode/p2j/ui/client/gui/GuiFontResolver.java
  src/com/goldencode/p2j/ui/client/gui/ImageGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/LabelGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/PopupMenu.java
  src/com/goldencode/p2j/ui/client/gui/PopupOption.java
  src/com/goldencode/p2j/ui/client/gui/RectangleGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/ScrollBarGuiButton.java
  src/com/goldencode/p2j/ui/client/gui/ScrollBarGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/ScrollableListGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/StatusLineGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/TextGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/ToolTip.java
  src/com/goldencode/p2j/ui/client/gui/WindowGuiImpl.java
  src/com/goldencode/p2j/ui/client/gui/WindowLayout.java
  src/com/goldencode/p2j/ui/client/gui/WindowTitleBar.java
  src/com/goldencode/p2j/ui/client/gui/WindowWorkSpace.java
  src/com/goldencode/p2j/ui/client/gui/driver/GuiDriver.java
  src/com/goldencode/p2j/ui/client/gui/driver/GuiOutputManager.java
  src/com/goldencode/p2j/ui/client/gui/driver/GuiPrimitives.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/SwingGuiDriver.java
  src/com/goldencode/p2j/ui/client/widget/AbstractWidget.java
  src/com/goldencode/p2j/ui/client/widget/Widget.java
  src/com/goldencode/p2j/util/ProcessDaemon.java
  src/com/goldencode/terminal/VT100AbstractKeyboard.java
  src/com/goldencode/terminal/VT100AbstractRenderer.java
  src/com/goldencode/p2j/ui/client/driver/swing/AbstractClient.java
  src/com/goldencode/p2j/ui/client/chui/driver/TerminalOptions.java
  src/com/goldencode/p2j/ui/client/chui/driver/VT100RendererImpl.java
  src/com/goldencode/p2j/web/AbstractHandlerCommons.java
  src/com/goldencode/p2j/web/DojoToolkitHandler.java
  src/com/goldencode/p2j/ui/client/driver/web/PushMessagesWorker.java
  src/com/goldencode/p2j/ui/client/driver/web/WatchdogTimer.java
  src/com/goldencode/p2j/ui/client/driver/web/EmbeddedWebServer.java
  src/com/goldencode/p2j/main/web_client.html
  src/com/goldencode/p2j/ui/client/driver/web/index.html
  src/com/goldencode/p2j/ui/client/driver/web/res/p2j.clipboard.js
  src/com/goldencode/p2j/ui/client/driver/web/res/p2j.js
  src/com/goldencode/p2j/ui/client/driver/web/res/p2j.keyboard.js
  src/com/goldencode/p2j/ui/client/driver/web/res/p2j.keymap.js
  src/com/goldencode/p2j/ui/client/chui/driver/web/res/p2j.screen.js
  src/com/goldencode/p2j/ui/client/driver/web/res/p2j.socket.js
  src/com/goldencode/p2j/ui/client/driver/web/res/p2j.sound.js
  src/com/goldencode/p2j/ui/client/driver/web/GenericWebSocketCreator.java
  src/com/goldencode/p2j/ui/client/driver/swing/ContentPane.java
  src/com/goldencode/p2j/ui/client/gui/driver/GuiConstants.java
  src/com/goldencode/p2j/ui/client/gui/driver/EditableWidget.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/SwingGuiClient.java
  src/com/goldencode/p2j/ui/client/gui/driver/swing/SwingEmulatedWindow.java
  src/com/goldencode/p2j/ui/client/gui/driver/MouseHandler.java
  src/com/goldencode/p2j/ui/client/gui/driver/MouseMoveable.java
  src/com/goldencode/p2j/ui/client/gui/driver/MousePopupable.java
  src/com/goldencode/p2j/ui/client/gui/driver/MouseResizeable.java
  src/com/goldencode/p2j/ui/client/gui/driver/MouseTooltipWorker.java
  src/com/goldencode/p2j/ui/client/gui/driver/MouseWidgetAction.java
  src/com/goldencode/p2j/ui/client/gui/driver/PaintPrimitives.java
  src/com/goldencode/p2j/ui/client/gui/driver/PaintStructure.java
  src/com/goldencode/p2j/ui/client/gui/driver/GuiPrimitivesImpl.java
  src/com/goldencode/p2j/ui/client/driver/swing/ColorMapper.java
  src/com/goldencode/p2j/ui/client/driver/swing/ContainerOps.java
  src/com/goldencode/p2j/ui/client/driver/swing/ImageXfer.java
  src/com/goldencode/p2j/ui/client/driver/swing/LinuxKeyboardReader.java
  src/com/goldencode/p2j/ui/client/driver/swing/PopupTrigger.java
  src/com/goldencode/p2j/ui/client/driver/swing/SwingHelper.java
  src/com/goldencode/p2j/ui/client/driver/swing/SwingKeyboardReader.java
  src/com/goldencode/p2j/ui/client/driver/swing/WinKeyboardReader.java

In addition to the list of refactoring in note 651, I had to significantly refactor interactive process launching, because it was deeply integrated between ProcessDaemon and the UI simulator classes.

I have tested the ConsoleDriver and SwingChuiDriver with some standalone testcases, including process launching. I don't know of any bugs there.

I then tested the ConsoleDriver, BatchDriver, SwingChuiDriver and ChuiWebDriver using the IM POC code. The known issues which I'm working on so far:

  • Although I can execute interactive child processes in the swing ChUI client using standalone testcases, trying to start batchmaster in IM POC does not work in the swing client. It does work from the native client (ConsoleDriver), so something is regressed there. It might also be a regression in BatchDriver.
  • The ChUI web client is broken right now. I did make some pretty deep changes there, including loading resources from a list of locations so I suspect that is the issue. I don't know what else is hiding behind that.

General use of the IM POC code (batchmaster, menus, address create, address lookup) all worked well except what is noted above.

I haven't tried MAJIC yet.

Hynek/Constantin: please review this code.

Anton: please apply this code to your web client environment and debug the failure in the ChUI web client. You should be able to use any simple converted testcase that you already have. The failure occurs after the web client process is started but before the main "terminal" window can be brought up.

I will look at the process launching failure.

#655 Updated by Greg Shah almost 6 years ago

Two other notes about my update:

1. The node "chuiWeb" in the directory needs to be changed to "webClient".

2. The client:driver:type for swing_chui_frame is now chui_swing.

3. The client:driver:type for swing_gui_frame is now gui_swing.

I haven't tested GUI yet.

#656 Updated by Greg Shah almost 6 years ago

Here is the update merged with bzr 10849.

#657 Updated by Constantin Asofiei almost 6 years ago

Greg Shah wrote:

Here is the update merged with bzr 10849.

There is a compilation problem in WidgetBrowserAspect:181:

               driver.drawHighlight(loc.x, loc.y, dim.width, dim.height, Color.RED);

#658 Updated by Constantin Asofiei almost 6 years ago

Another issue: FontDetails.key is never set - it should be set by FontManager$WorkArea.getFont.

#659 Updated by Greg Shah almost 6 years ago

Both are fixed with this update.

#660 Updated by Greg Shah almost 6 years ago

Merged with bzr 10850.

#661 Updated by Greg Shah almost 6 years ago

Merged with bzr 10853.

#662 Updated by Greg Shah almost 6 years ago

#663 Updated by Greg Shah almost 6 years ago

Merged with bzr 10855.

#664 Updated by Anton Breaur almost 6 years ago

After investigating the web Chui issue, I found that the ChuiWebDriver.getWelcomePageUrl method builds the wrong URL, so the index.html cannot be found.
I have fixed the URL and added a ContextLoader for "/*" path so index.html loads now.
This doesn't eliminate the whole issue, as the other resources (p2j.js and modules, sound files) still cannot be loaded.
Dojo toolkit does load.
This should be related to the resource handlers for "/client/*" and "/common/*"

#665 Updated by Anton Breaur almost 6 years ago

Adding another subfolder to res folders as such: com/goldencode/p2j/ui/client/chui/driver/web/res/client and com/goldencode/p2j/ui/client/driver/web/res/common solves the static resource loading issue.
Although all the docs and examples point out that wraping a ResourceHandler under a ContextHandler should enable loading resources for given path from a specific folder, the url path is added to location as well, so that is why we need the suggested folder structure.
This, along with the update from note 664 solves the web chui failure.

#666 Updated by Greg Shah almost 6 years ago

After investigating the web Chui issue, I found that the ChuiWebDriver.getWelcomePageUrl method builds the wrong URL, so the index.html cannot be found.

Please provide the specific result that is incorrect so that I can understand what is wrong.

In addition, post the code that is your proposed fix.

#667 Updated by Greg Shah almost 6 years ago

Adding another subfolder to res folders as such: com/goldencode/p2j/ui/client/chui/driver/web/res/client and com/goldencode/p2j/ui/client/driver/web/res/common solves the static resource loading issue.

The described solution is exactly what we don't want to do. The whole point of how I've split things allows client-specific and client-generic code to be split up.

I am very surprised that we can't make things load from 3 separate directories. That makes no sense. Please provide more details on the specific Jetty limitations that can't be overcome.

Although all the docs and examples point out that wraping a ResourceHandler under a ContextHandler should enable loading resources for given path from a specific folder,

We are not just limited to using this approach.

the url path is added to location as well,

Please provide specifics here to explain exactly what you mean.

This, along with the update from note 664 solves the web chui failure.

How much have you tested "beyond" just loading the client? Have you executed real converted 4GL code?

#668 Updated by Greg Shah almost 6 years ago

We are not just limited to using this approach.

To be clear, my preference is to use predefined classes from Jetty to implement this common, client (gui) and client (chui) approach. I'd be surprised if that can't be done.

But if it really can't be done, then we can just implement our own solution as a single handler. Right?

#669 Updated by Anton Breaur almost 6 years ago

After investigating the web Chui issue, I found that the ChuiWebDriver.getWelcomePageUrl method builds the wrong URL, so the index.html cannot be found.

Please provide the specific result that is incorrect so that I can understand what is wrong.

In addition, post the code that is your proposed fix.

The ChuiWebDriver.getWelcomePageUrl uses ChuiWebPageHandler package name to point to index.xml as such:

StringBuilder sb = new StringBuilder(pageContext);
String packageName = EmbeddedWebServerImpl.class.getPackage().getName();
return sb.append(packageName.replace('.', '/')).append(pageFile).toString();

This is poiting to "com.goldencode.p2j.ui.client.chui.driver.web". Instead, index.html is located at: "com.goldencode.p2j.ui.client.driver.web", so we could use WebPageHandler package for that:
String packageName = WebPageHandler.class.getPackage().getName();

The described solution is exactly what we don't want to do. The whole point of how I've split things allows client-specific and client-generic code to be split up.

I am very surprised that we can't make things load from 3 separate directories. That makes no sense. Please provide more details on the specific Jetty limitations that can't be overcome.

Yes, I do agree with that. I will further investigate the ResourceHandler code. As I see now, the path url is added to the location of the resource in the jar, so when trying to load "/common/p2j.js", the resource handler will look for it in "/com/goldencode/p2j/ui/client/driver/web/res/common/p2j.js".

How much have you tested "beyond" just loading the client? Have you executed real converted 4GL code?

I have run the ask.p and primes.p testcase and it is working fine.

#670 Updated by Anton Breaur almost 6 years ago

Although all the docs and examples point out that wraping a ResourceHandler under a ContextHandler should enable loading resources for given path from a specific folder, the url path is added to location as well, so that is why we need the suggested folder structure.

I was wrong about this. The ResourceHandlers behave as they should. It was another ContextHandler I added that was messing with the normal handling flow. I removed it and got other results.

Besides the index.html problem I mentioned in note 664, the real issue was related to the HandlerCollection in GenericWebServer.configWsContext method.
The handler collection kept all the needed specific handlers (web page, resource and dojo handlers) in the right order and it was "wrapped" in a ContextHandler with path "/" and resource base "." .
It looks like that context handler was intended to pass the requests to web page handlers, but it was sitting on top of the whole handler collection.
The requests never make it into the HandlerCollection and the result is a 404 not found.

I had removed this parent ContextHandler and wrapped each ChuiWebPageHandler instance under a ContextHandler("/"). This means that the server handler is the direct handler collection, so the GenericWebServer.configWsContext only has to return the handlerCollection.

In EmbeddedWebServerImpl.startupServer() :

// process each handler
for (int i = 0; i < hdlrs.length; i++) {
hdlrs[i].setAuthorizationToken(token);
ContextHandler contextHandler = new ContextHandler();
contextHandler.setHandler(hdlrs[i]);
server.addHandler(contextHandler);
}

In GenericWebServer :

private Handler configWsContext() {
// This commented code is not needed
// ContextHandler context = new ContextHandler();
// context.setContextPath(contextPath);
// context.setClassLoader(Thread.currentThread().getContextClassLoader());
//
// context.setHandler(handlerCollection);
return handlerCollection;
}

The method name and javadoc should be updated, but I left it as it is for review reasons.

Ask.p and primes.p are working as expected with these changes.

#671 Updated by Greg Shah almost 6 years ago

Please post your changed versions of my files (only include those files you have changed).

#672 Updated by Greg Shah almost 6 years ago

Merged to bzr 10863.

#673 Updated by Greg Shah almost 6 years ago

This includes a fix for the batch process launching from Swing clients.

I have also tried to implement the fixes for the ChUI web client, as described above. Those are included. Although I can now get to the index.html, successfully login to the OS and create a client. The web_client.html page that is loaded does not display anything.

Either I don't fully understand what was reported above or something is missing.

Anton, please post the necessary changes ASAP. The web client problems are the only known problems right now.

I'm going to test this update with MAJIC next.

#674 Updated by Anton Breaur almost 6 years ago

Greg, I don't understand the web_client.html issue. The web chui client does not need that resource.
If you applied my changes and index.html is loading, the rest should be loading as well.
I will post my changes asap.

#675 Updated by Greg Shah almost 6 years ago

The web_client.html is the login page. This works with my 0515b update. It allows the ChUI-specific index.html to be loaded. This part loads but is probably not loading all the associated resources as the result is a blank page.

#676 Updated by Greg Shah almost 6 years ago

I ran main regression testing on the 0515b update last night. GSO 418 did fail in addition to the expected failure in step 40 of tc_job_002. I manually retested GSO 418 and the failure could not be duplicated. I consider main regression testing passed.

I am running CTRL-C testing now.

My plan is to integrate the ChUI web client fix (it should be in code that can't be reached in regression testing) and check this in, assuming the CTRL-C testing passes.

I have also seen a potential regression in GUI where the fonts don't look right. I've asked Constantin to look at it. I'll integrate that change as well (it probably won't require any regression testing as it is likely in GUI-specific code) before check in.

#677 Updated by Anton Breaur almost 6 years ago

The web_client.html is the login page. This works with my 0515b update. It allows the ChUI-specific index.html to be loaded. This part loads but is probably not loading all the associated resources as the result is a blank page.

This works for me. I am packing the classes now and doing another test.

#678 Updated by Anton Breaur almost 6 years ago

This fixes the web chui failure. It was created on top of the merged with revision 10855 version, and not yet tested for the 10863 merge.

#679 Updated by Anton Breaur almost 6 years ago