FWD » Core Development » Conversion Tools

The scope of the reports should be chosen by the user:

- file-level
- a specific/arbitrary list of files
- an entire directory tree
- a list of directory trees
- the entire project

It would be nice to support any combination of the first 4 options.

The sorting of the output should be chosen by the user:

- alpha
- # matches
- ???

The user should also pick the report to view and optionally be able to control paging options.

It seems that there are these GWT capabilities http://www.gwtproject.org/doc/latest/tutorial/JSON.html, http://www.gwtproject.org/doc/latest/DevGuideCodingBasicsJSON.html and http://www.gwtproject.org/doc/latest/DevGuideCodingBasicsOverlay.html. Json objects can be shared between GWT generated code and hand-written code.

Thanks, I'm reading through the GWT project website now.

What I'm trying to get at is your impression of GWT and specifically the RPC functionality, based on the work you are doing in #3222. Good, bad, ugly? Pros, cons? If it's been a good choice on balance, is there anything I should look at in the 3222a branch that may be a candidate for reuse in this task, or is it best to just start with GWT directly? I don't have a need for FWD-specific data types or other code that is specific to FWD, but if there is any generic value-add code I should look at for UI or RPC, please point me to it.

Eric Faulhaber wrote:

Hynek, Sergey, I've added you as watchers to this issue because I would like your feedback, based on your use of GWT for #3222.

I am reworking FWD's reporting to be presented via a web application, instead of the static HTML generation we do today. We will be using JS on the front end and Java on the back end. I haven't decided on the framework(s) yet, but I wanted to get your impression of GWT as both an RPC mechanism and as a UI framework.

Eric, let me first ask about the design of the report app. Why is the backend needed? Is the data set to large to fit into the client JS completely, or are there new use cases to be implemented that would require a database?

Hynek Cihlar wrote:

Eric, let me first ask about the design of the report app. Why is the backend needed? Is the data set to large to fit into the client JS completely, or are there new use cases to be implemented that would require a database?

Yes and yes.

Today, we process the report criteria as a batch and store intermediate results in an H2 database. Then, in a second batch phase, we access that data to generate static HTML. This can get very large (multiple tens of GBs in some cases). There's currently no facility to integrate custom reports with the canned ones.

We want to keep the existing report criteria-crunching Java/TRPL code, since that's working well. But we will drop the second phase (static HTML generation). Instead, we will present the views dynamically, based on the database content. This will give the user a lot more flexibility in filtering and sorting data. We also want them to be able to define their own, custom reports, and retain and present those as first-class citizens alongside the canned reports. In addition, we will be using hand-written JS to present some interesting visualization/analytics.

As you can imagine, the presentation will be data-grid/table heavy. GWT seems to have good capabilities in that area. My main concern is getting boxed into the GWT "way", thereby potentially making it harder to do simple things in JS. GWT seems to have some reasonable answers there, but so far for me, that's just based on reading, not doing.

We will be using JSON between the back and front ends, so I want a simple way to do that, including the case where I need to drop into a more traditional use of JS.

I'd like to consider the RPC and UI aspects of GWT independently if it's feasible to use one without the other. If there's a good reason to use both, I'm open to it.

So, I just want to get your opinions, since you're both deep in GWT coding right now. In addition, it would be useful to know if there's any reusable code you already have written to solve common problems, to avoid reinventing the wheel.

Eric, 3222a exposes com.goldencode.p2j.admin.AdminServerImpl remotely to the browser GWT client. It implements an administration use case only. There are common dialogs and widgets that are used by GWT administration client. I think that you can find what libraries are used and how the gwt project can be built. It seems that there are no common code. The report task is new for me and it is not clear now. I need to read documentation if it is available.

Eric Faulhaber wrote:

What I'm trying to get at is your impression of GWT and specifically the RPC functionality

• We already had an existing application which needed to be ported with fair amount of generic code, code that could be to some extent reused (parts of the UI logic, data transfer objects, remote apis).
• The structure of the original Admin maps closely to the GWT's programming model.
• We had an existing RPC mechanism that was close to the GWT's servlet-based RPC mechanism.

The GWT bads:

• A bit outdated programming model - GWT uses a so called Model-View-Presenter architecture. It allows to decouple the UI views from UI logic. The idea is to have multiple views implementations to cover desktop browsers, mobile, etc. But with HTML5 the paradigm changed thanks to the responsive design. And so the advantage of Model-View-Presenter becomes questionable.
• Very verbose - the Model-View-Presenter architecture together with the GWT event model makes you to write a LOT of code compared to other more recent stacks.
• While GWT is very sophisticated, it lacks patterns to implement applications that require nested views (i.e. master view template with header, content, footer and dynamically changed content). We solve this by using a 3rd party addon called GWTP.
• It provides a thick layer between the app code and the web standards of HTML5. In other words it does things very differently and in its proprietary way.
• It is complex, and so if something goes wrong, you are in deep...
• Steep learning curve.

• Big community, although it diminishes in favor of the newer stacks.
• Plenty of 3rd party libraries, component libraries.
• You don't need to know much about HTML5, which may be good for Java developers.

I would probably look for a different stack in your situation. With new app, no legacy code, your requirement to use JS directly. I don't think the bads are worthy the JSON RPC which can be easily implemented anyway using other technologies like JAX-RS/Jersey, Spring REST, Play, etc.

I recently fell in love with Aurelia. It is very lightweight web framework trying to stay close to the HTML5 standards. It keeps things simple, not over-engineering like Angular (both 1 and 2) for example.

AngularJS can be use as a client of RESTful Web Services, https://docs.angularjs.org/tutorial/step_10, https://docs.angularjs.org/tutorial/step_13.

The objective for this task is to create a Java-based reporting engine that has an embedded web server (leveraging Jetty) whose primary purpose is to drive a web-based UI which will be written in HTML5/CSS/Javascript.

The reporting engine will leverage TRPL for generating the report results and will reuse as much of the current ReportDriver/ReportWorker infrastructure as makes sense.

Let's assume GWT is out. I think Hynek's comments are pretty compelling in this regard.

Some thoughts:

1. REST vs RPC

The idea of REST is "everything is a resource", where resources are identified by URLs and one can only operate on these resources via GET/PUT/DELETE/POST verbs.

Although some of the reporting support could be made to match into this idea, I'm not convinced that it is a natural fit.

I think a small API can be designed to handle the full set of functionality needed. Calling this API via some kind of RPC would be optimal.

I'm leaning away from REST.

2. Client Environment

We know that we have several important visualization requirements. The obvious ones are a "source view" (which will show the original 4GL source code and the linked AST, side by side), some possible charting and a call-graph visualizer. All of these will most likely be written in pure javascript, though we will be leveraging framework code where possible. Eric has found d3 to be a pretty impressive fit for these heavy visualization needs.

The rest of the UI will be built from more traditional UI elements, however we do not want to rely heavily on HTML forms or controls. Rather, we expect to use some javascript UI framework in a "single page application" mode. This will require deciding on a framework and there will be some ramp-up. I would like to pick something with less ramp-up.

It is easy to access Dojo from our FWD environment. I'm not sure if it is the right framework. One thing I like about it is that it has both declarative and imperative approaches to usage.

Angular, React and some other frmaes are popular. I worry that they are too heavy on the declarative side of design. I tend to like the imperative approach. In other words, I want the HTML to be minimal and for virtually everything to be in JS under our control. I see no intrinsic value to encoding significant parts of our web application in HTML.

The Aurelia approach seems heavy on declarative too, but perhaps that is just my misunderstanding.

3. JSON

Coding the UI in JS means that the best format for data exchange with the Java server will be JSON.

I think a simple answer for Java JSON usage is Jackson:

https://github.com/FasterXML/jackson
https://en.wikipedia.org/wiki/Jackson_(API)
http://wilddiary.com/serialize-java-objects-json-back/

4. RPC

One choice is go "old school" and pass JSON via AJAX. Then we can code the servlets to handle this and use Jackson to handle the JSON parts on the server side. This is reasonably simple and we can get started quickly. The learning curve is small. On the other hand, we end up doing things ourselves that might come for free. I'm not convinced yet that the stuff that comes for free is worth it.

I've looked into JSON-RPC. It seems simple/clean but it doesn't seem to be a vibrant community. Most tech associated with it seems old and somewhat dead:

https://en.wikipedia.org/wiki/JSON-RPC
http://software.dzhuvinov.com/json-rpc-2.0-server.html
https://dojotoolkit.org/reference-guide/1.10/dojox/rpc.html#dojox-rpc

Of the other common options, Play (https://www.playframework.com/) is probably the one that is popular, has a vibrant ecosystem and is reasonably close to our preferred design.

A few notes about the RPC. While I agree with Greg that the REST-like programming model isn't a good fit, I think we could benefit from one of the REST stacks. For an illustration this is how JAX-RS/Jersey "resource" handler looks like:

@Path("/api")
public class ReportServer
{
    @GET
    @Path("/queryFiles/{query}")
    @Produces(MediaType.APPLICATION_JSON)
    public List<FileEntry> queryFiles(@PathParam("query") String query,
                                      @QueryParam("maxResults") int maxResults)
    {
        // ...query results from backend
        return results;
    }
}

JAX-RS is a standard servlet api app with some extensions on top making REST application easier to develop. As seen in the example above, the API doesn't have to be modeled with the strict REST paradigm. With a simple configuration the above class is enough to serialize/deserialize JSON requests, the serialization is handled automatically by the JAX-RS implementation. In case of Jersey in particular, there are multiple JSON libraries to choose from, including Jackson.

Branch 1514a was rebased from trunk rev 11147.

Hynek Cihlar wrote:

A few notes about the RPC. While I agree with Greg that the REST-like programming model isn't a good fit, I think we could benefit from one of the REST stacks. For an illustration this is how JAX-RS/Jersey "resource" handler looks like:

[...]

JAX-RS is a standard servlet api app with some extensions on top making REST application easier to develop. As seen in the example above, the API doesn't have to be modeled with the strict REST paradigm. With a simple configuration the above class is enough to serialize/deserialize JSON requests, the serialization is handled automatically by the JAX-RS implementation. In case of Jersey in particular, there are multiple JSON libraries to choose from, including Jackson.

Hynek, I have a simple class which starts up a simple Jetty server with SSL, based on Greg's web server for the embedded hotel gui app. I would like to integrate Jersey and Jackson into this starting point, pulling in as few dependencies as possible. I'm trying to avoid pulling in Glassfish or Tomcat as the server, for example. I want to manage configuration programatically to the greatest degree possible and minimize the amount of declarative config necessary. In other words, I'm trying to make this server as much of a software appliance as possible. Have you already done something like this, or can you suggest a good tutorial or resource in this regard? Thanks.

Eric Faulhaber wrote:

Hynek, I have a simple class which starts up a simple Jetty server with SSL, based on Greg's web server for the embedded hotel gui app. I would like to integrate Jersey and Jackson into this starting point, pulling in as few dependencies as possible. I'm trying to avoid pulling in Glassfish or Tomcat as the server, for example. I want to manage configuration programatically to the greatest degree possible and minimize the amount of declarative config necessary. In other words, I'm trying to make this server as much of a software appliance as possible. Have you already done something like this, or can you suggest a good tutorial or resource in this regard? Thanks.

• http://nikgrozev.com/2014/10/16/rest-with-embedded-jetty-and-jersey-in-a-single-jar-step-by-step/
• https://www.acando.no/thedailypassion/200555/a-rest-service-with-jetty-and-jersey

AFAIK JAX-RS implementations (like Jersey) do a lot of bootstrap processing in order to auto-discover all the application components. If you want the full configuration control then this should be turned off. There is a property that controls the auto-discover behavior: https://jersey.java.net/apidocs/2.25/jersey/org/glassfish/jersey/CommonProperties.html#FEATURE_AUTO_DISCOVERY_DISABLE and https://jersey.java.net/documentation/latest/appendix-properties.html.

Regarding the Jersey's dependencies, I can imagine its (Maven) repository metadata declares a lot of stuff that may not be required. This should be explicitly excluded in the build script.

Branch 1514a has been rebased from trunk 11150. The latest revision is 11155.

The following is a description of the transport and API approach that we are using. Eric, please correct or add to this as needed.

• embedded Jetty web server in com.goldencode.p2j.report.server.ReportWebServer
• launcher in Hotel GUI deploy/server/report.sh, reporting (ant rpt) and call graph (ant callgraph) must be run before starting the report server
• single page web application is loaded from com.goldencode.p2j.report.web.res.index.html
• after the placeholder login dialog is past, a websocket is established to the report server
• on the server side, com.goldencode.p2j.report.ReportProtocol is the main websockets handler
• a generic API approach is provided with that class:
  • create a new class
  • implement the @WebApi annotation on each API method (see ReportApi which is already working and CallGraphApi which is a placeholder for the call graph work)
  • use message IDs that are unique to the API (each one gets a range of 1000 dedicated IDs)
    • report API is MSG_BASE + 1000
    • call graph API is MSG_BASE + 2000
    • AST API is MSG_BASE + 3000
  • call ReportProtocol.registerRequestApi() from ReportWebServer to register the API
  • all data in each API method which is passed as parameters or return values must be:
    • an object or array of objects of one of the basic data types like Integer, Long, Boolean, String
    • a javabean that aggregates/contains these same types
    • the collection classes do work, though it is probably a good idea to use them lightly to avoid the extra overhead
  • Jackson is used to automatically marshall/unmarshall the API parameters/return value to/from JSON
  • on the JS client side, the websocket helper functions are in wsbase.js, which allows upcalls (with callbacks to process the data) from report.js, callgraph.js or ast.js
  • right now we only use text mode for the websocket, if we need it we can always use binary mode later

Eric did give Jersey a try. RESTful support is pretty deeply encoded in Jersey and everything is a bit more complicated than needed as a result. The websocket approach is simpler and comes with more control and less baggage. We are not contorting ourselves into a REST approach and the result is pretty clean. After doing some prototype work in both approaches, it became clear that websockets was the right way to go.

Greg Shah wrote:

The following is a description of the transport and API approach that we are using. Eric, please correct or add to this as needed.

• W.r.t. using the collection classes, I haven't noticed any additional overhead at all of using List compared to arrays; Map works very well, too, and makes things convenient in the event creating a bean is overkill for certain data. The JSON produced is the same for a List as an array, and for a Map as a bean. I doubt the marshalling an unmarshalling itself has any significant overhead, at least for these two collection types.
• Java beans are not limited to the simple types; you can have more complex classes as well, as long as
  • every member you want to [un]marshall ultimately (directly or indirectly) is built from these types; and
  • you provide a getter/setter for each of those members (and within the objects referenced by those members).
• You can aggregate bean instances and more complex objects into arrays and collections as well, not just the simple types.
• Note that array elements need not be the same type (e.g., arrays of Object work well for passing parameters of different types).
• Avoid feeding deeply nested data structures to Jackson, as you will end up with a StackOverflowException. For example, I tried organizing AstNode into a true tree hierarchy that paralleled Aast, but the recursion was too much for Jackson. I ended up having to package an AST as a discrete list of AstNode objects, to be reconstituted into a tree structure on the JavaScript side.
• Since JSON is so duplicative with identifiers, I would recommend using the Jackson annotation @JsonProperty to provide short names for bean properties (e.g., see AstNode). It makes the JavaScript program a bit more terse to read, but I imagine repeating myReallyLongPropertyName in the JSON for long lists of marshalled objects adds significant overhead.
• You can use Jackson's @JsonIgnore annotation to ignore fields in a class.

Sergey: here are some details on the File Set functionality.

The idea is that we are creating arbitrary lists of files in a given project. Each one is called a "file set". There will be multiple file sets, each of which can be chosen by the user to look at the report or call graph results based on that list of AST files instead of the full project file set. There is always the "all files" set implicitly included in the list of file sets.

Right now, Eric has these file sets defined in the report database. We are needing two things:

1. (top priority) A web UI (JQueryUI based) to select one of the file sets from the list. This would also be used to select the "all files" (implicit) file set. This is the "File List" page of the analytics site. I would think that each file set would have a name and could be selected from this simple list.

2. A web UI to create/edit/view a file set. First, one has to name the file set. Also, a file set is either related to code ASTs or to schema ASTs. This is a boolean flag. To edit/view/select the file set contents (a list of files and directories), I would expect a side-by-side UI. On the left side, there would be a tree of all the (code or schema) AST files (and the directories containing them) in the project. The user should be able to expand and collapse the portions of this. Any file or directory should be selectable. Multiple selection would be useful. Then there would be an "Add" button that would add the selected items to the right side of the UI which would list all items in the file set. This could be done multiple times and both for editing and for creating the list. There would also be the ability to select items in the right side list and then you could press the "Remove" button to delete them from the file set (and they would no longer appear on the right side list). The meaning of selecting a directory is that all code/schema AST files that are contained in that directory tree (including subdirectories) would be included. It should be possible to select an arbitrary list of zero or more files and zero or more directories and that list will specify the exact list of code or schema ASTs that are in the file set.

It seems that the tree component that can be used as a file set creator is not implemented by JQuery UI framework. But there are 3rd party components free to use and we should select one of them for usage http://www.jqueryscript.net/tags.php?/tree%20view/.

Yes, the FancyTree looks pretty good. If it works well and is simple to use then it is OK. It has MIT license so it is OK to use.

The thing that is most urgent is item 1 in #1514-22. That is a much simpler UI.

• file_map - maps file and AST IDs to file names and type (schema, base file, cache file)
  • id identity - unique file ID
  • astid bigint - unique AST ID
  • type int - type of file (SRC_SCHEMA, SRC_CODE_BASE, SRC_CODE_CACHE)
  • name varchar - project relative, qualified, file name
• filter_profile - stores file set filter profile information
  • id identity - unique filter profile ID
  • default boolean - true if this is the default (include all) profile, else false
  • name varchar - descriptive name of the file set profile (UI should enforce this to be unique, so it is meaningful to a user, but there is no unique constraint at the database level, since this is not an identifier)
• file_filter - file names and wildcard specs of files included as part of a filter profile
  • pid bigint - foreign key to filter_profile primary key
  • type int - type of file/spec (SRC_SCHEMA, SRC_CODE_BASE, SRC_CODE_CACHE)
  • wild boolean - true if record represents a wildcard spec; false if it represents a file name
  • spec varchar - project-relative, qualified, file name or wildcard spec (trailing asterisk only supported at this time)

Note that the nature of the file_filter table is that it is inclusive (i.e., a white list) in nature. There is no way to specify an exclusion. So, only those files explicitly described by a file_filter.spec record will be included when the corresponding filter profile is chosen by the user.

One filter profile (the default, which includes all files) is generated when the report database is created. Others will need to be created by users.

Sergey, to get started, update to branch 1514a and to the latest hotel GUI project code. Run ant rpt to generate the report database. From hotel_gui/deploy/server, run ./report.sh and tail -f report_server.log.

The initial UI Greg refers to in #1514-22 (item 1) will just display the filter_profile.name items in a selection list and let the user choose the active one for the current session. To do this, we initially will need 2 new server APIs; one to retrieve the set of profile IDs and names from the server, and another to send the selected profile ID back to the server to make it active. I will add stubs for those if you take the UI part.

Later (not today), we will want to add the side-by-side tree view in the same UI, so the user can see (and edit) exactly which files are part of a filter profile.

Eric, is it correct that now you would like to get only read only filter_profiles list and to permit a user to select one of the predefined profiles. Is it possible for a user to add new profile to this list or update the selected profile from this list?
There is unclear issue. If filter_profile.default is true, then it includes all files. What is its file_filter? If filter_profile.default is false, then there is filter_profile.id and record from file_filter that defines files in this profile. Is it correct?

Sergey Ivanovskiy wrote:

Eric, is it correct that now you would like to get only read only filter_profiles list and to permit a user to select one of the predefined profiles.

Yes.

Is it possible for a user to add new profile to this list or update the selected profile from this list?

This is the idea, but I don't think it is possible to get this done today, so the first item is the high priority.

There is unclear issue. If filter_profile.default is true, then it includes all files. What is its file_filter?

It depends on the project, but I expect most will be based on the standard project layout we use. For example, for hotel GUI, the default profile looks like this:

select p.name, p.id, f.type, f.wild, f.spec from file_filter f join filter_profile p on f.pid = p.id where p.default = true;
NAME            ID      TYPE    WILD    SPEC  
Include All     1       1       TRUE    ./abl/*
Include All     1       1       TRUE    ./data/*
Include All     1       3       TRUE    ./abl/*
(3 rows, 0 ms)

If filter_profile.default is false, then there is filter_profile.id and record from file_filter that defines files in this profile. Is it correct?

I'm not sure what you mean, but I'll try to explain. Every record in file_filter has a foreign key reference (the pid column) to a single record in filter_profile. The records in file_filter describe the file or groups of files included in the filter profile.

Only one filter_profile record can be set as the default. This is set when reports are run. This restriction is not enforced at the database, but the server API will not change it. It is just a way that we can know which profile is the starting default. There is no other use for or difference based on the filter_profile.default column.

Thank you, it is clear now. will you provide stubs to get all profiles and to select one of the profiles? What are API for UI interfaces I can use for this task? I see that now Dojo, Bootstrap, D3 and JQuery-UI are used for the client.

Sergey Ivanovskiy wrote:

Thank you, it is clear now. will you provide stubs to get all profiles and to select one of the profiles? What are API for UI interfaces I can use for this task?

Yes, I've added these in rev 11244.

See ReportApi implementations for MSG_FILTER_INFO and MSG_SET_FILTER. The former is a full implementation, but it has not been tested. The latter is just a stub for now. See report.js for examples of how we use the API from the client side.

I see that now Dojo, Bootstrap, D3 and JQuery-UI are used for the client.

Yes, but the JQuery-UI implementation currently in use is minimal, limited only to those features needed by the Tabulator library. We will we need to pull in a larger set of features for general UI use.

Dojo is not really used at all beyond the initialization code in index.html, which was modelled after the Hotel GUI code. I was planning on removing it and replacing that initialization code with something else, but I haven't had time to look at it yet.

Bootstrap is only used for layout currently. I had no particular UI plans for it beyond this.

D3 is used heavily for visualization.

Ok, I will use JQuery UI. It seems that you forgot to commit new class FilterProfile. Is it a pure bean?

Sergey Ivanovskiy wrote:

Ok, I will use JQuery UI. It seems that you forgot to commit new class FilterProfile. Is it a pure bean?

Yes. Sorry, it's there now (rev. 11245).

Thanks, I get this list with one element followed by the active profile id at the end of this list. JQuery UI is very flexible, although I would like to fall back to unordered list and one button controls and then fix it by using the force of JQuery UI.

Sergey Ivanovskiy wrote:

Thanks, I get this list with one element followed by the active profile id at the end of this list. JQuery UI is very flexible, although I would like to fall back to unordered list and one button controls and then fix it by using the force of JQuery UI.

Using JQuery UI is not a hard requirement. If there is a simpler way to do it in the short term, let's do that.

I can't exactly picture what you mean with the unordered list and one button, but it might be quicker for you just to implement it and then I can see, rather than explaining it here.

FYI, the more I look at JQuery UI, the more I dislike it. The problems (in my opinion):

• It is very heavy in declarative syntax. Perhaps you can do everything programmatically, but all the examples show that most of the work is done by putting DOM elements into the tree and then running a function to make them look/work better. If I wanted to do that I wouldn't be using JQuery UI.
• The selection of widgets is quite weak. All the ones that I keep looking for not there and would need something else anyway.

I'm looking at Dojo now. It has a full progammatic approach for every Dijit widget. It seems closer to "the droid that I'm looking for".

Added Selected Profile label and selection control 11248. The styles for label and selector are not set accordingly to the parent view.

Sergey Ivanovskiy wrote:

Added Selected Profile label and selection control 11248. The styles for label and selector are not set accordingly to the parent view.

How does the selection control work? I can't seem to select anything. I added a second profile to the database manually, so there would be more than one item. On FF/Ubuntu, it appears with the first element highlighted in orange. On Chrome/Ubuntu, there is no background highlight, so the values are very hard to read. In both cases, it does not respond to any mouse action. Are there some CSS changes that were meant to accompany the last check-in?

Yes, I tried to change highlighted style, but without success and more JQuery ui should provide with $("#profileSelector").selectmenu(); selector widget, but it doesn't work with the custom version <script type="text/javascript" src="/3pl/jquery-ui-1.12.1.custom/jquery-ui.min.js"></script>.

Committed revision 11250 added my css tries.

Sergey Ivanovskiy wrote:

Yes, I tried to change highlighted style, but without success and more JQuery ui should provide with $("#profileSelector").selectmenu(); selector widget, but it doesn't work with the custom version <script type="text/javascript" src="/3pl/jquery-ui-1.12.1.custom/jquery-ui.min.js"></script>.

Please see Greg's note above (#1514-35) regarding Dojo and jQueryUI. Looks like Dojo is what we are going to be using instead of jQueryUI.

Eric Faulhaber wrote:

Sergey Ivanovskiy wrote:

Yes, I tried to change highlighted style, but without success and more JQuery ui should provide with $("#profileSelector").selectmenu(); selector widget, but it doesn't work with the custom version <script type="text/javascript" src="/3pl/jquery-ui-1.12.1.custom/jquery-ui.min.js"></script>.

Please see Greg's note above (#1514-35) regarding Dojo and jQueryUI. Looks like Dojo is what we are going to be using instead of jQueryUI.

To use dojo component it is required to use DJO loader, thus if some Dojo component is used by report.js, then it should be defined as AMD module listed required dependencies

define([dependencies declaration list], function (loaded list of dependencies) {
});

Really great! Thank you for fixing this. I didn't realize how badly broken the newer Dojo implementation is when you don't match the change to the new AMD module approach.

I want to make sure we are enabled for disconnected/offline usage. Eric has temporarily placed all our dependent 3rd party JS resources in src/com/goldencode/report/web/res/3pl/. Dojo is not there and I know we have it pulled into our build/lib/ during gradle dependency processing. Are we loading Dojo from the p2j/build/lib/dojo-1.10.0-distribution.zip?

Yes, it seems that dojo is loaded from this zip file by DojoToolkitHandler. If you need offline page, then you should place dojo in the application folder where main.js is located and
change folder name to dojo-toolkit. This folder should contain dojo, dijit, dojox folders. I committed these changes now

=== modified file 'src/com/goldencode/p2j/report/web/res/index.html'
--- src/com/goldencode/p2j/report/web/res/index.html    2017-06-03 13:51:48 +0000
+++ src/com/goldencode/p2j/report/web/res/index.html    2017-06-03 14:43:45 +0000
@@ -134,7 +134,7 @@
                        ]
          };
       </script>
-      <script data-dojo-config="async: true" src="/dojo-toolkit/dojo/dojo.js"></script>
+      <script data-dojo-config="async: true" src="./dojo-toolkit/dojo/dojo.js"></script>

       <!-- fwd analytics -->
       <link rel="stylesheet" href="/report.css" type="text/css">

Sergey, the combobox code for note 43 is not working in chrome.

It seems they are required for offline usages.

If we have it local to the install (in build/lib/), why do we need to make this change? Can't it continue to load from the classpath?

If it is needed for offline, I prefer to put it in the 3pl/ directory like everything else. But I think the classpath loading should be OK.

Constantin Asofiei wrote:

Sergey, the combobox code for note 43 is not working in chrome.

It is working for me in Chrome 58.0.3029.110 64-bit.

Greg Shah wrote:

Constantin Asofiei wrote:

Sergey, the combobox code for note 43 is not working in chrome.

It is working for me in Chrome 58.0.3029.110 64-bit.

I also checked this combobox for Google Chrome Version 49.0.2623.75 (64-bit), it seems that if open this popup view too early, then this combo box is not ready yet.

Please update report.js, committed revision 11259 changed selected profile to use dijit.Select.

Sergey: Please work on the file list editing next. It would be useful for us to be able to actually create some real file lists to use/test them.

Greg Shah wrote:

Constantin Asofiei wrote:

Sergey, the combobox code for note 43 is not working in chrome.

It is working for me in Chrome 58.0.3029.110 64-bit.

What I don't know how to do is to de-register the dijit widgets when the callgraph nav page is selected again: in this case, they need to be removed, otherwise I get this:

registry.js:6 Uncaught Error: Tried to register widget with id==startingPoints but that id is already registered
    at Object.add (registry.js:6)
    at Object.create (_WidgetBase.js:72)
    at Object._2eb (dojo.js:15)
    at Object.create (_FormWidgetMixin.js:96)
    at Object.postscript (_WidgetBase.js:50)
    at new <anonymous> (dojo.js:15)
    at callgraph.js:2848
    at report.js:679
    at wsbase.js:93
    at processMessage (wsbase.js:76)

         require(["dojo/_base/array", "dijit/registry", "dojo/domReady!"],
         function(arrayUtil, registry)
         {
            var formWidgets = registry.findWidgets("startingPoints");
            formWidgets.forEach(function (widget) 
            {
               widget.destroyRecursive();
            });

I used this code for destroy combobox

         var widget = dijit.byId("profilesSelector");
               if (widget)
               {
                  widget.destroy(true);
               }

May be this code can help

=== modified file 'src/com/goldencode/p2j/report/web/res/callgraph.js'
--- src/com/goldencode/p2j/report/web/res/callgraph.js    2017-06-05 17:28:51 +0000
+++ src/com/goldencode/p2j/report/web/res/callgraph.js    2017-06-05 19:26:55 +0000
@@ -2696,6 +2696,11 @@
          {
             if (filters && filters.length)
             {
+               var widget = dijit.byId("startingPoints");
+               if (widget)
+               {
+                  widget.destroy(true);
+               }
                var fileSets = [];

                fileSets.push({ "name": "Root List", "id": "-1" });

Sergey Ivanovskiy wrote:

May be this code can help
[...]

Yes, it helped, thanks!

Sergey, could you please help me debug a radio button issue?

I just checked in rev. 11278. On the "Customize" page, there is a set of 2 radio buttons. Each has an "onClick" function which seems to work properly in setting the srcType value. However, when I click the "Run" button and the code gets to report.js:261, the value is wrong. There must be some scope issue I am overlooking. Your urgent help is appreciated.

Eric Faulhaber wrote:

Sergey, could you please help me debug a radio button issue?

I just checked in rev. 11278. On the "Customize" page, there is a set of 2 radio buttons. Each has an "onClick" function which seems to work properly in setting the srcType value. However, when I click the "Run" button and the code gets to report.js:261, the value is wrong. There must be some scope issue I am overlooking. Your urgent help is appreciated.

Never mind. User error. Been staring at the debugger too long.

Eric Faulhaber wrote:

Eric Faulhaber wrote:

Sergey, could you please help me debug a radio button issue?

I just checked in rev. 11278. On the "Customize" page, there is a set of 2 radio buttons. Each has an "onClick" function which seems to work properly in setting the srcType value. However, when I click the "Run" button and the code gets to report.js:261, the value is wrong. There must be some scope issue I am overlooking. Your urgent help is appreciated.

Never mind. User error. Been staring at the debugger too long.

You can take a look in callgraph.js:createDirection and createLoadGraph for how I used the radio-button with dijit.

Constantin, I've found a flaw in our JS<-->Java API, in that exceptions on the server side are never reported to the client. In the event of a server-side error, the JS callback is simply never invoked. This can leave the client in a bad place. The code I am writing now accepts user input, and thus it is especially likely to cause server-side exceptions. I need to report these to the client.

Since our use of the API is still quite limited, I'd like to change the callback contract, such that each async callback function on the JS side will have an "err" argument as its first parameter. If that argument is null, it indicates success. If non-null, it will be a JSON representation of a Throwable (assuming Jackson can handle serializing this; if not, it will be a simple error message string, taken from the root cause exception on the Java side).

If you have no objection, I will add an "err" parameter as the first of each JS callback function, including in the callgraph API. I won't add any logic to process it for the callgraph functions, but it will be there in case you need to deal with it.

Eric Faulhaber wrote:

If you have no objection, I will add an "err" parameter as the first of each JS callback function, including in the callgraph API. I won't add any logic to process it for the callgraph functions, but it will be there in case you need to deal with it.

Yes, you can go ahead and add it.

Please review these small changes on the server side, revision 11289. It is required to get files specifications from file_filter table to display them for each filter in the tree view.

Sergey Ivanovskiy wrote:

Please review these small changes on the server side, revision 11289. It is required to get files specifications from file_filter table to display them for each filter in the tree view.

Sergey, this is a nice start. A few things to consider as we take this further:

1) We have some decisions to make on how to represent available directories and files.

One idea is to have two, side-by-side trees; the left would show available resources, the right would show those selected for a profile. There would be some buttons between the views to select/unselect resources. The tree that you have implemented would be closer to the tree representing selected resources, but with some changes (see below).

Another option is simply to represent the selections in a single tree, by changing the state and visualization of leaf and branch nodes to differentiate between selected and unselected resources.

A third would be to have a tree on the left of all available resources, and on the right a list that shows the text specs that would actually be stored in the database.

I am leaning toward the second approach as the cleanest and most straightforward.

In any case, the user's selections will need to be "translated" back to the paths and wildcard specs stored in the file_filter table.

2) In addition to "Add New Profile", we will need options to edit and delete an existing profile (both disabled for the "Include All" Profile), and an option to copy an existing profile as a starting point for a new one.

3) The "Include All" profile will not be allowed to be edited or deleted, since it is the default.

4) The descriptors "type=1" and "type=3" will mean nothing to the user, and only serve to confuse. I think instead, we will need a tabbed pane here with two tabs: "Code" and "Schema". Each tab will show the applicable tree(s) for types 3 and 1, respectively.

I would like to rebase branch 1514a soon. Let me know if you have pending updates, or if this is a bad time for any reason.

Eric Faulhaber wrote:

I would like to rebase branch 1514a soon. Let me know if you have pending updates, or if this is a bad time for any reason.

I have no objection.

Task branch 1514a has been rebased to trunk revision 11154. Current branch revision is 11310.

Sergey, would you please point me at code you have used in FWD that shows how to disable the browser's back button?

This code is in src/com/goldencode/p2j/ui/client/driver/web/res/p2j.js

   /**
    * Initialize module.
    * 
    * @param {object} cfg  configuration.
    */
   me.init = function(cfg)
   {
      /**
       * Prevent back navigation.
       */
      history.pushState({}, '', cfg.page);

      /**
       * Prevent back navigation.
       * 
       * @param {event} event object
       */
      window.onpopstate = function(event) 
      {
         if (event.type === 'popstate')
            history.pushState({}, '', cfg.page);
      };

• in the first (schema) command only, pass the default password for the initial FWD Analytics web app user (admin), via a system property (i.e., <jvmarg value="-DappAdminPass=${appname}"/>, which makes the value of the appname property the default password);
• no longer pass as an argument the file system directory in which to generate reports (since they are no longer statically generated), so this entry in the schema command was removed: <arg value ="${rpt.home}/schema"/>, as was this entry in the code command: <arg value ="${rpt.home}/code"/>.

I have made and committed these changes to the Hotel ChUI (rev 49) and Hotel GUI (rev 141) demo applications.

Sergey, I have changed the server-side implementation of the file filter feature, and I am re-working the filesview.js module to use a different UI tree control for the user to specify filters.

I need your help with integrating an external, Dojo-compatible, JavaScript package with this web application, which is compatible with the way we currently load Dojo resources from the dojo-1.10.0-distribution.zip file.

I am trying to use cbtree (http://thejekels.com/dojo/cbtree_AMD.html), which implements multi-state checkbox trees. The installation instructions for that project assume a local installation of the unpacked cbtree zip file within the unzipped Dojo toolkit hierarchy. Of course, we load from the Dojo zip file instead, and I don't really want to mix the files together -- I don't know whether this package ever will officially be part of Dojo, so it seems like commingling the files that way will cause upgrade problems when either package is updated, given the way we deal with Dojo modules today.

Can you please prototype a solution that loads cbtree modules in a way that is compatible with our current architecture w.r.t. loading Dojo modules, which integrates cbtree-v0.9.4-0.zip (http://thejekels.com/download/cbtree/cbtree-v0.9.4-0.zip)? The prototype doesn't have to do anything in particular (i.e., don't rewrite the existing filesview functionality -- it is changing). I just need the infrastructure code to load the cbtree package so that I can write UI code using the tree.

This is quite urgent. Thanks.

Please review committed revision 11342 that added this package to the repository and 11341 changed dojo loader configuration to use this package.

Sergey, thanks for the update. I need your help for two more library issues.

1) In addition to cbtree, we use a number of other third party libraries. Currently, these are in the src/com/goldencode/p2j/report/web/res/3pl directory, and the web server loads them directly from the file system. However, I only put them there as a short-cut for development. Ultimately, these need to be loaded from individual jar files (one per library), which are integrated into the build, in the same way Dojo is. These JavaScript libraries are not distributed as jar files of course; we would need to zip/jar them, and make them available for download by the gradle build, just as any other dependency.

2) I took another short-cut for development, which was to load the FWD-maintained resources (i.e., our JavaScript code, HTML, images, etc.) directly from the file system in ReportWebServer. These, too, should be built into a jar file by the gradle build, and loaded from the jar file by the web server, instead of from the file system. OTOH, it is extremely convenient to load these directly from the file system during development, as it avoids a jar build (which takes from 10-50 seconds on my system) to test every JS change. If we could we make it a command line option to the web server as to whether we load from the jar or the file system, it would be useful. This would default to loading from the jar.

Greg, any preference regarding whether we build the the analytics web resources into p2j.jar vs. a separate jar?

and make them available for download by the gradle build

I will place the jars in proj.goldencode.com/artifacts and the gradle build script would be modified to pull them from there.

Greg, any preference regarding whether we build the the analytics web resources into p2j.jar vs. a separate jar?

Yes, please put them in the p2j.jar.

Eric Faulhaber wrote:

1) In addition to cbtree, we use a number of other third party libraries. Currently, these are in the src/com/goldencode/p2j/report/web/res/3pl directory, and the web server loads them directly from the file system. However, I only put them there as a short-cut for development. Ultimately, these need to be loaded from individual jar files (one per library), which are integrated into the build, in the same way Dojo is. These JavaScript libraries are not distributed as jar files of course; we would need to zip/jar them, and make them available for download by the gradle build, just as any other dependency.

Does it need to manage these 3rd party software from the gradle configuration (download correct versions to the local file system and prepare them) or just to zip the existing sources in p2j.jar?

Now they are loaded by dojo loader via web server, so it doesn't matter where the web server find them from the local file system or from jar file.

2) I took another short-cut for development, which was to load the FWD-maintained resources (i.e., our JavaScript code, HTML, images, etc.) directly from the file system in ReportWebServer. These, too, should be built into a jar file by the gradle build, and loaded from the jar file by the web server, instead of from the file system. OTOH, it is extremely convenient to load these directly from the file system during development, as it avoids a jar build (which takes from 10-50 seconds on my system) to test every JS change. If we could we make it a command line option to the web server as to whether we load from the jar or the file system, it would be useful. This would default to loading from the jar.

It seems that if the web server will load these resources from its classpath, then the resources will be from the file system for development mode and from the jar file after deploying application for testing.

Sergey Ivanovskiy wrote:

Does it need to manage these 3rd party software from the gradle configuration (download correct versions to the local file system and prepare them) [...] ?

Yes, from the gradle configuration, but let's be clear what we mean by the "correct" versions. These must not be the latest versions from the various third party websites, but rather the current, known, working versions which we will archive at proj.goldencode.com/artifacts.

Sergey Ivanovskiy wrote:

Now they are loaded by dojo loader via web server, so it doesn't matter where the web server find them from the local file system or from jar file.

Actually, except for the dojo resources, the web server currently explicitly looks in ../../p2j/src/com/goldencode/p2j/report/web/res/ to pick up our own and most of the other third party resources. See the TODO at ReportWebServer.java:104.

Does it need to manage these 3rd party software from the gradle configuration (download correct versions to the local file system and prepare them) or just to zip the existing sources in p2j.jar?

It is my understanding that we will have to manually create our own jar file or zip file for each of these projects. Then the gradle build would just pull the pre-built jar/zip from the artifacts directory on proj.goldencode.com.

Please help. Is there any login/password changes related to the reporting tool? It seems that ReportApi.authenticate can't be passed. I tested jdbc:h2:../db/hotel;DB_CLOSE_DELAY=-1;MVCC=true;MV_STORE=FALSE via web client and fwd_admin/admin is there.

To authenticate with the Hotel GUI web app, use admin/hotel.

Sergey Ivanovskiy wrote:

Please help. Is there any login/password changes related to the reporting tool? It seems that ReportApi.authenticate can't be passed. I tested jdbc:h2:../db/hotel;DB_CLOSE_DELAY=-1;MVCC=true;MV_STORE=FALSE via web client and fwd_admin/admin is there.

The analytics database is completely separate from the demo application database, so that URL won't work. I am in the process of changing the analytics database URL in my current update, but as of rev 11353, you can use (relative to the project root directory) jdbc:h2:rptdb/rptdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE;AUTOCOMMIT=OFF;MV_STORE=FALSE;. Credentials are admin/admin.

Eric Faulhaber wrote:

To authenticate with the Hotel GUI web app, use admin/hotel.

Sorry, I should have been more clear: to authenticate with the Hotel GUI analytics web app, use admin/hotel. I haven't changed anything with the authentication of the converted demo application itself.

Now I am gathering officials js repositories that were installed in local repository com/goldencode/p2j/report/web/res/web/res/3pl. For an example,
jquery js was taken from official http://code.jquery.com/ version 3.2.1. This library adds a new repository to our gradle build

repositories {
........................
 ivy {
        url "http://code.jquery.com" 
        layout "pattern", {
            artifact "[module]-[revision](.[classifier]).[ext]" 
        }
    }
....................
}

Sergey Ivanovskiy wrote:

Eric, could you help to detect another dependencies for sources: jquery-ui-1.12.1.custom.

AFAIK, the only library jquery-ui-1.12.1.custom is dependent upon is jquery itself. The tabulator.js library is in turn dependent upon @jquery-ui-1.12.1.custom.

Sergey: Are you saying that we don't have to manually build custom artifacts in order to manage the js dependencies?

Eric Faulhaber wrote:

Sergey Ivanovskiy wrote:

Eric, could you help to detect another dependencies for sources: jquery-ui-1.12.1.custom.

AFAIK, the only library jquery-ui-1.12.1.custom is dependent upon is jquery itself. The tabulator.js library is in turn dependent upon @jquery-ui-1.12.1.custom.

Thank you, what is a repository for jquery-ui-1.12.1.custom? It seems that we need to deploy these packages into

    maven {
        name "goldencode" 
        url  "https://proj.goldencode.com/artifacts/" 
    }

Greg Shah wrote:

Sergey: Are you saying that we don't have to manually build custom artifacts in order to manage the js dependencies?

Yes, it seems that for jquery it isn't required, but for the other packages I don't know yet.

Sergey Ivanovskiy wrote:

Eric Faulhaber wrote:

Sergey Ivanovskiy wrote:

Eric, could you help to detect another dependencies for sources: jquery-ui-1.12.1.custom.

AFAIK, the only library jquery-ui-1.12.1.custom is dependent upon is jquery itself. The tabulator.js library is in turn dependent upon @jquery-ui-1.12.1.custom.

Thank you, what is a repository for jquery-ui-1.12.1.custom?

I'm not sure what you're asking.

It seems that we need to deploy these packages into
[...]
by some gradle tasks and then they can be used from this repository.

I think that is what Greg had in mind.

I don't know where is this custom build jquery-ui-1.12.1.custom comes from. I only know where to find official jquery-ui?

Please don't mind. It seems these info are in package.json files, so this custom package tabulator.js comes from https://github.com/olifolkerd/tabulator.git

Committed revision 11357 added a classpath to resources for the report web server. In order to start from IDE it needs to add this project folder src to classpath in order to resolve com/goldencode/p2j/report/web/res. Although this resource is zipped into p2j.jar if the project has been built. I think it wiil be changed as soon as all js dependencies will be available from https://proj.goldencode.com/artifacts/

Javascript libs are typically managed by the npm beast.

I prefer not to be dependent upon the node package manager.

There are gradle plugins for npm and for git. If we will not use npm, then it seems that it needs to select the best known gradle plugin for git and to use it.

My plan was to build the zips/jars manually and put it in the artifacts.

Adding a dependency on npm or git is more than I was wanting to do.

To be clear: every time we add a dependency to something like npm or git, we make the installation and support of our technology harder. Since Gradle's built-in support for dependency management has no concept of javascript resources, we can instead treat them as jar files or zip files. The resulting js resources can easily be loaded at runtime from our jetty environment. The related artifacts can easily be supplied from proj.goldencode.com. Since we rarely change these dependencies, it has little impact on our development process.

The overall benefit of eliminating dependence on npm or git is worth it.

Ok, understand. Is it correct to manage js libs inside different zips. Please look at this article https://blog.oio.de/2014/10/24/handling-javascript-dependencies-with-gradle/

Sergey Ivanovskiy wrote:

Ok, understand. Is it correct to manage js libs inside different zips. Please look at this article https://blog.oio.de/2014/10/24/handling-javascript-dependencies-with-gradle/

Sergey, we already manage Dojo with gradle. The added dependency to dojo zip will make the zip file appear in build/lib (together with other jar files). The zip file then appears on the class path and so the content is accessible to the Jetty handler (reading the zip content as classpath resource). Maybe it is enough to replicate this to the other js dependencies?

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

Ok, understand. Is it correct to manage js libs inside different zips. Please look at this article https://blog.oio.de/2014/10/24/handling-javascript-dependencies-with-gradle/

Sergey, we already manage Dojo with gradle. The added dependency to dojo zip will make the zip file appear in build/lib (together with other jar files). The zip file then appears on the class path and so the content is accessible to the Jetty handler (reading the zip content as classpath resource). Maybe it is enough to replicate this to the other js dependencies?

Dojo zip was taken from http://repo1.maven.org/maven2/ repository as a zip release distribution. If I understand correctly, there are 2 tasks. The first is to download the required js packages from different sources and second task is to deploy them into

maven {
        name "goldencode" 
        url  "https://proj.goldencode.com/artifacts/" 
    }

Do you propose to add several handlers, each one for a particular js package zipped in one file like dojo (DojoToolkitHandler)?

If this all resources are in this repository, then there are no problems to download then and to include in p2j.jar file as resources.

I don't want that code inside p2j.jar. I would like the resources loaded from the classpath.

Do you propose to add several handlers, each one for a particular js package zipped in one file like dojo (DojoToolkitHandler)?

Can't we have just a single handler for all of them?

Greg Shah wrote:

Do you propose to add several handlers, each one for a particular js package zipped in one file like dojo (DojoToolkitHandler)?

Can't we have just a single handler for all of them?

At this moment don't know, let me think about it.

It seems that the most simple way to manage all js zipped packages is to use dedicated resources but if they are all in the application bootstrap classpath, then it is required (at least I don't know another way) that the report web server knows about all of them like it knows about DojoToolkitHandler. The dedicated resources can be organized in different ways that finally will come to the collection of these zipped resources that will be added to the resource base of the root context for the report web server.

Greg, Hynek, what way would you advice to follow?

Sergey Ivanovskiy wrote:

It seems that the most simple way to manage all js zipped packages is to use dedicated resources but if they are all in the application bootstrap classpath,

Why do the js zip files have to be present on the bootstrap classpath? Can they appear on the regular classpath as the rest of p2j jars and zips?

then it is required (at least I don't know another way) that the report web server knows about all of them like it knows about DojoToolkitHandler.

When you say the server must know about all, do you mean that the zipped resources cannot be enumerated?

The dedicated resources can be organized in different ways that finally will come to the collection of these zipped resources that will be added to the resource base of the root context for the report web server.

Yes, this how we do it for the Admin web resources for example. We take the path from the http request, subtract the context path and add the resulting path to the base resource path to get the required web resource (html, js, image, etc). In Admin it is a bit more complicated because we need to merge two class paths together in order to provide the extensibility features.

Greg, Hynek, what way would you advice to follow?

No, I missed the definition of the bootstrap classpath, so I mean "application bootstrap" classpath is that resources are given by command line classpath.

Ok, let all js listed in application classpath.

Please review committed revision 11364 that added third party js dependencies handler.

Sergey Ivanovskiy wrote:

Please review committed revision 11364 that added third party js dependencies handler.

Will do.

Next task: please implement the client-side code to store a file locally. When looking at the source/ast view, we want the user to be able to press a key combination (similar to Ctrl+s) to be prompted to save on the local file system the current source file being viewed (i.e., preprocessed cache file). Browsers (at least FF and Chrome) seem to hook Ctrl+s already, so perhaps we hook Ctrl+Shift+s for this?

I will provide a server side API which returns an array of source code lines. The client code should save these out locally to the file specified by the user, with the appropriate line endings.

Let me know if you have questions.

As an alternative, copying to the clipboard would be very useful.

If to the clipboard, then I guess the key combination should be Ctrl+Shift+c (Ctrl+c should continue to work as it normally does).

Greg, which is the higher priority for you?

Clipboard.

Sergey Ivanovskiy wrote:

Please review committed revision 11364 that added third party js dependencies handler.

I've looked it over and this is not really my area of expertise, so I am a bit confused by the outcome. I see the handler is there, but we still have the loose js files under the 3pl directory, which we want to replace with the zip files. Is that the next step?

BTW, I noticed that my Bootstrap glyphicon use seems to be broken with this update.

Eric Faulhaber wrote:

BTW, I noticed that my Bootstrap glyphicon use seems to be broken with this update.

Please use the analytics web app a little bit, then look in the console. I am seeing entries like this:

Failed to load resource: the server responded with a status of 404 (Not Found)
glyphicons-halflings-regular.woff Failed to load resource: the server responded with a status of 404 (Not Found)
glyphicons-halflings-regular.ttf Failed to load resource: the server responded with a status of 404 (Not Found)
bootstrap.css Failed to load resource: the server responded with a status of 404 (Not Found)
jquery-ui.css Failed to load resource: the server responded with a status of 404 (Not Found)
tabulator.css Failed to load resource: the server responded with a status of 404 (Not Found)

Actually, it's much worse than just my missing glyphicons now. I guess the only reason anything was working at all for me at the time of my earlier post was that many js and css resources were still cached in my browser from before the update. Now everything fails to load with a 404 error and the app is completely unusable. Please urgently revert this change, or if something was missing from your 11364 update, please add it. I've backed out these changes locally to resume working.

Eric Faulhaber wrote:

Sergey Ivanovskiy wrote:

Please review committed revision 11364 that added third party js dependencies handler.

I've looked it over and this is not really my area of expertise, so I am a bit confused by the outcome. I see the handler is there, but we still have the loose js files under the 3pl directory, which we want to replace with the zip files. Is that the next step?

BTW, I noticed that my Bootstrap glyphicon use seems to be broken with this update.

Yes, zipped js modules are not ready because I'm not fluent in gradle. Planning to finish this task today. Now during development it is required to add src folder to the default application classpath, when starting the report tool, in order to get all resources from com/goldencode/p2j/report/web/res. Committed revision 11366 fixed build.xml in order that p2.jar doesn't contain 3pl and cbtree folders and their content. I tested that at least 11366 worked properly if src folder was added to classpath. Is this way fitted for you during development, otherwise I will roll back?

Please update to rev 11367. It moved cbtree under 3pl resource folder and fixed its path used in dojo configuration and direct css link. This src/com/goldencode/p2j/report/web/res/index.html is changed.

At that moment I can get versions for JQuery, Bootstrap and cbtree, but have difficulties with d3. There is a prepared package for d3 http://www.webjars.org/ , but it is required to change library paths since we use nonstandard web application layout with application classpath. Using webjars.org it is possible to get these definitions from maven central repository

dependencies {
    js group: 'webjars.org',     name: 'd3',      version: '4.9.1'
    js group: 'webjars.org', name: 'jquery',      version: '3.2.1', 
    js group: 'webjars.org',   name: 'bootstrap', version: '3.3.7'
}

repositories {
mavenCentral()
/** JQuery */

   ivy {
        url "http://code.jquery.com" 
        layout "pattern", {
            artifact "[module]-[revision](.[classifier]).[ext]" 
        }
   }

/** cbtree */
   ivy {
        url "http://thejekels.com/download/cbtree" 
        layout "pattern", {
            artifact "[module]-[revision](-[classifier]).[ext]" 
        }
   }

/** bootstrap */
   ivy {
        url 'https://www.github.com/'
        layout "pattern", {
            artifact "/[organisation]/[module]/archive/v[revision].[ext]" 
        }
    }

}

configurations {
    js
}

dependencies {
    js group: 'jquery', name: 'jquery',    version: '3.2.1', classifier: 'min', ext: "js" 
    js group: 'cbtree', name: 'cbtree',    version: 'v0.9.4-0', ext: "zip" 
    js group: 'twbs',   name: 'bootstrap', version: '3.3.7', ext: "zip" 
}

Greg, I can propose only to use something like ant get task that downloads these 3pl packages using custom urls and then zip them. Is it enough?

Sergey Ivanovskiy wrote:

Greg, I can propose only to use something like ant get task that downloads these 3pl packages using custom urls and then zip them. Is it enough?

If it is easy to do, yes that is fine.

But it is also OK to just manually build the zips and I can upload them into the artifacts directory.

Greg Shah wrote:

But it is also OK to just manually build the zips and I can upload them into the artifacts directory.

I tried to find suitable repositories that can be used by the automatic version maintenance, but didn't succeed yet. I will commit these zipped packages in 1541a branch under 3pl/packages sooner and please upload them into this artifacts repository. But the automatic version maintenance is an open task.

Do not check them in to the branch. Just upload them somewhere on devsrv01. I will move them from there to artifacts.

I already comitted them by revision 11368. I will roll back.

Sergey Ivanovskiy wrote:

Please update to rev 11367. It moved cbtree under 3pl resource folder and fixed its path used in dojo configuration and direct css link. This src/com/goldencode/p2j/report/web/res/index.html is changed.

Sergey, this is still not working for me. I have updated to rev 11367 and rebuilt with gradlew all. I am running hotel gui with the src directory added to the report server's classpath as noted in #1514-114, but I am still getting a 404 error for each third party web resource.

Specifically, my p2j directory is at /home/ecf/projects/hotel_gui/p2j/. I am launching the server from /home/ecf/projects/hotel_gui/deploy/server/, with a classpath of ../../p2j/build/lib/p2j.jar:../../p2j/src/:.:.

Have I missed something from your instructions?

Sergey, although the 1514a branch is a development branch, it is different than most development branches, in that we are actually already relying on it to maintain a working system for performing customer services, so it can't remain broken.

If you don't have a solution in the next hour or two that makes it functional again, please revert rev. 11364 until you do.

OTOH, if it works for you and you think the problem I'm seeing is due to the way I've configured things, please let me know.

Eric Faulhaber wrote:

Sergey, although the 1514a branch is a development branch, it is different than most development branches, in that we are actually already relying on it to maintain a working system for performing customer services, so it can't remain broken.

If you don't have a solution in the next hour or two that makes it functional again, please revert rev. 11364 until you do.

OTOH, if it works for you and you think the problem I'm seeing is due to the way I've configured things, please let me know.

I used to start from IDE Eclipse with this classpath and clean 1514a build

/home/sbi/projects/1514a/src:/home/sbi/projects/1514a/build/classes:/home/sbi/eclipse-workspace/1514a/bin:/home/sbi/projects/1514a/build/lib/antlr-2.7.7.jar:/home/sbi/projects/1514a/build/lib/asm-5.0.3.jar:/home/sbi/projects/1514a/build/lib/aspectjrt-1.8.10.jar:/home/sbi/projects/1514a/build/lib/axiom-api-1.2.13.jar:/home/sbi/projects/1514a/build/lib/axiom-impl-1.2.13.jar:/home/sbi/projects/1514a/build/lib/axis2-adb-1.6.2.jar:/home/sbi/projects/1514a/build/lib/axis2-kernel-1.6.2.jar:/home/sbi/projects/1514a/build/lib/axis2-transport-http-1.6.2.jar:/home/sbi/projects/1514a/build/lib/axis2-transport-local-1.6.2.jar:/home/sbi/projects/1514a/build/lib/batik-anim-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-awt-util-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-bridge-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-codec-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-css-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-dom-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-ext-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-extension-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-gui-util-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-gvt-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-parser-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-script-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-svg-dom-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-svggen-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-swing-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-transcoder-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-util-1.8.jar:/home/sbi/projects/1514a/build/lib/batik-xml-1.8.jar:/home/sbi/projects/1514a/build/lib/bcel-5.2.jar:/home/sbi/projects/1514a/build/lib/bcpkix-jdk15on-1.50.jar:/home/sbi/projects/1514a/build/lib/bcprov-jdk15on-1.50.jar:/home/sbi/projects/1514a/build/lib/mchange-commons-java-0.2.11.jar:/home/sbi/projects/1514a/build/lib/c3p0-0.9.5.2.jar:/home/sbi/projects/1514a/build/lib/cglib-2.1_3.jar:/home/sbi/projects/1514a/build/lib/common-image-3.1.2.jar:/home/sbi/projects/1514a/build/lib/common-io-3.1.2.jar:/home/sbi/projects/1514a/build/lib/common-lang-3.1.2.jar:/home/sbi/projects/1514a/build/lib/commons-beanutils-core-1.7.0.jar:/home/sbi/projects/1514a/build/lib/commons-codec-1.7.jar:/home/sbi/projects/1514a/build/lib/commons-collections-3.2.2.jar:/home/sbi/projects/1514a/build/lib/commons-configuration-1.10.jar:/home/sbi/projects/1514a/build/lib/commons-httpclient-3.1.jar:/home/sbi/projects/1514a/build/lib/commons-lang3-3.3.1.jar:/home/sbi/projects/1514a/build/lib/commons-lang-2.6.jar:/home/sbi/projects/1514a/build/lib/commons-logging-1.1.1.jar:/home/sbi/projects/1514a/build/lib/dojo-1.10.0-distribution.zip:/home/sbi/projects/1514a/build/lib/dom4j-1.6.1.jar:/home/sbi/projects/1514a/build/lib/ehcache-core-2.6.2.jar:/home/sbi/projects/1514a/build/lib/fwd-h2-1.4.192.jar:/home/sbi/projects/1514a/build/lib/fwd-hibernate-core-4.1.8.jar:/home/sbi/projects/1514a/build/lib/gremlin-core-3.2.3.jar:/home/sbi/projects/1514a/build/lib/gremlin-groovy-3.2.3.jar:/home/sbi/projects/1514a/build/lib/gremlin-shaded-3.2.3.jar:/home/sbi/projects/1514a/build/lib/groovy-2.4.7.jar:/home/sbi/projects/1514a/build/lib/groovy-2.4.7-indy.jar:/home/sbi/projects/1514a/build/lib/groovy-console-2.4.7.jar:/home/sbi/projects/1514a/build/lib/groovy-groovysh-2.4.7-indy.jar:/home/sbi/projects/1514a/build/lib/groovy-json-2.4.7-indy.jar:/home/sbi/projects/1514a/build/lib/groovy-jsr223-2.4.7-indy.jar:/home/sbi/projects/1514a/build/lib/groovy-swing-2.4.7.jar:/home/sbi/projects/1514a/build/lib/groovy-templates-2.4.7.jar:/home/sbi/projects/1514a/build/lib/groovy-xml-2.4.7.jar:/home/sbi/projects/1514a/build/lib/guava-18.0.jar:/home/sbi/projects/1514a/build/lib/hibernate-c3p0-4.1.8.Final.jar:/home/sbi/projects/1514a/build/lib/hibernate-commons-annotations-4.0.1.Final.jar:/home/sbi/projects/1514a/build/lib/hibernate-ehcache-4.1.8.Final.jar:/home/sbi/projects/1514a/build/lib/hibernate-jpa-2.0-api-1.0.1.Final.jar:/home/sbi/projects/1514a/build/lib/high-scale-lib-1.1.2.jar:/home/sbi/projects/1514a/build/lib/hppc-0.7.1.jar:/home/sbi/projects/1514a/build/lib/httpcore-4.0.jar:/home/sbi/projects/1514a/build/lib/imageio-bmp-3.1.2.jar:/home/sbi/projects/1514a/build/lib/imageio-core-3.1.2.jar:/home/sbi/projects/1514a/build/lib/imageio-metadata-3.1.2.jar:/home/sbi/projects/1514a/build/lib/ivy-2.3.0.jar:/home/sbi/projects/1514a/build/lib/jackson-annotations-2.8.0.jar:/home/sbi/projects/1514a/build/lib/jackson-core-2.8.8.jar:/home/sbi/projects/1514a/build/lib/jackson-databind-2.8.8.jar:/home/sbi/projects/1514a/build/lib/jackson-jaxrs-base-2.8.8.jar:/home/sbi/projects/1514a/build/lib/jackson-jaxrs-json-provider-2.8.8.jar:/home/sbi/projects/1514a/build/lib/jackson-module-jaxb-annotations-2.8.8.jar:/home/sbi/projects/1514a/build/lib/janusgraph-berkeleyje-0.1.0.jar:/home/sbi/projects/1514a/build/lib/janusgraph-core-0.1.0.jar:/home/sbi/projects/1514a/build/lib/janusgraph-lucene-0.1.0.jar:/home/sbi/projects/1514a/build/lib/javassist-3.16.1-GA.jar:/home/sbi/projects/1514a/build/lib/javatuples-1.2.jar:/home/sbi/projects/1514a/build/lib/javax.json-1.0.jar:/home/sbi/projects/1514a/build/lib/javax.servlet-api-3.1.0.jar:/home/sbi/projects/1514a/build/lib/jBCrypt-jbcrypt-0.4.jar:/home/sbi/projects/1514a/build/lib/jboss-logging-3.1.0.GA.jar:/home/sbi/projects/1514a/build/lib/jboss-transaction-api_1.1_spec-1.0.0.Final.jar:/home/sbi/projects/1514a/build/lib/jcabi-log-0.14.jar:/home/sbi/projects/1514a/build/lib/jcabi-manifests-1.1.jar:/home/sbi/projects/1514a/build/lib/jcl-over-slf4j-1.7.21.jar:/home/sbi/projects/1514a/build/lib/je-7.3.7.jar:/home/sbi/projects/1514a/build/lib/jetty-all-9.3.8.v20160314-uber.jar:/home/sbi/projects/1514a/build/lib/jetty-proxy-9.3.8.v20160314.jar:/home/sbi/projects/1514a/build/lib/jline-2.12.jar:/home/sbi/projects/1514a/build/lib/jta-1.1.jar:/home/sbi/projects/1514a/build/lib/lucene-analyzers-common-4.10.4.jar:/home/sbi/projects/1514a/build/lib/lucene-core-4.10.4.jar:/home/sbi/projects/1514a/build/lib/lucene-queries-4.10.4.jar:/home/sbi/projects/1514a/build/lib/lucene-queryparser-4.10.4.jar:/home/sbi/projects/1514a/build/lib/lucene-spatial-4.10.4.jar:/home/sbi/projects/1514a/build/lib/mail-1.4.jar:/home/sbi/projects/1514a/build/lib/metrics-core-3.0.1.jar:/home/sbi/projects/1514a/build/lib/msv-core-2010.2.jar:/home/sbi/projects/1514a/build/lib/neethi-3.0.2.jar:/home/sbi/projects/1514a/build/lib/xml-apis-1.3.04.jar:/home/sbi/projects/1514a/build/lib/xmlgraphics-commons-2.0.jar:/home/sbi/projects/1514a/build/lib/slf4j-jdk14-1.6.1.jar:/home/sbi/projects/1514a/build/lib/postgresql-9.4.1211.jar:/home/sbi/projects/1514a/build/lib/quartz-2.2.1.jar:/home/sbi/projects/1514a/build/lib/stax2-api-3.1.1.jar:/home/sbi/projects/1514a/build/lib/stax-utils-20070216.jar:/home/sbi/projects/1514a/build/lib/relaxngDatatype-20020414.jar:/home/sbi/projects/1514a/build/lib/woodstox-core-asl-4.2.0.jar:/home/sbi/projects/1514a/build/lib/xml-apis-ext-1.3.04.jar:/home/sbi/projects/1514a/build/lib/wsdl4j-1.6.2.jar:/home/sbi/projects/1514a/build/lib/XmlSchema-1.4.7.jar:/home/sbi/projects/1514a/build/lib/tinkergraph-gremlin-3.2.3.jar:/home/sbi/projects/1514a/build/lib/reflections-0.9.9-RC1.jar:/home/sbi/projects/1514a/build/lib/spatial4j-0.4.1.jar:/home/sbi/projects/1514a/build/lib/xsdlib-2010.1.jar:/home/sbi/projects/1514a/build/lib/snakeyaml-1.15.jar:/home/sbi/projects/1514a/build/lib/slf4j-api-1.7.21.jar:/home/sbi/projects/1514a/build/lib/p2j.jar:/home/sbi/projects/1514a/build/lib/p2jadmin.jar:/home/sbi/projects/1514a/build/lib/p2jpl.jar:/home/sbi/projects/1514a/build/lib/p2jspi.jar:/home/sbi/projects/hotel_gui/cfg:/home/sbi/projects/hotel_gui/deploy/server

/home/sbi/projects/hotel_gui/

-port 9443  -ks ./deploy/server/embedded-private-key.store -kspw 87ViGTQVp\<07zPw\(3i2A4%uCHYx3mIgSt\>Tk -kepw h8jNMO\>Vbn0\`cL\)b95dVn\$w03CePIB6eItL3 1> report_server.log 2>&1

We must be able to run outside of eclipse.

This path /home/sbi/eclipse-workspace/1514a/bin is empty folder.

Let me check a minute the script outside eclipse.

Eric, please approve that it works for you.

Greg Shah wrote:

Do not check them in to the branch. Just upload them somewhere on devsrv01. I will move them from there to artifacts.

Greg, you can download them from devsrv01:/home/sbi/packages.

Running from the filesystem is really not that important. It saves a 30 second jar operation. And none of those resources are related to the 3rd party code that needs to be moved out of fwd.

After I move the zips to artifacts, what is still needed to make this happen? Will the code in 1514a pull them down into the local installation yet? Will the report server script be able to run them from the classpath?

Greg Shah wrote:

Running from the filesystem is really not that important. It saves a 30 second jar operation. And none of those resources are related to the 3rd party code that needs to be moved out of fwd.

After I move the zips to artifacts, what is still needed to make this happen? Will the code in 1514a pull them down into the local installation yet? Will the report server script be able to run them from the classpath?

I can add these jars to build.gradle within 30 minutes and then they are wiil be used from classpath.

Sergey Ivanovskiy wrote:

Eric, please approve that it works for you.

Eric I forgot to generalize hotel_gui path. Please use this script.

#!/bin/bash

p2j=/home/sbi/projects/1514a
hotel_gui=/home/sbi/projects/hotel_gui

java -classpath $p2j/src:$p2j/build/lib/antlr-2.7.7.jar:$p2j/build/lib/asm-5.0.3.jar:$p2j/build/lib/aspectjrt-1.8.10.jar:$p2j/build/lib/axiom-api-1.2.13.jar:$p2j/build/lib/axiom-impl-1.2.13.jar:$p2j/build/lib/axis2-adb-1.6.2.jar:$p2j/build/lib/axis2-kernel-1.6.2.jar:$p2j/build/lib/axis2-transport-http-1.6.2.jar:$p2j/build/lib/axis2-transport-local-1.6.2.jar:$p2j/build/lib/batik-anim-1.8.jar:$p2j/build/lib/batik-awt-util-1.8.jar:$p2j/build/lib/batik-bridge-1.8.jar:$p2j/build/lib/batik-codec-1.8.jar:$p2j/build/lib/batik-css-1.8.jar:$p2j/build/lib/batik-dom-1.8.jar:$p2j/build/lib/batik-ext-1.8.jar:$p2j/build/lib/batik-extension-1.8.jar:$p2j/build/lib/batik-gui-util-1.8.jar:$p2j/build/lib/batik-gvt-1.8.jar:$p2j/build/lib/batik-parser-1.8.jar:$p2j/build/lib/batik-script-1.8.jar:$p2j/build/lib/batik-svg-dom-1.8.jar:$p2j/build/lib/batik-svggen-1.8.jar:$p2j/build/lib/batik-swing-1.8.jar:$p2j/build/lib/batik-transcoder-1.8.jar:$p2j/build/lib/batik-util-1.8.jar:$p2j/build/lib/batik-xml-1.8.jar:$p2j/build/lib/bcel-5.2.jar:$p2j/build/lib/bcpkix-jdk15on-1.50.jar:$p2j/build/lib/bcprov-jdk15on-1.50.jar:$p2j/build/lib/mchange-commons-java-0.2.11.jar:$p2j/build/lib/c3p0-0.9.5.2.jar:$p2j/build/lib/cglib-2.1_3.jar:$p2j/build/lib/common-image-3.1.2.jar:$p2j/build/lib/common-io-3.1.2.jar:$p2j/build/lib/common-lang-3.1.2.jar:$p2j/build/lib/commons-beanutils-core-1.7.0.jar:$p2j/build/lib/commons-codec-1.7.jar:$p2j/build/lib/commons-collections-3.2.2.jar:$p2j/build/lib/commons-configuration-1.10.jar:$p2j/build/lib/commons-httpclient-3.1.jar:$p2j/build/lib/commons-lang3-3.3.1.jar:$p2j/build/lib/commons-lang-2.6.jar:$p2j/build/lib/commons-logging-1.1.1.jar:$p2j/build/lib/dojo-1.10.0-distribution.zip:$p2j/build/lib/dom4j-1.6.1.jar:$p2j/build/lib/ehcache-core-2.6.2.jar:$p2j/build/lib/fwd-h2-1.4.192.jar:$p2j/build/lib/fwd-hibernate-core-4.1.8.jar:$p2j/build/lib/gremlin-core-3.2.3.jar:$p2j/build/lib/gremlin-groovy-3.2.3.jar:$p2j/build/lib/gremlin-shaded-3.2.3.jar:$p2j/build/lib/groovy-2.4.7.jar:$p2j/build/lib/groovy-2.4.7-indy.jar:$p2j/build/lib/groovy-console-2.4.7.jar:$p2j/build/lib/groovy-groovysh-2.4.7-indy.jar:$p2j/build/lib/groovy-json-2.4.7-indy.jar:$p2j/build/lib/groovy-jsr223-2.4.7-indy.jar:$p2j/build/lib/groovy-swing-2.4.7.jar:$p2j/build/lib/groovy-templates-2.4.7.jar:$p2j/build/lib/groovy-xml-2.4.7.jar:$p2j/build/lib/guava-18.0.jar:$p2j/build/lib/hibernate-c3p0-4.1.8.Final.jar:$p2j/build/lib/hibernate-commons-annotations-4.0.1.Final.jar:$p2j/build/lib/hibernate-ehcache-4.1.8.Final.jar:$p2j/build/lib/hibernate-jpa-2.0-api-1.0.1.Final.jar:$p2j/build/lib/high-scale-lib-1.1.2.jar:$p2j/build/lib/hppc-0.7.1.jar:$p2j/build/lib/httpcore-4.0.jar:$p2j/build/lib/imageio-bmp-3.1.2.jar:$p2j/build/lib/imageio-core-3.1.2.jar:$p2j/build/lib/imageio-metadata-3.1.2.jar:$p2j/build/lib/ivy-2.3.0.jar:$p2j/build/lib/jackson-annotations-2.8.0.jar:$p2j/build/lib/jackson-core-2.8.8.jar:$p2j/build/lib/jackson-databind-2.8.8.jar:$p2j/build/lib/jackson-jaxrs-base-2.8.8.jar:$p2j/build/lib/jackson-jaxrs-json-provider-2.8.8.jar:$p2j/build/lib/jackson-module-jaxb-annotations-2.8.8.jar:$p2j/build/lib/janusgraph-berkeleyje-0.1.0.jar:$p2j/build/lib/janusgraph-core-0.1.0.jar:$p2j/build/lib/janusgraph-lucene-0.1.0.jar:$p2j/build/lib/javassist-3.16.1-GA.jar:$p2j/build/lib/javatuples-1.2.jar:$p2j/build/lib/javax.json-1.0.jar:$p2j/build/lib/javax.servlet-api-3.1.0.jar:$p2j/build/lib/jBCrypt-jbcrypt-0.4.jar:$p2j/build/lib/jboss-logging-3.1.0.GA.jar:$p2j/build/lib/jboss-transaction-api_1.1_spec-1.0.0.Final.jar:$p2j/build/lib/jcabi-log-0.14.jar:$p2j/build/lib/jcabi-manifests-1.1.jar:$p2j/build/lib/jcl-over-slf4j-1.7.21.jar:$p2j/build/lib/je-7.3.7.jar:$p2j/build/lib/jetty-all-9.3.8.v20160314-uber.jar:$p2j/build/lib/jetty-proxy-9.3.8.v20160314.jar:$p2j/build/lib/jline-2.12.jar:$p2j/build/lib/jta-1.1.jar:$p2j/build/lib/lucene-analyzers-common-4.10.4.jar:$p2j/build/lib/lucene-core-4.10.4.jar:$p2j/build/lib/lucene-queries-4.10.4.jar:$p2j/build/lib/lucene-queryparser-4.10.4.jar:$p2j/build/lib/lucene-spatial-4.10.4.jar:$p2j/build/lib/mail-1.4.jar:$p2j/build/lib/metrics-core-3.0.1.jar:$p2j/build/lib/msv-core-2010.2.jar:$p2j/build/lib/neethi-3.0.2.jar:$p2j/build/lib/xml-apis-1.3.04.jar:$p2j/build/lib/xmlgraphics-commons-2.0.jar:$p2j/build/lib/slf4j-jdk14-1.6.1.jar:$p2j/build/lib/postgresql-9.4.1211.jar:$p2j/build/lib/quartz-2.2.1.jar:$p2j/build/lib/stax2-api-3.1.1.jar:$p2j/build/lib/stax-utils-20070216.jar:$p2j/build/lib/relaxngDatatype-20020414.jar:$p2j/build/lib/woodstox-core-asl-4.2.0.jar:$p2j/build/lib/xml-apis-ext-1.3.04.jar:$p2j/build/lib/wsdl4j-1.6.2.jar:$p2j/build/lib/XmlSchema-1.4.7.jar:$p2j/build/lib/tinkergraph-gremlin-3.2.3.jar:$p2j/build/lib/reflections-0.9.9-RC1.jar:$p2j/build/lib/spatial4j-0.4.1.jar:$p2j/build/lib/xsdlib-2010.1.jar:$p2j/build/lib/snakeyaml-1.15.jar:$p2j/build/lib/slf4j-api-1.7.21.jar:$p2j/build/lib/p2j.jar:$p2j/build/lib/p2jadmin.jar:$p2j/build/lib/p2jpl.jar:$p2j/build/lib/p2jspi.jar:$hotel_gui/cfg:$hotel_gui/deploy/server com.goldencode.p2j.report.server.ReportWebServer -port 9443  -ks embedded-private-key.store -kspw 87ViGTQVp\<07zPw\(3i2A4%uCHYx3mIgSt\>Tk -kepw h8jNMO\>Vbn0\`cL\)b95dVn\$w03CePIB6eItL3 1> report_server.log 2>&1

I've uploaded the zips:

See https://proj.goldencode.com/artifacts/javascript/ which contains:

adamwdraper-Numeral-js-7de892f.zip
bootstrap-3.3.7-dist.zip
cbtree-v0.9.4-0.zip
d3.zip
jquery-loading-overlay-1.5.3.zip
jquery-ui-1.12.1.custom.zip
jquery.zip
tabulator-master.zip
virtualscroller.zip

Ok, please wait 30 minutes. gradle.build should be committed within this time.

Greg Shah wrote:

Running from the filesystem is really not that important. It saves a 30 second jar operation.

That is a matter of opinion and I disagree. If I am trying to make a new client feature work, I often have many changes in rapid succession, and it may save dozens of 30 second jar operations over the course of an hour. It's about loading/testing my own code changes, not third party code.

Eric, please approve that it works for you.

Yes, a simpler form of it. The key issue was just that the src directory must come before p2j.jar in the classpath. Thanks.

Eric Faulhaber wrote:

I will provide a server side API which returns an array of source code lines. The client code should save these out locally to the file specified by the user, with the appropriate line endings.

Actually, you won't need to call a server API. The source lines are already stored on the client in the srcview module as variable sourceLines, an array of strings in line number order.

Sergey Ivanovskiy wrote:

Ok, please wait 30 minutes. gradle.build should be committed within this time.

Greg, I missed that versions are important here. Could you rename

d3.zip to d3-4.9.1.zip,
jquery.zip to jquery-3.2.1.zip
tabulator-master.zip to tabulator-master-2.12.0.zip

virtualscroller.zip to virtualscroller-1.zip

With these changes local resources under 3pl and src/com/goldencode/p2j/report/web/res/index.html should be changed accordingly.

I missed that versions are important here. Could you rename

Done.

Sergey Ivanovskiy wrote:

With these changes local resources under 3pl and src/com/goldencode/p2j/report/web/res/index.html should be changed accordingly.

Do we need a 3pl directory? At this point, don't we just need anything under the src/com/goldencode/p2j/report/web/res/ directory to be seen?

Thank you for changes

With these changes local resources under 3pl and src/com/goldencode/p2j/report/web/res/index.html should be changed accordingly.

Do we need a 3pl directory? At this point, don't we just need anything under the src/com/goldencode/p2j/report/web/res/ directory to be seen?

Yes, 3pl is for development mode and src/com/goldencode/p2j/report/web/res/ is required for the web application resources: javascripts and templates.

Sergey Ivanovskiy wrote:

Yes, 3pl is for development mode and src/com/goldencode/p2j/report/web/res/ is required for the web application resources: javascripts and templates.

I wanted to ask about that.

When you are ready, please document the process to add new third party libraries in the future.

My understanding so far is that if I want to try out a new library for development, I would drop it in 3pl temporarily and use it as I have been. Then, I would either discard it if I decide not to use it after all, or zip it up and push it to proj.goldencode.com/artifacts. However, I am not 100% clear on the changes needed to the build to pick it up from there.

Eric Faulhaber wrote:

Sergey Ivanovskiy wrote:

Yes, 3pl is for development mode and src/com/goldencode/p2j/report/web/res/ is required for the web application resources: javascripts and templates.

I wanted to ask about that.

When you are ready, please document the process to add new third party libraries in the future.

My understanding so far is that if I want to try out a new library for development, I would drop it in 3pl temporarily and use it as I have been. Then, I would either discard it if I decide not to use it after all, or zip it up and push it to proj.goldencode.com/artifacts. However, I am not 100% clear on the changes needed to the build to pick it up from there.

Let me do it, then I will answer.

Greg, I was running and archived packages incorrectly so they got additional parent folder. I fixed it and saved files in devsrv01:/home/sbi/packages-flat/ Please upload them again.

Sergey Ivanovskiy wrote:

Greg, I was running and archived packages incorrectly so they got additional parent folder. I fixed it and saved files in devsrv01:/home/sbi/packages-flat/ Please upload them again.

Done.

Greg, it seems that something is wrong with it now because trying to get https://proj.goldencode.com/artifacts/javascript/d3-4.9.1.zip leads to
Forbidden

You don't have permission to access /artifacts/javascript/d3-4.9.1.zip on this server.

Try it now.

Thank you, I am testing changes for commit.

Sergey Ivanovskiy wrote:

Greg, I was running and archived packages incorrectly so they got additional parent folder. I fixed it and saved files in devsrv01:/home/sbi/packages-flat/ Please upload them again.

Committed revision 11370 worked with zipped modules from devsrv01:/home/sbi/packages. We were correct at the second try. Greg, please could you upload devsrv01:/home/sbi/packages again to the server. The revision 11370 worked with this new zipped modules.

Eric Faulhaber wrote:

Sergey Ivanovskiy wrote:

Yes, 3pl is for development mode and src/com/goldencode/p2j/report/web/res/ is required for the web application resources: javascripts and templates.

When you are ready, please document the process to add new third party libraries in the future.

My understanding so far is that if I want to try out a new library for development, I would drop it in 3pl temporarily and use it as I have been. Then, I would either discard it if I decide not to use it after all, or zip it up and push it to proj.goldencode.com/artifacts. However, I am not 100% clear on the changes needed to the build to pick it up from there.

During the development mode we can use unzipped packages that should have version number in its folder following this pattern [module name]-[version](-[classifier]), where classifier in braces is not required but it can be there. For an example,

virtualscroller-1

Important if you are already using the last revision with incorrect repository, then after fixing the repositories zip files

[ ]    adamwdraper-Numeral-js-7de892f.zip    2017-07-21 14:12     125K     
[ ]    bootstrap-3.3.7-dist.zip    2017-07-21 14:12     364K     
[ ]    cbtree-v0.9.4-0.zip    2017-07-21 14:12     429K     
[ ]    d3-4.9.1.zip    2017-07-21 14:12     236K     
[ ]    jquery-3.2.1.zip    2017-07-21 14:12     78K     
[ ]    jquery-loading-overlay-1.5.3.zip    2017-07-21 14:12     40K     
[ ]    jquery-ui-1.12.1.custom.zip    2017-07-21 14:12     191K     
[ ]    tabulator-master-2.12.0.zip    2017-07-21 14:12     89K     
[ ]    virtualscroller-1.zip    2017-07-21 14:12     4.3K     

The development mode is working without these zipped modules as we realized the src folder should be first in the classpath.

Sergey, are you still planning to remove the remaining contents of the 3pl directory? I thought this normally would remain empty (even for development mode), and we would use it only on a transient basis while testing/developing/evaluating new third party libraries.

Sergey Ivanovskiy wrote:

Sergey Ivanovskiy wrote:

Greg, I was running and archived packages incorrectly so they got additional parent folder. I fixed it and saved files in devsrv01:/home/sbi/packages-flat/ Please upload them again.

Committed revision 11370 worked with zipped modules from devsrv01:/home/sbi/packages. We were correct at the second try. Greg, please could you upload devsrv01:/home/sbi/packages again to the server. The revision 11370 worked with this new zipped modules.

OK, I've re-uploaded the code.

Eric Faulhaber wrote:

Sergey, are you still planning to remove the remaining contents of the 3pl directory? I thought this normally would remain empty (even for development mode), and we would use it only on a transient basis while testing/developing/evaluating new third party libraries.

Yes, we can remove all content in this folder, but before we should check that artifacts repository contains the correct zip archives from which we can restore the content of 3pl.

Greg Shah wrote:

Sergey Ivanovskiy wrote:

Sergey Ivanovskiy wrote:

Greg, I was running and archived packages incorrectly so they got additional parent folder. I fixed it and saved files in devsrv01:/home/sbi/packages-flat/ Please upload them again.

Committed revision 11370 worked with zipped modules from devsrv01:/home/sbi/packages. We were correct at the second try. Greg, please could you upload devsrv01:/home/sbi/packages again to the server. The revision 11370 worked with this new zipped modules.

OK, I've re-uploaded the code.

Thank you, the zip archives are correct. It seems that due to my manual built each of them contains its zipped content inside.

Sergey Ivanovskiy wrote:

Eric, please confirm that save to file and copy to clipboard must be accessed only from this screen
[...]

Yes. I think your implementation will go in srcview.js. It probably doesn't matter for your implementation, but note that this screen can be accessed from several places (code reports, schema reports, call graph reports, and visualize call graph).

Committed revision 11373 added copy to clipboard functionality. The usage is that press CTRL+SHIFT+C, then Copy to Clipboard dialog prompt appears. Press CTRL+C or close this prompt.
If CTRL+C is pressed, then the target sources are copied into the system clipboard and finally the opened dialog prompt is closed.

Code review 1514a/11373:

• Ctrl+Shift+C is already hooked by Chrome to bring up the debugging view, so we need to find some other key combination for this feature that works across browsers.
• Why is the intermediate dialog needed?
• In buildSourceCode, we are hard-coding the line ending to \n. Don't we need to use a platform-specific line ending here?
• Please follow the same "javadoc" convention used throughout the rest of the file.

Note that I've cleaned up the code just a little bit. Please resume work from the latest revision in bzr.

Sergey, next is a find feature for the same source view.

The scrolling viewport only buffers enough rows to display what is needed, not the entire document, so the browser's Ctrl+F won't search the whole source/cache file. We need a find next/previous feature.

The idea is to have a small icon in the top right of the source view. When clicked, it is replaced by a text box with up and down arrows at its right edge. The feature should work in a somewhat similar way to the browser's Ctrl+F feature, but scoped only to the source code view. As the user types into the text box, a key handler searches downward through the source lines to match the typed text, starting at the line currently positioned at the top of the viewport.

If a match is found, it should be highlighted the same way text is highlighted when drilling down from a report. However, it should only be highlighted, not boxed. Also, to make things simpler than the browser's Ctrl+F feature, only one match should be highlighted at a time. Clicking the up or down arrows would highlight the previous or next match (if any), respectively. If there is only one match, or none, the arrows would be disabled. The viewport should scroll to the line number of the match.

The arrows should be flat, not buttons; the cursor should change to a point over them as a cue. The text box should be relatively plain, probably just white with drop shadowing to match the edges of the viewport. The text box is dismissed (and the icon returns) if the user clicks anywhere on the source or AST view, or if the user presses ESC.

The search should wrap to the top of the file and conclude at the line (if any) just before the line at which the it started.

Let's use an orange-red color for the highlighting, to differentiate from the other three types of highlighting we already do. Please use the existing code as a guide w.r.t. to the highlighted text visualization.

Why is the intermediate dialog needed?

This is due to a security feature in all browsers (except IE). They do not allow javascript code to directly copy text into the system clipboard. Only a direct user clipboard action on a browser-level input field with the highlighted text to be copied will work. If your code doesn't have such a setup, then the cross-browser way to do this is to create an intermediate dialog with the full text.

Eric Faulhaber wrote:

Code review 1514a/11373:

• Ctrl+Shift+C is already hooked by Chrome to bring up the debugging view, so we need to find some other key combination for this feature that works across browsers.
• Why is the intermediate dialog needed?

Can we use ALT+C keys to open this Copy dialog? This intermediate dialog is used to select and to focus a hidden input field holding the target value that must be copied into the system clipboard.

Working on the find next/previous feature.

Sergey Ivanovskiy wrote:

Can we use ALT+C keys to open this Copy dialog?

That's fine with me, as long as this combination is not already assigned to some other purpose by a browser. Which browsers have you tested?

I tested Firefox and Google Chrome. The Alt key's action is to open the Browser's menu but it seems that the default action can be blocked by intercepting key press event on this event like event.preventDefault() and event.stopPropogation(). It looks that for these browsers Alt + C is not assigned by default.

In addition to Firefox and Chrome, please test IE, Edge, and Safari.

If this can be implemented so that it does not interfere with the normal operation of any of these browsers (at the save versions that we support generally for FWD), please move ahead with this plan.

We already developed cross-browsing keys processing in p2j.keyboard.js that implements Progress keystrokes. This code can't be reused but this approach can be used for detecting key strokes for our cases. The current implementation used to detect keystrokes in srcview.js is incorrect for Chrome, since CTRL+SHIFT+C doesn't generate keypress.

Eric Faulhaber wrote:

In addition to Firefox and Chrome, please test IE, Edge, and Safari.

1.The current web report doesn't work with IE-Edge browser running on Windows 7. 'user' is undefined script error in source code index.html line 463 column 16
2. On Linux Array.includes doesn't work for Firefox 38.0.5 (not modern), but IE doesn't support Array.includes function too. source code event.js

It seems that it can be fixed by

=== modified file 'src/com/goldencode/p2j/report/web/res/index.html'
--- src/com/goldencode/p2j/report/web/res/index.html    2017-07-25 04:28:03 +0000
+++ src/com/goldencode/p2j/report/web/res/index.html    2017-07-25 20:21:36 +0000
@@ -454,13 +454,13 @@
                if (evt.which == '13')
                {
                   evt.preventDefault();
-                  doLogin(user.value, pass.value);
+                  doLogin($("#user").val(), $("#pass").val());
                }
             });
             $("#submitLogin", "#loginContainer").on("click", function(evt) 
             {
                evt.preventDefault();
-               doLogin(user.value, pass.value);
+               doLogin($("#user").val(), $("#pass").val());
             });

             // set initial focus to user name field

Sergey Ivanovskiy wrote:

It seems that it can be fixed by
[...]
I will commit this diff if there are no objections.

No objection.

Sergey Ivanovskiy wrote:

Eric Faulhaber wrote:

Sergey, are you still planning to remove the remaining contents of the 3pl directory? I thought this normally would remain empty (even for development mode), and we would use it only on a transient basis while testing/developing/evaluating new third party libraries.

Yes, we can remove all content in this folder, but before we should check that artifacts repository contains the correct zip archives from which we can restore the content of 3pl.

Please do this.

We have all zipped modules stored in the artifacts repository and each of these zip archives has its own zipped content inside. Thus they can be squeezed by removing these extra contents. And now we use uncompressed js modules for jquery, bootstrap.js, d3.js, adamwdraper-Numeral-js-7de892f, tabulator and virtualscroller. These modules: jquery, tabulator and virtualscroller, have no minimized versions inside their zipped modules.

      <!-- jquery -->
      <script type="text/javascript" src="./3pl/jquery-3.2.1/jquery-3.2.1.js"></script>
      <!-- bootstrap -->
      <link rel="stylesheet" href="./3pl/bootstrap-3.3.7-dist/css/bootstrap.css" type="text/css">
      <script type="text/javascript" src="./3pl/bootstrap-3.3.7-dist/js/bootstrap.js"></script>
      <!-- Numeral.js -->
      <script type="text/javascript" src="./3pl/adamwdraper-Numeral-js-7de892f/numeral.js"></script>
      <!-- d3 -->
      <script type="text/javascript" src="./3pl/d3-4.9.1/d3.js"></script>
      <!-- virtualscroller -->
      <script type="text/javascript" src="./3pl/virtualscroller-1/virtualscroller.js"></script>

The copy into the clipboard feature requires to add some OS and Browsers detection flags and query command supported, but there is the following issue. We have two independent clients: FWD web client and FWD report web client. These two applications can share common js codes between them. At this moment I would like to reuse OS and Browsers detection flags and query command supported, but couldn't do it because we have no shared codes.

Committed rev 11381 fixed Chrome copy into clipboard functionality, but added some common code from p2j.js into main.js that should be refactored according to the system design for the shared modules that can be used by these clients.
Committed rev 11382 removed 3pl content. From this revision these archives must be in the classpath of the report application

-rw-rw---- 1 sbi sbi 267820 Jul 21 22:03 adamwdraper-Numeral-js-7de892f.zip
-rw-rw---- 1 sbi sbi 746350 Jul 21 22:03 bootstrap-3.3.7-dist.zip
-rw-rw---- 1 sbi sbi 886350 Jul 21 22:03 cbtree-v0.9.4-0.zip
-rw-rw---- 1 sbi sbi 483752 Jul 21 22:03 d3-4.9.1.zip
-rw-rw---- 1 sbi sbi 159342 Jul 21 22:03 jquery-3.2.1.zip
-rw-rw---- 1 sbi sbi  83430 Jul 21 22:04 jquery-loading-overlay-1.5.3.zip
-rw-rw---- 1 sbi sbi 392530 Jul 21 22:04 jquery-ui-1.12.1.custom.zip
-rw-rw---- 1 sbi sbi 183876 Jul 21 22:04 tabulator-master-2.12.0.zip
-rw-rw---- 1 sbi sbi   7072 Jul 21 22:06 virtualscroller-1.zip

As of rev 11381, branch 1514a still has the following features/defects to be addressed:

• Make "change password" prompt more user friendly:
  • <Enter> in a text field to submit [SBI: done, rev 11387-11388]
  • Change text of last field to "Re-enter New Password:" [ECF: done, rev 11385]
• Attributions [ECF: done, rev 11405]
• Link to Redmine doc (https://proj.goldencode.com/projects/p2j/wiki/Code_Analytics) [ECF: done, rev 11405]
  • This documentation needs to be improved significantly; currently, it points to documentation about the older versions of this feature set and the most useful information is the PUG Challenge presentation
• LE: Copyright Notice [ECF: done, rev 11405]
• Source view "Find" feature
• Source file list screen (and menu item) with links to source view [ECF: done, rev 11401]
• New database CONNECT statement standard report [ECF: done, rev 11397]
• Visualize Call Graph not honoring some file filter profiles (error "Non-AST entry point received:<profile name>" appears in report server log). "Include All" default profile seems to be honored, but produces very dense visualization after a long load time. [CA: done, rev 11395]

• Metadata tables/fields not represented in schema reports and source view
• Source view highlight not right for some schema artifacts (ranges not correct for tables and fields in many cases)
• Source view highlight not correct (2 lines off vertically) when viewing very bottom of any source file
• Source/AST view layout not correct when linked from Visualize Call Graph screen (extends beyond containing border horizontally)
• Old height algorithm used when resizing call graph view (new algorithm used for initial view) -- perhaps we add a skip list or selector parameter to getAvailableHeight function in main.js, so this algorithm can be used when components are already added to DOM.
• Loading overlay is stilted/blocked when used with a long-running client-side operation (e.g., when loading filter profiles for large projects), due to single threading on client
• AST view is very closely packed vertically and is illegible for broad trees (e.g., when viewing level 0 for most external procedures). Should extend beyond top and bottom view boundaries, requiring panning or zooming.
• Multi-user use is problematic in some cases
  • The active_file global temp-table is used in read mode for nearly every report; it is written to whenever a user logs in. If a long-running report is using this table in one session, a user can be blocked from logging in at all, since the request to acquire the write lock on this table will time out.
  • Activating a filter profile also causes writes to the shared active_file temporary table. So, though I have not tested this, I assume we will have similar issue when trying to activate a filter profile in one session while a long-running report is being built in another.
• LE: Importing a file list that references source files that no longer exist in the latest version of the application may be a problem. [ECF: struck this item; I had intentially omitted, as I don't think there's a referential integrity issue after all]
• LE: Clicking on the expand/collapse "twisty" in the tablulator can be tricky. It seems so sensitive as to be error prone.
• Performance issues
  • Reports for large projects can take a very long time to build, especially with non-default file filter profiles
  • Ad-hoc search and custom reports can take a very long time to run with large projects -- they are single-threaded and visit every AST (except when active filter is applied for search)
  • Loading filter profiles causes a long-running client operation for large projects, as it leverages the checkbox tree's logic to manage nested branches/nodes
• Investigate: certain server APIs write to the database (e.g., adding a custom report, updating search history, etc.). Do we need to explicitly synchronize these APIs or is the database's normal locking behavior enough?

• Integrate preprocessor information (i.e., *.pphints data) with source view, so user can see where included content originated
• Investigate changing default filter profile
  • Note: the concept of the default profile has fairly complex/deep implementation implications; the definition of this profile is hard-coded in the batch report run which creates the initial database. Making this changeable is not recommended.
  • As an alternative, we may just want to remember which profile a user had active in their last session and apply that when they log back in; however, this has its own performance implications, since applying a non-default profile causes all overview reports to be rebuilt, which is very expensive.
• Disable browser back button
• LE: Remove the red background from the menu icons, replace it with something more neutral (or make it transparent).
• Improve error output for search and custom reports.
  • The error messages for invalid condition and multiplex expressions currently reported come directly from the TRPL engine and aren't particularly user friendly
  • Currently, we display only the most specific error message from the Java exception chain
  • Making TRPL error messages significantly more user-friendly may not be feasible in the short run, but perhaps we can at least provide more of the original error output to help with some context
• Support multiple user accounts in addition to the initial admin account
  • Requires an administration page in the web app for the admin user to
    • create a user
    • delete a user
    • reset a password
• Add on-demand facility to change a password (currently, it is required only once, when logging in with the default password)
• Automatic password aging
• Pluggable valid password policy (low priority)
• Review APIs to determine if administrative authority should be checked before performing
  • Can probably be implemented by augmenting existing WebApi annotation with security information

LE: GES made some additions marked LE: above.

Eric Faulhaber wrote:

• Make "change password" prompt more user friendly:
  • <Enter> in a text field to submit

Sergey, please take this item next.

• It disrupts the browser's default handling of Ctrl+Shift+C.
  • I thought we had decided to use Alt+C instead? Then again, you mentioned the browser hooks Alt to open the menu, so disrupting this is no better.
  • To recreate:
    • Log into the web app and hit Ctrl+Shift+C before visiting the source file view. In both Firefox and Chrome, this brings up the inspector. Close the inspector.
    • Now drill down to the source file view and hit Ctrl+Shift+C again. This time, it brings up the copy dialog instead. I understand this is intentional, however, we don't want to disrupt the default browser behavior, so we need to choose an otherwise unused key combination.
• The key handler survives the source file view. After using it once, the Ctrl+Shift+C key combination brings up the copy to clipboard dialog anywhere in the web app. This is confusing and without meaning anywhere but the source view.

Let's take a step back and rethink this. The key combination trigger solution is becoming too complicated. There is no hard requirement for this to be a key combination. Let's use a much simpler way to do this by using a graphical cue instead. We can add an icon to the right of the "back button" (i.e., the left arrow), above the source file viewer. This also will make it more obvious to the user that the feature exists.

I think a good icon for this would be a bootstrap glyph icon (glyphicon glyphicon-duplicate). Similarly, when we add the download file feature, we could use glyphicon glyphicon-download for that.

Ok, the tool icons are good to represent these functionality. I will remove CTRL+SHIFT+C keys handler and then add this tool button with icon glyphicon glyphicon-duplicate. ALT+C should work too because I used CTRL+SHIFT+C and ALT+C to invoke Copy to Clipboard dialog.
If the Copy to clipboard dialog is displayed in a different screen, then it means that srcview.cleanup() is not invoked during its life cycle. I supposed that this method would be invoked if srcview became hidden, but didn't check this.
Are there functionality that can be used if one screen becomes hidden? If there are no such functionality, then the view itself can do the required cleanups depending on its use cases.

No, there is a more simple way to check if srcview is visible $('#srcview').is(':visible').

Committed revision 11386 added the copy to clipboard tool button, fixed duplicated event listeners and the Copy to Clipboard dialog incorrect appearances.

Eric Faulhaber wrote:

Make "change password" prompt more user friendly:
<Enter> in a text field to submit

Sergey, please take this item next.

The target dialog is created by wsbase.changePassword, correct? How to access to the administration functionality?

Sergey Ivanovskiy wrote:

The target dialog is created by wsbase.changePassword, correct?

Yes.

How to access to the administration functionality?

Do you mean the server call? It is done by the function defined in the rpc variable at line 268. This variable is registered with the dijit ConfirmDialog as its onExecute (i.e., OK button clicked) callback.

I didn't see this dialog until all Hotel GUI report application had been rebuilt with ant rpt and ant callgraph. Committed revision 11387 fixed the Change Password dialog to be user-friendly.

Sorry, I should have mentioned this.

Sergey Ivanovskiy wrote:

I didn't see this dialog until all Hotel GUI report application had been rebuilt with ant rpt and ant callgraph. Committed revision 11387 fixed the Change Password dialog to be user-friendly.

Sergey, the new functionality is nice, but there is a regression. The dialog now remains visible after you've successfully changed the password, and you have to close it explicitly. It was closing automatically before.

This is an SQL snippet I've been using to create and populate the active_file temporary table to represent an "Include All" file filter profile for a standard project. I had this in the ReportApi.java file, but I want to drop it from there. Saving it here for reference.

drop table if exists active_file;
create temp table active_file (fid bigint, sid bigint,
constraint fk_actfile_fid foreign key(fid) references file(id));
insert into active_file (fid, sid)
select distinct(f.id), 1 from file f join ast_map a on a.fid = f.id where f.name like './abl/%' union
select distinct(f.id), 1 from file f join ast_map a on a.fid = f.id where f.name like './data/%';

It seems that this regression was fixed by committed revision 11388. It take more time than expected due to an exception that was probably thrown by fadeOut.play() and fadeOut.cancel() concurrent conflicts. dialog.hide() invokes deferred actions, so onhide event must be used.

Sergey, Constantin, I am planning to rebase 1514a to the latest trunk shortly. Is this going to cause a problem for either of you w.r.t. your current work?

Eric Faulhaber wrote:

Sergey, Constantin, I am planning to rebase 1514a to the latest trunk shortly. Is this going to cause a problem for either of you w.r.t. your current work?

Is OK for me to rebase.

I have rebased task branch 1514a to trunk rev 11156. The current 1514a task branch revision is 11391.

Greg, I've made the callgraph to work with the filter profiles. A question: I don't need to add the filter profiles to the callgraph reports, too, correct?

Hmmm. Good question. For now, we won't worry about adding filter profiles to the CG reports.

Greg Shah wrote:

Hmmm. Good question. For now, we won't worry about adding filter profiles to the CG reports.

OK, see revision 11393 for the changes.

Greg, I've made the callgraph to work with the filter profiles.

It is strange but it is not working for me. I rebuilt 1514a at rev 11394. I did an ant clean and ant report_server in Hotel GUI. I can login and use the call graph visualization.

But when I try to load graph on a file list (I have one called "Top Level" that has includes these: abl/emain.p, abl/ehotel.p, abl/start.p, I just get a blank page in the CG visualization. All that appears in the log file is:

Non-AST entry point received:Top Level

Ideas?

Greg Shah wrote:

Greg, I've made the callgraph to work with the filter profiles.

It is strange but it is not working for me. I rebuilt 1514a at rev 11394. I did an ant clean and ant report_server in Hotel GUI. I can login and use the call graph visualization.

OK, my problem was that I tested with report starting dir the same as the dir where the AST files reside. If started from deploy/server, it will not see them as the paths are relative to ./abl. I was using the file system to generate the file list, but I think I need to use the rpt DB to get them.

Greg, see 1514a revision 11395 for a revised filter profile for callgraph view.

Loading from a file list works now.

In my case the load leaves everything off-screen right now, but if you pan/zoom you see it. It is a something to fix later.

Sergey, w.r.t. the find task, I want to make sure we are taking the simplest/cleanest approach to this implementation.

There should be minimal "new" work using d3 for this feature. What I mean by that is that there are already examples of the viewport scrolling and highlighting (including the creation of paths and gradients). We need to leverage these existing techniques and/or functions, instead of inventing new ways to do this.

We can use the Bootstrap glyphicons again (e.g., glyphicon glyphicon-search) in what is becoming our source view "tool bar" for the initial trigger to display a "floating" panel of Dojo controls in the top right corner of the source viewer.

This panel should be minimalist, containing only a Dojo text box and perhaps two small buttons to move to the next and previous match. Maybe we don't even need buttons for this; perhaps just clickable glyphicons (glyphicon glyphicon-triangle-bottom and glyphicon glyphicon-triangle-top) would suffice. Model this floating panel after what you did for the controls in the "Visualize Callgraph" screen.

Please let me know what d3 issues you are having. We can probably work around or eliminate them by leveraging existing code or choosing a different approach.

Also, I should note that the find feature doesn't need to interact with the AST viewer at all, so this should cut down on the complexity.

Let's keep the find itself simple, too. By only allowing a match within a single line (i.e., not across lines), the search algorithm can be relatively simple. I do think it should be case insensitive, however.

Ok, understand, the task is clear for me, thank you. Planning to fix this functionality sooner.

Eric Faulhaber wrote:

We can use the Bootstrap glyphicons again (e.g., glyphicon glyphicon-search) in what is becoming our source view "tool bar" for the initial trigger to display a "floating" panel of Dojo controls in the top right corner of the source viewer.

This panel should be minimalist, containing only a Dojo text box and perhaps two small buttons to move to the next and previous match. Maybe we don't even need buttons for this; perhaps just clickable glyphicons (glyphicon glyphicon-triangle-bottom and glyphicon glyphicon-triangle-top) would suffice. Model this floating panel after what you did for the controls in the "Visualize Callgraph" screen.

Please look at the committed revision 11399, it reuses the same tool bar panel for search field, previous and next source buttons. Now the search functionality should work in a forward direction only if the enter key is pressed on the search field, the next pressed enter moves to the next sources forward. I will proceed if you agree that we can use the same tool bar for search control. If the focus is moved out of the search control, then the search control becomes hidden and the search tool bar button appears again.
Committed rev. 11400.

• Please make the placeholder text "Find..." instead of "Search...". I don't know why, but at least to me, "Search..." connotes a more complex operation, possibly involving database access, while "Find..." to me suggests a simpler operation involving only the current document. Also, it is consistent with a similar feature in the browser and most editors and word processors.
• Please add a few pixels of horizontal padding between the tool bar items; it seems a bit crowded.
• If it is straightforward to add tooltips to these tool bar items (e.g., "Back", "Copy to Clipboard", "Find"), this would be useful.

• Find should only highlight the exact phrase typed by the user in the "Find..." text box, not the full text representing an AST node. This will require some refactoring of the code inside createAstSourcePath into a new function that allows one to pass in the starting and ending line and column numbers. This new function should be called from createAstSourcePath and from whatever new code path you create for the Find highlighting.
• The highlight color for Find should differ from the other selection types. Please create a distinctive, new gradient (using appendRadialGradient) to create an orange-red gradient for this purpose. Add a new selectType.FIND constant for this use.
• Find shouldn't change the (pink) highlight associated with the selectType.MATCH selection.
• Find should un-highlight the previous Find highlight when the next/previous one is visited, but no other selection types should be affected.

Eric Faulhaber wrote:

• If it is straightforward to add tooltips to these tool bar items (e.g., "Back", "Copy to Clipboard", "Find"), this would be useful.

Yes, at first I used https://www.w3schools.com/css/css_tooltip.asp solution example, and then decided to use Bootstrap tooltip plugin.

Find should only highlight the exact phrase typed by the user in the "Find..." text box, not the full text representing an AST node.
Find shouldn't change the (pink) highlight associated with the selectType.MATCH selection.

Please approve that FIND results are only highlighted on the Source panel and its peer, AST panel, is not used for the AST representation of FIND results.

Sergey Ivanovskiy wrote:

Find should only highlight the exact phrase typed by the user in the "Find..." text box, not the full text representing an AST node.
Find shouldn't change the (pink) highlight associated with the selectType.MATCH selection.

Please approve that FIND results are only highlighted on the Source panel and its peer, AST panel, is not used for the AST representation of FIND results.

Yes, the AST view is not involved at all in the Find feature.

Sergey Ivanovskiy wrote:

Eric Faulhaber wrote:

• If it is straightforward to add tooltips to these tool bar items (e.g., "Back", "Copy to Clipboard", "Find"), this would be useful.

Yes, at first I used https://www.w3schools.com/css/css_tooltip.asp solution example, and then decided to use Bootstrap tooltip plugin.

Does the title attribute not work for the glyphicon spans?

Thank you, adding title attribute displays a tooltip properly with a different style. I will fix this issue too.

As you make this change, please ensure the tooltip appears for each use of the back "button". For some reason, it is not appearing for the detail report use case, for instance.

Also, there seems to be too much real estate reserved for these tool bar items now, shifting the previously centered titles too far to the right. Perhaps the bootstrap row/column approach is not the best in this case. The titles should remain centered, or at least very close to centered, despite the presence of the glyphs.

Ok, will do this. I just found that Schema Reports>Tables By Database with Database _temp can open an empty dropdown report for

./abl/common/adecomm/as-utils.w 82 26 appsrv-tt [TEMP_TABLE] @82:26 partition [FIELD_CHAR] @83:28 ...

TypeError: sourceLines[i] is undefined[Learn More]  srcview.js:1492:10
    createAstSourcePath https://localhost:9443/srcview.js:1492:10
    highlightAst https://localhost:9443/srcview.js:1419:21
    highlightCode https://localhost:9443/srcview.js:1360:13
    me.show https://localhost:9443/srcview.js:953:10
    showSourceAstView/< https://localhost:9443/report.js:1428:10
    me.sendMessage/< https://localhost:9443/wsbase.js:204:51
    processMessage https://localhost:9443/wsbase.js:172:13
    me.init/socket.onmessage https://localhost:9443/wsbase.js:156:10

OK, thanks. This is likely a regression due to a recent change I made. I'll look into it.

Eric Faulhaber wrote:

OK, thanks. This is likely a regression due to a recent change I made. I'll look into it.

This regression is fixed in 1514a/11408.

Committed rev 11409 fixed highlight of the currently matched element. Does it make sense to display a message with all matches like CTRL+F panel of Firefox? It selects all matches at first pressed enter and changes this selection if the filter is updated.

I ran conversion regression testing on devsrv01 (ChUI app) for 1514a rev 11404. It passed. The code was identical to that generated by FWD trunk.

If the changes after 11404 are only in the reporting web server/client code, then conversion regression testing will not have to be repeated.

Sergey Ivanovskiy wrote:

Does it make sense to display a message with all matches like CTRL+F panel of Firefox? It selects all matches at first pressed enter and changes this selection if the filter is updated.

No, let's keep it simple for this release. We can enhance later if needed.

I have not tried this revision yet. Please set focus to the Find text box automatically when it appears, if you haven't done this already.

Committed revision 11410 fixed the focus on the Find text box and added look-ahead all matches.
For this rev. there is a bug that it seems related to the virtual scroll component. If the scroll thumb reaches its lowest position, then the search highlight rectangle covers incorrectly place near the matched text.

This is a known problem (see third bullet point in defect punch list in #1514-177). If you can fix it quickly (an hour or two), go ahead. Otherwise, please leave it for now, and switch back to your other tasks.

Sergey Ivanovskiy wrote:

Committed revision 11410 fixed the focus on the Find text box and added look-ahead all matches.
For this rev. there is a bug that it seems related to the virtual scroll component. If the scroll thumb reaches its lowest position, then the search highlight rectangle covers incorrectly place near the matched text.

• The " x of matches " text doesn't make sense. Normally, one would expect, " x of y matches ", where x is the current match and y is the total number of matches in the file.
• The " x of matches " text seems to improperly change the count as the text box loses then regains focus.
• The find operation should begin at the line of code currently at the top of the viewport for a "next" find, or at the line at the bottom of the viewport for a "previous" find, not at the first line in the file (if they are different). It should wrap when it reaches the bottom of the file. Currently, it jumps to the first match in the file, even if the viewport already contains the text being sought.
• Currently, you can only find the initial match by hitting <Enter> after typing the target text. The next/previous buttons are disabled until after the first match is found using the <Enter> key. The user should be able to click a button immediately after typing the target text, without having to hit <Enter> first.

If the " x of matches " issues are problematic to fix or will take more than an hour or so, it is ok to remove this feature temporarily; we can fix it later. It is a nice feature, but it was not part of the original requirements, and we need to get this branch into testing.

Eric Faulhaber wrote:

• The " x of matches " text doesn't make sense. Normally, one would expect, " x of y matches ", where x is the current match and y is the total number of matches in the file.

Yes, I shortened x of y matches because the string becomes long and doesn't fit into the available space. I will check x of y matches template and if it is fitted the horizontal space, then will commit, otherwise will remove it completely.

Is it ok if the template x of y will be used instead of x of y matches? It seems that "matches" word doesn't add new meaning to this message.

• The " x of matches " text seems to improperly change the count as the text box loses then regains focus.

Please explain more thoroughly. I tested it seems that the count reflects matched cases properly.

• The find operation should begin at the line of code currently at the top of the viewport for a "next" find, or at the line at the bottom of the viewport for a "previous" find, not at the first line in the file (if they are different). It should wrap when it reaches the bottom of the file. Currently, it jumps to the first match in the file, even if the viewport already contains the text being sought.

The search is started from the first line on pressed ENTER and then prev and next searches by prev and next buttons are started from the current matched string.

• Currently, you can only find the initial match by hitting <Enter> after typing the target text. The next/previous buttons are disabled until after the first match is found using the <Enter> key. The user should be able to click a button immediately after typing the target text, without having to hit <Enter> first.

I will fix it.

Sergey Ivanovskiy wrote:

Eric Faulhaber wrote:

• The " x of matches " text doesn't make sense. Normally, one would expect, " x of y matches ", where x is the current match and y is the total number of matches in the file.

Yes, I shortened x of y matches because the string becomes long and doesn't fit into the available space. I will check x of y matches template and if it is fitted the horizontal space, then will commit, otherwise will remove it completely.

Is it ok if the template x of y will be used instead of x of y matches? It seems that "matches" word doesn't add new meaning to this message.

Yes, good.

• The " x of matches " text seems to improperly change the count as the text box loses then regains focus.

Please explain more thoroughly. I tested it seems that the count reflects matched cases properly.

1. open web app for hotel GUI
2. select "Language Statement Usage" report from "Base Language" in Code Reports section
3. select "find [KW_FIND]" in summary report
4. select "./abl/common/adecomm/as-utils.w" (310:1) in detail report
5. bring up Find text box and type "available", then <Enter>
   • message says "8 of matches"
6. click in text box again
   • message says "15 of matches"
7. keep clicking mouse in text box
   • count advances to 21, 26, 30, 33, 35, 36

Note: this is with Chrome; haven't tested with other browsers.

• The find operation should begin at the line of code currently at the top of the viewport for a "next" find, or at the line at the bottom of the viewport for a "previous" find, not at the first line in the file (if they are different). It should wrap when it reaches the bottom of the file. Currently, it jumps to the first match in the file, even if the viewport already contains the text being sought.

The search is started from the first line on pressed ENTER and then prev and next searches by prev and next buttons are started from the current matched string.

Yes, but this was not the requirement. This is not so important, and it seems it may take a while to fix. Let's leave it as is for now.

Thank you, I recreated this bug. Clicking on the Find text box initiates a look-ahead search with position moved to the next match and these new found matches are collected in the common array. I will fix it too.

Committed revision 11415 should fix these issues.

This revision is very close. The fixes are good.

• Please make the tool tips "Find Next" and "Find Previous" (i.e., remove the word "Source").
• When you reach the last match, "Find Next" becomes disabled. Likewise, when you reach the first match, "Find Previous" becomes disabled. Clicking on a disabled button causes the text box to lose focus and disappear, replaced by the magnifying glass icon. This is disconcerting. Please don't disable these buttons and instead wrap the search to the first match or to the last match, respectively.
• The wrapping of the Find Next from the last item to the first item also should apply to using the <Enter> key.

• Please "javadoc" all the code you added.

I think with these final changes, we will be done with the first release.

After this, please test the core features across Chrome, Firefox, Safari, IE 11, and Edge. Post any defects you find, which are not already documented in #1514-177.

Committed revision 11416 fixed jsdocs and find next/previous logic. Planning to complete manual testing at the beginning of the next working day.

Committed revision 11418 fixed copy to clipboard for IE.

I found these issues:
1) JS client throws these exceptions if resize or refresh the application web page:

jquery-3.2.1.js:273 Uncaught Error: cannot call methods on tabulator prior to initialization; attempted to call method 'redraw'

Task branch 1514a is frozen for regression testing. Please do not make any further changes to this branch.

1514a rev 11419 has passed ChUI conversion regression testing. There are no conversion changes.

Main runtime regression testing is running now.

Main regression testing failed but the 4 unexpected failures were all in some common problem spots. It is most likely OK, but I am running another round of testing now.

Greg Shah wrote:

Main regression testing failed but the 4 unexpected failures were all in some common problem spots. It is most likely OK, but I am running another round of testing now.

1514a/11419 has passed full search and developer ETF testing, so I will merge to trunk if/when you get a positive result with your runtime tests.

My 2nd run passed everything except for 1 test. Unfortunately, the same test failed in the 1st run. I ran it individually and that test passed.

We don't have any changes that would affect CTRL-C. I think this branch can be merged to trunk.

Task branch 1514a was merged to trunk and committed as revision 11157. The team was notified via email.