Project

General

Profile

Feature #3701

create a logic/control flow chart "view" for top-level blocks which will be available as part of Code Analytics

Added by Greg Shah almost 6 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Target version:
-
Start date:
Due date:
% Done:

100%

billable:
No
vendor_id:
GCD
version_reported:
version_resolved:

missing_call_graph_3701a_rev_11283.png (203 KB) Greg Shah, 09/10/2018 05:28 PM

missing_flow_chart_3701a_revision_11283.png (140 KB) Greg Shah, 09/10/2018 05:31 PM

webcola-3.3.9.zip (54.8 KB) Constantin Asofiei, 09/11/2018 08:58 AM

flow_chart_for_external_proc_start.p_branch_3701a_revision_11284.png (106 KB) Greg Shah, 09/11/2018 12:25 PM

improved_call_graph_branch_3701a_revision_11284.png (341 KB) Greg Shah, 09/11/2018 12:29 PM

start.p_external_proc_logic_snapshot_branch_3701a_rev_11289.png (94.1 KB) Greg Shah, 09/17/2018 02:41 PM

start.p_external_proc_flow_chart_branch_3701a_rev_11289.png (201 KB) Greg Shah, 09/17/2018 02:41 PM

expanded_zoomed_flow_chart_branch_3701a_rev_11296.png (23.8 KB) Greg Shah, 09/19/2018 07:02 PM

fully_collapsed_flow_chart_branch_3701a_rev_11296.png (6.7 KB) Greg Shah, 09/19/2018 07:02 PM

partially_expanded_flow_chart_branch_3701a_rev_11296.png (21.2 KB) Greg Shah, 09/19/2018 07:02 PM

partially_expanded_zoomed_flow_chart_branch_3701a_rev_11296.png (34.7 KB) Greg Shah, 09/19/2018 07:02 PM

fwd_code_analytics_code_control_flow_logic_main_window_external_proc_subset.png (22.9 KB) Greg Shah, 09/24/2018 08:41 PM

fwd_code_analytics_code_control_flow_logic_main_window_external_proc.png (401 KB) Greg Shah, 09/24/2018 08:41 PM

History

#1 Updated by Greg Shah almost 6 years ago

  • Subject changed from create a logic flow chart "view" for top-level blocks which will be available as part of Code Analytics to create a logic/control flow chart "view" for top-level blocks which will be available as part of Code Analytics

All 4GL applications we have ever worked with have almost zero technical documentation. One of the most important artifacts that can be provided for a program is a flow chart of the logic. Although we could consider some UML artifacts for this usage, a simple chart of the logic/flow of control can greatly aid in the understanding of code with which one is unfamiliar.

The "Control Flow" report in Analytics already identifies the AST nodes that would be represented by symbols in the flow chart. We don't want a symbol per line of code. Rather, we are looking for the overall structure of the flow of control (FOR, DO, REPEAT, IF/THEN/ELSE, LEAVE, NEXT, STOP, QUIT, RUN...). I would expect blocks, loops, decisions and so forth to be charted. Things that are not flow of control will not be included. Some of these overlap with call graph but the others are "internal" to a top-level block.

The idea is to calculate the flow chart on the fly for a given top-level block. Then we would create a visualization. I think the cola.js (http://ialab.it.monash.edu/webcola/) provides the basis for flow charts using D3. We would then display that in the Analytics, either on its own page or as part of the Source View.

I think the hardest part of this is the visualization. In particular, we have to make sure that it is usable, even for really large procedures. This suggests we might need an expand/collapse feature for sections of the chart.

#2 Updated by Constantin Asofiei almost 6 years ago

The estimate for this is ~1 to 1.5 weeks of work, assuming #3699 is already worked (i.e. webcola is implemented already in the callgraph, so we know how to use it).

#3 Updated by Constantin Asofiei almost 6 years ago

  • Status changed from New to WIP
  • Assignee set to Constantin Asofiei
  • % Done changed from 0 to 30

I've created 3701a with the changes for this task - see 11283.

The current state is this:
  • the flow diagram calculation is mostly working, but I still have some cases to determine the correct 'exit' nodes (i.e. how the next statement is linked to, when there are nested IFs, blocks, etc). For example, an IF with no ELSE will link to the next statement via itself and the last statement in the THEN branch, but there are problems if the THEN branch contains multiple nested/complex blocks.
  • the node/links are not 'prettified' yet (with i.e. labels and so on), I want to finish the chart to be computed and layout correctly.
  • currently, there is no grouping of the nested blocks or THEN/ELSE branches, the diagram will be just a flow on the X axis, which is pretty annoying, as it just a long enumeration of nodes. This is part of the layout, will look how to force it maybe to go on multiple rows.

The state is not yet usable for a demo.

#4 Updated by Constantin Asofiei almost 6 years ago

To access the diagram chart, use the icon on the left-side of the i.e. external program block's text.

#5 Updated by Greg Shah almost 6 years ago

I tested 3701a revision 11283 using Hotel GUI revision 173. I ran ant report_server and looked at the Analytics on the Call Graph Visualization screen:

If I click on the small drawing artifacts in the top left, I get this:

Any ideas about what I am doing wrong?

#6 Updated by Constantin Asofiei almost 6 years ago

Hmm, the com/goldencode/p2j/report/web/res/3pl/ folder is excluded by the build process, and I use 3pl/ folder to (temporarily) hold the cola.js sources. I'll change to (temporarily) use the report/web/res/cola folder, until I have the cola.zip archive (uploaded the GCD artifacts folder) and the change to build.xml.

#7 Updated by Greg Shah almost 6 years ago

It's easy for me to update the artifacts, so if you want to implement that part now go ahead.

#8 Updated by Constantin Asofiei almost 6 years ago

Greg Shah wrote:

It's easy for me to update the artifacts, so if you want to implement that part now go ahead.

OK, please upload this to the artifacts folder.

#9 Updated by Greg Shah almost 6 years ago

Done.

#10 Updated by Constantin Asofiei almost 6 years ago

Greg, please check rev 11284.

#11 Updated by Greg Shah almost 6 years ago

Wow! This is moving along nicely.

The following is the Hotel GUI call graph:

I manually moved things around to make them layout a bit better. By default, there should be some space between the containers. This is true at the top level containers but you can also see it in the intermediate levels. For example, each type of node tends to cluster together and layout such that they are touching. Please see if you can get:

  • Some padding between sibling nodes at all levels; and
  • Avoid grouping like-nodes in a single space (e.g. all events in one group, all internal procedures in another group).

Then, by clicking on the flow chart icon (nice touch!):

You can definitely see some structure start to appear even from this early version. It is really cool! The following are priorities from my perspective.

1. Different flow control mechanisms should be mapped to standard flow chart symbols. Some useful examples and descriptions:

https://en.wikipedia.org/wiki/Flowchart
http://www.breezetree.com/article-excel-flowchart-shapes.htm
http://duckmedia.co/editor/
https://creately.com/blog/diagrams/flowchart-guide-flowchart-tutorial/

2. Inner blocks/loops (REPEAT,FOR,DO,EDITING) should be be a container that groups the stuff inside. This has an example:

https://www.researchgate.net/figure/Flow-chart-of-principal-abdominal-aortic-aneurysm-AAA-monitoring-trajectory-The_fig1_49840317

I would expect some details for the block such as the language statement that defines the block/loop, label (if it exists), condition, tables referenced in FOR) to be displayed at the top of the container.

These same containers would be the obvious place to expand/collapse to make very large flow charts fit the screen.

You may already be thinking these same things, I just want to make sure the ideas get discussed here.

#12 Updated by Greg Shah almost 6 years ago

In the flow chart, clicking on the specific node should take you to the source code view with that node highlighted.

#13 Updated by Constantin Asofiei almost 6 years ago

Greg Shah wrote:

2. Inner blocks/loops (REPEAT,FOR,DO,EDITING) should be be a container that groups the stuff inside.

Yes, I'm considering this, but I haven't implemented yet it because the cola.js's powergraph support (which allows routing the edges so that they do not overlap) doesn't support explicitly specified groups. I'm still trying to use at least the edge routing part (which might allow me to define my own groups), but it doesn't seem to be working yet.

#14 Updated by Constantin Asofiei almost 6 years ago

Greg, for the IF, WHILE, FOR, expressions, I need a way to anti-parse the expression (to show it in the chart). At this time I have something simple which works with basic binary expressions and functions calls, but I'd like to have a Java method which gets a generic EXPRESSION node and produces the 4GL counterpart (maybe part of the #3696 task).

#15 Updated by Greg Shah almost 6 years ago

As soon as #2110 is done, I will work start work on this task. I will make sure we have a method that can output an arbitrary subtree.

#16 Updated by Greg Shah almost 6 years ago

It's a good idea that you had.

#17 Updated by Constantin Asofiei almost 6 years ago

Greg, the v = expr1 to expr2 [by k] and while expr cases for REPEAT, DO and FOR statements - do you want to expand this into:
  • var = expr1 to expr2 [by k]
    [v = expr1]
        |
        |    /------------------------------------------------------\
        v    V                                                       |
      /-----------\                                                  |
     | v <= expr2  | -- yes -> [the block's code] --> [ v = v + k ] /
      \-----------/
           |
           no
           v 
        [next statement after the block]
    
  • WHILE expr
             /-------------------------------------\
             V                                      |
      /-----------\                                 |
     |    expr    | -- yes -> [the block's code] --/
      \-----------/
           |
           no
           v 
        [next statement after the block]
    

    This has impact on how NEXT statement behaves, which would i.e. skip to v = v + k instead of the i.e. REPEAT block.

Otherwise, I can leave them as plain text in the SVG element associated with this block.

#18 Updated by Greg Shah almost 6 years ago

I think that when the block is expanded, it is better to have it broken out explicitly like you do here. This seems easier to understand and more in line with the idea of the flow chart.

Perhaps we use the "plain text in the SVG element associated with this block" for the "collapsed" state?

#19 Updated by Constantin Asofiei almost 6 years ago

I've rebased 3701a from trunk rev 11284. Also, I've committed some other improvements for the flow chart - see rev 11289.

Greg, please take a look. Currently, I have start (simple circle), end (thicker circle) and exit (filled circle) nodes - the END node marks a block exit, and the EXIT node marks the procedure termination (i.e. RETURN and normal exit will link to it). What remains:
  • done I plan to have multiple EXIT nodes (at least one for each block) - these will mark program termination. Otherwise, if there are multiple ways to terminate and only a single EXIT node, the links don't look so good. I think the links to END nodes can be routed cleanly.
  • done there are some issues with routing links for WHILE/TO loops - working on this
  • I need to fix fill colors and borders for blocks/statements, plus link lengths so that nodes are closer together.
  • discarded when multiple links converge to the same target, currently they overlap (closer they get to the target). I'm not sure yet how to route them without overlapping, this is low priority for now.
  • discarded for longer flows, I think we could 'fold them', i.e. if the flow reaches a certain width, go to another line, still from left-to-right, but with the outgoing links from the top flow properly routed to the first node in the flow bellow it. Again, this is low priority, I want to have a good version first.
  • OO flow test

#20 Updated by Constantin Asofiei almost 6 years ago

To add to the previous list:
  • done the support for collapse/expand block

#21 Updated by Greg Shah almost 6 years ago

Wow! This is really getting quite close. I captured the flow chart for the external procedure for start.p:

Here is the matching code:

  • There are 2 RUN statements that don't appear in the flow chart.
  • I think the WAIT-FOR statements should appear too.
  • The APPLY would be better if it had some details of the event and target (of available).
  • The labels on the arrows are pretty hard to read. I think they may need to be larger.
  • It probably makes sense to put TRUE/FALSE labels on the exit points from an IF "diamond".
  • The arrowheads probably should be a bit larger (not too much), so they are easier to see.
  • We may need to bias blocks to be vertical instead of horizontal so that the brain can more easily process the similarity to the top to bottom ordering of the code.

This is really cool!

#22 Updated by Constantin Asofiei almost 6 years ago

Greg Shah wrote:

  • There are 2 RUN statements that don't appear in the flow chart.
  • I think the WAIT-FOR statements should appear too.

I haven't considered explicitly including these - should all call-sites appear, and if so, should they link somewhere or just as code snippets?

  • The APPLY would be better if it had some details of the event and target (of available).

This doesn't appear intentionally in the flow chart, it appears just because the code looks like IF ... THEN APPLY, and not IF ... THEN DO: APPLY - the idea is, if the THEN doesn't have a child block, the single statement will be shown. This will be fixed when I have a routine to anti-parse some generic AST. But again, there is nothing hard-coding APPLY to be included in the flow chart.

  • The labels on the arrows are pretty hard to read. I think they may need to be larger.
  • It probably makes sense to put TRUE/FALSE labels on the exit points from an IF "diamond".
  • The arrowheads probably should be a bit larger (not too much), so they are easier to see.

Thanks, will improve it.

  • We may need to bias blocks to be vertical instead of horizontal so that the brain can more easily process the similarity to the top to bottom ordering of the code.

OK, unfortunately until now I worked on the left-to-right flow and I have some assumptions (like IF) which hard-code the border 'pin' of the source and target links; but shouldn't be hard to change (and keep the left-to-right, too).

#23 Updated by Constantin Asofiei almost 6 years ago

Greg, I need some input for #3701-22.

#24 Updated by Greg Shah almost 6 years ago

should all call-sites appear, and if so, should they link somewhere or just as code snippets

By definition, most call sites invoke targets that are outside of the scope of the current top-level block. Where some specific call/return is occurring, I think we should indicate that in some way.

An APPLY exits the local scope and then returns, possibly having executed some event logic. For this, it probably makes sense to have some exit terminator (possibly labeled with the event name) and then an arrow to and an arrow from to show that the control flow exits and returns.

A RUN statement does the same thing but with a procedure target.

On the other hand, a WAIT-FOR doesn't need to link to anything specific because there is no specific call/return and no specific target. I do think we might want something on the WAIT-FOR block to indicate an internal event processing loop, but otherwise that is it.

Does that make sense? If there is a specific target, provide some linkage and handle any call/return flow. If no specific target, then just show it.

#25 Updated by Constantin Asofiei almost 6 years ago

Greg Shah wrote:

Does that make sense? If there is a specific target, provide some linkage and handle any call/return flow. If no specific target, then just show it.

OK, so any statement which can affects the flow should be enhanced, to provide details about the call target.

#26 Updated by Greg Shah almost 6 years ago

OK, so any statement which can affects the flow should be enhanced, to provide details about the call target.

Yes, I think so. This is an opportunity for the individual symbols in the flow chart to be augmented so that the viewer gets a powerful amount of information at a glance.

#27 Updated by Constantin Asofiei almost 6 years ago

Greg Shah wrote:

Yes, I think so. This is an opportunity for the individual symbols in the flow chart to be augmented so that the viewer gets a powerful amount of information at a glance.

What about this: we output the entire 4GL code for such statements, and add a way to show to the target AST for i.e. external program, internal procedure, etc (if is not missing)?

For the block header (i.e. loop, ON phrases, etc) I've added a way to show them in the lower-left corner, with the AST details - do we need more, at the actual block?

#28 Updated by Greg Shah almost 6 years ago

What about this: we output the entire 4GL code for such statements, and add a way to show to the target AST for i.e. external program, internal procedure, etc (if is not missing)?

I'll have to see this to know how well it will work.

Are you trying to avoid statement-specific nodes?

For the block header (i.e. loop, ON phrases, etc) I've added a way to show them in the lower-left corner, with the AST details - do we need more, at the actual block?

This sounds like what I was asking for, though I would probably prefer the text at the top-left. Let me see how this looks and we will go from there.

#29 Updated by Constantin Asofiei almost 6 years ago

Greg Shah wrote:

Are you trying to avoid statement-specific nodes?

Yes, I want to avoid adding other nodes, and just rely on augmenting/showing info for the node.
Please see rev 11296 - I think is close to be done.

This sounds like what I was asking for, though I would probably prefer the text at the top-left. Let me see how this looks and we will go from there.

Please see rev 11296.

#30 Updated by Greg Shah almost 6 years ago

Rev 11296 captures.

On first entry:

When the repeat block is expanded, full size:

Zoomed in portion of partially expanded repeat:

Zoomed in portion of fully expanded:

Very cool! Thoughts:

  • The vertical flow is very helpful. Eric and I have been discussing the idea of being able to display this in the source view (in the right pane). If we do that, the vertical approach will also be needed.
  • I like the expand/collapse. We may want a different symbol for the expand/collapse than a circle, which has many alternative meanings in these flow charts.
  • I do think we need node-specific text and for some, customization of the symbol being used. Anti-parsing the code won't yield a good result because of the inconsistencies and irregularities of 4GL source code. In addition, some statements are usually massive (UPDATE) and we won't want to show all that detail. I know it is more work but the result will be worth it.
  • We should probably upper case the statement text like DO instead of do.
  • The orientation of the link title is often upside down or angled strangely.
  • The exit labels on IF diamonds are sometimes misplaced. They can overlay the diamond contents instead of being just outside of the diamond. This seems to happen when the larger of the two DO blocks is expanded.
  • When both DO blocks are expanded, they touch instead of having some buffer between them.
  • Even when the DO blocks are collapsed, shouldn't they indicate that the contents loops back up to the top of the REPEAT?
  • We will probably need a legend.

#31 Updated by Constantin Asofiei almost 6 years ago

Greg, please see 3701a rev 11299. You may want to check ./abl/fwd-embedded-driver.p.ast, it contains more complex code.

At this time, the chart is stable - I couldn't find any more mishaps on terms of computing the constraints for the flow levels, and left/right separation for conditional branches. I'm cleaning up the code and see how the callgraph can be improved next.

#32 Updated by Greg Shah almost 6 years ago

The result is really impressive! I love the extra controls to expand all blocks and choose horizontal/vertical. It is working quite well. I also like the ability to drill in to RUN targets. Very cool.

By the way, the flow generally seems best in vertical mode. The overall approach will work in its current form. I did find some places where there is some crossing of links, but we can leave it alone for now.

I should start the anti-parser this week. I'll let you know when that is ready.

One thing to mention is that the link text still sometimes is upside-down. It would be good to fix that.

Great work!

#34 Updated by Constantin Asofiei over 5 years ago

  • % Done changed from 30 to 100
  • Status changed from WIP to Review

One thing to mention is that the link text still sometimes is upside-down. It would be good to fix that.

If you are talking about a middle link text (and not a start link text), then I don't think I can fix that.

#35 Updated by Greg Shah over 5 years ago

Constantin Asofiei wrote:

One thing to mention is that the link text still sometimes is upside-down. It would be good to fix that.

If you are talking about a middle link text (and not a start link text), then I don't think I can fix that.

Yes, I was talking about the middle link text. We won't worry about it for now.

#36 Updated by Constantin Asofiei over 5 years ago

3701a was merged to trunk rev 11285 and archived.

#37 Updated by Greg Shah over 5 years ago

  • Status changed from Review to Closed

Also available in: Atom PDF