FWD » Core Development » User Interface

A customer has reported that the web client renders in a low resolution when run using a high dpi screen (e.g. a MacBook Pro).

It seems that canvas image smoothing (anti-aliasing) properties can cause this effect of low graphics resolution. In FWD anti-aliasing effects are switched off. Thus, turning on anti-aliasing makes text visual presentation to have smooth glyphs instead of sharp glyphs. Any way, these settings (p2j.screen.js) can control these visual effects

      /** Canvas 2D graphics context. */
      this.ctx = this.canvas.getContext('2d', {alpha : true});

      setImageSmoothing(this.ctx, false);

....................................................

   /**
    * Enable or disable image smoothing for the given graphics context.
    * 
    * @param    {CanvasRenderingContext2D} graphicsContext
    *           The given graphics context
    * @param    {boolean} enabled
    *           The enable flag that must be true to enable image smoothing and false to disable.
    */
   function setImageSmoothing(graphicsContext, enabled)
   {
      if (graphicsContext.imageSmoothingEnabled)
      {
         graphicsContext.imageSmoothingEnabled = enabled;
      }
      else
      {
         if (graphicsContext.mozImageSmoothingEnabled)
         {
            graphicsContext.mozImageSmoothingEnabled = enabled;
         }

         if (graphicsContext.webkitImageSmoothingEnabled)
         {
            graphicsContext.webkitImageSmoothingEnabled = enabled;
         }

         if (graphicsContext.msImageSmoothingEnabled)
         {
            graphicsContext.msImageSmoothingEnabled = enabled;
         }
      }
   }

and if image smoothing is on, then ctx.imageSmoothingQuality = "low" || "medium" || "high"; can have an effect.
It is an issue for me how to check these functionality on high resolution screens at this moment.

Please provide the display resolution parameters for which the low graphics quality has been observed.
These parameters can be important:

Resolution: 1920 x 1080 pixel, diagonal_size: 21.5 inch and the calculated pixels density
PPI (pixels per inch) = SQRT(width^2 + height^2)/diagonal_size = SQRT(1920^2+1080^2)/21.5 = 102.46

From https://en.wikipedia.org/wiki/Retina_display it can be found that the retina displays have PPI more than 218. Thus they have more than twice as many pixels per inch as the display PPI above.

Please provide the display resolution parameters for which the low graphics quality has been observed.

The customer saw this low resolution result on a Macbook (Pro?). I will ask for the details.

Perhaps the solution is to detect the high resolution mode and turn on anti-aliasing for this mode? As I recall, the reason we turned off anti-aliasing was because it made line drawing and text "fuzzy". Perhaps that does not occur in high resolution mode.

Sergey, Chrome has a feature where, after you open the Developer Tools, you can test a site how it looks on different resolutions - there you can add a retina display config (2880 x 1800 at 220 pixels per inch) and see how FWD Web client behaves.

The settings to switch resolutions are bellow the address bar.

From the customer:

This is seen on all high resolution displays, both Mac, Ubuntu and Windows. Although almost all modern Mac systems have high resolution (retina) displays as a standard. My MacBook Pro has a resolution of 2880x1800, but Mac OS is scaling everything 200% as if it were 1440x900 to prevent fonts, buttons etc. to be too small. The result of this is a very sharp display, on which you cannot see the pixels on a normal working distance.

I also think anti aliasing will make things worse, fuzzy as you said.

I have a partial screenshot attached of a screen and Virtual Box. As you can see the fonts and icons in the virtual box window are much sharper.

Maybe google on 'retina java’ or 'HiDPI java’ can help?

The screenshot is proprietary to the customer and is not being made available for public viewing.

Constantin, thanks, I will test changes with Chrome and then provide the patch for this issue. There is another vision of this issue https://developer.mozilla.org/en-US/docs/Web/API/Window/devicePixelRatio

It seems that for high resolution screens we should follow this example from https://developer.mozilla.org/en-US/docs/Web/API/Window/devicePixelRatio

var canvas = document.getElementById('canvas');
var ctx = canvas.getContext('2d');

// Set display size (css pixels).
var size = 200;
canvas.style.width = size + "px";
canvas.style.height = size + "px";

// Set actual size in memory (scaled to account for extra pixel density).
var scale = window.devicePixelRatio; // Change to 1 on retina screens to see blurry canvas.
canvas.width = size * scale;
canvas.height = size * scale;

// Normalize coordinate system to use css pixels.
ctx.scale(scale, scale);

Sergey Ivanovskiy wrote:

It seems that for high resolution screens we should follow this example from https://developer.mozilla.org/en-US/docs/Web/API/Window/devicePixelRatio

[...]

At this moment we can't use this approach because we use these low level functions getImageData and setImageData. If the underlined offscreen canvas size is 2 times wider than its css size, then canvas scaling has no effect on the image returned by getImageData because this function works with logical pixels, but not with css pixels. We probably should think about similar approach in order to change our drawing api so that it can use 2 times wider canvas for high resolution screen and put it finally into canvas having css size 2 times less than its canvas size. All coordinates should be scaled by factor 2 and the pixel lines should be smoothed somehow. We can draw a one pixel width line but the pixel line that 4 times wider should be drawn differently than CanvasRenderer.prototype.drawSlopedLineSegment does now.

Created a new task branch 4135a for these changes. It needs to add new functionality to the java client that should send 2 times wider binary images than original ones if the javascript client has a high resolution screen.

Can you just scale the smaller images into larger ones on the Javascript side? It seems useless to have Java send more data when this is really just a scaling problem that is specific to JS.

Greg Shah wrote:

Can you just scale the smaller images into larger ones on the Javascript side? It seems useless to have Java send more data when this is really just a scaling problem that is specific to JS.

The javascript client gets a binary image and puts it on the canvas. There is no Canvas API that can scale the binary image. I will google if there are java script open source libraries that can do this scaling.

Sergey Ivanovskiy wrote:

Greg Shah wrote:

Can you just scale the smaller images into larger ones on the Javascript side? It seems useless to have Java send more data when this is really just a scaling problem that is specific to JS.

The javascript client gets a binary image and puts it on the canvas. There is no Canvas API that can scale the binary image. I will google if there are java script open source libraries that can do this scaling.

drawImage() of the Canvas API can do scaling but it needs an image source but the client has only an image data that is drawn upon the existing background. The image data can be transformed into image if we put the image data into the delegated image offscreen canvas and then transforms it to image with help of this API

         if (imageOffscreenCanvas instanceof HTMLCanvasElement)
         {
            img = new Image();
            img.src = imageOffscreenCanvas.toDataURL("image/png");
         }
         else
         {
            // OffscreenCanvas : get ImageBitmap
            img = imageOffscreenCanvas.transferToImageBitmap();
         }

I suspect that this approach can be slower than if the client gets an original image of proper quality. The other side is that a bitmap image can lost its quality when it is scaled. If you would like to test this approach, then I will implement this way first.

Where would "an original image of proper quality" come from? Even if we provided high res bitmaps for all of our existing resources, the customer's application would still have many of their own low res images. Anytime a low res image is used, we will need to scale.

What was your plan as an alternative?

Now debugging this way to scale the given image on the javascript client, but as an alternative solution we can scale on the java client side within VirtualScreenImpl if the client has high resolution screen.

Committed revision 11338 (4135a) has dirty changes that show the kind of changes that should be done. If we uncommented scaling factor for getScaledImageDataFromUint8ClampedArray of p2j.screen.js module and scaling factor for drawImage of p2j.canvas_renderer.js the images became scaled properly but some artifacts could be observed. Thus the image data regions were drawn incorrect. It seems that extra data were drawn on to the canvas.
This approach still needs to draw one pixel width line correctly. Please evaluate these changes. I need to take a short time-out to think more about this issue.

In this article another method to scale images is described. https://phoboslab.org/log/2012/09/drawing-pixels-is-hard

I checked a similar approach in which the offscreen canvas remains unscaled and when it is drawn onto the screen canvas it is zoomed while the screen canvas is adapted to the same css dimension as the dimension of the offscreen canvas. We can't follow this way because all drawings and text are antialiased and don't have sharp outlines. Thus there is only one way to draw on the scaled offscreen canvas and then to copy it on to the screen canvas that has the same logical dimension as the offscreen canvas, but its css dimension equals to the dimension of the unscaled offscreen canvas. Following this approach we will get sharp outlines but it can be observed white drawing artifacts that corresponds to some clipping regions and this regions have incorrect background colors. At this moment I don't know how to resolve this issue. It can be that it is related to half pixel conception that a logical point having fractional coordinates.

I came to the idea that x and y coordinates values of a pixel should be whole numbers. If these values are not integers, then some approximations are probably applied and we can't rely on our calculations. If I understand correctly we developed sub pixel conception for these half integers, but I don't know any articles or documentations that support this conception.

Committed revision 11342 (4135a) fixed the task bar resize. Please look at demo_widgets.p with this revision in responsive design mode for Laptop with HDPI screen (DPI>=2)

If I understand correctly we developed sub pixel conception for these half integers

No. The reason for the shift of a half pixel in each direction is to eliminate problems with anti-aliasing. Please see #1811-744 for the details. You can try the implementation without the shift and check the results. If I recall we implement the transform in just a single place, early in the creation/init of the canvas, so it should be easy to check.

Following this approach we will get sharp outlines but it can be observed white drawing artifacts that corresponds to some clipping regions and this regions have incorrect background colors.

I suspect that you must have the same .5 shift transform implemented for the offscreen canvas so that the results will match.

Please look at demo_widgets.p with this revision in responsive design mode for Laptop with HDPI screen (DPI>=2)

Can you please post the screenshot here?

Committed revision 11343 (4135a) tried to fix the drawing issue from #4135-25. The root cause seems to be related to the fact that two neighbour lines become separated by a new line on the scaled canvas so clipping regions can have gaps that produce such drawing artifacts. I tried to fix this issue by extending line width but it seems that the most correct solution is to draw the same drawing by scaling its dimension.

The correct way to draw on the high resolution screen is to draw scaled drawings on the java side.
The java script client can't fix all possible artifacts that can be observed after scaling due to scaling produces gaps that should be filled but it can be done only on the java side for each widget class separately. If we have a bitmap image of low dimensions, then we can't zoom it preserving the image quality unless the bitmap can be represented as a sequence of vector drawings. Unfortunately, we have bitmap drawings. Zooming produces gaps that must be filled. Two parallel lines of one pixel width become separated by a line of undefined color.

Can you post an image with e.g. Hotel GUI on a normal screen and on a HDPI screen?

Related to your current scaling approach: the issue in #4135-30, is it related to drawing 4GL images, or actual FWD primitives (line strokes, text, etc)?

Constantin Asofiei wrote:

Related to your current scaling approach: the issue in #4135-30, is it related to drawing 4GL images, or actual FWD primitives (line strokes, text, etc)?

Yes, it is related to actual FWD primitives because they are drawn on the java side by using knowledge about screen resolution. Some widget drawings take into account pixel widths or heights. The matter of this issue can be observed for the simple example of two lines but you can use another example of two neighbour areas to observe that this way doesn't work or to be more precise can't work properly.

Can you post an image with e.g. Hotel GUI on a normal screen and on a HDPI screen?

Yes, I will post these images later.

Which one is with normal DPI? Also, please rebase 4135a when you can.

Planning to rebase now. The attached png files have comments so that DPR2 in their names means device pixel ratio 2 and it represents a screen shot for a high resolution screen. I used Firefox responsive design mode for testing. Two of them were produced by pressing Print Screen key and the last one was produced by Firefox itself by pressing on "Take a Screen Shots of the viewport" button.

HotelDemoDPR2PrintScreen.png - Print Screen key - high resolution screen (295 KB)

HotelDemo.png - Print Screen key - normal resolution (339 KB)

HotelDemoDPR2.png - Take a Screen Shots of the viewport (830 KB) - high resolution screen

4135a was updated up to rev. 11344 over trunc rev 11338.

From the customer:

The file HotelDemoDPR2.png is looking good. Text looks sharp.

As mentioned earlier in this threat the icons/images are not sharp, because they are scaled up. This the way to solve this in a normal web environment is to make a 200% version of an image and add @2x to the file name. E.g. icon.jpg (could be 32x32 px) and icon@2x.jpg (would be 64x64 px in this example). The HDPi browser will take the second version. We would need to generate a @2x image for all the used icons and images to make this work.

Created 4135b, moving effective changes of 4135a over 4135b.

Is this ready for review?

Greg Shah wrote:

Is this ready for review?

No, it is not ready. I am just applying the changes manually. From the other side there were issues in 4135a but I did not recall them due to this thread is 4 years old.

Moved changes over the current versions. Please review revision 15792 (4135b). I am not sure if the goal is reached or not but the review can start.

Guessed what happened from rev 4135a to rev 4135b. Now images are drawn with help of drawImage that takes into account transform matrix and scaling but 4135a uses putImageData. It needs to rewrite the code correctly in order to use scaling transform for images.

Hynek Cihlar wrote:

Code review 4135b.

How is the scale factor of 2 determined for high resolution screens? Why don't you use window.devicePixelRatio?

You can read references in #4135-5 and #4135-12 but window.devicePixelRatio can be changed by zooming. Please investigate the corresponding MDN page devicePixelRatio.

In DefaultPathRenderer.strokeLine shouldn't the the width and height of the line be also scaled?

How about strokePoint, DotsPathRenderer.strokeLine, should these be scaled, too?

They are attached from p2j.canvas_renderer.js, where methods that update pixels manually should be scaled. Although I can miss something but at this moment I did not find regressions. The large customer application looks properly but please help here I need that some another team member tests this branch with the customer application too.

There are missing file history entries.

Yes, I will add them.

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Code review 4135b.

How is the scale factor of 2 determined for high resolution screens? Why don't you use window.devicePixelRatio?

You can read references in #4135-5 and #4135-12 but window.devicePixelRatio can be changed by zooming. Please investigate the corresponding MDN page devicePixelRatio.

Shouldn't we use Math.ceil(devicePixelRatio) to also cover the screens with higher ratios?

In DefaultPathRenderer.strokeLine shouldn't the the width and height of the line be also scaled?

How about strokePoint, DotsPathRenderer.strokeLine, should these be scaled, too?

They are attached from p2j.canvas_renderer.js, where methods that update pixels manually should be scaled. Although I can miss something but at this moment I did not find regressions. The large customer application looks properly but please help here I need that some another team member tests this branch with the customer application too.

Yes, I will try to help.

Eugenie, you worked on the XOR line mode, could you please test that?

I think we still need to use media query because we should detect screen density. devicePixelRatio increases with screen zoom factor but we should not react on zooming.

Please wait until I rebased this branch. These changes do not support webgl renderer so I need to clean code taking into account that renderer can be switched.

Sergey Ivanovskiy wrote:

I think we still need to use media query because we should detect screen density. devicePixelRatio increases with screen zoom factor but we should not react on zooming.

Actually I think we should also cover zooming for this issue. Without canvas scaling the renderings are very blurry even on normal screens when the zoom level is increased. Proper scaling will fix this, too.

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

I think we still need to use media query because we should detect screen density. devicePixelRatio increases with screen zoom factor but we should not react on zooming.

Actually I think we should also cover zooming for this issue. Without canvas scaling the renderings are very blurry even on normal screens when the zoom level is increased. Proper scaling will fix this, too.

I did not think about zooming. Although MDN supposes to use the same method to scale to css units. Should the zoom level be bounded by the upper value? Can we support the large amount of pixels?

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

I think we still need to use media query because we should detect screen density. devicePixelRatio increases with screen zoom factor but we should not react on zooming.

Actually I think we should also cover zooming for this issue. Without canvas scaling the renderings are very blurry even on normal screens when the zoom level is increased. Proper scaling will fix this, too.

I did not think about zooming. Although MDN supposes to use the same method to scale to css units. Should the zoom level be bounded by the upper value? Can we support the large amount of pixels?

I only tried a simple JS fiddle but the result was good. Try to apply the scaling factor when zoomed in FWD and check the result. I used the following formula I found somewhere on the net:

const zoom = window.innerWidth / document.documentElement.clientWidth;
const scale = window.devicePixelRatio * zoom;

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

I think we still need to use media query because we should detect screen density. devicePixelRatio increases with screen zoom factor but we should not react on zooming.

Actually I think we should also cover zooming for this issue. Without canvas scaling the renderings are very blurry even on normal screens when the zoom level is increased. Proper scaling will fix this, too.

I did not think about zooming. Although MDN supposes to use the same method to scale to css units. Should the zoom level be bounded by the upper value? Can we support the large amount of pixels?

I only tried a simple JS fiddle but the result was good. Try to apply the scaling factor when zoomed in FWD and check the result. I used the following formula I found somewhere on the net:
[...]

Could we do this in another task? I supposed that this task is about "high dpi screens display in low dpi" but you proposed to make changes on zoom operations. I do not understand why this should use multiplication.
https://stackoverflow.com/questions/995914/catch-browsers-zoom-event-in-javascript
https://gist.github.com/abilogos/66aba96bb0fb27ab3ed4a13245817d1e
The author of the last document proposed

//for zoom detection
px_ratio = window.devicePixelRatio || window.screen.availWidth / document.documentElement.clientWidth;

$(window).resize(function(){isZooming();});

function isZooming(){
    var newPx_ratio = window.devicePixelRatio || window.screen.availWidth / document.documentElement.clientWidth;
    if(newPx_ratio != px_ratio){
        px_ratio = newPx_ratio;
        console.log("zooming");
        return true;
    }else{
        console.log("just resizing");
        return false;
    }
}

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

I only tried a simple JS fiddle but the result was good. Try to apply the scaling factor when zoomed in FWD and check the result. I used the following formula I found somewhere on the net:
[...]

Could we do this in another task? I supposed that this task is about "high dpi screens display in low dpi" but you proposed to make changes on zoom operations.

Both the use cases share the same solution, the only difference with the zoom factor is to come up with the correct formula to calculate the resulting scale. If we address zooming together with high dpi screens then we will save a lot of effort on testing and stabilizing the implementation.

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

I only tried a simple JS fiddle but the result was good. Try to apply the scaling factor when zoomed in FWD and check the result. I used the following formula I found somewhere on the net:
[...]

Could we do this in another task? I supposed that this task is about "high dpi screens display in low dpi" but you proposed to make changes on zoom operations.

Both the use cases share the same solution, the only difference with the zoom factor is to come up with the correct formula to calculate the resulting scale. If we address zooming together with high dpi screens then we will save a lot of effort on testing and stabilizing the implementation.

We have methods that operates over pixels so scaling increases the client memory usages and makes it draw slower because of scaling factor. If you have powerful hardware, then the tests results can be accepted but the performance should degrade because the number of pixels increases in scaleFactor*scaleFactor times. So scaleFactor should be reasonably bounded and discrete. It can be no more than 1, 2 or 4. Why scaleFactor should have continuous range?

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

I only tried a simple JS fiddle but the result was good. Try to apply the scaling factor when zoomed in FWD and check the result. I used the following formula I found somewhere on the net:
[...]

Could we do this in another task? I supposed that this task is about "high dpi screens display in low dpi" but you proposed to make changes on zoom operations.

Both the use cases share the same solution, the only difference with the zoom factor is to come up with the correct formula to calculate the resulting scale. If we address zooming together with high dpi screens then we will save a lot of effort on testing and stabilizing the implementation.

We have methods that operates over pixels so scaling increases the client memory usages and makes it draw slower because of scaling factor. If you have powerful hardware, then the tests results can be accepted but the performance should degrade because the number of pixels increases in scaleFactor*scaleFactor times. So scaleFactor should be reasonably bounded and discrete. It can be no more than 1, 2 or 4. Why scaleFactor should have continuous range?

Agree, scaleFactor should be a discrete number. Performance is a good point. We can make the scaling configurable, allow/disallow based on configuration (default in directory), overridable by the end-user.

Hynek, did you test switching renderer with the trunk recently? It seems when switching from 2d to webgl the window is moved to upper left corner and its title becomes unresponsive.

Sergey Ivanovskiy wrote:

Hynek, did you test switching renderer with the trunk recently? It seems when switching from 2d to webgl the window is moved to upper left corner and its title becomes unresponsive.

No, I didn't. Any error in the JS console?

Hynek, could you reproduce the issue in #4135-69 with the trunk version?

First I have the trunk issue #4135-69 when switching between 2d and webgl renderers and the second issue is to find a way to repaint windows if all windows are scaled on zooming. If the web client has loaded with zoom, then it is not required and windows are scaled from the web client loading.

I have not succeeded yet in attempts to apply scaling for zooming windows. It seems that one of the root cause is that resize window event does not happened when zooming. desktop resized event has been sent to the java web client but for scaling we need to replay all drawings.

I tried two ways to create new renderer and to reset the existing renderer. In both cases I encountered issues related to that window drawings are not replayed.

Sergey Ivanovskiy wrote:

I have not succeeded yet in attempts to apply scaling for zooming windows. It seems that one of the root cause is that resize window event does not happened when zooming. desktop resized event has been sent to the java web client but for scaling we need to replay all drawings.

I tried a simple jsfiddle and resize event does come when zoom is changed by the user. If you do it in the content frame try in the root document.

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

I have not succeeded yet in attempts to apply scaling for zooming windows. It seems that one of the root cause is that resize window event does not happened when zooming. desktop resized event has been sent to the java web client but for scaling we need to replay all drawings.

I tried a simple jsfiddle and resize event does come when zoom is changed by the user. If you do it in the content frame try in the root document.

You incorrectly interpreted my issues. The browser resize events happens correctly but the application should have the methods that can redraw the current application state on the new screen.

Sergey Ivanovskiy wrote:

I tried a simple jsfiddle and resize event does come when zoom is changed by the user. If you do it in the content frame try in the root document.

You incorrectly interpreted my issues. The browser resize events happens correctly but the application should have the methods that can redraw the current application state on the new screen.

Are you saying we register the screen resize, send it to Java client, but we should also invalidate all the visible top-level windows to repaint them?

Yes, please look at the video attached in #4135-74. We cannot use cached drawings for new high density screen. It follows that each window should be repainted somehow. But I switched to another task as you know and planning to continue this task later.

Sergey Ivanovskiy wrote:

Yes, please look at the video attached in #4135-74. We cannot use cached drawings for new high density screen. It follows that each window should be repainted somehow. But I switched to another task as you know and planning to continue this task later.

Invalidating all the visible top-level window should be trivial. Extending MSG_DESKTOP_RESIZED with a flag "invalidate windows" and just invalidate in Java client.

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

Yes, please look at the video attached in #4135-74. We cannot use cached drawings for new high density screen. It follows that each window should be repainted somehow. But I switched to another task as you know and planning to continue this task later.

Invalidating all the visible top-level window should be trivial. Extending MSG_DESKTOP_RESIZED with a flag "invalidate windows" and just invalidate in Java client.

Please point on the referenced methods that can invalidate windows?

Sergey Ivanovskiy wrote:

Please point on the referenced methods that can invalidate windows?

Please see GuiWebDriver.desktopResized.

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

Please point on the referenced methods that can invalidate windows?

Please see GuiWebDriver.desktopResized.

      this.taskBarStyle = taskBarStyle;
      config.setConfigItem(ConfigItem.TASKBAR_STYLE, taskBarStyle.name());

      updateDesktopDimensions(width, height);

      boolean forceMax = isEmbeddedClient() && DynamicLayoutConfig.getInstance().isWindowMaximize();
      Window<?> currWin = WindowManager.getCurrentWindow();

      WindowManager.windowList().stream()
         .filter(w ->
            (w instanceof TopLevelWindow &&
             w.isVisible() &&
             ((TopLevelWindow<?>) w).isMaximized() && !w.isModal()) ||
            (forceMax && currWin == w))
         .forEach(w ->
         {
            WindowConfig wcfg = w.config();
            wcfg.maxWidthPixels = Math.max(width, wcfg.maxWidthPixels);
            wcfg.maxHeightPixels = Math.max(height, wcfg.maxHeightPixels);

            int wid = w.getId().asInt();
            setWindowLocation(wid, 0, 0);
            setWindowDimension(wid, width, height);
         });

Sergey Ivanovskiy wrote:

Found only the code that resized maximized windows but I need to repaint all of them.

The invalidation logic must be added. But it will be simple, take all the visible windows (that were not previously resized) and call repaint on them in a drawing bracket.

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

Found only the code that resized maximized windows but I need to repaint all of them.

The invalidation logic must be added. But it will be simple, take all the visible windows (that were not previously resized) and call repaint on them in a drawing bracket.

Thank you. I asked to be sure that there are no special invalidation methods already implemented.

Committed revision 15820 (4135b) added zooming. Hynek, please review these changes. I did not succeed in fixing 2d->webgl renderer switching that also does not work in the trunk version. There are another issues that we can discuss. One of them is images resolution. The visual presentations of scaled existing binary images and icons are not improved but the text presentation can be improved.

Eugenie, could you post references to standalone testcases that can help to test direct manipulation functionality?

Sergey Ivanovskiy wrote:

Eugenie, could you post references to standalone testcases that can help to test direct manipulation functionality?

I have updated new testcases project to revision 1732. Please see drag-n-drop subdirectory. With testcases you can see the movable/sizeable/selectable widgets testcases. They are pretty simple so you can adjust them to your local needs.

Eugenie Lyzenko wrote:

Sergey Ivanovskiy wrote:

Eugenie, could you post references to standalone testcases that can help to test direct manipulation functionality?

I have updated new testcases project to revision 1732. Please see drag-n-drop subdirectory. With testcases you can see the movable/sizeable/selectable widgets testcases. They are pretty simple so you can adjust them to your local needs.

Thank you, I will test them with 4135b (rev. 15821).

Found NUM-SELECTED-WIDGETS regressions of drag-n-drop testcases for the trunk version.
For example, ./drag_n_drop_new/dnd_test0_0.p - the number of selected widgets is incorrectly increased. If a user clicks the frame and the button "Num Selected?", then the reported value of FRAME a:NUM-SELECTED-WIDGETS is increased.

4135b was rebased up to rev 15848. I tested drag-n-drop testcases with 4135b and did not find new regressions.

Code review 4135b.

ChuiWebSimulator.java, ClientProtocolHooks.java, GuiWebDriver.java, GuiWebSocket.java, p2j.socket.js are missing history entries.

The ScreenManager constructor has a parameter mismatch in JSDoc: it lists initPixelRatio twice, but the second one is never used in the function signature.

For isHighResolution function, does it make sense to use window.devicePixelRatio if available? This check would be a lot more efficient than the media query.

repaintVisibleWindows should be moved in the private section.

Instead of ThinClient.getInstance().eventDrawingBracket(w, w::draw); please do ThinClient.getInstance().eventDrawingBracket(w, w::repaint);. Invalidating the widgets prevents potential double drawing, first when windows resized and second when w:draw.

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Code review 4135b.

Instead of ThinClient.getInstance().eventDrawingBracket(w, w::draw); please do ThinClient.getInstance().eventDrawingBracket(w, w::repaint);. Invalidating the widgets prevents potential double drawing, first when windows resized and second when w:draw.

This method is called only when zoom is happened but not on resize. We need to redraw all GUI elements on zoom.

What about maximized windows? Won't they resize when the virtual desktop size changes due to zoom level change?

On zoom event all windows preserve their dimensions.

Sergey Ivanovskiy wrote:

On zoom event all windows preserve their dimensions.

See the following test case:

message "Press i".
// maximize the window
default-window:window-state = 1.
on 'i' anywhere do:
message default-window:width.
end.
wait-for close of this-procedure.

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

On zoom event all windows preserve their dimensions.

See the following test case:
[...]

I think that maximize window is buggy now.

Sergey Ivanovskiy wrote:

I think that maximize window is buggy now.

What is the issue?

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

I think that maximize window is buggy now.

What is the issue?

There is an issue that maximized windows occupy only the browser's view because their dimensions based on desktop size calculated as window.innerWidth and window.innerHeight.
But for the current version of 4135b there is an issue related to zooming functionality when the browser's zoom is reset. Not maximized windows are not affected because they do not change their dimensions on zooming.

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

I think that maximize window is buggy now.

What is the issue?

There is an issue that maximized windows occupy only the browser's view because their dimensions based on desktop size calculated as window.innerWidth and window.innerHeight.

Maximized windows should always occupy the browser's view. I.e. when the browser window is resized or scale changed the window should resize to always fit.

Hynek Cihlar wrote:

Maximized windows should always occupy the browser's view. I.e. when the browser window is resized or scale changed the window should resize to always fit.

In the practical experience it means that zooming causes a maximized window to be smaller and its content becomes hidden. We should discuss this requirement.

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Maximized windows should always occupy the browser's view. I.e. when the browser window is resized or scale changed the window should resize to always fit.

In the practical experience it means that zooming causes a maximized window to be smaller and its content becomes hidden. We should discuss this requirement.

This is comparable to OS screen resolution or screen scale change. In that case the windows also resize.

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Maximized windows should always occupy the browser's view. I.e. when the browser window is resized or scale changed the window should resize to always fit.

In the practical experience it means that zooming causes a maximized window to be smaller and its content becomes hidden. We should discuss this requirement.

This is comparable to OS screen resolution or screen scale change. In that case the windows also resize.

The sizes of windows are changed because windows are zoomed. Repeating windows are not resized because they are zoomed. Css dimensions are the same as the logical business 4GL dimensions but canvas logical sizes are scaled. But maximized windows are resized because they fill decreased view space that in my view is not correct and makes issues on reset zooming.

If the browser content is zoomed, then browser adds scroll bars to extend its view size.

So the maximized windows behavior is not the same as the browser zooms its content.

I tested zoom functionality with graphicsCache = FALSE setting but enabling this cache adds new painting issues. Initially this task was created to support "high dpi screens display in low dpi" but Hynek proposed to add zooming support that makes this task changes not working properly because of repainting issues that I did not resolve at this moment.

Sergey Ivanovskiy wrote:

I tested zoom functionality with graphicsCache = FALSE setting but enabling this cache adds new painting issues. Initially this task was created to support "high dpi screens display in low dpi" but Hynek proposed to add zooming support that makes this task changes not working properly because of repainting issues that I did not resolve at this moment.

If this is not trivial to fix, just defer the zoom support.

Didn't the customer already complain about zooming? I worry that we do need to fix this now.

Greg Shah wrote:

Didn't the customer already complain about zooming? I worry that we do need to fix this now.

I'm not aware of a customer complaint about zooming. I complained myself multiple times though, since I use zooming often :-).

Greg, it's your call, if you think it should be fixed now, let's do it.

It is a valid use case that must work properly.

Hynek Cihlar wrote:

Greg Shah wrote:

Didn't the customer already complain about zooming? I worry that we do need to fix this now.

I'm not aware of a customer complaint about zooming. I complained myself multiple times though, since I use zooming often :-).

Greg, it's your call, if you think it should be fixed now, let's do it.

I have another idea. What should be if we enable double scale by the directory settings. In this case zooming will work with canvases that have high density points and the zoomed screens should look better. Finally, we should not write complex code that only intend to fix painting issues. What do you think?

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Greg Shah wrote:

Didn't the customer already complain about zooming? I worry that we do need to fix this now.

I'm not aware of a customer complaint about zooming. I complained myself multiple times though, since I use zooming often :-).

Greg, it's your call, if you think it should be fixed now, let's do it.

I have another idea. What should be if we enable double scale by the directory settings. In this case zooming will work with canvases that have high density points and the zoomed screens should look better. Finally, we should not write complex code that only intend to fix painting issues. What do you think?

Won't there be performance impact if double scale enabled in the directory and no zoom or retina display in use?

My impression is that we only need to detect zooming is active in order to enable double scale. The rest of the code should be the same with scaling for retina displays. In other words the only additional work needed for zooming is the zooming detection, no?

Sergey Ivanovskiy wrote:

If the browser content is zoomed, then browser adds scroll bars to extend its view size.

This happens for unmaximized windows and is expected.

Hynek Cihlar wrote:

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Greg Shah wrote:

Didn't the customer already complain about zooming? I worry that we do need to fix this now.

I'm not aware of a customer complaint about zooming. I complained myself multiple times though, since I use zooming often :-).

Greg, it's your call, if you think it should be fixed now, let's do it.

I have another idea. What should be if we enable double scale by the directory settings. In this case zooming will work with canvases that have high density points and the zoomed screens should look better. Finally, we should not write complex code that only intend to fix painting issues. What do you think?

Won't there be performance impact if double scale enabled in the directory and no zoom or retina display in use?

My impression is that we only need to detect zooming is active in order to enable double scale. The rest of the code should be the same with scaling for retina displays. In other words the only additional work needed for zooming is the zooming detection, no?

No, zooming detection is already present in the code changes. The problems if my view is correct are that in some testcases related to maximize/restore window the old scaled or original image is used to draw original or scaled image and the graphics cache system on the java web client interferes the drawing of scaled canvases. So I proposed to use directory defined scaling. In this case I have no issues.

Sergey Ivanovskiy wrote:

No, zooming detection is already present in the code changes. The problems if my view is correct are that in some testcases related to maximize/restore window the old scaled or original image is used to draw original or scaled image and the graphics cache system on the java web client interferes the drawing of scaled canvases. So I proposed to use directory defined scaling. In this case I have no issues.

Can you just invalidate the whole cache on scale change?

Found at least one bug in my code changes that affects canvas proportion. If scale factor is greater than 1, then the canvas css width and height are set, but if scale factor has been reset, then the canvas css width and height are not reset. This affects the screen view.

I still have issues if graphics cache is on.

Committed revision 15864 (4135b) this version seems working with graphics cache but it needs new fresh eyes. There is an issue when maximize is clicked from the system menu. For the system menu restore button is disabled in all cases.

I tested zooming on the latest branch and the results are pretty impressive. The improvement in display clarity is huge! Btw I tested on a full hd display.

I noticed one issue. When I first login to an app the UI is blurry (even on 100% zoom), but when I change zooming level (even back to 100%) the UI becomes sharp.

Sergey Ivanovskiy wrote:

Committed revision 15864 (4135b) this version seems working with graphics cache but it needs new fresh eyes. There is an issue when maximize is clicked from the system menu. For the system menu restore button is disabled in all cases.

What's the issue? Do you mean the window restore?

Hynek Cihlar wrote:

I noticed one issue. When I first login to an app the UI is blurry (even on 100% zoom), but when I change zooming level (even back to 100%) the UI becomes sharp.

I have no hd display but tried to test with help of browser responsive design mode with DPR = 2. I would change

   function isHighResolution()
   {
      return window.matchMedia &&
               (window.matchMedia(
                     'only screen and (min-resolution: 192dpi), only screen and ' +
                     '(min-resolution: 2dppx), ' +
                     'only screen and (min-resolution: 75.6dpcm)').matches ||
                window.matchMedia(
                     'only screen and (-webkit-min-device-pixel-ratio: 2), only screen and ' +
                     '(-o-min-device-pixel-ratio: 2/1), only screen and ' +
                     '(min--moz-device-pixel-ratio: 2), only screen and ' +
                     '(min-device-pixel-ratio: 2)').matches);
   }

   function isHighResolution()
   {
      return window.devicePixelRatio > 1;
   }

Sergey Ivanovskiy wrote:

Could you please check if the initial screen would be blur for this new isHighResolution(). I can commit this change into 4135b if you do not object.

The change does resolve it for me. Btw the screen looks sharper even on 100% zoom (full hd and qhd displays).

Sergey Ivanovskiy wrote:

I recorded this video to clarify the issue.

Please compare the actions executed by the maximize window icon and the Maximize menu item in the window menu.

Hynek: are you OK with moving #4135-124 to another task?

I'd like to get 4135b into trunk as soon as it is ready. Since #4135-124 also exists in trunk, I think we can split it off.

I tested these changes

=== modified file 'src/com/goldencode/p2j/ui/client/gui/WindowTitlePopupGuiImpl.java'
--- src/com/goldencode/p2j/ui/client/gui/WindowTitlePopupGuiImpl.java    2025-01-06 19:08:03 +0000
+++ src/com/goldencode/p2j/ui/client/gui/WindowTitlePopupGuiImpl.java    2025-04-28 14:54:42 +0000
@@ -212,8 +212,12 @@
                @Override
                public void mouseClicked(MouseEvent e)
                {
-                  window.maximize(true);
-                  menu.restoreOption.setEnabled(true);
+                  ThinClient tc = ThinClient.getInstance();
+                  tc.independentEventDrawingBracket(window, () ->
+                  {
+                     window.maximize(true);
+                     menu.restoreOption.setEnabled(true);
+                  });
                }
             });

Hynek, I think that we need to WindowGuiImpl::draw but not repaint that

   public void repaint(Rectangle ur)
   {
      // TODO: unrealized widgets must never be repainted.  code bellow needs to be enabled only
      // when all widgets which have configs (part of the window decorations or frames) have the
      // realized state activated properly
      if (!isDisplayed() || ur.empty() /* || (config() != null && !isRealized())*/)
         return;

      // do not draw the widget, if it should not be repainted
      if (isRepaintDisabled())
      {
         return;
      }

      Container<O> ct = parent(true);

      // do not readjust coordinates if the root window is reached
      while (ct != null && !(ct instanceof TopLevelWindow))
      {
         if (!ct.isVisible())
         {
            return;
         }

         if (ct.clipping())
         {
            ur = ur.intersection(ct.getClipRect());

            if (ur == null || ur.empty())
            {
               return;
            }
         }

         Point parentLoc = prepParentLocation(ct);

         // insets must be translated the same way they are in 
         // screenLocation/screenPhysicalLocation so that we end up
         // with the correct screen location of the update rect
         if (ct instanceof BorderedPanelGuiImpl)
         {
            parentLoc = parentLoc.translate(ct.insets());
         }

         ur = ur.translate(parentLoc);

         ct = ct.parent(true);
      }

      EventManager.postEvent(new PaintEvent(this, ur));
   }

repaint must be done in an event-drawing bracket, otherwise no draw will be executed.

repaint is needed so that the windows are not drawn twice. Once when repainiting resized windows and then when draw is called direcrly.

I decided to work around #4135-135 because debugging and draw logging do not help me to find out its root cause. This version committed rev. 15897 (4135b) disables the graphics cache while zooming windows are redrawn. It looks more stable.

Hynek Cihlar wrote:

Code review 4135a.

When the page is zoomed, MSG_DESKTOP_RESIZED is sent from browser to Java driver. When the message is dispatched Java driver then sends MSG_ZOOM message to the browser. Can this network call be eliminated? JS driver should have all this info anyway, when it sends MSG_DESKTOP_RESIZED, no?

The idea was to zoom windows with the disabled graphics cache. The java client starts zooming only if its jscript peer sends MSG_DESKTOP_RESIZED with
invalidate windows flag. At this moment the java client disables the graphics cache and sends MSG_ZOOM with windows that require redraw. It gives the jscript client side to enable the graphics cache only after all required windows are redrawn.

MSG_DESKTOP_RESIZED still doesn't resize maximized windows when zooming.

I think you should compare these two variants in the video. I will prepare them now. As we discussed before the latest explanation was in #4135-135, but you did not respond on my notes and repeated the same issue.

Please also test embedded mode.

The Hotel Gui is only known to me application where the embedded mode is used.

Sergey Ivanovskiy wrote:

This video 4135b_let_max_win_zooming.mp4 shows the implemented case when maximized windows are permitted to be zoomed and 4135b_let_max_win_resized_on_zooming.mp4 shows the case when maximized windows are resized on zooming so they dimensions are decreased on zooming.

Thanks for the comparison video.

I think we should keep the behavior of the native platform. When you resize (or change scale/zoom) any Windows desktop, then all the maximized windows will still take the visible desktop area, they won't spill over. Hence we should keep this behavior, too.

From the UX perspective it makes sense. I don't think the user would be glad to resize all the maximized windows to see the whole window (consider message and status areas, for example) when scaling or browser window size changed.

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

When the page is zoomed, MSG_DESKTOP_RESIZED is sent from browser to Java driver. When the message is dispatched Java driver then sends MSG_ZOOM message to the browser. Can this network call be eliminated? JS driver should have all this info anyway, when it sends MSG_DESKTOP_RESIZED, no?

OK, this is not critical. We can improve this in the future if needed. But please document the need for sending MSG_ZOOM in the code.

Hotel_Gui embedded application is not compiled because of this issue

BUILD FAILED
/home/sbi/projects/hotel_gui/embedded/build.xml:234: src '/home/sbi/projects/hotel_gui/p2j/build/lib/bootstrap-3.3.7-dist.zip' doesn't exist.

Adding some old zip archives gives

prepare:
    [unzip] Expanding: /home/sbi/projects/hotel_gui/p2j/build/lib/bootstrap-3.3.7-dist.zip into /home/sbi/projects/hotel_gui/embedded/run/embedded
    [unzip] Expanding: /home/sbi/projects/hotel_gui/p2j/build/lib/d3-4.9.1.zip into /home/sbi/projects/hotel_gui/embedded/run/embedded
    [unzip] Expanding: /home/sbi/projects/hotel_gui/p2j/build/lib/dojo-1.10.0-distribution.zip into /home/sbi/projects/hotel_gui/embedded/run/embedded
    [unzip] Expanding: /home/sbi/projects/hotel_gui/p2j/build/lib/font-bundle-v1.zip into /home/sbi/projects/hotel_gui/embedded/run/embedded
    [unzip] Expanding: /home/sbi/projects/hotel_gui/p2j/build/lib/jquery-3.2.1.zip into /home/sbi/projects/hotel_gui/embedded/run/embedded
    [unzip] Expanding: /home/sbi/projects/hotel_gui/p2j/build/lib/jquery-ui-1.12.1.custom.zip into /home/sbi/projects/hotel_gui/embedded/run/embedded
    [unzip] Expanding: /home/sbi/projects/hotel_gui/p2j/build/lib/tabulator-master-2.12.0.zip into /home/sbi/projects/hotel_gui/embedded/run/embedded

compile:
    [javac] /home/sbi/projects/hotel_gui/embedded/build.xml:271: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
    [javac] Compiling 5 source files to /home/sbi/projects/hotel_gui/embedded/build/classes
    [javac] /home/sbi/projects/hotel_gui/embedded/src/com/goldencode/testcases/embedded/CustomP2JClientApp.java:24: error: cannot find symbol
    [javac]    private WebClientLauncher launcher = null;
    [javac]            ^
    [javac]   symbol:   class WebClientLauncher
    [javac]   location: class CustomP2JClientApp
    [javac] /home/sbi/projects/hotel_gui/embedded/src/com/goldencode/testcases/embedded/EmbeddedWebAppDriver.java:4: error: cannot find symbol
    [javac] import com.goldencode.p2j.main.WebHandler;
    [javac]                               ^
    [javac]   symbol:   class WebHandler
    [javac]   location: package com.goldencode.p2j.main
    [javac] /home/sbi/projects/hotel_gui/embedded/src/com/goldencode/testcases/embedded/CustomP2JClientApp.java:155: error: cannot find symbol
    [javac]             launcher = (WebClientLauncher)
    [javac]                         ^
    [javac]   symbol:   class WebClientLauncher
    [javac]   location: class CustomP2JClientApp
    [javac] /home/sbi/projects/hotel_gui/embedded/src/com/goldencode/testcases/embedded/CustomP2JClientApp.java:156: error: cannot find symbol
    [javac]                RemoteObject.obtainNetworkInstance(WebClientLauncher.class, session);
    [javac]                                                   ^
    [javac]   symbol:   class WebClientLauncher
    [javac]   location: class CustomP2JClientApp
    [javac] /home/sbi/projects/hotel_gui/embedded/src/com/goldencode/testcases/embedded/EmbeddedWebAppDriver.java:367: error: cannot find symbol
    [javac]                                     WebHandler.getRequestParameters(request));
    [javac]                                     ^
    [javac]   symbol:   variable WebHandler
    [javac]   location: class EmbeddedPageHandler
    [javac] /home/sbi/projects/hotel_gui/embedded/src/com/goldencode/testcases/embedded/EmbeddedWebAppDriver.java:369: error: cannot find symbol
    [javac]             if (url == null || url.startsWith(WebHandler.ERROR_PREFIX))
    [javac]                                               ^
    [javac]   symbol:   variable WebHandler
    [javac]   location: class EmbeddedPageHandler
    [javac] 6 errors

BUILD FAILED
/home/sbi/projects/hotel_gui/embedded/build.xml:271: Compile failed; see the compiler error output for details.

Hynek, I tested Hotel Gui and Metacom applications except embedded mode. The current version rev 15932 should fix all review issues.

Sergey, please create a new issue for the broken Hotel GUI embedded.

Added new issue for the embedded Hotel Gui application #10045.

Hynek Cihlar wrote:

Is there any remaining work expected? If not we can schedule this for merge.

No, it seems that this version can be merged.

Hynek Cihlar wrote:

Please merge 4135b to trunk.

Let me recheck Hotel GUI and Metacom. I cannot access the web application after the rebase. The web client is crashed. If they are OK, then I will merge this version into the trunk.

Sergey Ivanovskiy wrote:

Hynek Cihlar wrote:

Please merge 4135b to trunk.

Let me recheck Hotel GUI and Metacom. I cannot access the web application after the rebase. The web client is crashed. If they are OK, then I will merge this version into the trunk.

The trunk 15928 has regression that breaks the Web client and other spawned processes starting. We need to resolve this or undo ASAP.

Eugenie, thank you, but I did not find that this issue is reported in #7143?

Sergey Ivanovskiy wrote:

Eugenie, thank you, but I did not find that this issue is reported in #7143?

Yes, I reported this in #10000 instead. It is the fix origin and I hope this is the fastest way to deliver this message.

Let me recheck Hotel GUI and Metacom. I cannot access the web application after the rebase. The web client is crashed. If they are OK, then I will merge this version into the trunk.

I will merge now the fix for it.

Branch 10000e was merged to trunk rev 15933 and archived. With it, Hotel GUI should now work correctly. Please rebase.

Alexandru Lungu wrote:

Sergey, I am getting this off the merge queue to merge pending branches. Let me know when you are ready to merge 4135b.

I should test with the embedded Hotel Gui application, then this branch can be merged.

I tested with the embedded Hotel Gui application so this branch can be merged into the trunk.

4135b was merged into trunk as rev. 15940 and archived.

Sergey Ivanovskiy wrote:

Hynek, 4135b broke p2j.perf_tests.js tests. I did not check them at the moment. This patch could fix them.

Good catch. Please create new branch for the changes.

Hynek, could this branch 4135c with minor changes be merged into the trunk? The current rev 4135c is 16018.

Sergey Ivanovskiy wrote:

Hynek, could this branch 4135c with minor changes be merged into the trunk? The current rev 4135c is 16018.

All regression testing is done?

Yes, the changes are done only for p2j.perf_tests.js. So I tested Hotel Gui in the virtual desktop mode.