FWD » Core Development » User Interface

I have some Sikuli scripts to check into the hotel_gui project. However, I am unsure of the location in the directory structure. I was going to place all the directories (there is a directory per sikuli testcase) in a new "testing" directory in the project. Will that suffice?

For now:

------------------------------------------------------------
revno: 255
committer: Roger F Borrello <rfb@goldencode.com>
branch nick: hotel_gui
timestamp: Tue 2022-11-08 17:45:23 -0500
message:
  Added the testing directory which contains Sikuli scripts for:
  - hotel_gui_os_logon.sikuli : Perform OS logon
  - hotel_gui_os_hotel_logon.sikuli : Perform OS and hotel application logon
  - hotel_gui_hotel_logon.sikuli : Peform hotel application logon
  - hotel_gui_hotel_logoff.sikuli : Perform hotel application logoff
  - hotel_gui_navigate_main_tabs.sikuli : Click through main tabs
  - hotel_gui_search_for_room.sikuli : Perform a search for a Double - Sea View room 2 weeks out for 3 nights

  You will need to add the sikulixide-2.0.5.jar to your configuration and run: java -jar <path_to_sikulixide-2.0.5.jar>
added:
  testing/
  testing/hotel_gui_hotel_logoff.sikuli/
  testing/hotel_gui_hotel_logoff.sikuli/chrome_hotel_login_screen.png
  testing/hotel_gui_hotel_logoff.sikuli/chrome_hotel_main_menu.png
  testing/hotel_gui_hotel_logoff.sikuli/chrome_hotel_main_screen_controls.png
  testing/hotel_gui_hotel_logoff.sikuli/hotel_gui_hotel_logoff.py
  testing/hotel_gui_hotel_logon.sikuli/
  testing/hotel_gui_hotel_logon.sikuli/chrome_hotel_login_screen.png
  testing/hotel_gui_hotel_logon.sikuli/chrome_hotel_main_menu.png
  testing/hotel_gui_hotel_logon.sikuli/hotel_gui_hotel_logon.py
  testing/hotel_gui_navigate_main_tabs.sikuli/
  testing/hotel_gui_navigate_main_tabs.sikuli/chrome_hotel_available_rooms_tab.png
  testing/hotel_gui_navigate_main_tabs.sikuli/chrome_hotel_guests_tab.png
  testing/hotel_gui_navigate_main_tabs.sikuli/chrome_hotel_main_menu.png
  testing/hotel_gui_navigate_main_tabs.sikuli/chrome_hotel_main_screen_controls.png
  testing/hotel_gui_navigate_main_tabs.sikuli/chrome_hotel_rates_tab.png
  testing/hotel_gui_navigate_main_tabs.sikuli/chrome_hotel_reservations_tab.png
  testing/hotel_gui_navigate_main_tabs.sikuli/chrome_hotel_rooms_tab.png
  testing/hotel_gui_navigate_main_tabs.sikuli/hotel_gui_navigate_main_tabs.py
  testing/hotel_gui_os_hotel_logon.sikuli/
  testing/hotel_gui_os_hotel_logon.sikuli/hotel_gui_os_hotel_logon.py
  testing/hotel_gui_os_logon.sikuli/
  testing/hotel_gui_os_logon.sikuli/chrome_hotel_login_screen.png
  testing/hotel_gui_os_logon.sikuli/chrome_logon_info.png
  testing/hotel_gui_os_logon.sikuli/chrome_maximized.png
  testing/hotel_gui_os_logon.sikuli/chrome_not_private_connection.png
  testing/hotel_gui_os_logon.sikuli/chrome_not_private_connection_advanced.png
  testing/hotel_gui_os_logon.sikuli/hotel_gui_os_logon.py
  testing/hotel_gui_search_for_room.sikuli/
  testing/hotel_gui_search_for_room.sikuli/chrome_available_room_type_double_seaview.png
  testing/hotel_gui_search_for_room.sikuli/chrome_available_rooms_room_type_selection.png
  testing/hotel_gui_search_for_room.sikuli/chrome_hotel_available_rooms_tab.png
  testing/hotel_gui_search_for_room.sikuli/chrome_hotel_find_available_rooms_button.png
  testing/hotel_gui_search_for_room.sikuli/chrome_hotel_main_screen_controls.png
  testing/hotel_gui_search_for_room.sikuli/hotel_gui_search_for_room.py

Yes, that is fine.

Please post the previous discussion about calling Java from the Sikuli python scripts.

Would I just need to add harness.jar to the classpath, or all the jars in the harness's build/lib directory?

Considering that the manifest/harness.mf is intended to have all the dependencies defined, one would expect to only have to add the harness.jar and the other dependencies will be automatically found (assuming they are in place).

So much of the harness is driven by XML... is there a quick start on calling something like text-file-comparison directly?

You can look at com.goldencode.harness.TestFactory.processStepDefinition() to see how we load each specific test directive. You will find that com.goldencode.harness.test.TextFileComparison is used to implement this directive. A quick look at TextFileComparison will be quite clear to see how it is used.

I have the testcase to export guests. I am filtering the list by room, comparing the screen, then filtering by guest ("Smith") which yields 5 entries. Then I export as XLS.

As my baseline I saved off the first export, which had 6 pages. Then I added the comparison tool (Python filecmp.cmp), and the first time I tried to compare the xls that was exported had 16 pages... There may be a bug, because as I exported by hand, the number of pages just grows. 21 pages, then 25...

Greg Shah wrote:

You can look at com.goldencode.harness.TestFactory.processStepDefinition() to see how we load each specific test directive. You will find that com.goldencode.harness.test.TextFileComparison is used to implement this directive. A quick look at TextFileComparison will be quite clear to see how it is used.

In the first example I am trying to code, I am comparing the XLS exported to the ~/Downloads directory. I have code to determine the latest file saved to that directory, so I have a valid file accessible from the Sikuli script:

# This function will find the latest file (of the given blob and path) to the given file
def compareLatestFile(file_blob, baseline_file):
    # The max file is the most recently created
    print("Blobbing: " + file_blob)
    print("Baseline file:" + baseline_file)
    testcase_file = max(glob.glob(file_blob), key=os.path.getctime)
    print("Testcase file:" + testcase_file)
    # filecmp.cmp will return True for matching files. Use Deep comparison to match more than size and timestamp
    return filecmp.cmp(baseline_file, testcase_file, shallow=False)

# Baseline files
baseline_file_path = getBundlePath() + path_sep
export_by_guest_baseline=baseline_file_path+"guests_export_smith_baseline.xls" 
...
# Generate the export
click(Pattern("chrome_hotel_guests_export_area_xls.png").targetOffset(10,0))
# Find xls file that was generated in the downloads directory (platform specific, assumes Chrome)
xls_files = downloads_dir + path_sep + '*.xls'
time.sleep(10)
assert(compareLatestFile(xls_files, export_by_guest_baseline))

That currently works, but I would like to replace it with com.goldencode.harness.test.BinaryFileComparison.

Step 1 was to gain access to the classes in harness.jar. To do this, I updated the Sikuli extensions file in the home directory to contain the absolute path to the harness.jar:

rfb@rfb:~/projects/hotel_gui/testing$ cat ~/.Sikulix/Extensions/extensions.txt 
# add absolute paths one per line, that point to other jars,
# that need to be available on Java's classpath at runtime
# They will be added automatically at startup in the given sequence
/home/rfb/projects/harness/build/lib/harness.jar

# empty lines and lines beginning with # or // are ignored
# delete the leading # to activate a prepared keyword line

# pointer to a Jython install outside SikuliX
# jython = c:/jython2.7.1/jython.jar

# pointer to a JRuby install outside SikuliX
# jruby = c:/jruby/jruby.jar

On Windows, this is the %APPDATA%\Sikulix\Extensions\extensions.txt file

This allowed the script to be updated with the import line:

from com.goldencode.harness.test import BinaryFileComparison

Prior to updating the extension file, I was receiving an error when trying to run the import:

[error] script [ hotel_gui_export_guest_list ] stopped with error in line 12
[error] ImportError ( No module named goldencode )
[error] --- Traceback --- error source first
line: module ( function ) statement 
12: main (  <module> )     from com.goldencode.harness.test import BinaryFileComparison
[error] --- Traceback --- end --------------

Use of the IDE results in some additional configuration items. How this affects the use of the IDE to create a runnable jar is TBD.

The next step is to understand how to invoke BinaryFileComparison:

   /**
    * Create an instance.
    *
    * @param    baseline
    *           Identifies the file containing the expected data.
    * @param    actual
    *           Identifies the file containing the actual data generated during
    *           the test.
    * @param    downloadDir 
    *           The local directory in which a downloaded file will be placed
    *           or <code>null</code> if no downloading should occur.
    * @param    sftp
    *           Helper for downloading the comparison file via SFTP. The
    *           <code>actual</code> parameter must specify the absolute or
    *           relative filename of the file on the server system. The file
    *           will be downloaded into the download directory as specified
    *           in the project index.
    * @param    excludes
    *           A list of portions of the byte stream to be ignored for
    *           comparison purposes or <code>null</code> if there are no
    *           exclusion regions.
    */
   public BinaryFileComparison(String            baseline,
                               String            actual,
                               String            downloadDir,
                               SFTPSession       sftp,
                               ExclusionRegion[] excludes)

The first thing I'll try is the below, and add getLatestFile to return the latest file of the given blob:

BinaryFileComparison(getLatestFile(xls_files),export_by_guest_baseline,null,null,null)

Roger Borrello wrote:

The first thing I'll try is the below, and add getLatestFile to return the latest file of the given blob:
[...]

That resulted in an error, since "null" wasn't defined. This did not get an error:

BinaryFileComparison(getLatestFile(xls_files),export_by_guest_baseline,None,None,None)

• Waiting for image to appear for n seconds, n defaults to 3.
• Waiting for more precise image to appear (default is 70% hit rate, we can go up to 99%). Sometimes it is to our benefit that a screen with some differences will match, especially when there are minor data differences that don't affect the testcase.
• Clicking on an image (default is 0,0, which is center of the image)
• Clicking on an (x,y) offset of an image:
  • x moves you right/left and y moves you up/down
  • x/y > 0 moves you right/down, x/y < 0 moves you left/up
• Logic for if a pattern exists or not
• Typing in text
• Flow control via calling other scripts
• Conditional logic based upon OS
• User input (visible and hidden)
• Modular organization via import statement to include functions and variables used by multiple scripts

• I need to add some text recognition.
• My use of modularity introduced some scoping oddities where the external functions utilizing normal Sikuli functions needed to have additional qualifications added for them to be found.

It is important to move to a relative path for the harness.jar. I would copy it into the project as part of the setup and reference it there.

In regard to the NPE, changes can be made to the Harness code to refactor it such that SFTP support is not tightly coupled. That would be a better approach overall.

Greg Shah wrote:

It is important to move to a relative path for the harness.jar. I would copy it into the project as part of the setup and reference it there.

It is constrained by the Sikuli IDE's design. There isn't any command line parameter for that, and the usage of the extensions.txt configuration file doesn't allow anything but absolute paths:

      File extFile = new File(extPath);
      if (extFile.isAbsolute()) {
        if (!extFile.exists() || !extFile.getName().endsWith(".jar")) {
          RunTime.startLog(-1, "extension path not valid: %s", line);
          continue;
        }
      } else {
        extFile = new File(sxExtensions, extFile.getPath());
        if (!extFile.exists() || !extFile.getName().endsWith(".jar")) {
          RunTime.startLog(-1, "extension path not valid: %s", line);    // <-- This error is given when using a relative path to harness.jar
          continue;
        }
      }

I tried every which way to get the classpath involved, but it only appears to be used via the extensions.txt. I scoured the LaunchPad for similar questions, but none so far. So I will pose it.

Sikuli is open source and written in Java, right?

Greg Shah wrote:

In regard to the NPE, changes can be made to the Harness code to refactor it such that SFTP support is not tightly coupled. That would be a better approach overall.

There isn't any NPE... I just can't figure out how to get results from the issuance of: BinaryFileComparison(getLatestFile(xls_files),export_by_guest_baseline,null,null,null)

In TestFactory.processStepDefinition(), when it is a binary file command (ELEM_BINARY_FILE_COMPARISON), the code creates a new Testable instance:

         step = new BinaryFileComparison(baseline,
                                         actual,
                                         index.getDownloadDirectory(),
                                         sftp,
                                         gaps);

It's not obvious to me how to perform the execute of that via this type of interface.

Greg Shah wrote:

Sikuli is open source and written in Java, right?

Yes... I was investigating the source code that was available for download (without the build configuration). I'm trying to get the code off LaunchPad now.

I just can't figure out how to get results from the issuance of: BinaryFileComparison(getLatestFile(xls_files),export_by_guest_baseline,null,null,null)

Look at the compare() method.

Roger Borrello wrote:

Yes... I was investigating the source code that was available for download (without the build configuration). I'm trying to get the code off LaunchPad now.

The LaunchPad code was terminated, so I registered at GitHub and pulled the code. The branch master is the 2.1.0 version, which I tried to build using:

PATH=/usr/lib/jvm/java-11-openjdk-amd64/bin:$PATH mvn package

The result of running java11 -jar ~/projects/sikuli/lib/sikulixide-2.1.0-SNAPSHOT.jar -d5 -r ./hotel_gui_export_guest_list.sikuli was:

no main manifest attribute, in /home/rfb/projects/sikuli/lib/sikulixide-2.1.0-SNAPSHOT.jar

However, the README.md shows: 2.1.0 (branch master) currently not useable - development suspended

I tried to git clone the 2.0.6 branch, since the README.md shows: 2.0.6 (branch release_2.0.x) preparing for release - snapshots available

Using the same maven build, I received the same results. So there is a knowledge gap about building this. Just comparing the distributed sikulixide-2.0.5-lux.jar contents to one of the builds shows that the distributed jar contains all the dependencies within it and the MANIFEST.MF generated doesn't have the Main-Class. So there is definitely something I am missing in the build.

so I registered at GitHub and pulled the code

Generally, you should not have to register for github to download code.

From:

https://github.com/RaiMan/SikuliX1

You can just click on the green "Code" drop-down button and then select "Download ZIP".

2.1.0 and even 2.0.6 would be considered unstable. I think it is safest to work with 2.0.5.

Greg Shah wrote:

so I registered at GitHub and pulled the code

Generally, you should not have to register for github to download code.
You can just click on the green "Code" drop-down button and then select "Download ZIP".

2.1.0 and even 2.0.6 would be considered unstable. I think it is safest to work with 2.0.5.

I actually did the download for 2.0.6 because I didn't know how to specify the older branch. I'll try 2.0.5

Roger Borrello wrote:

I'll try 2.0.5

It took searching through the issues, but I found this: https://github.com/RaiMan/SikuliX1/issues/513

mvn -P complete-xxx-jar clean install (xxx: win, mac, macm1, lux is target system to build for)

With this update, I was able to build Sikuli that can handle our requirements:

--- /home/rfb/projects/sikuli/SikuliX1-2.0.5-final/API/src/main/java/org/sikuli/script/support/ExtensionManager.java
+++ /home/rfb/projects/sikuli/SikuliX1-2.0.5-gcd/API/src/main/java/org/sikuli/script/support/ExtensionManager.java
@@ -236,6 +236,16 @@
         extPath = lineParts[1].trim();
       }
       File extFile = new File(extPath);
+      // RFB2022110: Check for relative path
+      if (extPath.charAt(0) == '.') {
+          try {
+              extPath = extFile.getCanonicalPath();
+          } catch (Exception e) {
+              RunTime.startLog(-1, "extension path not valid (2): %s", line);
+              continue;
+          }
+          extFile = new File(extPath);
+      }
       if (extFile.isAbsolute()) {
         if (!extFile.exists() || !extFile.getName().endsWith(".jar")) {
           RunTime.startLog(-1, "extension path not valid: %s", line);

Now we can include a relative path in the extensions file, assuming we start from a known directory.

I configured the hotel_gui project with the testing directory. At that location, I created a symbolic link: ln -s ~/projects/harness/build harness.

Now I can include ./harness/lib/harness.jar in the extensions.txt file, and run the IDE from the testing directory:

java11 -jar ~/projects/sikuli/SikuliX1-2.0.5-gcd/IDE/target/sikulixide-2.0.5-complete-lux.jar -d3 -r hotel_gui_export_guest_list.sikuli

So the harness classes are available. If I were to update the harness with interfaces for calling in from Sikuli, where would they go? For example, would I create:

public String compare(String baseName, String otherName)
{
   File base = new File(baseName);
   File other = new File(otherName);

   return compare(base, other);
}

I found this: https://www.jython.org/jython-old-sites/archive/21/docs/usejava.html

If I were to update the harness with interfaces for calling in from Sikuli, where would they go? For example, would I create:

 public String compare(String baseName, String otherName)
{
   File base = new File(baseName);
   File other = new File(otherName);

   return compare(base, other);
}

in BinaryFileCompare.java?

Not exactly. It is important that you do not need to instantiate the BinaryFileCompare instance itself. Move the core comparison logic into a new static method in BinaryFileCompare that has this signature public static String compareWorker(File base, File other, ExclusionRegion[] excludes). You'll notice that the static version will need to also have the exclusion logic moved to a private static boolean excludeWorker(long num, ExclusionRegion[] excludes). This excludeWorker() would be called from BinaryFileComparison.exclude(long) as well as from compareWorker().

By making BinaryFileComparison.compareWorker() a static, you can then call it without instantiating the BinaryFileComparison instance. This will avoid all the other Harness stuff.

Greg Shah wrote:

Not exactly. It is important that you do not need to instantiate the BinaryFileCompare instance itself. Move the core comparison logic into a new static method in BinaryFileCompare that has this signature public static String compareWorker(File base, File other, ExclusionRegion[] excludes). You'll notice that the static version will need to also have the exclusion logic moved to a private static boolean excludeWorker(long num, ExclusionRegion[] excludes). This excludeWorker() would be called from BinaryFileComparison.exclude(long) as well as from compareWorker().

By making BinaryFileComparison.compareWorker() a static, you can then call it without instantiating the BinaryFileComparison instance. This will avoid all the other Harness stuff.

Where does the ExclusionRegion[] excludes come from that gets passed to the workers?

Where does the ExclusionRegion[] excludes come from that gets passed to the workers?

If you need one (because you have exclusions to process), then you would instantiate it locally and pass it in. Otherwise, just pass null.

I have this update.

It seems good.

I got around this last try by having the function receive String, and instantiating File for the compare.

Can't you just instantiate java.io.File instances in the Jython code? You will need to be able to do that kind of thing in order to integrate with Java code.

After working for a few days on the customer testcases, and making sure I was able to get the Region(x,y,w,h) values (See https://sikulix-2014.readthedocs.io/en/latest/region.html#region-rectangular-pixel-area-on-a-screen) I found that while the settings worked on my Linux Mint dual monitor configuration (laptop is the primary 1920x1080), they did not on the customer's Windows system I RDP to (single monitor 1920x1080).

TBD: determine if there is a ratio that can be determined between monitor configurations so that adjustments can be calculated.

This chunk of code will show monitor information:

s=Screen()
s.showMonitors()
num_screens=s.getNumberScreens()
print("number of screens=", num_screens)
for x in range(0, num_screens):
    s=Screen(x)
    print("bounds of screen ("+str(x)+")=", s.getBounds())

On my laptop results in:

*** monitor configuration [ 2 Screen(s)] ***
*** Primary is Screen 0
Screen 0: R[0,0 1920x1080]@S(0)
Screen 1: R[1920,0 1920x1080]@S(1)
*** end monitor configuration ***
('number of screens=', 2)
('bounds of screen (0)=', java.awt.Rectangle[x=0,y=0,width=1920,height=1080])
('bounds of screen (1)=', java.awt.Rectangle[x=1920,y=0,width=1920,height=1080])

And on the dev system:

*** monitor configuration [ 1 Screen(s)] ***
*** Primary is Screen 0
Screen 0: R[0,0 1920x1080]@S(0)
*** end monitor configuration ***
('number of screens=', 1)
('bounds of screen (0)=', java.awt.Rectangle[x=0,y=0,width=1920,height=1080])

I made all my calculations while the application was running on the laptop. At least I can use the number of monitors to determine ranges.