Automating GUI Testing¶

• Automating GUI Testing
  • Environment Setup
    • Fonts and GUI Sizes
    • GUI Testing Source Code
  • SikuliX Basics
    • Load the IDE
    • Regions
      • Text from a Region
    • Patterns
      • Matching
      • Similarity
      • Target Offset
      • Masking
    • Manipulation of the Mouse and Keyboard
  • Other Testing Considerations
    • File Comparison
    • Invoking the tests
    • Drivers

This is an Open Source project that is gaining popularity. The main page is http://sikulix.com/. It utilizes other technologies to provide the graphics comparisons (OpenCV: https://opencv.org/) and Optical Character Recognition (tesseract: https://github.com/tesseract-ocr/).

The fastest way to get started is by using the IDE. Download: https://launchpad.net/sikuli/sikulix/2.0.5/+download/sikulixide-2.0.5.jar and run: java -jar path-to/sikulixide-2.0.5.jar

Environment Setup¶

For best results, GUI test development should be done on the same system (or an equivalent configuration) as will be used for executing the testing. Definitely, the browser should be the same. The testcases should be able to run on both Windows 10 and most Linux's Chrome.

Fonts and GUI Sizes¶

Best results have been seen when as clean a font as possible is chosen, such as Arial, with sizes being as large as feasible for the application. The success rate for OCR when looking into a region of the screen for data is higher when you can avoid inclusion of lines (such as in a grid) within it.

Tesseract training is available to increase accuracy. Read more here: https://tesseract-ocr.github.io/tessdoc/tess4/TrainingTesseract-4.00.html

GUI Testing Source Code¶

The GUI SikuliX files are checked into the Hotel GUI configuration project in hotel_gui/testing. Each directory within being an individual test:

• hotel_gui_os_logon.sikuli
• hotel_gui_os_logoff.sikuli
• hotel_gui_os_hotel_logon.sikuli
• hotel_gui_hotel_logon.sikuli
• hotel_gui_hotel_logoff.sikuli
• hotel_gui_navigate_main_tabs.sikuli
• hotel_gui_export_guest_list.sikuli
• hotel_gui_search_for_room.sikuli

Each of those directories contains:

• a Python file, which drives the tests
• all the PNG screen captures, which are used for comparisons

SikuliX Basics¶

This wiki is not intended to be a complete documentation for SikuliX. The most complete documentation is available here: https://sikulix-2014.readthedocs.io/en/latest/index.html while https://sikulix.github.io/docs is attempting to replace it, but is not completed yet. Where links are placed, I will place both.

This wiki is intended to help you work through the existing testcases, understand them, and create additional testcases. There is a lot to know.

Load the IDE¶

Start the IDE. When it starts, it will open whatever projects you had open the last time it closed. NOTE: Make sure to use at least Java 11

cd c:\projects\hotel_gui\testing
java.exe -jar <path_to_jar>\sikulixide-2.0.5.jar

or if you are running on Linux:

cd ~/projects/hotel_gui/testing
java -jar <path_to_jar>/sikulixide-2.0.5.jar

You have the option of displaying an actual thumbnail of the images referred to in the IDE or just the filenames themselves. This is controlled by the View->Toggle ThumbNails choice in the IDE.

When thumbnails are turned on, the display of them will either be a "psuedo" thumbnail (the filename with a box around it) which activates the actual image as a popup, or an actual thumbnail of the image itself. This is controlled by a setting within the preference's more-options (see more-options) named ImageThumbs(on)/ImageLabels(off). With the checkbox on, it will turn on thumbnails and turn off image labels. With that checked, you will see the actual thumbnails of images. With it unchecked, you will see the "psuedo" thumbnails as noted above.

Carefully navigate over the icon representation of the image file to avoid too many float over images. Below are images comparing the 2 settings of ImageThumbs(on)/ImageLabels(off).

CHECKED	UNCHECKED

The IDE is useful for gathering screens, computing offsets for mouse clicks, running quick tests (as well as big ones) and managing the files.

To open a project, the File->Open menu will open a dialogue for you to select a .sikuli directory, which will contain the script and all the screen grabs for that project. That is where the .png files will be stored as you gather them.

Preferences
From the IDE, application preferences can be accessed from File->Preferences.... Below are the settings that have been successfully used:

Preferences for screen captures

Preferences for text editing

Preferences for general settings

Preferences for more options

Screenshots
While running the IDE and Hotel, you can grab screen captures directly into your script using the "Take screenshot" icon, or pressing Cntrl+Shift+2 hotkey.

Once you activate, the IDE will disappear, revealing the screen area with a capture "crosshair". Drag with the left mouse button down to create a square around the screen you want to capture, and release when you are satisfied. It will "drop" an icon into the IDE at the location the cursor was when you activate the capture if you have the View->Toggle thumbnails checked (see thumbnail info), or the filename of the newly created file if you don't. It will attempt to name the file from text found in the image (when this is the preference as shown in preferences for screen capture. You can also let it use a timestamp, or force you to enter the name as it is captured.

TIP: Leave the thumbnails off unless you are actively working with them, as will be shown below.

Images
You can use any tool to capture the screens and enter the filenames manually into the script. The "Insert image" lets you pull in the file, or you can simply type it in.

Regions¶

Regions define a rectangular area of the screen. Once a region is defined, you can act upon it, such as search for an image (.find()), highlight an area (.highlight()), or pull in the text that is within (.text()). It utilizes (x, y, w, h) points where x=0, y=0 is the upper-left point on your screen. If your resolution is 1920x1080, the lower-right point would be x=1920, y=1080. So region r=Region(0,0,1920,1080) would encompass the entire screen.

From within the IDE, you can define a region in your script using the Region icon at the top. It will allow you to use the mouse to draw a rectangle around the region, then enter the corresponding coordinates into the Region(x,y) method. If you are showing thumbnails, it will show a thumbnail of the selected region as it appears on the screen. When you turn thumbnails off, you will see the API.

If you do not enter region information, the subsequent methods will operate against the entire screen, which is find in most cases of these tests, since the Chrome browser should be maximized to cover the entire screen.

See https://sikulix-2014.readthedocs.io/en/latest/region.html or https://sikulix.github.io/docs/api/region for details and a full range of methods.

Text from a Region¶

One of the techniques used often is trying to collect text from a region on the screen. This is used to verify things like the correct date showing up on the screen, or values. Unfortunately, there are some inaccuracies in the OCR that is used (tesseract) by SikuliX. There are some "best practices" that can help.

For example. hotel_gui_navigate_main_tabs.py, we want to see what text is on a couple of widgets on the screen. The code snippet below the image shows how to get some data.

The screen below was captured while the room_type_r.highlight(10) was being performed (highlight the region for 10 seconds). This helps you determine if your .text() method will be retrieving the correct region contents.

In this snippet, we make sure to click the "Available Rooms" tab, so it is showing:

# Make sure Available Rooms is showing
click(Pattern("chrome_hotel_main_screen_controls.png").targetOffset(available_rooms_x,chrome_hotel_main_screen_controls_y))
wait("chrome_hotel_available_rooms_tab.png",5)

# See what some of the text is
room_type_r=Region(95,191,73,17)
room_type_r.highlight(10)
room_type=room_type_r.text()
num_days_r=Region(92,285,73,17)
num_days_r.highlight(10)
num_days=num_days_r.text()
print("INFO: room_type=" + room_type + " and num_days=" + num_days)

room_type=room_type_r.text() pulls in the text and interprets it. From that point on we can do whatever we need to do against the Python variable (which is a string). The output in the message area of the IDE is:

[log] CLICK on L[57,171]@S(0) (553 msec)
[log] highlight R[95,191 73x17]@S(0) for 10.0 secs
[log] highlight R[92,285 73x17]@S(0) for 10.0 secs
INFO: room_type=All rooms and num_days=7

Patterns¶

Patterns are created so a image file can be utilized in a useful manner. This would be for something more than just matching up with a Region. You might want to assign a similarity factor so that a "partial" match on the image can be found. You might want to determine an offset into an image for purposed of targeting a mouse click, or even create an image that can either obscure certain areas of the Region that might differ in a known way, or only look at a small portion of the Region. These are detailed below.

See https://sikulix-2014.readthedocs.io/en/latest/pattern.html or https://sikulix.github.io/docs/api/pattern for details and a full range of methods.

Matching¶

For a simple case of matching, we can use find() where the Region is searched for the given pattern. Consider this snippet from hotel_gui_os_logon.py where we want to sign into the OS sign-on screen of the web client, but in certain circumstances, a Private Connection warning is given by Chrome if the certificate authority is not yet cached. We want to notice that, and react to it. (Below IP address is blurred).

Rather than fail, we can check for it appearing using find(), and click on the advanced button so we can proceed through the "danger" to get to the logon. In this example, we don't care what the IP address is (since it will change often) and we obscure it with a mask (see Masking for more):

msk=Pattern("chrome_not_private_connection_mask.png").mask()
try:
    m=find(Pattern("chrome_not_private_connection_mask.png").exact().mask())
    if m != None:
        #m.highlight(5)
        print("WARNING: NET::ERR_CERT_AUTHORITY_INVALID warning found, score="+str(m.getScore()))
        click(Pattern("chrome_not_private_connection_mask.png").targetOffset(-280,175).mask())
        msk=Pattern("chrome_not_private_connection_advanced_mask.png").mask()
        wait(msk,5)
        click(msk.targetOffset(-260,260))
except FindFailed as e:    
    pass

# Wait for the user name text to appear
wait("chrome_logon_info.png",5)
click(Pattern("chrome_logon_info.png").targetOffset(10,-15))

In the IDE, you can determine if something matches by using the icon thumbnails. The below example has a simple image that is the set of icons in the top/left of the Windows desktop:

After clicking on the thumbnail, the pattern settings screen showing the image is displayed, and there are tabs for matching and offsets.

The "Matching Preview" tab will compare the image with what is on the screen at the moment you clicked on the thumbnail. So make sure the desktop is clear of other things, or the matching preview will take those into account. When working with Hotel, you need to be at the screen in the application you are working with for the particular testcase. We'll be using the simple example of the Windows desktop here.

Matching Preview Example

See how the matching preview highlights the 3 icon, indicating a match.

By using this method, you are able to feel confident your testcase will find a match, as well.

Similarity¶

There are times when a pattern "could" match something in the region based upon how stringent you desire the pattern to match. This is controlled by the "similar" option. By increasing the similarity factor, the algorithm requires more of the data to match. By decreasing the similarity factor, the algorithm requires less of the data to match.

Take the example above. The captured image has Acrobat in it. But in your testcase you don't care if the Adobe Acrobat icon were available, but rather that Google Chrome and Firefox were in that location, you could scale back the similarity to allow for that image to match with a desktop missing Acrobat. It wouldn't make any difference to the validity of the testcase.

Below is the matching preview with Acrobat missing.

The similarity slidebar is at the default of .7. (This default is controlled by Settings.MinSimilarty. See https://sikulix-2014.readthedocs.io/en/latest/scripting.html#Settings.MinSimilarity). By clicking it to the left to .69, the match preview now shows a match:

When you click on "Apply" and then "Save", a new attribute is added to the Pattern in the IDE:

If you turnoff thumbnails, the code is revealed:

p=Pattern(Pattern("DPGoogledian.png").similar(0.69))

This can work the other way around. If the capture image has Acrobat within it but matches the desktop that doesn't have it, and you definitely want it to be there, you can increase the similarity factor to find the point at which it will no longer match.

The screen below show that the desktop doesn't have the Acrobat icon in position, but the captured image on the right does have it in the capture (along with a large amount of space to the right):

When we look at the matching preview, we see that we have to slide the similarity indicator all the way past 0.94 to ensure it no longer matches. The GIF below illustrates that action of going from matching at .94 to no longer matching at .95 and back:

A typical occurrence is that the captured screen might be too small or too large for what is required, and the similarity can be used to adjust. You can also enforce an exact match by using .exact() instead of .similar(similarity).

Exceptions

We cannot discuss matching without introducing the need to press on when we fail to find one. In general, something that doesn't match when request a match using find() will throw a FindFailed exception and the script execution will stop. However, in many cases you would want to log a warning or an error and continue. The try/except is the Python construct to use. A complete example is below, which will also illustrate how to use getScore() so you can determine how exactly something matches:

l=Location(1920, 1080)
r=Region(0,0,1920,1080)

p=Pattern("small_with_acrobat.png")
fn=p.getFilename()
try:
    sim=0.69
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

try:
    print("Looking for "+fn+" on the desktop score=default")
    m=r.find(Pattern(p))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

try:
    sim=0.94
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))
print("")

p=Pattern("small_without_acrobat.png")
fn=p.getFilename()
try:
    sim=0.68
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

try:
    print("Looking for "+fn+" on the desktop score=default")
    m=r.find(Pattern(p))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

try:
    sim=0.94
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))
print("")

p=Pattern("large_with_acrobat.png")
fn=p.getFilename()
try:
    sim=0.69
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

try:
    print("Looking for "+fn+" on the desktop score=default")
    m=r.find(Pattern(p))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

try:
    sim=0.95
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))
print("")

p=Pattern("large_without_acrobat.png")
fn=p.getFilename()
try:
    sim=0.69
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

try:
    print("Looking for "+fn+" on the desktop score=default")
    m=r.find(Pattern(p))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

try:
    sim=0.95
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

Results when the desktop contains Acrobat:

• Small image with Acrobat matches every time
• Small image without Acrobat never matches unless similarity decreases (< 0.68)
• Large image with Acrobat matches every time
• Large image without Acrobat matches until similarity increases (> 0.94)
  

  Looking for C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.69
  +++ Found C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.987895131111
  Looking for C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=default
  +++ Found C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.987895131111
  Looking for C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.94
  +++ Found C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.987895131111
  
  Looking for C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.68
  +++ Found C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.682968258858
  Looking for C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=default
  --- Did not find C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.682968258858
  Looking for C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.94
  --- Did not find C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.682968258858
  
  Looking for C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.69
  +++ Found C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.999118030071
  Looking for C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=default
  +++ Found C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.999118030071
  Looking for C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.95
  +++ Found C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.999118030071
  
  Looking for C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=0.69
  +++ Found C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=0.947983443737
  Looking for C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=default
  +++ Found C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=0.947983443737
  Looking for C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=0.95
  --- Did not find C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=0.947983443737
  

Results when the desktop is missing Acrobat

• Small image with Acrobat never matches unless similarity decreases (< 0.70)
• Small image without Acrobat matches every time
• Large image with Acrobat matches until similarity increases (> 0.94)
• Large image without Acrobat matches every time
  

  Looking for C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.69
  +++ Found C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.690889000893
  Looking for C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=default
  --- Did not find C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.690889000893
  Looking for C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.94
  --- Did not find C:\gcd\wiki.sikuli\small_with_acrobat.png on the desktop score=0.690889000893
  
  Looking for C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.68
  +++ Found C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.99085021019
  Looking for C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=default
  +++ Found C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.99085021019
  Looking for C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.94
  +++ Found C:\gcd\wiki.sikuli\small_without_acrobat.png on the desktop score=0.99085021019
  
  Looking for C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.69
  +++ Found C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.946603298187
  Looking for C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=default
  +++ Found C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.946603298187
  Looking for C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.95
  --- Did not find C:\gcd\wiki.sikuli\large_with_acrobat.png on the desktop score=0.946603298187
  
  Looking for C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=0.69
  +++ Found C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=1.0
  Looking for C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=default
  +++ Found C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=1.0
  Looking for C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=0.95
  +++ Found C:\gcd\wiki.sikuli\large_without_acrobat.png on the desktop score=1.0
  

Target Offset¶

When you want to pinpoint a specific location within a Pattern, you can use .targetOffset(dx,dy) method. This is most useful when attempting to click the mouse in a certain area of a matched pattern. The (0,0) coordinates correspond to the center of the pattern.

For example, in keeping with the Adobe Acrobat appearing on the screen or not, if we have determined with appropriate certainty that the Acrobat icon does appear on the desktop using:

l=Location(1920, 1080)
r=Region(0,0,1920,1080)

p=Pattern("small_with_acrobat.png")
fn=p.getFilename()

try:
    sim=0.94
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

We'd like do "double-click" on it to launch it. This requires us to know the location of the icon within the small_with_acrobat.png image

Using the IDE for this is very easy, if you have the thumbnails turned on. While you have the actual screen in place that the image should match, click on the thumbnail:

Then you will be able to click on the Target Offset table. The bottom shows the (dx,dy) of the selection you make within the tab. You can also use the up/down arrows to move the crosshair to the desired location. Below it is placed in (0,85)

Once you click "apply" and "OK", the offset will be added as the parameters to targetOffset(). The IDE will do other things on your behalf, like add the Pattern() generation to the image file, if it wasn't already there. So you have to be aware of which thumbnail you are clicking since it doesn't have any intelligence around adding the additional Pattern("image").targetOffset(dx,dy). And you cannot perform this if you are using a variable to hold the pattern match. The IDE doesn't allow you to open the Target Offset settings against it. You can alway explicitly use the image, then adjust your code accordingly.

With the code snippet above, I would actually have to add a line such as:

    doubleClick("small_with_acrobat.png")

By itself, the line above would actually doubeClick() on the center of the image (0,0) by default.

Then with thumbnails viewable, click on it, and apply/close the settings. When I turn off thumbnails, I see:

    doubleClick(Pattern("small_with_acrobat.png").targetOffset(0,85))

At this point, I can re-add the elegancy desired:

try:
    sim=0.94
    print("Looking for "+fn+" on the desktop score="+str(sim))
    m=r.find(Pattern(p).similar(sim))
    print("+++ Found "+fn+" on the desktop score="+str(m.getScore()))
    doubleClick(p.targetOffset(0,85))
except FindFailed as e:
    print("--- Did not find "+fn+" on the desktop score="+str(m.getScore()))

There are other tools to use which would help you find the offset, but you need to keep in mind that the (dx,dy) coordinates are centered on the middle point of the image. Tools like GIMP typically utilize the upper-left point as (0,0) and move from there.

This is another warning if you are using the something typical, notebook tabs like a screen grab below from the main hotel screen so that you can navigate:

It is very convenient because the dy will be fixed for each of the tabs, and the dx will "move" across from left to right (negative to positive), so you can use variables to hold the coordinates. But, the notebook tab image may not match the screen as the individual tabs are clicked on and highlighted, since there will be slight changes from tab-to-tab. In those cases you can take a more exact screen capture with the correct tab highlighted, or make adjustment to the similarity so it would match the screen with a lower number.

Show Actions
By default, actions the mouse performs are done "silently", where the movements look as normal mouse movements that you would be performing yourself. However, sometimes it does help to have the mouse's point visible. Use setShowActions(False|True) to control this. See http://doc.sikuli.org/globals.html#controlling-sikuli-scripts-and-their-behavior for details.

Masking¶

You have already seen an example of when masking would have helped. In the case where we don't really care if there is an Acrobat icon on the desktop, there is a technique in which a layer of black can be added over a spot on image to allow the matching to ignore that spot on the image. In the below case:

You see that the image has an area filled with black. This was accomplished by using GIMP, and adding a new layer to the PNG file (I call it "mask"), then making a selection of the area we "don't care" about, and fill it with the color black. Then save it as a new image file (such as add "_mask" to the filename).

So the code snippet, which will now "wait" for an image to appear in 5 seconds or fail, is:

p=Pattern("small_with_acrobat_mask.png")
fn=p.getFilename()
print("Looking for "+fn+" on the desktop")
if wait(p.mask(), 5):
    print("+++ Found "+fn+" on the desktop with mask() option")
else:
    print("--- Did not find "+fn+" on the desktop with mask() option")
print("")

The addition of .mask() indicates the patter contains a mask to ignore. More details can be found https://sikulix-2014.readthedocs.io/en/latest/pattern.html#Pattern.mask or https://sikulix.github.io/docs/api/pattern#masking. There is also a "tutorial" here: https://sikulix-2014.readthedocs.io/en/latest/tutorials/masking/masking.html

Manipulation of the Mouse and Keyboard¶

You have already seen some ways to manipulate the mouse using doubleClick() (click() works in the same manner for a single click).

For drag-and-drop, below is a little example showing usage:

# Drag items to new folder
r = find("image_you_want_to_drag.png")
dropRegion = Location(411, 313)
dragDrop(r, dropRegion)
keyUp()

There are some video tutorials online:

• https://youtu.be/SEJmoZMHPgQ
• https://youtu.be/RAgFmJdo0B4?t=418
• https://youtu.be/QjdKY1KrDmo

Typing Values
The type method allows you to enter data into a region of the screen. Using the standard Python string constructs, you can build a string to enter, including tabs ("\t") and enter ("\n"). There are many other special characters as well, and they are all outlined here: https://sikulix-2014.readthedocs.io/en/latest/region.html#Region.type where there is also a link to special characters and modifiers.

For example, to enter the login information for "hotel":

type("hotel")
type("\t") # type("\n")
# Turn off ActionLogs so the text doesn't appear in the log
Settings.ActionLogs = False
type("hotel")
Settings.ActionLogs = True

Other Testing Considerations¶

Some information pertaining to testing. A few examples make use of the helper functions in includes/sikuli_helpers.py to handle different OS environments. That file is shown here for reference:

#
# Module   : sikuli_helpers.py
# Abstract : Python Helper functions for Sikuli scripts
#
# Copyright (c) 2022, Golden Code Development Corporation.
#
# -#- -I- --Date-- ------------------------------Description----------------------------------
# 001 RFB 20221109 Initial version.

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.
#
# You may find a copy of the GNU Affero GPL version 3 at the following
# location: https://www.gnu.org/licenses/agpl-3.0.en.html
#
# Additional terms under GNU Affero GPL version 3 section 7:
#
#   Under Section 7 of the GNU Affero GPL version 3, the following additional
#   terms apply to the works covered under the License.  These additional terms
#   are non-permissive additional terms allowed under Section 7 of the GNU
#   Affero GPL version 3 and may not be removed by you.
#
#   0. Attribution Requirement.
#
#     You must preserve all legal notices or author attributions in the covered
#     work or Appropriate Legal Notices displayed by works containing the covered
#     work.  You may not remove from the covered work any author or developer
#     credit already included within the covered work.
#
#   1. No License To Use Trademarks.
#
#     This license does not grant any license or rights to use the trademarks
#     Golden Code, FWD, any Golden Code or FWD logo, or any other trademarks
#     of Golden Code Development Corporation. You are not authorized to use the
#     name Golden Code, FWD, or the names of any author or contributor, for
#     publicity purposes without written authorization.
#
#   2. No Misrepresentation of Affiliation.
#
#     You may not represent yourself as Golden Code Development Corporation or FWD.
#
#     You may not represent yourself for publicity purposes as associated with
#     Golden Code Development Corporation, FWD, or any author or contributor to
#     the covered work, without written authorization.
#
#   3. No Misrepresentation of Source or Origin.
#
#     You may not represent the covered work as solely your work.  All modified
#     versions of the covered work must be marked in a reasonable way to make it
#     clear that the modified work is not originating from Golden Code Development
#     Corporation or FWD.  All modified versions must contain the notices of
#     attribution required in this license.

# Import helpers
import os
from datetime import time
from datetime import date
import glob
import filecmp
import org.sikuli.basics.Settings as Settings
import org.sikuli.script.support.Runner as Runner

# Setup some OS specific things so that we can deal with generated files easier
if Settings.isWindows() == True:
   path_sep = '\\'
   downloads_dir = os.environ['HOMEDRIVE'] + os.environ['HOMEPATH']
elif Settings.isLinux() == True:
   path_sep = '/'
   downloads_dir = os.environ['HOME']

downloads_dir = downloads_dir + path_sep + 'Downloads'
logoff_script = "hotel_gui_hotel_logoff" 
logon_script = "hotel_gui_hotel_logon" 

today_val = date.today()

# This function will find the latest file (of the given blob and path) to the given file
def getLatestFile(file_blob):
    # The max file is the most recently created
    print("Blobbing: " + file_blob)
    latest_file = max(glob.glob(file_blob), key=os.path.getctime)
    print("Latest file:" + latest_file)
    return latest_file

# This function will find the latest file (of the given blob and path) to the given file
def compareLatestFile(file_blob, baseline_file):
    # The max file is the most recently created
    print("Blobbing: " + file_blob)
    print("Baseline file:" + baseline_file)
    testcase_file = getLatestFile(file_blob)
    print("Testcase file:" + testcase_file)
    # filecmp.cmp will return True for matching files. Use Deep comparison to match more than size and timestamp
    return filecmp.cmp(baseline_file, testcase_file, shallow=False)

# Function to run given script using the root of all our scripts
def runMyScript(scr,bundle):
    print("runMyScript: bundle="+bundle)
    script_bundle = os.path.abspath(os.path.join(bundle, os.pardir)) + path_sep + scr + ".sikuli" 
    #script_bundle = script_root + scr + ".sikuli" 
    print("runMyScript: script_bundle="+script_bundle)
    Runner.runScript(script_bundle,None,None)

File Comparison¶

Often there times where output generated to a file should be validated. File comparison against a known file is a good way to do this. hotel_gui_export_guest_list.py has such an example.

The testcase navidates to the guests tab, and then filters the guest data by the name "Smith" and exports as XLS.

# Baseline files
bundle_path=getBundlePath()
baseline_file_path = bundle_path + shikuli.path_sep
export_by_guest_baseline=baseline_file_path+"guests_export_smith_baseline.xls" 

# Get room information
room = "303" 
guest = "Smith" 
...
# Apply name filter
click("chrome_hotel_guests_reset_filters_button.png")
click("chrome_hotel_guests_filter_by_guest.png")
type(guest)
click("chrome_hotel_guests_search_button.png")
wait("chrome_hotel_guests_guest_filtered.png",5)

# Make sure we are creating xls
if exists(Pattern("chrome_hotel_guests_export_area_pdf.png")):
    click(Pattern("chrome_hotel_guests_export_area_pdf.png").targetOffset(-35,0))
wait(Pattern("chrome_hotel_guests_export_area_xls.png").similar(0.73),5)

# Generate the export
click(Pattern("chrome_hotel_guests_export_area_xls.png").targetOffset(10,0))
# Find xls file that was generated in the downloads directory (platform specific, assumes Chrome)
xls_files = shikuli.downloads_dir + shikuli.path_sep + '*.xls'
time.sleep(5)
if shikuli.compareLatestFile(xls_files, export_by_guest_baseline) != True:
    print("WARNING: files don't match")

By keeping a baseline file (in this case guests_export_smith_baseline.xls) we have to make sure the data being utilized in the database contains a known set of data so that the test can be validated properly.

This code snippet will compare a generated report, passing it through the "more" command on Windows, or the "tail" command on Linux to that areas of the file can be ignored (everthing but the top lines):

# This function redirects output of the given command to the specified file
def redirOutput(cmd, outfile):
    split_cmd = shlex.split(cmd)
    print("redirOutput: redirecting command="+cmd+" to file="+outfile)
    #print(split_cmd)
    process = subprocess.Popen(split_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()
    f = open(outfile, "wb")
    f.write(stdout)
    f.close()
...
csv_files = downloads_dir + path_sep + 'pbamdprod-'+today_fmt_yyyymmdd+'-*.csv'
removeFiles(csv_files)
click(Pattern("chrome_extract_stuff.png").targetOffset(-250,190))
time.sleep(15)
baseline_file = baseline_file_path+'extract_stuff_baseline_truncated.csv'
if Settings.isWindows() == True:
    tmp_file=os.getenv('TEMP')+"\\result_file.txt" 
    bashCommand = "cmd.exe /c more +3 " 
else:    
    tmp_file="/tmp/result_file.txt" 
    bashCommand = "tail -n +4 " 
result_file = max(glob.glob(csv_files), key=os.path.getctime)
bashCommand = bashCommand + repr(result_file).strip("'")
print("bashCommand="+bashCommand)
# Skip the top 4 lines, since we don't care about that part
os.remove(tmp_file)
redirOutput(bashCommand, tmp_file)
a = open(baseline_file, 'r').read()
b = open(tmp_file, 'r').read()
assert(a==b)
# TODO: filecmp.cmp cannot handle the end of line being different on Unix v Windows
#assert(filecmp.cmp(baseline_file, tmp_file, shallow=False))

Invoking the tests¶

The command line help for Sikuli:

C:\projects\hotel_gui\testing>java -jar \sikuli\lib\sikulixide-2.0.5.jar -h
usage:
 [-c] [-d <arg>] [-f <arg>] [-h] [-l <arg>] [-m] [-p] [-q] [-r <arg>] [-s <arg>]
       [-u <arg>] [-v]
----- Running SikuliX -------------
 -c,--console         print all output to commandline (IDE message area)
 -d,--debug <arg>     positive integer (1)
 -f,--logfile <arg>   a valid filename (WorkingDir/SikuliLog.txt)
 -h,--help            print this help message
 -l,--load <arg>      preload scripts in IDE
 -m,--multi           two or more IDE instances are allowed
 -p,--pythonserver    use SikuliX features from Python
 -q,--quiet           show nothing
 -r,--run <arg>       run script
 -s,--server <arg>    run as server on optional port
 -u,--userlog <arg>   a valid filename (WorkingDir/UserLog.txt)
 -v,--verbose         Debug level 3 and elapsed time during startup
-----
<foobar.sikuli> (.sikuli might be omitted, is assumed)
path relative to current working directory or absolute path
though deprecated: so called executables .skl can be used too
------
anything after --
or after something beginning with --
go to script as user parameters (respecting enclosing ")
------
-d use this option if you encounter any weird problems
DebugLevel=3 and all output goes to <workingFolder>/SikuliLog.text
----------------------------------------------------------------

If you'd like to run hotel_gui_search_for_room.sikuli:

cd \projects\hotel_gui\testing
java -jar \sikuli\lib\sikulixide-2.0.5.jar hotel_gui_search_for_room

Jumping into a test like that requires that you understand what need to be setup already and what the test assumes to be setup, and what it can handle itself. In this particular case, you would need to be logged in.

Drivers¶

Putting all the tests together so that they can be "driven" through one-by-one also requires a robustness to handles certain setups that may occur on the test environment, such as are all the desktops where the tests are going to be run the same, or do you need to find things. The more standard the environments are, they easier to design the tests.

The below is hotel_gui_driver.py which shows the layout of invoking the other scripts that have been created:

thescript = "./hotel_gui_os_logon" 
exitValue = runScript(thescript)
if exitValue > 1:
   print "there was a special case with " + thescript
   exit(exitValue)
elif exitValue == 1:
   print "there was an exception with " + thescript
   exit(exitValue)
else:
   print "successfully ran " + thescript

thescript = "./hotel_gui_hotel_logon" 
exitValue = runScript(thescript)
if exitValue > 1:
   print "there was a special case with " + thescript
   exit(exitValue)
elif exitValue == 1:
   print "there was an exception with " + thescript
   exit(exitValue)
else:
   print "successfully ran " + thescript

thescript = "./hotel_gui_navigate_main_tabs" 
exitValue = runScript(thescript)
if exitValue > 1:
   print "there was a special case with " + thescript
   exit(exitValue)
elif exitValue == 1:
   print "there was an exception with " + thescript
   exit(exitValue)
else:
   print "successfully ran " + thescript

thescript = "./hotel_gui_search_for_room" 
exitValue = runScript(thescript)
if exitValue > 1:
   print "there was a special case with " + thescript
   exit(exitValue)
elif exitValue == 1:
   print "there was an exception with " + thescript
   exit(exitValue)
else:
   print "successfully ran " + thescript

thescript = "./hotel_gui_export_guest_list" 
exitValue = runScript(thescript)
if exitValue > 1:
   print "there was a special case with " + thescript
   exit(exitValue)
elif exitValue == 1:
   print "there was an exception with " + thescript
   exit(exitValue)
else:
   print "successfully ran " + thescript

thescript = "./hotel_gui_hotel_logoff" 
exitValue = runScript(thescript)
if exitValue > 1:
   print "there was a special case with " + thescript
   exit(exitValue)
elif exitValue == 1:
   print "there was an exception with " + thescript
   exit(exitValue)
else:
   print "successfully ran " + thescript

thescript = "./hotel_gui_os_logoff" 
exitValue = runScript(thescript)
if exitValue > 1:
   print "there was a special case with " + thescript
   exit(exitValue)
elif exitValue == 1:
   print "there was an exception with " + thescript
   exit(exitValue)
else:
   print "successfully ran " + thescript

exit(exitValue)

---

© 2023-2023 Golden Code Development Corporation. ALL RIGHTS RESERVED.