Automation of any GUI - Sikuli

Sikuli is a software with internal IDE to manipulate GUI automatically. The automation and image recognition help us saves a lot of time to do many regular and trivial tasks. It is based on Java and its package Jython. In this article, you will learn the installation on the Windows platform, commonly used operations, execution of the script without launching GUI, an example of fully automatic execution. That’s get start it.

Installation

  1. Download and install through .exe from Java offline version. (version 8 update 281 is feasible)
  2. Download SikuliX IDE (Click download the ready to use sikulix.jar) to certain directory. (version 2.0.4 is feasible)
  3. Download Jython to the same directory as above.
  4. Type the command java -jar your_path_to_the_file and it will launch the GUI if your installation succeed.

Operations

Quick start

  1. In the beginning, we have to switch the resolution to 1024*768
  2. Launch the Sikuli and the Windows calculator as example
  3. Take a screenshot to a button of the calculator
  4. Type the command click(YourPicture) and execute it.

Common operations

The sikuli IDE supports not only the intrinsic commands (e.g.click) but also the Python commands.

Code example

If you are familiar with python, you can write a sikuli script easily by simple commands such as if-else, for, while. If not, you can learn it from many resources in just several ten minutes cuz its easy. Here I list the commonly used command as following.

CommandsInputOutput
click()screenshotClick the picture (Error while the picture does not exist)
doubleClick()screenshotDouble click the picture (Error while the picture does not exist)
wait()intWait the seconds
exists()screenshotReturn whether the picture exsists as bool
region()regionReturn an object with several attributes such as click.
type()stringEnter the string
Key.ENTER, KEY.SPACE, etc.Press the keyboard

Export the script as a program

Once you want to execute the program without artificial works (launch IDE and press the execute button), we can export the script as a program.

  1. Click file -> Export skl, then select the path that the Sikuli is installed. You can choose the other path but the command should specify both Sikuli and the script in next step.

  1. Open CMD and type the following command to execute the script.
1
java -jar sikulixide-2.0.4.jar -r myscript.skl

The script will be executed by the command. Furthermore, if you know how to manipulate batch file or shell script as well as the setting of automatic execution after login the desktop. You can do the whole process automatically, that is, the only thing you need to do is to turn on the computer. See more in the following example.

An example

Here is an example that I write for automatically login a game “Tower of Saviors”. To let the debug step become more convenient, I make the script write the logs. Moreover, to ensure the target picture shown on the screen, we have to wait for a few seconds before executing a next Sikuli command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import os

log = "C:\Users\your_name\Desktop\Sikuli\log.txt"
os.system("echo start %date% %time% >> "+log)

wait(2)
doubleClick("1612020451846.png")
wait(75)

R = None
def myclick(pic, name, delay=7, rai=False):
    global R
    exist = R.exists(pic)
    if exist:
        R.click(pic)
        wait(delay)
    else:
        os.system("echo {} not found >> ".format(name)+log)
        if rai:
            raise ValueError(name+" not found")
        else:
            print(name+" not found")

R = Region(Region(261,40,446,687))
myclick("1612020693302.png", "enlarge")
myclick("1612020801498.png", "start")
myclick("1612017836334.png", "bigNews")
myclick("1612018419083.png", "smallNews")
myclick("1612018508468.png", "loginDays")
myclick("1612056546155.png", "monthlyPack")

os.system("echo finish %date% %time% >> "+log)
os.system("echo ---------------------------------- >> "+log

P.S. The screenshots are saved in the folder C:\Users\yoor_name\Documents\first.sikuli

After we suceed to execute the script and export the script into a .skl file in a folder (same as Sikuli folder is recommended), to execute the script automatically as you login your desktop:

  1. Create a text file and alter the extension file name from .txt to .bat at the same folder as Sikuli.

  2. Right click the .bat and edit it by text editor. Type the following code.

1
2
cd C:\Users\your_name\Desktop\Sikuli
java -jar sikulixide-2.0.4.jar -r myscript.skl

  1. Go to C:\Users\your_name\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup and create a text file followed by altering the extension file name from .txt to .vbs.

    • Scripts (.bat, .sh, .vbs, etc.) in this folder will be executed after you login.
    • .vbs is the caller of .bat which let it executes without CMD windows. (It’s important cuz we don’t want the CMD windows cover the picture that detected in Sikuli script)

  2. Right click the .vbs and edit it by text editor. Type the following code.

    1
    2
    3
    Set WshShell = CreateObject("WScript.Shell") 
    WshShell.Run chr(34) & "C:\Users\your_name\Desktop\Sikuli\startTOS.bat" & Chr(34), 0
    Set WshShell = Nothingl

Now, you can logout then login. You will see the Sikuli scripts executes automatically.