Main Page
TESTAR
On this wiki we will describe how to install and use Testar. If you want/need an account to contribute please send an e-mail to wiki@testar.org (Consult the User's Guide for information on using the wiki software.)
Automated testing at the Graphical User Interface Level
GUI Testing is a testing technique where one tests the System Under Test (SUT) solely through its Graphical User Interface (GUI). The only way to find errors is to thoroughly observe the status of the GUI throughout the test. This type of testing is usually still carried out manually, where a tester just follows a previously written test case and verifies whether the application responds to all inputs as expected.
On the one hand GUI testing is a relatively straightforward process, since one does not need to read or test the source code of the SUT. On the other hand it is quite laborious, time consuming and, well... boring...
Approaches for automation of GUI testing are for example Capture & Replay or Visual Testing. These approaches use test scripts to automate the steps of each test case. These type of tests, whether manual or automated through scripts, test the sunny day use cases and mostly follow the happy paths. But what about the less sunny test scenarios? Those scenarios executed by the rogue user?
TESTAR
TESTAR is an open source tool for automated testing through GUI<ref>paper</ref>. It automatically interacts with the SUT by clicking and typing on the controls of the GUI. It is able to recognize abnormal SUT behaviour, such as crashing or freezing. Moreover, you can add application specific test oracles that are able to detect domain specific faults. TESTAR reports test sequences that lead to failure.
Virtual Environment
We have a Virtual environment at the UPV in Valencia.
Github open source community
How to get started (handson manual)
There exists a hands-on manual that will permit you to get started with automation of GUI testing using TESTAR. You can always get the latest version of this manual here:
TESTAR download the hands-on manual
In the manual it is explained how to get started with the latest version and a virtual machine.
How to use the experiments architecture to launch worker machines using a central distribution "TESTAR + suts"
The software architecture establishes guidelines on the structure, operation and interaction between the parts of the software. In this case, the needed instructions are executed from the controlling machine. It coordinates the operations of each worker machine and sends them the requests that they must attend.
The file server is known as Qsamba server. It represents the center of operations of this architecture, located in the N directory of the controlling machine. It contains:
- A folder associated to the IP address of each worker, where TESTAR output information will be stored.
- The metrics folder:
There are all the coverage results for each SUT (System Under Test) and algorithm.
- The demo_files folder:
It is a custom folder, and contains a distribution of TESTAR and OrientDB that will be copied by all the workers to their own environment, to be used for the executions.
- The scheduler folder:
It stores the batch data files. Here we can find folders whose name is the IP address of each worker, that contain the current, done and queue folders. These represent the status of the work for each worker.
All workers execute the runner.sh script, located in the scheduler folder. This script makes them sleep for a minute and then check their queue folder to see if they have any .bat file to execute (Image "Runner.sh").
The central structure of the folders can be seen in the image below right.
Preparation of a centralized TESTAR distribution
A single, stable TESTAR distribution must be used for all workers. This avoids unexpected behavior differences that might appear when using different versions.
- Clone experiments branch.
- Prepare a distribution. First, open a cmd command line in the project folder. Then, run gradle installDist or gradle distZip.
- Add the experiments SUTs from https://testar.org/images/development/experiments_suts.zip to the “testar\bin\suts” folder in the project.
- Copy TESTAR distribution to "N:\demo_files".
Preparation of a centralized OrientDB distribution
As was done with TESTAR, a centralized OrientDB distribution must be prepared for all workers.
- Download OrientDB 3.0.34 from https://s3.us-east-2.amazonaws.com/orientdb3/releases/3.0.34/orientdb-3.0.34.zip.
- Execute the server.bat file to launch the OrientDB server and create a new database (testar).
- Add a new user with testar\testar credentials with admin permissions.
- Stop the server.
- Copy this OrientDB distribution to "N:\demo_files".
Batch files
As stated before, the central Qsamba server contains a folder "N:\scheduler" that is used by workers to check their pending tasks. Each worker checks the IP address + queue folder to get the "task.bat" to be executed (for example, "N:\scheduler\10.101.0.101\queue").
For Each SUT + Protocol we will need a customized batch file to launch TESTAR with the desired settings. Between every execution, to have the same initial state, each worker will:
- Restart the system (Image "1_Restart_System.bat"):
Each worker needs a new and stable version of the system to run the experiments reliably. For this, they go to the desktop and execute an instruction to shutdown the system and start it again.
- Set a specific resolution (Image "2_Prepare_Screen_Resolution.bat"):
We need to always have the same resolution in the system, so we execute the VMwareResolution.exe file with the appropriate parameters. This file is stored in the "C:\Program Files\VMware\VMware Tools" folder.
- Clean and copy TESTAR (Image "3_Clean_And_Copy_TESTAR.bat"):
The workers go to the desktop and remove any TESTAR distribution. Then, they copy the TESTAR distribution located in the "N:\demo_files" folder to the desktop.
- Clean and copy OrientDB (Image "4_Clean_And_Copy_OrientDB.bat"):
If there is any version of OrientDB in the desktop, it will be removed. After that, the version of Orientdb that is stored in the "N:\demo_files" folder will be copied to the desktop.
How to launch multiple batch and workers
- Open a Windows command line in "N:\scheduler".
- Type "bash".
- Enter the appropriate commands in the new bash command line.
This is an example of the general steps to follow:
for i in 10.10* ; do echo $i; cp -v batch_demo/1_Restart_System.bat $i/queue/a_restart_system.bat; done
for i in 10.10* ; do echo $i; cp -v batch_demo/2_Prepare_Screen_Resolution.bat $i/queue/b_resolution.bat; done
for i in 10.10* ; do echo $i; cp -v batch_demo/3_Clean_And_Copy_TESTAR.bat $i/queue/c_prepare_testar.bat; done
for i in 10.10* ; do echo $i; cp -v batch_demo/4_Clean_And_Copy_OrientDB.bat $i/queue/d_prepare_orientdb.bat; done
After that, for a specific SUT (Spaghetti) and protocol (Random), this must be run:
for i in 10.10* ; do echo $i; cp -v batch_demo/TEST_SpaghettiRandom.bat $i/queue/e_TEST_SpaghettiRandom.bat; done
As there are multiple batch files to execute, we use alphabetical order by placing a letter at the beginning of the name of the new files.
TEST_SpaghettiRandom.bat, mentioned above, is an example of a specific batch file for the Spaghetti SUT and the Random ASM. In the image on the left you can be seen the content of two files corresponding to the Spaghetti SUT, one of them with the Random ASM, and the other with the Reinforcement Learning ASM. The main parameters that these type of files contain are:
- sse:
The application to be run
- ShowVisualSettingsDialogOnStartup:
It is set to False to not run the GUI of TESTAR.
- ApplicationName:
TESTAR uses this name to save the output files and the metrics in the folder associated with each worker.
- ApplicationVersion:
It is not very important, as there is a new version of the SUT in each execution.
- Sequences:
Number of sequences to be executed.
- >:
Everything about the command line is stored in the specified temporary file.
As we can see in the second example, there is a parameter called ExtendedSettingsFile, An XML file that contains additional settings is referenced by it, and looks like the next image. In this case, its content consists mainly of attributes related to the Reinforcement Learning technique.
Workers logs and outputs
Each "N:\scheduler\10.10X.0.1XX" worker contains a runner.log that prints the tasks results. They also read TESTAR err output buffer, so we can check these logs to see if TESTAR throws an exception. This command prints the last 5 lines of all logs:
tail -n 5 10.10*/*.log
The compressed TESTAR output for each worker + SUT + protocol are stored in “N:\10.10X.0.1XX\SUT_Protocol”, and the TESTAR metrics (jacoco + time) are saved in “N:\metrics\SUT_Protocol”.
FAQ
java.lang.RuntimeException: JDK required (running inside of JRE)
If you get this error this means that your JAVA_HOME path does not point correctly to the installation path for the Java Development Kit.. To set your JAVA_HOME look for example here:
https://stackoverflow.com/questions/2619584/how-to-set-java-home-on-windows-7
Which platforms are supported?
Testar needs Java Standard Edition (SE) to run and we support the Java SE LTS versions that are currently under Oracle Extended support [1]. Other versions might work .
| Java SE version | Testar Support Until |
|---|---|
| 8 | December 2030 |
| 11 | September 2026 |
| 17 | September 2029 |
The TESTAR binaries that are downloaded currently supports Windows Desktop applications. When using the WebDriver TESTAR can also run on MacOS, Linux, Docker and Android.
What is required from the System Under Test (SUT)?
Not much, just that the application respects the accesibility API of the underlying Operating System. For web applications you can use the WebDriver.
How to set TESTAR with different SUT?
TESTAR main version offers three options to connect with the SUT. Depending on the application that acts as SUT and the way we want to interact with it, we must choose one or the other.
TESTAR WebDriver version also includes an additional option to connect to the desired SUT using the Selenium webdrivers.
How to set TESTAR with different SUT
ISSUE: Windows screen settings affects to Screenshots and Actions coordinates
If TESTAR doesn't correctly take the coordinates of your environment to execute mouse actions or take screenshots, please go to https://github.com/TESTARtool/TESTAR_dev/issues/222
The feedback about the Windows screen setting of your environment will be welcome to improve TESTAR tool.
How is TESTAR licensed?
TESTAR is distributed FREE of charge as an open source project under the BSD3 license, some components are licence under the Eclipse Public License 2.0.
Publications
From other people using TESTAR
- Mauro Pezzè, Paolo Rondena, and Daniele Zuddas. 2018. Automatic GUI testing of desktop applications: an empirical assessment of the state of the art. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops (ISSTA '18). ACM, New York, NY, USA, 54-62. DOI: https://doi.org/10.1145/3236454.3236489 (https://dl.acm.org/citation.cfm?id=3236489)
<references />