SDL/visualtest/docs/latex/index.tex

\hypertarget{index_license_sec}{}\section{License}\label{index_license_sec}
Check the file {\ttfamily C\-O\-P\-Y\-I\-N\-G.\-txt} for licensing information.\hypertarget{index_intro_sec}{}\section{Introduction}\label{index_intro_sec}
The goal of this G\-So\-C project is to automate the testing of testsprite2. testsprite2 takes 26 parameters which have thousands of valid combinations and is used to validate S\-D\-L's window, mouse and rendering behaviour. By having a test harness that runs testsprite2 with various command line argument strings and validates the output for each run, we can make testing an easier task for maintainers, contributors and testers. The test harness can be used by a continuous integration system (like buildbot or jenkins) to validate S\-D\-L after checkins.

S\-D\-L Homepage\-: \href{http://libsdl.org/}{\tt http\-://libsdl.\-org/}\hypertarget{index_build_sec}{}\section{Building}\label{index_build_sec}
\hypertarget{index_build_linux}{}\subsection{Building on Linux/\-Cygwin}\label{index_build_linux}
{\ttfamily ./autogen.sh; ./configure; make;}\hypertarget{index_build_windows}{}\subsection{Building on Windows}\label{index_build_windows}
Use the Visual Studio solution under {\ttfamily S\-D\-L/\-Visual\-C/visualtest}.\hypertarget{index_docs_sec}{}\section{Documentation}\label{index_docs_sec}
Documentation is available via Doxygen. To build the documentation, cd to the S\-D\-L/visualtest/docs directory and run {\ttfamily doxygen}. A good starting point for exploring the documentation is {\ttfamily S\-D\-L/visualtest/docs/html/index.\-html} \hypertarget{index_usage_sec}{}\section{Usage}\label{index_usage_sec}
To see all the options supported by the test harness, just run {\ttfamily testharness} with no arguments.

At the moment the following options are supported\-: \begin{DoxyItemize}
\item {\ttfamily sutapp} -\/ Path to the system under test (S\-U\-T) application \item {\ttfamily sutargs} -\/ Launch the S\-U\-T with the specified arguments string \item {\ttfamily timeout} -\/ The maximum time after which the S\-U\-T process will be killed; passed as hh\-:mm\-:ss; default 00\-:01\-:00 \item {\ttfamily variator} -\/ Which variator to use; see \hyperlink{index_variators_sec}{Variators} \item {\ttfamily num-\/variations} -\/ The number of variations to run for; taken to be 1 for the random variator and A\-L\-L for the exhaustive variator by default \item {\ttfamily no-\/launch} -\/ Just print the arguments string for each variation without launching the S\-U\-T or performing any actions \item {\ttfamily parameter-\/config} -\/ A config file that describes the command line parameters supported by the S\-U\-T; see \hyperlink{index_paramconfig_sec}{The S\-U\-T Parameters File} or the sample $\ast$.parameters files for more details \item {\ttfamily action-\/config} -\/ A config file with a list of actions to be performed while the S\-U\-T is running; see \hyperlink{index_actionconfig_sec}{The Actions File} or the sample $\ast$.actions files \item {\ttfamily output-\/dir} -\/ Path to the directory where screenshots should be saved; is created if it doesn't exist; taken to be \char`\"{}./output\char`\"{} by default \item {\ttfamily verify-\/dir} -\/ Path to the directory with the verification images; taken to be \char`\"{}./verify\char`\"{} by default\end{DoxyItemize}
Paths can be relative or absolute.

Alternatively, the options can be passed as a config file for convenience\-:

{\ttfamily testharness -\/-\/config testsprite2\-\_\-sample.\-config}

For a sample, take a look at the $\ast$.config files in this repository.

We can also pass a config file and override certain options as necessary\-: {\ttfamily testharness -\/-\/config testsprite2\-\_\-sample.\-config -\/-\/num-\/variations 10}

Note\-: You may find it convenient to copy the S\-U\-T executable along with any resources to the test harness directory. Also note that testsprite2 and its resources (icon.\-bmp) are automatically copied when using the Visual Studio solution.\hypertarget{index_usageexamples_subsec}{}\subsection{Usage examples\-:}\label{index_usageexamples_subsec}
Passing a custom arguments string\-: {\ttfamily testharness -\/-\/sutapp testsprite2 -\/-\/sutargs \char`\"{}\textbackslash{}-\/\textbackslash{}-\/cyclecolor \textbackslash{}-\/\textbackslash{}-\/blend mod
\textbackslash{}-\/\textbackslash{}-\/iterations 2\char`\"{} -\/-\/action-\/config xyz.\-actions}

Using the random variator\-: {\ttfamily testharness -\/-\/sutapp testsprite2 -\/-\/variator random -\/-\/num-\/variations 5 -\/-\/parameter-\/config xyz.\-parameters -\/-\/action-\/config xyz.\-actions}\hypertarget{index_config_subsec}{}\subsection{Config Files}\label{index_config_subsec}
Config files are an alternate way to pass parameters to the test harness. We describe the paramters in a config file and pass that to the test harness using the -\/-\/config option. The config file consists of lines of the form \char`\"{}x=y\char`\"{} where x is an option and y is it's value. For boolean options, we simply give the name of the option to indicate that it is to be passed to the testharness.

The hash '\#' character can be used to start a comment from that point to the end of the line.\hypertarget{index_paramconfig_sec}{}\section{The S\-U\-T Parameters File}\label{index_paramconfig_sec}
To generate variations we need to describe the parameters the will be passed to the S\-U\-T. This description is given in a parameters file. Each line of the parameters file (except the blank lines) represents one command line option with five comma separated fields\-: {\ttfamily name, type, values, required, categories}

\begin{DoxyItemize}
\item {\ttfamily name} is the name of the option, e.\-g., {\ttfamily -\/-\/cyclecolor}. \item {\ttfamily type} can have one of three values -\/ integer, boolean and enum. \item {\ttfamily values} -\/ for integer options this is the valid range of values the option can take, i.\-e., \mbox{[}min max\mbox{]}. For enum options this is a list of strings that the option can take, e.\-g., \mbox{[}val1 val2 val3\mbox{]}. For boolean options this field is ignored. \item {\ttfamily required} -\/ true if the option is required, false otherwise. \item {\ttfamily categories} -\/ a list of categories that the option belongs to. For example, \mbox{[}video mouse audio\mbox{]}\end{DoxyItemize}
Just like with config files, hash characters can be used to start comments.\hypertarget{index_additionalnotes_subsec}{}\subsection{Additional Notes}\label{index_additionalnotes_subsec}
\begin{DoxyItemize}
\item If you want to have an option that always takes a certain value, use an enum with only one value. \item Currently there isn't any way to turn an option off, i.\-e., all options will be included in the command line options string that is generated using the config. If you don't want an option to be passed to the S\-U\-T, remove it from the config file or comment it out.\end{DoxyItemize}
\hypertarget{index_variators_sec}{}\section{Variators}\label{index_variators_sec}
Variators are the mechanism by which we generate strings of command line arguments to test the S\-U\-T with. A variator is quite simply an iterator that iterates through different variations of command line options. There are two variators supported at the moment\-: \begin{DoxyItemize}
\item {\bfseries Exhaustive} -\/ Generate all possible combinations of command line arguments that are valid. \item {\bfseries Random} -\/ Generate a random variation each time the variator is called.\end{DoxyItemize}
As an example, let's try a simple .parameters file\-:\par
 {\ttfamily  -\/-\/blend, enum, \mbox{[}add mod\mbox{]}, false, \mbox{[}\mbox{]} \par
 -\/-\/fullscreen, boolean, \mbox{[}\mbox{]}, false, \mbox{[}\mbox{]} }

The exhaustive variator would generate the following four variations\-:\par
 {\ttfamily  -\/-\/blend add \par
 -\/-\/blend mod \par
 -\/-\/blend add -\/-\/fullscreen \par
 -\/-\/blend mod -\/-\/fullscreen \par
 }

The random variator would simply generate a random variation like the following\-:\par
 {\ttfamily -\/-\/blend mod}\hypertarget{index_actionconfig_sec}{}\section{The Actions File}\label{index_actionconfig_sec}
Once the S\-U\-T process has been launched, automated testing happens using a mechanism called actions. A list of actions is read from a file and each action is performed on the S\-U\-T process sequentially. Each line in the actions file describes an action. The format for an action is {\ttfamily hh\-:mm\-:ss A\-C\-T\-I\-O\-N\-\_\-\-N\-A\-M\-E additional parameters}. There are five actions supported at the moment\-: \begin{DoxyItemize}
\item {\bfseries S\-C\-R\-E\-E\-N\-S\-H\-O\-T} -\/ Takes a screenshot of each window owned by the S\-U\-T process. The images are saved as {\ttfamily }\mbox{[}hash\mbox{]}\-\_\-\mbox{[}i\mbox{]}.bmp where {\ttfamily }\mbox{[}hash\mbox{]} is the 32 character long hexadecimal M\-D5 hash of the arguments string that was passed to the S\-U\-T while launching it and {\ttfamily i} is the window number. i = 1 is an exceptional case where the {\ttfamily \-\_\-}\mbox{[}i\mbox{]} is dropped and the filename is simply {\ttfamily }\mbox{[}hash\mbox{]}.bmp\par
 Note\-: The screenshots are only of the window's client area. \item {\bfseries V\-E\-R\-I\-F\-Y} -\/ Verifies the screenshots taken by the last S\-C\-R\-E\-E\-N\-S\-H\-O\-T action by comparing them against a verification image. Each {\ttfamily }\mbox{[}hash\mbox{]}\-\_\-i.\-bmp image output by the S\-C\-R\-E\-E\-N\-S\-H\-O\-T action is compared against a {\ttfamily }\mbox{[}hash\mbox{]}.bmp image in the verify-\/dir. \item {\bfseries Q\-U\-I\-T} -\/ Gracefully quits the S\-U\-T process. On Windows this means sending a W\-M\-\_\-\-C\-L\-O\-S\-E message to each window owned by the S\-U\-T process. On Linux it means sending a S\-I\-G\-Q\-U\-I\-T signal to the S\-U\-T process. \item {\bfseries K\-I\-L\-L} -\/ Forcefully kills the S\-U\-T process. This is useful when the S\-U\-T process doesn't respond to the Q\-U\-I\-T action. \item {\bfseries L\-A\-U\-N\-C\-H \mbox{[}/path/to/executable\mbox{]} \mbox{[}args\mbox{]}} -\/ Runs an executable with {\ttfamily }\mbox{[}args\mbox{]} as the arguments string.\end{DoxyItemize}
Just like with config files, hash characters can be used to start comments.\hypertarget{index_contint_sec}{}\section{Continuous Integration (\-C\-I)}\label{index_contint_sec}
One of the goals of the project was to create a test harness that integrates with C\-I systems to provide automated visual and interactive testing to S\-D\-L.

At the moment the test harness can be run in two modes that are useful for C\-I\-: \begin{DoxyItemize}
\item Crash testing mode -\/ launch the S\-U\-T with every variation and all parameters, report to the C\-I if there's a crash \item Visual testing mode -\/ launch and visually verify the S\-U\-T for a smaller subset of the parameters\end{DoxyItemize}
Look at the launch\-\_\-harness.\-sh/launch\-\_\-harness.cmd for an example scripts that run the test harness for all variations with all parameters and report an error on a crash. The script uses the testsprite2\-\_\-crashtest config, so remember to copy those files over to the test harness executable directory along with the script.\hypertarget{index_todo_sec}{}\section{T\-O\-D\-Os}\label{index_todo_sec}
\begin{DoxyItemize}
\item Allow specifying a clipping box along with the V\-E\-R\-I\-F\-Y action, i.\-e., hh\-:mm\-:ss V\-E\-R\-I\-F\-Y x, y, w, h \item Add support for spaces between the equals sign in test harness config files \item Implement the S\-C\-R\-E\-E\-N\-S\-H\-O\-T action on Linux \item Add a pairwise variator \item Add actions to inject keyboard/mouse events \item Add actions to manipulate the S\-U\-T window, e.\-g., minimize, restore, resize \item Add support to load and save screenshots as .pngs instead of .bmps\end{DoxyItemize}
\hypertarget{index_issues_sec}{}\section{Known Issues}\label{index_issues_sec}
\begin{DoxyItemize}
\item The Q\-U\-I\-T action does not work on a testsprite2 process with multiple windows. This appears to be an issue with testsprite2. \item The S\-C\-R\-E\-E\-N\-S\-H\-O\-T action doesn't capture the testsprite2 window correctly if the --fullscreen option is supplied. It works with --fullscreen-\/desktop, however.\end{DoxyItemize}
\hypertarget{index_moreinfo_sec}{}\section{More Information}\label{index_moreinfo_sec}
Author Contact Info\-:\par
 Apoorv Upreti {\ttfamily $<$\href{mailto:apoorvupreti@gmail.com}{\tt apoorvupreti@gmail.\-com}$>$} 

Other useful links\-:
\begin{DoxyItemize}
\item Project Repository\-: \href{https://bitbucket.org/nerdap/sdlvisualtest}{\tt https\-://bitbucket.\-org/nerdap/sdlvisualtest}
\item Project Wiki\-: \href{https://github.com/nerdap/autotestsprite2/wiki}{\tt https\-://github.\-com/nerdap/autotestsprite2/wiki}
\item Project Blog\-: \href{http://nerdap.github.io}{\tt http\-://nerdap.\-github.\-io}
\item Verification images for testsprite2\-\_\-blendmodes\-: \href{https://www.dropbox.com/s/nm02aem76m812ng/testsprite2_blendmodes.zip}{\tt https\-://www.\-dropbox.\-com/s/nm02aem76m812ng/testsprite2\-\_\-blendmodes.\-zip}
\item Verification images for testsprite2\-\_\-geometry\-: \href{https://www.dropbox.com/s/csypwryopaslpaf/testsprite2_geometry.zip}{\tt https\-://www.\-dropbox.\-com/s/csypwryopaslpaf/testsprite2\-\_\-geometry.\-zip} 
\end{DoxyItemize}