E2E (e2etest)

Central Script

The central script is e2e.sh.
The syntax for calling it can be obtained with e2e.sh -h

The script is integrated into the makefile as targets named "e2e" and "dry".

make e2e is the central command in the Gitlab pipeline to test the
artifacts. It must succeed for a deployment.

Test Suites

A complete test consists of a colletion of test suites.

There are test suits for basic tests, regressions, vocabularies and edge cases.

Test Suite Structure

A test suite is bundled in a directory named TS/[0-9][0-9]*.
Optionally there can be a file suite.inc per test suite. This shell
script is sourced in, allowing you to set environment variables. These
environment variables can be used to configure the behavior of e2e.sh.

SUITE_CHECKLEON=0|1 .... controls whether to compare the parser output
with a given .leon file. The default value is 1.
SUITE_BOOTLEO=0|1 ...... controls whether to source in the boot.leo
file. The default value is 0.
SUITE_WILDCARD=0|1 ..... controls whether to accept bash wildcards in
test*.out files. The default value is 0.
SUITE_VOCABULARY=voc ... controls whether to source in the vocabulary
called voc. voc can be a comma-separated list of names.
The default value is empty.
TESTxxx_EC_PARSER=0|y .. parser should honour test case xxx with
exit code y. Default is 0.
TESTxxx_EC_LEON=0|y .... leon should honour test case xxx with
exit code y. Default is 0.

Test Cases

Each test suite consists of multiple test cases. e2e.sh always runs
all test cases within a given test suite directory.

Each test case consists of two or three files:

test*.leo contains the input for Leonardus.
test*.leon (optional) contains the expected output of the parser pass
(if SUITE_CHECKLEON=1).
test*.out contains the expected output of the interpreter.

Comments in Test Cases

The first line of each .leo file should contain a comment. This comment
is used for progress reporting during an e2e.sh run.

Test Execution

Iterating Through Test Suites: First, e2e.sh iterates through all
TS/[0-9][0-9]* test suite directories provided on the command line.
Iterating Through Test Cases: Within each test suite, e2e.sh
then iterates through all individual test cases.
Executing Test Cases:
- The test*.leo file is executed as a LeoScript by the parser.
- The exit code is checked to be 0 or equal to the given TEST*_EC_PARSER value.
- The output is optionally compared to the test*.leon file (if it exists).
- The parser's output is executed as leon-format by leon.
- The exit code is checked to be 0 or equal to the given TEST*_EC_LEON value.
- Finally, the leon output is compared to the test*.out file.

BOOST Unit Tests (utest)

make generates a utest executable. It is used to test the system on
a class member and function level.

With build type PROFILE utest generates additional data for the lcov
coverage statistics.

The source code files are named UTest_*.cpp

The following Usage Variant is applied: "Static library usage variant".

A Classification of Test Scopes

Unit testing: the subject of testing is a single component of the software,
or a module or a class
Integration testing: the subject of testing are the interactions between
the modules of a system
System testing: the subject of testing is the software system as a whole
in order to see if it meets its requirements
Acceptance testing: performed for determining whether the customer should
accept the program or not
Regression testing: re-running tests after making changes to the software

A Classification of Unit Test Scopes

Scope	Unit Abbreviation	Boost.Test Level	C++ Terminology
method tests	MUTs - methods under test	boost test case	method, constructor, destructor, friend, ...
free function tests	FUTs - functions under test	boost test case	function
class tests	CUTs - classes under test	boost test suite	class, struct
module tests	MOUTs - modules under test	boost test suite	compilation unit, shared header file, collection of free functions, ...
library tests	LUTs - libraries under test	boost test module

The project Leonardus approach is: Use the level boost test suite and have
a single source file for every CUT and every MOUT.

Testing exit()

Testing functions that exit() the program needs a fork() approach.
There is a shared fixture SFxExit supporting this.

Special note: the used approach fails for member functions defined inline in
the class header file. It is neccessary to move the code to the corresponding
'*.cpp' file.

Command Line Tests (ctest)

The script Tools/ctest.sh tests command line options of Leonardus binaries
and all further features which can not not be tackled with a call of lb.

On one hand ctest.sh serves as component for the coverage tests
and on the other hand it is used stand alone to check features.

Examples (xtest)

A test run of all examples is part of the test stage in the pipelines.
One can execute this task manually with make runxtest.
See EXAMPLES

This category only tests the return codes of the example runs and ignores
the console output.

This categroy is not used by the the coverage tests.

Test Coverage of Functions, Lines and Branches

The central script is Tools/dolcov.sh.
The syntax for calling it can be obtained with Tools/dolcov.sh -h

The script is integrated into the makefile as target named lcov.

Coverage tests only runs with the build type PROFILE.

The command make lcov generates profile data by running

e2e.sh checks (e2etest)
BOOST unit tests (utest)
additional command line checks (ctest)

and using lcov to generate HTML documents in ./public/lcov.

A manual inspection of the HTML output is needed to assess the quality of the tests.

GitLab Coverage Integration

The script dolcov.sh additionally produces a console output
lines..... : xx.x% ( etc.. This string is parsed by the GitLab coverage
collecting feature.

GitLab accumulates the collected values and presents them as graph under
Analyze/Repository analytics/Code coverage statistics

Source Code Exclusion Markers

Following exclusion markers for lcov are in use. See man page of geninfo.

LCOV_EXCL_LINE - Lines containing this marker will be excluded.
LCOV_EXCL_START - Marks the beginning of an excluded section.
The current line is part of this section.
LCOV_EXCL_STOP - Marks the end of an excluded section. The current line
is not part of this section.