You are here

Model Based Testing with GraphWalker 2

I wrote prior to the release of GraphWalker 3. So keep in mind the not all information in this article is valid for GW3. I'll make an updated version soon.

Model-based testing is a new and evolving technique for generating a suite of test cases from requirements.

Testers using this approach concentrate on a data model and generation infrastructure instead of hand-crafting individual tests.

Model-based testing depends on three key technologies:

the notation used for the data model, the test-generation algorithm, and the tools that generate supporting infrastructure for the tests (including
expected outputs). Unlike the generation of test infrastructure, model notations and test generation algorithms are portable across projects.

The Figure shows gives an overview of the problem; it shows the data flows in a generic test-generation system.


Model Based Testing with GraphWalker 101

GraphWalker is a tool for generating test sequences from Finite State Machines and Extended Finite State Machines.

To understand the modeling; there are some simple rules to follow. File format should be designed using GraphML. GraphML is a comprehensive and easy-to-use
file format for graphs.

It consists of a language core to describe the structural properties of a graph and a flexible extension mechanism to add application-specific data. The
model is saved as a graphml file and the model has to be a directed graph, or digraph.

Extended FSM (EFSM)

EFSMs are like FSMs but more expressive (internalvariables encode more detailed state information)

In EFSM: one Enter state + timeouts variable to count nr of timeouts

It seems to have a small nr. of visible states: in reality a much larger nr. of internal states!

Mapping large set of internal states of an EFSM into the smaller set of visible states: abstraction

”An EFSM can model an SUT more accurately than an FSM, and its visible states define a 2nd layer of abstraction (an FSM) that drives test generation”

Source: M. Utting and B. Legeard, Practical Model-Based Testing

The two levels of abstractions give better control: used for different purpose:

Medium-size state space of EFSM (and code in transitions) used to model the SUT behavior more accurately and thus generate more precise inputs and oracles
for the SUT

- Smaller nr. of visible states of EFSM: defines an FSM used to drive test generation (eg, algorithm for transition tour)


Graphs

Every graph must have one (and only one) vertex with the label 'Start'. This shows MBT [4, 5] the starting point in the graph.

Vertex

A vertex is some kind of state that is possible to verify using some kind of oracle. A vertex must

always have a label. The label of a vertex is mapped against a method, function or sub routine during test execution. As a rule of thumb, the best practice
is to start the name with 'v_'.

The reason is that it is easier to recognize the function in the test execution tool, as a vertex

A vertex is some kind of state that is possible to verify using some kind of oracle.

A vertex must always have a label.

Every vertex in a graph must have a name.

The label of a vertex is mapped against a method, function or sub routine during test execution.

As a rule of thumb, the best practice is to start the name with 'v_'. The reason for this, is that it is easier to recognize the function in the test
execution tool, as a vertex. For example:

v_ApplicationStarted or v_BasicView

The same name of a vertex, can be re-used in the model.

Edge

The model in the graphwalker is built using a finite-state machine [FSM].A FSM (Finite-State Machine) is a model with vertices (nodes) and edges (arrows)
which connects the vertices with each other. A FSM model has no data domain, and no conditional paths to walk. There are certain set of standards which
graphwalker uses:

· A model consists of vertices and edges.

· A vertex represents a state in the system under test.

· A vertex must have a defined label.

· You can have multiple vertices with the same name.

· An edge represents the transition between 2 states.

· An edge can have a have a label, but it is not mandatory.

· A model must have one and only one vertex with the label Start.

· The Start vertex must have one and only one out-edge which has a mandatory label.

· The Start vertex must have no in-edges.

An edge is a transition. Could be a click on a button, a timeout, a server API call.

An edge can have a label, but it is not mandatory.

The label of an edge will map against a method, function or sub routine during test execution.

As a rule of thumb, the best practice is to start the name with 'e_'. The reason for this, is that it is easier to recognize the function in the test
execution tool, as an edge.

Example:

e_StartApplication or e_EnterBasicView

The same name of an edge, can be re-used in the model.

Label [cond. expr.] / Statement1;Statement2;

White spaces not allowed.

Follow the naming convention for the implementing programming language.

Edges usually have names, but are not required to have them. The name shall follow whatever naming convention is required of the test execution engine.

Labels

Label's represents the name of the vertex or edge.

Action/KeyWords

Define transitions of the EFSM (also sends test inputs to SUT and check answers)

Keywords are written within the label of a vertex or an edge.

They must exist on their own line in the label, but not the first line. yEd is another open source tool used to design the model.


Guard

Guard of the action method; actions with no guard defined have an implicit true guard.

The guard indicates if the edge is accessible in the current state of the machine. The language is either Java or JavaScript.

Most often, however, theorem provers are used to check the satisfiability of formulas that directly occur as guards of transitions in state-based models

Label [cond.expr.] / Statement1;Statement2;

Conditional expression that returns a Boolean value.

The guard indicates if the edge is accessible in the current state of the machine. The language is either Java or JavaScript.

Example:

Example:

e_Checkout [shoppingBasket.contains("Cerials")]


GraphWalker - Statements

Label [cond.expr.] / Statement1;Statement2;

Any applicable Java or JavaScript statements

Example:

e_Calculate / x=5;y++;

Example:

e_AddCerials / shoppingBasket.add("Cerials");

GraphWalker – Keyword Start

Used by Vertices only.

Each graph must have a start vertex, and that vertex holds this keyword. There is only one vertex in a graph, holding this keyword.

The Start vertex can only have one out-edge.

If the graph is the main-graph, that edge must have a label.

If the graph is a sub-graph, that edge cannot have a label.

GraphWalker - REQTAG

Used by Vertices

Used by GraphWalker for keeping track of which requirements are fulfilled during testing.

In the example above, if v_BookInfomation is passed, it would mean the requirement with tag UC01 2.2.3 has passed.

GraphWalker - Weight

Used by Edges only.

Used by MBT during random walks during test sequence generation. It holds a real value between 0 and 1, and represents the probability that a specific edge
should be chosen. A value of 0.05, would mean a 5% chance of that edge to be selected during a run.

Note: It only works with the RANDOM generator


GraphWalker – Keyword Blocked

Used by both Vertices and Edges.

A vertex or an edge with the key word BLOCKED, will be excluded from the model when walked.

Sometimes it can be useful, to mark vertices or edges as BLOCKED due to bugs in the system


GraphWalker use 101

· Generate a test sequence offline. The sequence is printed to the standard output

  • -a

o Prints the statistics of the test, at the end of the run.

o The file (or folder) containing graphml formattedfiles.

o The generator to be used when traversing the model. At least 1 generator must be given. To separate multiple generators, the separator pipe-character |
is used. A list of valid generators are:

§ RANDOM

§ A_STAR

§ SHORTEST_NON_OPTIMIZED

§ ALL_PATH_PERMUTATIONS

  • -j

o Enable JavaScript engine

o Optional report filename to save report to. (Alsorequires option -t)

o Defines the stop condition(s).

o Halts the generation after the specified stop-conditon(s) has been met. At least 1 condition must be given. If more than 1 is given, the condition that
meets it's stop-condition first, will cause the generation to halt. To separate multiple conditions, the separator pipe-character | is used. A list of
valid stop-conditions are:

§ NEVER

o Optional report template to use. (Also requires option -r)

  • -w,--weighted

o Use weighted values if they exist in the model, and the generator is RANDOM.

  • -x

o Use an extended finite state machine to handle the model.


Stopping a test with a stop criteria

When modeling is might not always be the best implementation of the model to force all the test to end in the same state. (But is it more manageable to
model with this thought in mind.)

Sometime the test need to stop when a specific condition arises.

A stop-condition, is a stop criteria for a given generator. It will halt the generation of a test sequence when the given criteria is reached.

GraphWalker support the following stop conditions.

REACHED_EDGE, REACHED_VERTEX, EDGE_COVERAGE, VERTEX_COVERAGE, TEST_LENGTH, TEST_DURATION, REACHED_REQUIREMENT, REQUIEMENT_COVERAGE and NEVER.

These stop conditions are used through the GraphWalker Command line:

Command-line reference for stop conditions, as from the GraphWalker website, with one or two notes.

REACHED EDGE

The stop criteria is a named edge. When, during execution, the edge is reached, the test is stopped. Make sute that the edge is reachable in the model. Or
exception is thrown by GraphWalker.

java -jar graphwalker.jar offline -f [file|folder] -g [generator] -s reached_edge:[Edge labe]

REACHED VERTEX

The stop criteria is a named vertex. When, during execution, the vertex is reached, the test is stopped.Make sute that the vertex is reachable in the
model. Or exception is thrown by GraphWalker.

java -jar graphwalker.jar offline -f [file|folder] -g [generator] -s reached_vertex:[Vertex label]

EDGE COVERAGE

The stop criteria is a percentage number. When, during execution, the percentage of traversed edges is reached, the test is stopped. If an edge is
traversed more than one time, it still counts as 1, when calculating the percentage coverage. In some models covering, getting a large percentage coverage,
is very difficult. Due to the complexity of the model. When working with this. You will get experience in what number of coverage, that can be applicable
for your project.And do note. That the test does not end in a predetermined state.

java -jar graphwalker.jar offline -f [file|folder] -g [generator] -s edge_coverage:[Coverage in %, between 1 and 100]

VERTEX COVERAGE

The stop criteria is a percentage number. When, during execution, the percentage of traversed states is reached, the test is stopped. If vertex is
traversed more than one time, it still counts as 1, when calculating the percentage coverage.In some models covering, getting a large percentage coverage,
is very difficult. Due to the complexity of the model. When working with this. You will get experience in what number of coverage, that can be applicable
for your project.And do note. That the test does not end in a predetermined state.

java -jar graphwalker.jar offline -f [file|folder] -g [generator] -s vertex_coverage:[Coverage in %, between 1 amd 100]

TEST LENGTH

The stop criteria is a number, representing the total numbers of pairs of vertices and edges generated by a generator. For example, if the TEST_LENGTH is
110, the test sequence would be 220 lines long. (including 110 pairs of edges and vertices). This too is dependant of the model. If the model is modelled
in a way which makes it hard to traverse recursively, then generating a large number of steps in the test, will be rare or impossible.And do note. That the
test does not end in a predetermined state.

java -jar graphwalker.jar offline -f [file|folder] -g [generator] -s test_length:[Number of edge and vertex pairs to execute]

TEST DURATION

The stop criteria is a time, representing the number of seconds that the test generator is allowed to execute.This too is dependant of the model. If the
model is modelled in a way which makes it hard to traverse recursively, then running the generation for a long sequence, will be rare or impossible. And do
note. That the test does not end in a predetermined state.

java -jar graphwalker.jar offline -f [file|folder] -g [generator] -s test_duration:[Time in seconds]

REQUIREMENT COVERAGE

The stop criteria is a percentage number. When, during execution, the percentage of traversed requirements is reached, the test is stopped. If an
requirement is traversed more than one time, it still counts as 1, when calculating the percentage coverage. This too is dependant of the model. If the
model is modelled in a way which makes it hard to traverse recursively, then generating a sequence covering a large number of the requirements, will be
rare or impossible.And do note. That the test does not end in a predetermined state.Again, mind how you model. the stop criteria, much be achievable.

java -jar graphwalker.jar offline -f [file|folder] -g [generator] -s requirement_coverage:[Coverage in %, between 1 and 100]

REACHED REQUIREMENT

The stop criteria is a named requirement. When the requirement is reached, the test is stopped. The same as "reached vertex" or "reached edge".Make sute
that the requirement is reachable in the model. Or exception is thrown by GraphWalker

java -jar graphwalker.jar offline -f [file|folder] -g [generator] -s reached_requirement:[Requirement id]

NEVER

This special stop critera will never halt the generator. I hve never used this. But I would not say, it is not usable.

java -jar graphwalker.jar offline -f [file|folder] -g [generator] -s never

Using the GraphWlaker help

java -jar graphwalker-2.6.3.jar help

mand.

Valid commands are: analyze, help, manual, merge, methods, offline, online, requirements, soap, source, xml

Type 'java -jar graphwalker.jar -v (--version)' for version information.