(Also see the Randoop Manual.)
Prerequisites for compiling and testing:
Under Unix:
make all, ORmake build under Randoop's home
directory (the directory containing this file).make clean first.Under Eclipse:
make tests under Randoop's home directory.Your classpath must include bin/ and lib/* (either the literal text “lib/*”, or explicitly listing all the .jar files under lib/*, should work).
To avoid repeatedly setting up the classpath, you can do either of the following:
RANDOOP_HOME ?= directory include $(RANDOOP_HOME)/common.mk run: java -ea randoop.main.Main help gentests
To modify the Randoop manual, edit directly the HTML files:
index.html has the user manual.dev.html has the developer manual with instructions
for hacking RandoopThen, run under Randoop's home directory:
% make manual
RANDOOP_VERSION in src/randoop/Globals.java).make clean followed by make all. Make sure all tests pass.hg tag v1.3.2).dist/ directory to the appropriate version number.This section describes Randoop's main concepts, data structures and internal algorithms. To this end, it helps to understand our model of unit tests.
At a general level, a unit test is just a snippet of code that checks some expected property of some classes under test. A more detailed description of a unit test (and one useful for the purpose of automatically generating tests) is as consisting of two things:
As a simple example, suppose we're testing the JDK's collections
classes (LinkedList, Collections, etc.) The
following unit test checks that the size of a linked list is computed
correctly after an element is added.
1. // Tests that the size of a list is 1 after adding an element.
2. public void testAdd() {
3. LinkedList list = new LinkedList();
4. boolean b = list.add(null);
5. assert list.size() == 1;
6. }
The test input consists of lines 3-4, which creates a list and calls the
add method, exercising the code of the list class under
test. The test check is in line 5.
In Randoop, test inputs are represented using Sequences, which describe a sequence of Statement}s (like the statements in lines 3-4 above) exercising the code under test, and test checks are represented using Checks.
Randoop's generation algorithms basically create collections of Sequences, add the appropriate Checks to them, and output the results to the user as unit tests.
Randoop's main data structure is the Sequence. A sequence describes the syntactic structure of a test input. In Randoop, all test inputs are sequences of operations that use the methods and constructors of the classes under test.
Continuing with our running example (testing the JDK collections classes), here is an example of a sequence of statements that calls methods from the classes under test, which each statement in a separate line:
LinkedList l = new LinkedList(); Object o = new Object(); l.addFirst(o); int i = l.size(); TreeSet t = new TreeSet(l); Set s = Collections.synchronizedSet(t);
Notice from the examples that each statement has three elements:
l,
o,
i,
t, and
s).Rearranging the above sequence, we can see these elements more clearly:
Result StatementKind Inputs
====== ============= ======
statement 0: l = new LinkedList ()
statement 1: o = new Object ()
statement 2: addFirst (l, o)
statement 3: i = size (l)
statement 4: t = new TreeSet (l)
statement 5: s = synchronizedSet (t)
The three elements of a statement map to the following classes in Randoop:
A StatementKind represents the kind of operation that the statement performs.
Randoop statements include:
Method class, and adds additional
functionality useful to Randoop.
null value (in code, int x =
1, String s = "s", Foo f = null,
etc.).
int[] x = new int[] { var0, var1
};).
s, the variable s.getVariable(i)
represents the value produced by the i-th statement in the sequence.
You can create sequences in a variety of ways. The first way is via extension: take a sequence and add a new statement at the bottom. Recall the example sequence:
LinkedList l = new LinkedList(); Object o = new Object(); l.addFirst(o); int i = l.size(); TreeSet t = new TreeSet(l); Set s = Collections.synchronizedSet(t);
Below is code that creates this sequence via extension.
// First, create the StatementKinds we'll need.
StatementKind newLL = RConstructor.getRConstructor(LinkedList.class.getConstructor());
StatementKind newOb = RConstructor.getRConstructor(Object.class.getConstructor());
StatementKind addFist = RMethod.getRMethod(LinkedList.class.getMethod("addFirst", Object.class));
StatementKind size = RMethod.getRMethod(LinkedList.class.getMethod("size"));
StatementKind newTS = RConstructor.getRConstructor(TreeSet.class.getConstructor(Collection.class));
StatementKind syncS = RMethod.getRMethod(Collections.class.getMethod("synchronizedSet", Set.class));
// Now, create the sequence by repeated extension.
Sequence s = new Sequence();
s = s.extend(newLL);
s = s.extend(newOb);
s = s.extend(addFist, s.getVariable(0), s.getVariable(1));
s = s.extend(size, s.getVariable(0));
s = s.extend(newTS, s.getVariable(0));
s = s.extend(syncS, s.getVariable(4));
A couple notable points:
extend operation
returns a new sequence rather than modifying its receiver.
A second way to create a sequence is using concatenation:
given sequences s1, s2 and s3,
you can create a new sequence that is the concatenation of the three
sequences.
List<Sequence> seqs = new ArrayList<Sequence>(); seqs.add(s1); seqs.add(s2); seqs.add(s3); Sequence newSeq = Sequence.concatenate(seqs);
If you're wondering why concatenation is useful, it is actually how
Randoop combines sequences to create new ones. To create a new
sequence that tests method m(A a, B b), Randoop first
finds and concatenates previously-generated sequences that create
objects of type A and B, and then extends
the concatenated sequence with a call of m. See
Section
2.2. of this paper for more details.
A third way to create a sequence is by parsing it from a String. For
example, given s string parseable with the following contents:
var0 = cons : java.util.LinkedList.<init>() : var1 = cons : java.lang.Object.<init>() : var2 = method : java.util.LinkedList.addFirst(java.lang.Object) : var0 var1 var3 = method : java.util.LinkedList.size() : var0 var4 = cons : java.util.TreeSet.<init>(java.util.Collection) : var0 var5 = method : java.util.Collections.synchronizedSet(java.util.Set) : var4
The following call creates a sequence corresponding to our running-example sequence:
Sequence seq = Sequence.parse(parseable);
An ExecutableSequence
wraps around a Sequence and adds two pieces of
functionality:
ExecutableSequence can be augmented with
checks of expected properties. A Check is
an object that represents some expected property of a sequence;
for a example, that a specific method call in the sequence returns
normally. When an executable sequence is executed, any checks that are
present in the sequence are checked at runtime, and the
passing/failing status of the checks is available for the client to
inspect.
ExecutableSequence can be executed.
Randoop uses Java's reflection mechanism to call the methods and
constructors in the sequence, and uses the structure of the
underlying Sequence to determine what inputs to pass to
them.
Suppose we have created a Sequence s The following two
lines will execute the sequence:
ExecutableSequence es = new ExecutableSequence(s); es.execute(null);
After the execute method returns, you can access the
runtime objects created during execution via the getResult(int
i) method, which returns the result of executing
statement i. For example, assuming s refers
to our running-example sequence from above, the following code prints
out
1, i.e. the return value of the call to
method LinkedList.size().
ExecutableSequence es = new ExecutableSequence(s);
es.execute(null);
// Assuming statement at index 3 returned normally, print the runtime value
ExecutionOutcome resultAt3 = es.getResult(3);
if (resultAt3 instanceof NormalExecution) {
System.out.println(((NormalExecution)resultAt3).getRuntimeValue());
}
To dig more into execution results, explore the classes Execution and ExecutionOutcome.
ExecutableSequence.executeStatement. This is
accomplished by changing stdout and stderr to a memory based
printsteam and recording the results.
Writing a list of sequences to a serialized file:
List<ExecutableSequence> sequences = ...;
FileOutputStream fileos = new FileOutputStream("MyFile.gz");
ObjectOutputStream objectos = new ObjectOutputStream(new GZIPOutputStream(fileos));
objectos.writeObject(sequences);
objectos.close();
fileos.close();
Any Checks decorating the sequences will also be serialized.
Reading a list of sequences from a serialized file:
FileInputStream fileis = new FileInputStream("MyFile.gz");
ObjectInputStream objectis = new ObjectInputStream(new GZIPInputStream(fileis));
List<ExecutableSequence> seqsfromfile = (List<ExecutableSequence>) objectis.readObject();
You can replay the execution of a deserialized sequence (including evaluation of its Checks) by executing it with a ReplayVisitor.
Writing a list of executed sequences as JUnit files:
List<ExecutableSequence> sequences = ...; JunitFileWriter jfw = new JunitFileWriter(output_dir, junit_package_name, junit_classname, testsperfile); jfw.createJunitFiles(seq);
If you want to modify Randoop's JUnit-generating code, here are laces to look:
randoop.JunitFileWriter figures out how many JUnit classes/files
to write, how many tests to put in each, what to name them, etc.
randoop.ExecutableSequence.toCodeString(),randoop.ExecutableSequence.toCodeString(),StatementKind.appendCode(...)A Check is an object that represents an expected property of a sequence. In terms of unit test concepts, a check represents some piece of checking code (or “test oracle”) of a unit test.
For example, consider the following unit test:
1. // Tests that the size of a list is 1 after adding an element.
2. public void testAdd() {
3. LinkedList l = new LinkedList(); // Create a list.
3. assertTrue(l.equals(l)); // List should be equal to itself.
4. Object o = new Object();
5. boolean b = l.add(o); // Add an element to it.
6. assertTrue(l.size() == 1); // List should have size 1.
7. assertTrue(l.equals(l)); // List should still be equal to itself.
8. int i = 10;
9. try {
10. Object o2 = l.remove(i); // Removing from invalid index
11. fail(); // should throw exception.
12. } catch (IndexOutOfBoundsException e) {
13. // expected exception.
14. }
15. }
Like a typical unit test, this test combines test input code
(the code creating a list, adding an element to it, removing an
element, etc.) with checking code (the code in lines 6-7
creating assertions, and the code in lines 9, 11-14 checking that the
correct exception is thrown). In Randoop, test input code is
represented as a Sequence and checking code is
represented with Checks associated with the sequence.
More specifically, each check is associated with an index in the sequence, where the index represents the time at which the check should be performed. For example, rearranging the above test to better see its structure, we have:
Checks performed
Statement after statement
========= ===============
l = new LinkedList()
l.equals(l) returns true.
o = new Object()
no checks.
b = l.add(o)
l.size() returns 1.
l.equals(l) returns true.
i = 10;
no checks.
o2 = l.remove(i)
throws IndexOutOfBoundsException.
The example illustrates how Randoop represents a test input along with its correctness checks.
Looking at the above example, you may reasonably ask: why do we draw a
distinction between “test input code” (the statements on the left) and
“checking code” (the statements on the right)? After all, aren't the
calls to size and equals in the checking
column also calls of the classes under test, and why not consider them
part of the test input?
We offer an answer in terms of how Randoop works. Randoop generates test inputs (Sequences) randomly, by combining and extending previously-generated sequences. On the other hand, Randoop performs its checks deterministically. For every sequence it creates, it performs checks on all the objects of the sequence.
Given how Randoop treats sequences and checks differently, it makes
more sense for a call like l.equals(l) to be expressed as
a Check, so that it is always
(not randomly) performed.
(Under construction)
handle is the
main GenTests
entrypoint for Randoop. (This is not strictly true, as Randoop's true
entrypoint is
class randoop.main.Main
But GenTests is where all the action starts with test
generation.) The handle method is long and mostly deals with
setting up things before the generation process, and doing things like
outputting tests after generation.