Randoop Developer's Manual

(Also see the Randoop Manual.)


Compiling, testing, releasing, etc.

Prerequisites for compiling and testing:

Compiling Randoop

Under Unix:

  1. To do everything (build and run all tests), type make all, OR
  2. From a command prompt, type make build under Randoop's home directory (the directory containing this file).
  3. If you want to rebuild from scratch, type make clean first.
  4. The generated classes are placed in bin/ directory.

Under Eclipse:

  1. If you are in Eclipse, Randoop compiles out of the box.
  2. The generated classes are also placed in bin/ directory.

Running unit and system tests

Randoop classpath

Your classpath must include bin/ and lib/* (either the literal text “lib/*”, or explicitly listing all the .jar files under lib/*, should work).

To avoid repeatedly setting up the classpath, you can do either of the following:

Modifying the manual

To modify the Randoop manual, edit directly the HTML files:

Then, run under Randoop's home directory:

% make manual

Making a new distribution

  1. Increase version number (modify variable RANDOOP_VERSION in src/randoop/Globals.java).
  2. Compile and test: run make clean followed by make all. Make sure all tests pass.
  3. Commit all changes. Tag the repository tip with the version number (e.g. hg tag v1.3.2).
  4. Rename the zip and jar files created under the dist/ directory to the appropriate version number.
  5. Upload the zip and jar files to google code.

Randoop Internals

This section describes Randoop's main concepts, data structures and internal algorithms. To this end, it helps to understand our model of unit tests.

Unit test concepts

At a general level, a unit test is just a snippet of code that checks some expected property of some classes under test. A more detailed description of a unit test (and one useful for the purpose of automatically generating tests) is as consisting of two things:

As a simple example, suppose we're testing the JDK's collections classes (LinkedList, Collections, etc.) The following unit test checks that the size of a linked list is computed correctly after an element is added.

1.   // Tests that the size of a list is 1 after adding an element.
2.   public void testAdd() {
3.      LinkedList list = new LinkedList();
4.      boolean b = list.add(null);
5.      assert list.size() == 1;
6.   }

The test input consists of lines 3-4, which creates a list and calls the add method, exercising the code of the list class under test. The test check is in line 5.

In Randoop, test inputs are represented using Sequences, which describe a sequence of Statement}s (like the statements in lines 3-4 above) exercising the code under test, and test checks are represented using Checks.

Randoop's generation algorithms basically create collections of Sequences, add the appropriate Checks to them, and output the results to the user as unit tests.

Sequences

Randoop's main data structure is the Sequence. A sequence describes the syntactic structure of a test input. In Randoop, all test inputs are sequences of operations that use the methods and constructors of the classes under test.

Continuing with our running example (testing the JDK collections classes), here is an example of a sequence of statements that calls methods from the classes under test, which each statement in a separate line:

  LinkedList l = new LinkedList();
  Object o = new Object();
  l.addFirst(o);
  int i = l.size();
  TreeSet t = new TreeSet(l);
  Set s = Collections.synchronizedSet(t);

Notice from the examples that each statement has three elements:

  1. A specific method (or constructor) that is being called.
  2. A value (primitive or object) returned by each call (l, o, i, t, and s).
  3. Inputs to the call, all of which come from values produced in earlier statements.

Rearranging the above sequence, we can see these elements more clearly:

                  Result      StatementKind        Inputs
                  ======      =============        ======
statement 0:      l       =   new LinkedList       ()
statement 1:      o       =   new Object           ()
statement 2:                  addFirst             (l, o)
statement 3:      i       =   size                 (l)
statement 4:      t       =   new TreeSet          (l)
statement 5:      s       =   synchronizedSet      (t)
The three elements of a statement map to the following classes in Randoop:

Creating sequences

You can create sequences in a variety of ways. The first way is via extension: take a sequence and add a new statement at the bottom. Recall the example sequence:

  LinkedList l = new LinkedList();
  Object o = new Object();
  l.addFirst(o);
  int i = l.size();
  TreeSet t = new TreeSet(l);
  Set s = Collections.synchronizedSet(t);

Below is code that creates this sequence via extension.


  // First, create the StatementKinds we'll need.
  StatementKind newLL = RConstructor.getRConstructor(LinkedList.class.getConstructor());
  StatementKind newOb = RConstructor.getRConstructor(Object.class.getConstructor());
  StatementKind addFist = RMethod.getRMethod(LinkedList.class.getMethod("addFirst", Object.class));
  StatementKind size = RMethod.getRMethod(LinkedList.class.getMethod("size"));
  StatementKind newTS = RConstructor.getRConstructor(TreeSet.class.getConstructor(Collection.class));
  StatementKind syncS = RMethod.getRMethod(Collections.class.getMethod("synchronizedSet", Set.class));
  
  // Now, create the sequence by repeated extension.
  Sequence s = new Sequence();
  s = s.extend(newLL);
  s = s.extend(newOb);
  s = s.extend(addFist, s.getVariable(0), s.getVariable(1));
  s = s.extend(size,    s.getVariable(0));
  s = s.extend(newTS,   s.getVariable(0));
  s = s.extend(syncS,   s.getVariable(4));

A couple notable points:

A second way to create a sequence is using concatenation: given sequences s1, s2 and s3, you can create a new sequence that is the concatenation of the three sequences.

List<Sequence> seqs = new ArrayList<Sequence>();
seqs.add(s1);
seqs.add(s2);
seqs.add(s3);
Sequence newSeq = Sequence.concatenate(seqs);

If you're wondering why concatenation is useful, it is actually how Randoop combines sequences to create new ones. To create a new sequence that tests method m(A a, B b), Randoop first finds and concatenates previously-generated sequences that create objects of type A and B, and then extends the concatenated sequence with a call of m. See Section 2.2. of this paper for more details.

A third way to create a sequence is by parsing it from a String. For example, given s string parseable with the following contents:

var0 =  cons : java.util.LinkedList.<init>() : 
var1 =  cons : java.lang.Object.<init>() : 
var2 =  method : java.util.LinkedList.addFirst(java.lang.Object) : var0 var1 
var3 =  method : java.util.LinkedList.size() : var0 
var4 =  cons : java.util.TreeSet.<init>(java.util.Collection) : var0 
var5 =  method : java.util.Collections.synchronizedSet(java.util.Set) : var4 

The following call creates a sequence corresponding to our running-example sequence:

Sequence seq = Sequence.parse(parseable);

Executable Sequences

An ExecutableSequence wraps around a Sequence and adds two pieces of functionality:

Executing a sequence

Suppose we have created a Sequence s The following two lines will execute the sequence:

ExecutableSequence es = new ExecutableSequence(s);
es.execute(null);

After the execute method returns, you can access the runtime objects created during execution via the getResult(int i) method, which returns the result of executing statement i. For example, assuming s refers to our running-example sequence from above, the following code prints out 1, i.e. the return value of the call to method LinkedList.size().

  
  ExecutableSequence es = new ExecutableSequence(s);
  es.execute(null);

  // Assuming statement at index 3 returned normally, print the runtime value
  ExecutionOutcome resultAt3 = es.getResult(3);
  if (resultAt3 instanceof NormalExecution) {
    System.out.println(((NormalExecution)resultAt3).getRuntimeValue());
  }

To dig more into execution results, explore the classes Execution and ExecutionOutcome.

Miscellaneous Notes

Writing/reading sequences to file

Writing/reading serialized sequences

Writing a list of sequences to a serialized file:

  List<ExecutableSequence> sequences = ...;
  FileOutputStream fileos = new FileOutputStream("MyFile.gz");
  ObjectOutputStream objectos = new ObjectOutputStream(new GZIPOutputStream(fileos));
  objectos.writeObject(sequences);
  objectos.close();
  fileos.close();

Any Checks decorating the sequences will also be serialized.

Reading a list of sequences from a serialized file:

  FileInputStream fileis = new FileInputStream("MyFile.gz");
  ObjectInputStream objectis = new ObjectInputStream(new GZIPInputStream(fileis));
  List<ExecutableSequence> seqsfromfile = (List<ExecutableSequence>) objectis.readObject();

You can replay the execution of a deserialized sequence (including evaluation of its Checks) by executing it with a ReplayVisitor.

Writing a sequence as a JUnit test

Writing a list of executed sequences as JUnit files:

  List<ExecutableSequence> sequences = ...;
  JunitFileWriter jfw =
  new JunitFileWriter(output_dir, junit_package_name, junit_classname, testsperfile);
  jfw.createJunitFiles(seq);

If you want to modify Randoop's JUnit-generating code, here are laces to look:

Checks

A Check is an object that represents an expected property of a sequence. In terms of unit test concepts, a check represents some piece of checking code (or “test oracle”) of a unit test.

For example, consider the following unit test:

 1.  // Tests that the size of a list is 1 after adding an element.
 2.  public void testAdd() {
 3.     LinkedList l = new LinkedList();         // Create a list.
 3.     assertTrue(l.equals(l));                 // List should be equal to itself.
 4.     Object o = new Object(); 
 5.     boolean b = l.add(o);                    // Add an element to it.
 6.     assertTrue(l.size() == 1);               // List should have size 1.
 7.     assertTrue(l.equals(l));                 // List should still be equal to itself.
 8.     int i = 10;                              
 9.     try {
10.       Object o2 = l.remove(i);               // Removing from invalid index
11.       fail();                                // should throw exception.
12.     } catch (IndexOutOfBoundsException e) {
13.       // expected exception.
14.     }
15.  }

Like a typical unit test, this test combines test input code (the code creating a list, adding an element to it, removing an element, etc.) with checking code (the code in lines 6-7 creating assertions, and the code in lines 9, 11-14 checking that the correct exception is thrown). In Randoop, test input code is represented as a Sequence and checking code is represented with Checks associated with the sequence.

More specifically, each check is associated with an index in the sequence, where the index represents the time at which the check should be performed. For example, rearranging the above test to better see its structure, we have:


                          Checks performed
Statement                 after statement
=========                 ===============
l  = new LinkedList()
                           l.equals(l) returns true.
o  = new Object()
                           no checks.
b  = l.add(o)
                           l.size() returns 1.
                           l.equals(l) returns true.
i = 10;
                           no checks.
o2 = l.remove(i)
                           throws IndexOutOfBoundsException.

The example illustrates how Randoop represents a test input along with its correctness checks.

Statements vs. Checks

Looking at the above example, you may reasonably ask: why do we draw a distinction between “test input code” (the statements on the left) and “checking code” (the statements on the right)? After all, aren't the calls to size and equals in the checking column also calls of the classes under test, and why not consider them part of the test input?

We offer an answer in terms of how Randoop works. Randoop generates test inputs (Sequences) randomly, by combining and extending previously-generated sequences. On the other hand, Randoop performs its checks deterministically. For every sequence it creates, it performs checks on all the objects of the sequence.

Given how Randoop treats sequences and checks differently, it makes more sense for a call like l.equals(l) to be expressed as a Check, so that it is always (not randomly) performed.

Executing Checks

(Under construction)

Main entry points

Test generator classes