Randoop Manual

(Also see the Randoop Developer's Manual.)

Introduction

Writing tests is a difficult and time-consuming activity, and yet it is a crucial part of good software engineering. Randoop automatically generates unit tests for Java classes. For example, here is a JUnit test case generated by Randoop that reveals an error in Sun's JDK (comments added by hand):


  // This test shows that the JDK collection classes
  // can create an object that is not equal to itself. 
  public static void test1() {

    LinkedList list = new LinkedList();
    Object o1 = new Object();
    list.addFirst(o1);

    // A TreeSet is an ordered collection. According to Sun's
    // API, this constructor call should throw a ClassCastException
    // because the list element is not Comparable. But
    // the constructor silently (and problematically) accepts the list.
    TreeSet t1 = new TreeSet(list);
  
    Set s1 = Collections.synchronizedSet(t1);
  
    // At this point, we have successfully created a set (s1)
    // that violations reflexivity of equality: it is not equal
    // to itself! This assertion fails at runtime on Sun's JDK.
    Assert.assertTrue(s1.equals(s1));
  }

This tutorial will help you get started with Randoop by describing the kinds of tests it generates, its command-line interface and walking you examples of using the tool to generate unit tests.

Vision, successes and challenges

The vision of our work is a testing system that at the push of a button creates a solid set of tests that provide good coverage of arbitrary classes, with no manual effort on your part. Randoop is a step in this direction and it has had many successful applications, especially with library classes (such as java.util). Randoop is used at companies like ABB and Microsoft, and on open-source projects.

lightbulb Like any tool attempting to solve a challenging problem, there are also many areas for improvement. Below, we will point out potential limitations and opportunities for improvement for Randoop, indicated with the light bulb image on the left. Also, take a look at our ProjectIdeas page if you are interested to contributing to Randoop or the field of automated test generation.

Getting Randoop

You have three options:

Running Randoop

Run Randoop by invoking its main class randoop.main.Main:

java -ea -classpath randoop.jar randoop.main.Main command args...

You don't need the -classpath argument if you add randoop.jar to your classpath. We recommend using the -ea (enable assertions) option to java when running Randoop: if something goes wrong and you send us an error report, we will be able better to help you diagnose the cause of the problem.

Randoop supports two commands:

Command-line options

[+] marked option can be specified multiple times

Getting help

There are three sources for getting help on Randoop:

Troubleshooting

Randoop does not terminate

If Randoop does not terminate, run it with the --log=filename command-line option, and either examine the log or send it to the Randoop developers for diagnosis.

You can also determine which calls are executing by examining the stack trace. On Unix this can be done by typing Ctrl-\ (hold down the Control key, then press the Backslash key).

Generating tests

WARNING: Testing code that modifies your file system might result in Randoop generating tests that modify your file system! Be careful when choosing classes and methods to test.

Randoop can be used to generate two kinds of unit tests. Error-revealing tests are tests that fail when executed, indicating a potential error in one or more classes under test. Regression tests are tests that pass when executed, and can be used to augment a regression test suite. The following sections describe error-revealing and regression tests in more detail and how you can use Randoop to generate them.

Imagine we want to generate tests for the class java.util.Collections, a utility class that defines several methods for manipulating collections.

The first important thing to keep in mind is that Randoop will only generate tests using classes you specify. In order to effectively test Collections, You should probably also specify some helper classes, including classes that generates collections. For this example, we will add java.util.TreeSet to the mix.

Invoke Randoop as follows (all in a single line):

java -classpath randoop.jar randoop.main.Main gentests --testclass=java.util.TreeSet --testclass=java.util.Collections --timelimit=60

Alternatively, you can create a file that lists the names of the classes under test, and use the --classlist option:

java -classpath randoop.jar randoop.main.Main gentests --classlist=myclasses.txt --timelimit=60

Where the contents of myclasses.txt are as follows.

java.util.Collections
java.util.TreeSet

After 60 seconds, Randoop stops generating tests. The last thing Randoop prints out is the name of the JUnit files containing the tests it generated. You should see a message similar to the following:

Created file: my/home/directory/RandoopTest0.java
Created file: my/home/directory/RandoopTest.java
done.

The main test driver is in class RandoopTest. You can now compile and run the tests. Don't forget to include junit.jar in the classpath when you compile and run the tests. In this example, you should also add the current directory (".") when you run the tests.

javac -classpath randoop.jar RandoopTest*.java
java -classpath .:randoop.jar junit.textui.TestRunner RandoopTest

JUnit will execute the generated tests. Most will pass, and some will fail. The next section describes the kinds of tests that Randoop generates, and the reason why they may fail.

Specifying methods and constructors under test

Randoop creates unit tests by calling a set of methods and constructors under test. A method or constructor M is considered a member under test if:

  1. M is declared in a public class via --testclass or --classlist, or a member via via --methodlist, AND
  2. M is public, has public return type and arguments, and does not match the pattern given via --omitmethods.

Additional notes:

Error-revealing Tests

Failing tests show a specific use of a class under test that leads to a runtime violation of a contract; a contracts is just a property that a class, object or method is expected to preserve, e.g. an object invariant or a method postcondition. Contract violations suggest errors in the code. For example, the test shown in the Introduction represents a violation of reflexivity of equality, an expected property of Java objects stating that for every object o, o.equals(o) should return true.

// This test shows that the JDK collection classes
// can create an object that is not equal to itself.
public static void test1() {

  LinkedList list = new LinkedList();
  Object o1 = new Object();
  list.addFirst(o1);

  // A TreeSet is an ordered collection. According to Sun's
  // API, this constructor call should throw a ClassCastException
  // because the list element is not Comparable. But
  // the constructor silently (and problematically) accepts the list.
  TreeSet t1 = new TreeSet(list);

  Set s1 = Collections.synchronizedSet(t1);

  // At this point, we have successfully created a set (s1)
  // that violations reflexivity of equality: it is not equal
  // to itself! This assertion fails at runtime on Sun's JDK.
  Assert.assertTrue(s1.equals(s1));
}

lightbulb The test shown above is nicely succinct: it shows a small number of method calls leading up to the assertion violation. Because Randoop's generation is random, the actual test revealing the error can be significantly longer, and contain many "random" calls leading up to the violation. An older version of Randoop used to have a minimizer built in, i.e. a module that attempted to reduce the number of calls in a failing test to ease debugging. The current version of the tool does not have a minimizer -- but we are hoping to implement it for the next release.

This test shows a scenario that makes it possible to create a TreeSet that violates reflexivity of equality. Test test reveals an error in the TreeSet constructor, which does not properly check its arguments (and possibly a second error in the TreeSet.equals method, which should always check if its argument is the same object as itself and return true in such a case).

What kinds of errors does Randoop check for?

Currently, Randoop checks for a small set of contracts:

lightbulb The set of contracts that Randoop checks is relatively small; we wanted to be conservative and only check for contracts whole violation is highly likely to indicate an error. A future version of the tool should let the user specify more contracts to check by writing a special "contract-checking" class.

Regression tests

Most of the tests that Randoop generates will probably pass. They are regression tests: they record the current behavior of the classes under test. Regression tests are useful because they can alert you in the future if you make a change to your code that changes the external behavior of the classes. Here is an example of a regression test for the TreeSet? class we generated tests for in the previous section.

// This test passes when executed
public void test10() throws Throwable {

  java.util.TreeSet var0 = new java.util.TreeSet();
  java.lang.Short var1 = new java.lang.Short((short)100);
  boolean var2 = var0.contains(var1);
    
  // Regression assertion (captures the current behavior of the code)
  assertTrue(var2 == false);
}

This test will pass when you run it right after executing it. But notice that it captures an important behavior of the method TreeSet?.contains: it returns false if the set is empty. If later, as the developers of this class, we introduced an error that caused contains to return true on an empty set, the test would fail and thus alert us to the error.

Customizing Randoop's behavior to your application

Avoiding calls to specific methods

To avoid Randoop calling certain methods in the code under test, use the --omitmethods=regexp command-line option. The rest of this section of the manual discusses how to cope with undesired calls by the code under test.

The code under test may make calls to certain methods that you do not want Randoop to execute. Two examples are System.exit (which will terminate not just the target program but also Randoop!) and methods that require user interaction, such as popping up a modal dialog box.

You can provide a different implementation of a method that will be used when running under Randoop. Randoop transforms the code under test, replacing each call to a undesired method by a call to a new implementation that you provide.

To transform calls to undesired methods, pass the following command-line option to java when running Randoop:

    -javaagent:$(randoop)/randoop_agent.jar=--map_calls=map_calls.txt

(Note: If you supply the -javaagent option to java when running Randoop, and there is any chance that Randoop may spawn another process (for example, if the --compare-checks command-line option is specified), then then you should also specify the --agent argument to Randoop.)

You need to supply the map_calls.txt file, and also a definition of the replacement methods. When you run the generated tests, you will also need to supply the same -javaagent:... argument. (In the future, Randoop may provide a way to generate standalone tests even when generating tests using the -javaagent:... argument.)

The map_calls.txt file consists of sections, where each section has the format

class-regex
  old-method-name (arg-type, ...) new-class-name
  old-method-name (arg-type, ...) new-class-name
  ...

Each section gives a regular expression that matches class names, and one or more replacements that will be done in classes that match the regexp. The replacement is the fully-qualified method name, fully-qualified argument types, and a new class name. A call to the old method will be replaced by a call to a method with identical name and arguments in the new class.

Blank lines and // comments are ignored in the map_calls.txt file.

See an example map_calls.txt file and an example definition of replacement methods.

Determining which calls to place in the map_calls file

If Randoop is calling methods that it should not (for example, while generating tests, a dialog box pops up), then you need to add more calls to you map_calls file. While the dialog box is displayed, determine which calls are executing by examining the stack trace. On Unix this can be done by typing Ctrl-\ (hold down the Control key, then press the Backslash key).

To determine the specific method, examine the application's code where it calls the method that puts up the box (or other unwanted activity). If it is a virtual call, the map must specify the name of the type used in the call.

Specifying representation invariant ("checkRep") methods

A representation invariant method checks that a class's internal representation satisfies expected properties. For example, a rep invariant method for a sorted list data structure may check that the elements in the list are indeed sorted; a rep invariant method for a balanced binary tree may check that the tree is balanced, etc.

You can tell Randoop that some methods in your classes under test are rep invariant methods, and Randoop will call these methods on objects of the appropriate type, outputting violations of the rep invariant as failing unit tests. To tell Randoop that a method in a class under test is a rep invariant method, add the @CheckRep annotation to the method declaration (don't forget to import randoop.*).

A method annotated with @CheckRep must have one of two allowed signatures:

Specifying additional primitive values

By default, Randoop uses the following pool of primitive values as inputs to methods:

This is the initial set of seed values used by Randoop: in addition to the above values, any primitive value returned at runtime from a method call under test may be later used as input to another method call.

There are two ways of specifying additional seed values to Randoop.

Speeding up test generation

If you're interested in increasing the number of tests that Randoop generates in a given amount of time, you can try running Randoop with the option --usethreads=false. This will cause Randoop to use a single thread to execute all tests, and can speed up the tool by an order of magnitude.

By default, Randoop executes generated tests in separate threads, so that it can kill non-terminating threads that might otherwise prevent Randoop from making progress. For this reason, if you set --usethreads to false and Randoop appears to hang, it may be due to a randomly-generated test that is taking too long to execute and cannot be killed by Randoop.