(Also see the Randoop Developer's Manual.)
Writing tests is a difficult and time-consuming activity, and yet it is a crucial part of good software engineering. Randoop automatically generates unit tests for Java classes. For example, here is a JUnit test case generated by Randoop that reveals an error in Sun's JDK (comments added by hand):
// This test shows that the JDK collection classes
// can create an object that is not equal to itself.
public static void test1() {
LinkedList list = new LinkedList();
Object o1 = new Object();
list.addFirst(o1);
// A TreeSet is an ordered collection. According to Sun's
// API, this constructor call should throw a ClassCastException
// because the list element is not Comparable. But
// the constructor silently (and problematically) accepts the list.
TreeSet t1 = new TreeSet(list);
Set s1 = Collections.synchronizedSet(t1);
// At this point, we have successfully created a set (s1)
// that violations reflexivity of equality: it is not equal
// to itself! This assertion fails at runtime on Sun's JDK.
Assert.assertTrue(s1.equals(s1));
}
This tutorial will help you get started with Randoop by describing the kinds of tests it generates, its command-line interface and walking you examples of using the tool to generate unit tests.
The vision of our work is a testing system that at the push of a button creates a solid set of tests that provide good coverage of arbitrary classes, with no manual effort on your part. Randoop is a step in this direction and it has had many successful applications, especially with library classes (such as java.util). Randoop is used at companies like ABB and Microsoft, and on open-source projects.
Like any tool attempting to solve a challenging problem,
there are also many areas for improvement. Below, we will point out
potential limitations and opportunities for improvement for Randoop,
indicated with the light bulb image on the left. Also, take a look at
our ProjectIdeas page if
you are interested to contributing to Randoop or the field of
automated test generation.
You have three options:
Run Randoop by invoking its main class randoop.main.Main:
java -ea -classpath randoop.jar randoop.main.Main command args...
You don't need the -classpath argument if you add
randoop.jar to your classpath.
We recommend using the -ea (enable assertions)
option to java when running Randoop: if something goes wrong and you
send us an error report, we will be able better to help you diagnose
the cause of the problem.
Randoop supports two commands:
java -ea -classpath randoop.jar randoop.main.Main help java -ea -classpath randoop.jar randoop.main.Main help gentests
java -classpath randoop.jar randoop.main.Main gentests --testclass=java.util.TreeSet --timelimit=60
java.lang.reflect.Method#toString() matches the regular expression
given. This does not prevent indirect calls to such methods from
other, allowed methods. [no default]randoop.LiteralFileReader.
The special value "CLASSES" (with no quotes) means to read literals from all classes under test. [default []]java -javaagent:jarpath=args randoop.main.Main gentests --agent="-javaagent:jarpath=args"[no default]
There are three sources for getting help on Randoop:
java -classpath randoop.jar randoop.main.Main helpTo print out help for test generation (gentests command), type:
java -ea -classpath randoop.jar randoop.main.Main help gentestsTo also include unpublicized (due to being experimental or unsupported) options for a command, use the --unpub flag:
java -ea -classpath randoop.jar randoop.main.Main help --unpub gentests
If Randoop does not terminate, run it with the --log=filename command-line option, and either examine the log or send it to the Randoop developers for diagnosis.
You can also determine which calls are executing by examining the stack trace. On Unix this can be done by typing Ctrl-\ (hold down the Control key, then press the Backslash key).
WARNING: Testing code that modifies your file system might result in Randoop generating tests that modify your file system! Be careful when choosing classes and methods to test.
Randoop can be used to generate two kinds of unit tests. Error-revealing tests are tests that fail when executed, indicating a potential error in one or more classes under test. Regression tests are tests that pass when executed, and can be used to augment a regression test suite. The following sections describe error-revealing and regression tests in more detail and how you can use Randoop to generate them.
Imagine we want to generate tests for the class java.util.Collections, a utility class that defines several methods for manipulating collections.
The first important thing to keep in mind is that Randoop will only generate tests using classes you specify. In order to effectively test Collections, You should probably also specify some helper classes, including classes that generates collections. For this example, we will add java.util.TreeSet to the mix.
Invoke Randoop as follows (all in a single line):
java -classpath randoop.jar randoop.main.Main gentests --testclass=java.util.TreeSet --testclass=java.util.Collections --timelimit=60
Alternatively, you can create a file that lists the names of the classes under test, and use the --classlist option:
java -classpath randoop.jar randoop.main.Main gentests --classlist=myclasses.txt --timelimit=60
Where the contents of myclasses.txt are as follows.
java.util.Collections java.util.TreeSet
After 60 seconds, Randoop stops generating tests. The last thing Randoop prints out is the name of the JUnit files containing the tests it generated. You should see a message similar to the following:
Created file: my/home/directory/RandoopTest0.java Created file: my/home/directory/RandoopTest.java done.
The main test driver is in class RandoopTest. You can now compile and run the tests. Don't forget to include junit.jar in the classpath when you compile and run the tests. In this example, you should also add the current directory (".") when you run the tests.
javac -classpath randoop.jar RandoopTest*.java java -classpath .:randoop.jar junit.textui.TestRunner RandoopTest
JUnit will execute the generated tests. Most will pass, and some will fail. The next section describes the kinds of tests that Randoop generates, and the reason why they may fail.
Randoop creates unit tests by calling a set of methods and constructors under test. A method or constructor M is considered a member under test if:
--testclass or --classlist, or a
member via via --methodlist, AND
--omitmethods.
Additional notes:
A class list file used with --classlist consists of a
text file with the fully-qualified name of a class in each line; see
a class_list_example.txt file.
A method list file used with --methodlist consists of a
text file with the name of a method in each line. See
a method_list_example.txt file.
A constructor line begins with "cons :" followed by the
classname, the string <init> and the constructor's
parameter types, enclosed in parentheses. Methods are specified in a
similar way. The following example shows informally the form of a
constructor and method specification:
cons : Type0.<init>(Type1, Type2, ..., TypeN) method : Type0.method_name(Type1, Type2, ..., TypeN)
Type0, Type1,
..., TypeN must be fully-qualified (include package names).
Randoop does not currently support generics. Use raw types when specifying classes or methods.
When determining the set of members under test from the
--testclass, --classlist or
--methodlist options, non-public classes or members are
ignored.
Failing tests show a specific use of a class under test that leads to a runtime violation of a contract; a contracts is just a property that a class, object or method is expected to preserve, e.g. an object invariant or a method postcondition. Contract violations suggest errors in the code. For example, the test shown in the Introduction represents a violation of reflexivity of equality, an expected property of Java objects stating that for every object o, o.equals(o) should return true.
// This test shows that the JDK collection classes
// can create an object that is not equal to itself.
public static void test1() {
LinkedList list = new LinkedList();
Object o1 = new Object();
list.addFirst(o1);
// A TreeSet is an ordered collection. According to Sun's
// API, this constructor call should throw a ClassCastException
// because the list element is not Comparable. But
// the constructor silently (and problematically) accepts the list.
TreeSet t1 = new TreeSet(list);
Set s1 = Collections.synchronizedSet(t1);
// At this point, we have successfully created a set (s1)
// that violations reflexivity of equality: it is not equal
// to itself! This assertion fails at runtime on Sun's JDK.
Assert.assertTrue(s1.equals(s1));
}
The test shown above is nicely succinct: it shows a small number of method calls leading up to the assertion violation. Because Randoop's generation is random, the actual test revealing the error can be significantly longer, and contain many "random" calls leading up to the violation. An older version of Randoop used to have a minimizer built in, i.e. a module that attempted to reduce the number of calls in a failing test to ease debugging. The current version of the tool does not have a minimizer -- but we are hoping to implement it for the next release.
This test shows a scenario that makes it possible to create a TreeSet that violates reflexivity of equality. Test test reveals an error in the TreeSet constructor, which does not properly check its arguments (and possibly a second error in the TreeSet.equals method, which should always check if its argument is the same object as itself and return true in such a case).
Currently, Randoop checks for a small set of contracts:
The set of contracts that Randoop checks is relatively
small; we wanted to be conservative and only check for contracts whole
violation is highly likely to indicate an error. A future version of
the tool should let the user specify more contracts to check by
writing a special "contract-checking" class.
Most of the tests that Randoop generates will probably pass. They are regression tests: they record the current behavior of the classes under test. Regression tests are useful because they can alert you in the future if you make a change to your code that changes the external behavior of the classes. Here is an example of a regression test for the TreeSet? class we generated tests for in the previous section.
// This test passes when executed
public void test10() throws Throwable {
java.util.TreeSet var0 = new java.util.TreeSet();
java.lang.Short var1 = new java.lang.Short((short)100);
boolean var2 = var0.contains(var1);
// Regression assertion (captures the current behavior of the code)
assertTrue(var2 == false);
}This test will pass when you run it right after executing it. But notice that it captures an important behavior of the method TreeSet?.contains: it returns false if the set is empty. If later, as the developers of this class, we introduced an error that caused contains to return true on an empty set, the test would fail and thus alert us to the error.
To avoid Randoop calling certain methods in the code under test, use the --omitmethods=regexp command-line option. The rest of this section of the manual discusses how to cope with undesired calls by the code under test.
The code under test may make calls to certain methods that you do not want Randoop to execute. Two examples are System.exit (which will terminate not just the target program but also Randoop!) and methods that require user interaction, such as popping up a modal dialog box.
You can provide a different implementation of a method that will be used when running under Randoop. Randoop transforms the code under test, replacing each call to a undesired method by a call to a new implementation that you provide.
To transform calls to undesired methods, pass the following command-line option to java when running Randoop:
-javaagent:$(randoop)/randoop_agent.jar=--map_calls=map_calls.txt
(Note: If you supply the -javaagent option to java when running Randoop, and there is any chance that Randoop may spawn another process (for example, if the --compare-checks command-line option is specified), then then you should also specify the --agent argument to Randoop.)
You need to supply the map_calls.txt file, and also a definition of the replacement methods. When you run the generated tests, you will also need to supply the same -javaagent:... argument. (In the future, Randoop may provide a way to generate standalone tests even when generating tests using the -javaagent:... argument.)
The map_calls.txt file consists of sections, where each section has the format
class-regex old-method-name (arg-type, ...) new-class-name old-method-name (arg-type, ...) new-class-name ...
Each section gives a regular expression that matches class names, and one or more replacements that will be done in classes that match the regexp. The replacement is the fully-qualified method name, fully-qualified argument types, and a new class name. A call to the old method will be replaced by a call to a method with identical name and arguments in the new class.
Blank lines and // comments are ignored in the map_calls.txt file.
See an example map_calls.txt file and an example definition of replacement methods.
If Randoop is calling methods that it should not (for example, while generating tests, a dialog box pops up), then you need to add more calls to you map_calls file. While the dialog box is displayed, determine which calls are executing by examining the stack trace. On Unix this can be done by typing Ctrl-\ (hold down the Control key, then press the Backslash key).
To determine the specific method, examine the application's code where it calls the method that puts up the box (or other unwanted activity). If it is a virtual call, the map must specify the name of the type used in the call.
A representation invariant method checks that a class's internal representation satisfies expected properties. For example, a rep invariant method for a sorted list data structure may check that the elements in the list are indeed sorted; a rep invariant method for a balanced binary tree may check that the tree is balanced, etc.
You can tell Randoop that some methods in your classes under test are
rep invariant methods, and Randoop will call these methods on objects
of the appropriate type, outputting violations of the rep invariant as
failing unit tests. To tell Randoop that a method in a class under
test is a rep invariant method, add the @CheckRep
annotation to the method declaration (don't forget to import
randoop.*).
A method annotated with @CheckRep must have one of two
allowed signatures:
true as the rep invariant being satisfied, and a
return value of false or an exception escaping the
method as the rep invariant being violated.
By default, Randoop uses the following pool of primitive values as inputs to methods:
This is the initial set of seed values used by Randoop: in addition to the above values, any primitive value returned at runtime from a method call under test may be later used as input to another method call.
There are two ways of specifying additional seed values to Randoop.
Alternatively, you can simply add a @TestValue annotation to a field in one of your classes under test, as long as it adheres to the above requirements.
Below is an example helper class that demonstrates the use of the @TestValue annotation. For Randoop to use the primitive values described in the class, you would add this class to the list of classes under test (e.g. via the option --testclass=example.TestValueExamples).
package example;
import randoop.*;
public class TestValueExamples {
@TestValue
public static int i = 0;
@TestValue
public static boolean b = false;
@TestValue
public static byte by = 3;
@TestValue
public static char c = 'c';
@TestValue
public static long l = 3L;
@TestValue
public static float f = (float) 1.3;
@TestValue
public static double d = 1.4;
@TestValue
public static String s1 = null;
@TestValue
public static String s2 = "hi";
@TestValue
public static int[] a1 = new int[] { 1, 2, 3 };
}
If you're interested in increasing the number of tests that Randoop
generates in a given amount of time, you can try running Randoop with
the option --usethreads=false. This will cause Randoop to
use a single thread to execute all tests, and can speed up the tool by
an order of magnitude.
By default, Randoop executes generated tests in separate threads,
so that it can kill non-terminating threads that might otherwise
prevent Randoop from making progress. For this reason, if you
set --usethreads to false and Randoop
appears to hang, it may be due to a randomly-generated test that is
taking too long to execute and cannot be killed by Randoop.