TestNG

… and improving use of JUnit

Dec 11, 2008

C. Gordon Huffman
 Agenda (not chronological)
• TestNG testing framework

• Improving use of JUnit

• Testing integrated classes vs. 1 class

2
 What is TestNG?
• Automated testing framework
• NG = Next Generation
• Similar to JUnit (especially JUnit 4)
• Not a JUnit extension (but inspired by JUnit)
• Designed to be better than JUnit, especially
when testing integrated classes
• Created by Dr. Cédric Beust (of Google)
• Open source (http://testng.org)

3
 History
• sUnit for Smalltalk (~1998, Kent Beck)
• JUnit (~2000, Kent Beck & Erich Gamma)
– Latest 3.x: 3.8.2 (March 2006)

• TestNG (~2004)
– Latest: 5.8 (March 2008)

• JUnit 4.x (2006)

– Latest: 4.5 (August 2008)
4
 TestNG philosophy
• Use more Java and OO features
• Feature-rich (JUnit: simplicity is valued)
• Support testing integrated classes (e.g., by
default, don’t create a new test class
instance for every test method).
• Separate compile-time test code from run-
time configuration/data info.

5
 Basic Testing Terms

Test class Class(es) under test

(“Production” class)
“CUT” (class under test)
“SUT” (system under test)

6
 “Rhythm” of an Automated Test
• Setup (most of it in a separate method)

• Exercise (call) production class(es)

• Assertion(s); true => silence; false => fail

• Teardown (if needed, in a separate method)

7
 Test class/method (JUnit 3 vs. 4)
// JUnit 3
import junit.framework.TestCase;
public class MyTestClass extends TestCase {
public void testSomething1() throws ... { ... }
public void testSomething2() throws ... { ... }
}

// JUnit 4
import org.junit.Test;
public class MyTestClass {
@Test public void aTest1() throws ... { ... }
@Test public void aTest2() throws ... { ... }
}

8
Test class/method (JUnit 4 vs. TestNG)
import org.junit.Test; // JUnit 4
... or
import org.testng.annotations.Test; // TestNG

public class MyTestClass {

@Test
public void aTestMethod() throws ... { ... }
}

9
 TestNG Assertions
import static org.testng.Assert.*;
import org.testng.annotations.Test;

public class MyTest {

@Test
public void myTestMethod() {
// ... Possibly some setup
// ... Call production method(s)
assertTrue(boolExpression);
// ... more assertions
}
}
10
 Assertions: TestNG vs. JUnit 3/4
• JUnit 3 assertions inherited by base TestCase
• JUnit 4: similar to TestNG
• TestNG assertEquals(…) assertion signatures
are different than JUnit’s:
– JUnit: [msg,] expected, actual
– TestNG: actual, expected [, msg]
• Can use JUnit 3/4 assertions (library issues)
• Can use TestNG’s copy of JUnit 3 assertions
(org.testng.AssertJUnit): migration aid.
• JUnit4/TestNG assertions richer than JUnit 3;
each contains features that the other doesn’t.
11
 Expected Exceptions
JUnit 4 (can simplify test code)
@Test(expected = AThrowable.class)

TestNG (can have more than one exception)

@Test(expectedExceptions = AThrowable.class)
Or ...
@Test(expectedExceptions = { T1.class, ... })

12
 TestNG “Groups”
• Each test method is tagged with any # of groups.
– @Test // no groups
– @Test (groups = “group1”)
– @Test (groups = { “g1”, “g2”, ... })
• A group therefore contains any # of test methods.
• Can also annotate test class this way; test method groups are union
of any explicit test method groups, and any of its test class.
• Groups can span classes.
• Groups can also be externally defined (TestNG xml configuration
file).
• A group is identified by a unique string (don’t use white space).
– There are no pre-defined group names.
– E.g., “slow”, “db”, “ui”, “unit”, “integration”, “broken.unknownReason”,
“check-in”, “week-end”, “functional.billing”

13
 TestNG Groups (continued)
• TestNG community suggests hierarchical
names from more general to less. E.g.:
– db.table.CUSTOMER
– state.broken.fix-in-progress
• Design group names so that you can
select them with prefix patterns.
• Groups complement other features (as
we’ll see).

14
 Setup/Teardown (JUnit 3/4)
JUnit 3
• protected void setUp() throws Exception { ... }
• protected void tearDown() throws Exception { ... }

JUnit 4 (also, @After*; annotation package: org.junit)

• @BeforeClass public static void before1() ...
• @Before public void before2() ...
• Multiple per class (order not defined)
• These 4 annotations are “inherited” (order is [kind of] defined).

15
 Setup/Teardown (TestNG)
@BeforeMethod
@BeforeClass (no need to be static)
@BeforeGroups ({“group1”, …})
@BeforeTest
@BeforeSuite

• And @After*

16
 TestNG run-time concepts
• A org.testng.TestNG instance runs tests.
– Invoked by IDE, ant, or on command line.
– Run params specified as setters, or in xml.

• Hierarchy of suite/test/class/method.

• A TestNG run executes 1 or more suites.

– Can be transparent, e.g., run 1 test method via Eclipse.

17
 testng.xml
• Controls test runs
• Optional
• File can have any name (.xml suffix advised)
– TestNG creates “testng-failed.xml”, for easy re-run.
• Root: <suite> (then <test>/<classes>/<method>)
• Suite can also point to a list of additional xml suite files.
• Some of the available specifications are described next.

18
 testng.xml (continued)
• Which potential test methods to run: suite/test: list of
java package name patterns to include/exclude; test: list
of fully qualified class names to include; within a class,
optional (default = include all) list of method names to
include/exclude; test: list of group name patterns to
include/exclude (and can define new groups for this file
with a list of existing group names).
• Parameter name/value pairs: Suite/test: (primitive
values) for @Parameters (test values override suite
values).
• JUnit semantics: test-level flag (default: false);
migration aid.

19
 Disable Tests
• TestNG
– @Test(enabled = false)
– Add to a group which is excluded
– Exclude in other ways in testng.xml

• JUnit 4: @Ignore, @Ignore(“reason”)

– Can also annotate test class.
– Runners report # ignored
20
 Test Timeouts
TestNG
• @Test(timeout = 1000)
• testng.xml <suite|test> time-out attribute

JUnit 4
• @Test(timeout = 1000)

21
 The Costs of Testing
• Time, money, schedule, production scope.
• Doubled (very roughly) code base.
– Maintenance (production and test).
• Sloppily written tests??
• Is test code quality as good as production?
• Slow tests?
• True buy in from business sponsors/owners,
project managers, user reps?

22
 “High View” of Test Benefits
• “High View”: using tests to best advantage
• “Do the thing right”
• “Do the right thing” (acceptance tests)
• Safety net for modifications/refactoring
• Defect localization (1 fail is more info than 100)
• Documentation/training
• Minimize costs of testing
• This view leads to better tests
23
 Integration vs. Unit Tests
• Unit test: tests 1 invocation of 1 method of 1
isolated class with 1 thread
• Integration test: tests more than 1 method
invocation of an isolated class, often much more,
or more than 1 thread
• Distinction: unit test vs. a test written in JUnit
• Examples
– Test hits db
– Life-cycle of an object (e.g., a customer order)
– Testing units by testing integrated classes

24
 Integration vs. Unit (continued)
• Should both be tested?
• Unit
– How big do you want the holes in your
refactoring safety net?
– How many permutations can integration tests
handle?
• Integration
– If the units work independently, will they work
together?

25
 If Unit Tests are Really Integration Tests

• Slow tests.
• False sense of security for refactoring safety net.
• Permutations of business rules likely not
covered.
• Poor defect isolation: 100 tests failing is less
information than 1 test failing.
• High maintenance for test/production
modification.
• Counter to the goals of TDD and Agility.
26
 Test Method Dependencies
• JUnit 3: Antithetical concept; each test method
gets its own test class instance; high isolation;
unit oriented by intentional design
• TestNG:
– @Test(dependsOnMethods = {“m1”, “m2”, ...})
– @Test(dependsOnGroups = {“g1”, “g2”, ...})
– E.g.: configuration dependencies; production lifecycle
• JUnit 4: tests ignored if “assumptions” (similar to
assertions) are violated: assumeTrue(bool),
assumeNoException(Throwable), assumeNotNull(refs
…), assumeThat(…) high-readability method

27
 Parallel (multiple threads)
@Test(threadPoolSize = #,
invocationCount = #,
timeout = #)
testng.xml:
– <suite|test
parallel =“tests|methods|none”
thread-count=“#” (5); 1 if not parallel
time-out =“#” ms (default: 0=none)
...>
• Testing production code with multiple threads
(unless not supported); stress testing; speeding up
tests! 28
 Test Method Parameters I
@Test
@Parameters({“name1”, “name2”})
public void aTest(String name1, int name2) {
...
}

• Values come from testng.xml (<test> is

checked first, then <suite>).
• Only basic types (String, [Bb]ool, #’s, …).
29
 Test Method Parameters II
@Test(dataProvider = “myDataProvider”)
public void aTestMethod(/* params! */) {
...
}

@DataProvider(name = “myDataProvider”)
public Object[][] getSomeTestData() {
...
}
• Runs (possibly multiple: first []) parameterized tests.
• Multiple parameters: second []
• Return type can also be Iterator<Object[]>
• Data provider itself can take 0, 1, or 2 args: Method, and test context

30
 Test Class Factories
@Factory
public static Object[] myTestFactory() {
// return instances of test classes
}

• Those instances will also be interrogated for @Factory

methods, and so on. (Each such method invoked once.)
• Example use: you want some of your test classes to
have non-standard constructors; the factory constructs
them.

31
 Migrating JUnit 3 => TestNG
• TestNG tool that modifies source code by
adding imports, @Test, @BeforeTest,
@AfterTest, and an optional list of desired group
names; tool accessible via command line, ant, or
Eclipse/IDEA.
• Can use testng.xml <test junit=“true”> for
tests that need further migration analysis.
• That’s not necessarily enough; for other
warnings and tips, see Appendix D of [TestNG].

32
 More TestNG and JUnit 4
• TestNG and JUnit 4 each have other
features not mentioned here.

• For more TestNG features, see [TestNG].

• For more JUnit 4 features, see JUnit 4

docs and the source code (e.g.,
Hamcrest).
33
 Reference
• [TestNG] Next Generation JavaTM Testing;
TestNG and Advanced Concepts; Cédric Beust and Hani
Suleiman; Addison-Wesley; ©2008

34