1 % Testing the JDK
   3 ## Using "make test" (the run-test framework)
   5 This new way of running tests is developer-centric. It assumes that you have
   6 built a JDK locally and want to test it. Running common test targets is simple,
   7 and more complex ad-hoc combination of tests is possible. The user interface is
   8 forgiving, and clearly report errors it cannot resolve.
  10 The main target `test` uses the jdk-image as the tested product. There is
  11 also an alternate target `exploded-test` that uses the exploded image
  12 instead. Not all tests will run successfully on the exploded image, but using
  13 this target can greatly improve rebuild times for certain workflows.
  15 Previously, `make test` was used invoke an old system for running test, and
  16 `make run-test` was used for the new test framework. For backward compatibility
  17 with scripts and muscle memory, `run-test` (and variants like
  18 `exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system
  19 can still be accessed for some time using `cd test && make`.
  21 Some example command-lines:
  23     $ make test-tier1
  24     $ make test-jdk_lang JTREG="JOBS=8"
  25     $ make test TEST=jdk_lang
  26     $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
  27     $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
  28     $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
  29     $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
  30     $ make exploded-test TEST=tier2
  32 ### Configuration
  34 To be able to run JTReg tests, `configure` needs to know where to find the
  35 JTReg test framework. If it is not picked up automatically by configure, use
  36 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
  37 Note that this option should point to the JTReg home, i.e. the top directory,
  38 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
  39 environment variable to point to the JTReg home before running `configure`.)
  41 To be able to run microbenchmarks, `configure` needs to know where to find
  42 the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
  43 containing the core JMH and transitive dependencies. The recommended dependencies 
  44 can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which 
  45 `--with-jmh=build/jmh/jars` should work.
  47 ## Test selection
  49 All functionality is available using the `test` make target. In this use case,
  50 the test or tests to be executed is controlled using the `TEST` variable. To
  51 speed up subsequent test runs with no source code changes, `test-only` can be
  52 used instead, which do not depend on the source and test image build.
  54 For some common top-level tests, direct make targets have been generated. This
  55 includes all JTReg test groups, the hotspot gtest, and custom tests (if
  56 present). This means that `make test-tier1` is equivalent to `make test
  57 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
  58 test runs, the `test TEST="x"` solution needs to be used.
  60 The test specifications given in `TEST` is parsed into fully qualified test
  61 descriptors, which clearly and unambigously show which tests will be run. As an
  62 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
  63 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
  64 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can
  65 always submit a list of fully qualified test descriptors in the `TEST` variable
  66 if you want to shortcut the parser.
  68 ### JTReg
  70 JTReg tests can be selected either by picking a JTReg test group, or a selection
  71 of files or directories containing JTReg tests.
  73 JTReg test groups can be specified either without a test root, e.g. `:tier1`
  74 (or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`,
  75 `test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test
  76 root can be specified either as an absolute path, or a path relative to the
  77 JDK top directory, or the `test` directory. For simplicity, the hotspot
  78 JTReg test root, which really is `hotspot/jtreg` can be abbreviated as
  79 just `hotspot`.
  81 When specified without a test root, all matching groups from all test roots
  82 will be added. Otherwise, only the group from the specified test root will be
  83 added.
  85 Individual JTReg tests or directories containing JTReg tests can also be
  86 specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or
  87 `hotspot/jtreg/native_sanity`. Just like for test root selection, you can
  88 either specify an absolute path (which can even point to JTReg tests outside
  89 the source tree), or a path relative to either the JDK top directory or the
  90 `test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as
  91 well.
  93 As long as the test groups or test paths can be uniquely resolved, you do not
  94 need to enter the `jtreg:` prefix. If this is not possible, or if you want to
  95 use a fully qualified test descriptor, add `jtreg:`, e.g.
  96 `jtreg:test/hotspot/jtreg/native_sanity`.
  98 ### Gtest
 100 Since the Hotspot Gtest suite is so quick, the default is to run all tests.
 101 This is specified by just `gtest`, or as a fully qualified test descriptor
 102 `gtest:all`.
 104 If you want, you can single out an individual test or a group of tests, for
 105 instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This
 106 can be particularly useful if you want to run a shaky test repeatedly.
 108 For Gtest, there is a separate test suite for each JVM variant. The JVM variant
 109 is defined by adding `/<variant>` to the test descriptor, e.g.
 110 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
 111 variant present (e.g. server, client). So if you only have the server JVM
 112 present, then `gtest:all` will be equivalent to `gtest:all/server`.
 114 ### Microbenchmarks
 116 Which microbenchmarks to run is selected using a regular expression
 117 following the `micro:` test descriptor, e.g., `micro:java.lang.reflect`. This
 118 delegates the test selection to JMH, meaning package name, class name and even
 119 benchmark method names can be used to select tests.
 121 Using special characters like `|` in the regular expression is possible, but
 122 needs to be escaped multiple times: `micro:ArrayCopy\\\\\|reflect`.
 124 ### Special tests
 126 A handful of odd tests that are not covered by any other testing framework are
 127 accessible using the `special:` test descriptor. Currently, this includes
 128 `failure-handler` and `make`.
 130   * Failure handler testing is run using `special:failure-handler` or just
 131     `failure-handler` as test descriptor.
 133   * Tests for the build system, including both makefiles and related
 134     functionality, is run using `special:make` or just `make` as test
 135     descriptor. This is equivalent to `special:make:all`.
 137     A specific make test can be run by supplying it as argument, e.g.
 138     `special:make:idea`. As a special syntax, this can also be expressed as
 139     `make-idea`, which allows for command lines as `make test-make-idea`.
 141 ## Test results and summary
 143 At the end of the test run, a summary of all tests run will be presented. This
 144 will have a consistent look, regardless of what test suites were used. This is
 145 a sample summary:
 147     ==============================
 148     Test summary
 149     ==============================
 150        TEST                                          TOTAL  PASS  FAIL ERROR
 151     >> jtreg:jdk/test:tier1                           1867  1865     2     0 <<
 152        jtreg:langtools/test:tier1                     4711  4711     0     0
 153        jtreg:nashorn/test:tier1                        133   133     0     0
 154     ==============================
 157 Tests where the number of TOTAL tests does not equal the number of PASSed tests
 158 will be considered a test failure. These are marked with the `>> ... <<` marker
 159 for easy identification.
 161 The classification of non-passed tests differs a bit between test suites. In
 162 the summary, ERROR is used as a catch-all for tests that neither passed nor are
 163 classified as failed by the framework. This might indicate test framework
 164 error, timeout or other problems.
 166 In case of test failures, `make test` will exit with a non-zero exit value.
 168 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
 169 where TEST_ID is a path-safe conversion from the fully qualified test
 170 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is
 171 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the
 172 test run.
 174 Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For
 175 some frameworks, this directory might contain information that is useful in
 176 determining the cause of a failed test.
 178 ## Test suite control
 180 It is possible to control various aspects of the test suites using make control
 181 variables.
 183 These variables use a keyword=value approach to allow multiple values to be
 184 set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTReg
 185 concurrency level to 1 and the timeout factor to 8. This is equivalent to
 186 setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that
 187 the `JTREG` variable is parsed and verified for correctness, so
 188 `JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just
 189 pass unnoticed.
 191 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
 192 normally eats `;`, the recommended usage is to write the assignment inside
 193 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
 194 as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
 196 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`.
 197 Also, as a special technique, the string `%20` will be replaced with space for
 198 certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
 199 This can be useful if you have layers of scripts and have trouble getting
 200 proper quoting of command line arguments through.)
 202 As far as possible, the names of the keywords have been standardized between
 203 test suites.
 205 ### JTReg keywords
 207 #### JOBS
 208 The test concurrency (`-concurrency`).
 210 Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
 211 JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but
 212 never more than 12.
 214 #### TIMEOUT
 215 The timeout factor (`-timeoutFactor`).
 217 Defaults to 4.
 219 #### TEST_MODE
 220 The test mode (`-agentvm`, `-samevm` or `-othervm`).
 222 Defaults to `-agentvm`.
 224 #### ASSERT
 225 Enable asserts (`-ea -esa`, or none).
 227 Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except
 228 for hotspot.
 230 #### VERBOSE
 231 The verbosity level (`-verbose`).
 233 Defaults to `fail,error,summary`.
 235 #### RETAIN
 236 What test data to retain (`-retain`).
 238 Defaults to `fail,error`.
 240 #### MAX_MEM
 241 Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none).
 243 Limit memory consumption for JTReg test framework and VM under test. Set to 0
 244 to disable the limits.
 246 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
 248 #### OPTIONS
 249 Additional options to the JTReg test framework.
 251 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
 253 #### JAVA_OPTIONS
 254 Additional Java options to JTReg (`-javaoption`).
 256 #### VM_OPTIONS
 257 Additional VM options to JTReg (`-vmoption`).
 259 ### Gtest keywords
 261 #### REPEAT
 262 The number of times to repeat the tests (`--gtest_repeat`).
 264 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
 265 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
 266 problem.
 268 #### OPTIONS
 269 Additional options to the Gtest test framework.
 271 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
 273 ### Microbenchmark keywords
 275 #### FORK
 276 Override the number of benchmark forks to spawn. Same as specifying `-f <num>`.
 278 #### ITER
 279 Number of measurement iterations per fork. Same as specifying `-i <num>`.
 281 #### TIME
 282 Amount of time to spend in each measurement iteration, in seconds. Same as
 283 specifying `-r <num>`
 285 #### WARMUP_ITER
 286 Number of warmup iterations to run before the measurement phase in each fork.
 287 Same as specifying `-wi <num>`.
 289 #### WARMUP_TIME
 290 Amount of time to spend in each warmup iteration. Same as specifying `-w <num>`.
 293 Specify to have the test run save a log of the values. Accepts the same values
 294 as `-rff`, i.e., `text`, `csv`, `scsv`, `json`, or `latex`.
 296 #### VM_OPTIONS
 297 Additional VM arguments to provide to forked off VMs. Same as `-jvmArgs <args>`
 299 #### OPTIONS
 300 Additional arguments to send to JMH.
 302 ---
 303 # Override some definitions in the global css file that are not optimal for
 304 # this document.
 305 header-includes:
 306  - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
 307 ---