< prev index next >

doc/testing.md

Print this page




   6 built a JDK locally and want to test it. Running common test targets is simple,
   7 and more complex ad-hoc combination of tests is possible. The user interface is
   8 forgiving, and clearly report errors it cannot resolve.
   9 
  10 The main target `test` uses the jdk-image as the tested product. There is
  11 also an alternate target `exploded-test` that uses the exploded image
  12 instead. Not all tests will run successfully on the exploded image, but using
  13 this target can greatly improve rebuild times for certain workflows.
  14 
  15 Previously, `make test` was used to invoke an old system for running tests, and
  16 `make run-test` was used for the new test framework. For backward compatibility
  17 with scripts and muscle memory, `run-test` (and variants like
  18 `exploded-run-test` or `run-test-tier1`) are kept as aliases.
  19 
  20 Some example command-lines:
  21 
  22     $ make test-tier1
  23     $ make test-jdk_lang JTREG="JOBS=8"
  24     $ make test TEST=jdk_lang
  25     $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
  26     $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT_FACTOR=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
  27     $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
  28     $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
  29     $ make exploded-test TEST=tier2
  30 
  31 ### Configuration
  32 
  33 To be able to run JTReg tests, `configure` needs to know where to find the
  34 JTReg test framework. If it is not picked up automatically by configure, use
  35 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
  36 Note that this option should point to the JTReg home, i.e. the top directory,
  37 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
  38 environment variable to point to the JTReg home before running `configure`.)
  39 
  40 To be able to run microbenchmarks, `configure` needs to know where to find
  41 the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
  42 containing the core JMH and transitive dependencies. The recommended dependencies
  43 can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which
  44 `--with-jmh=build/jmh/jars` should work.
  45 
  46 ## Test selection


 163 error, timeout or other problems.
 164 
 165 In case of test failures, `make test` will exit with a non-zero exit value.
 166 
 167 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
 168 where TEST_ID is a path-safe conversion from the fully qualified test
 169 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is
 170 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the
 171 test run.
 172 
 173 Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For
 174 some frameworks, this directory might contain information that is useful in
 175 determining the cause of a failed test.
 176 
 177 ## Test suite control
 178 
 179 It is possible to control various aspects of the test suites using make control
 180 variables.
 181 
 182 These variables use a keyword=value approach to allow multiple values to be
 183 set. So, for instance, `JTREG="JOBS=1;TIMEOUT_FACTOR=8"` will set the JTReg
 184 concurrency level to 1 and the timeout factor to 8. This is equivalent to
 185 setting `JTREG_JOBS=1 JTREG_TIMEOUT_FACTOR=8`, but using the keyword format means that
 186 the `JTREG` variable is parsed and verified for correctness, so
 187 `JTREG="TMIEOUT_FACTOR=8"` would give an error, while `JTREG_TMIEOUT_FACTOR=8` would just
 188 pass unnoticed.
 189 
 190 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
 191 normally eats `;`, the recommended usage is to write the assignment inside
 192 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
 193 as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
 194 
 195 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT_FACTOR=8`.
 196 Also, as a special technique, the string `%20` will be replaced with space for
 197 certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
 198 This can be useful if you have layers of scripts and have trouble getting
 199 proper quoting of command line arguments through.)
 200 
 201 As far as possible, the names of the keywords have been standardized between
 202 test suites.
 203 
 204 ### General keywords (TEST_OPTS)
 205 
 206 Some keywords are valid across different test suites. If you want to run
 207 tests from multiple test suites, or just don't want to care which test suite specific
 208 control variable to use, then you can use the general TEST_OPTS control variable.
 209 
 210 There are also some keywords that applies globally to the test runner system,
 211 not to any specific test suites. These are also available as TEST_OPTS keywords.
 212 
 213 #### JOBS
 214 
 215 Currently only applies to JTReg.


 288 to disable the limits.
 289 
 290 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
 291 
 292 #### KEYWORDS
 293 
 294 JTReg kewords sent to JTReg using `-k`. Please be careful in making sure that
 295 spaces and special characters (like `!`) are properly quoted. To avoid some
 296 issues, the special value `%20` can be used instead of space.
 297 
 298 #### EXTRA_PROBLEM_LISTS
 299 
 300 Use additional problem lists file or files, in addition to the default
 301 ProblemList.txt located at the JTReg test roots.
 302 
 303 If multiple file names are specified, they should be separated by space (or, to
 304 help avoid quoting issues, the special value `%20`).
 305 
 306 The file names should be either absolute, or relative to the JTReg test root of
 307 the tests to be run.
 308 
 309 #### RUN_PROBLEM_LISTS
 310 
 311 Use the problem lists to select tests instead of excluding them.
 312 
 313 Set to `true` or `false`.
 314 If `true`, JTReg will use `-match:` option, otherwise `-exclude:` will be used.
 315 Default is `false`.
 316 
 317 
 318 #### OPTIONS
 319 Additional options to the JTReg test framework.
 320 
 321 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
 322 
 323 #### JAVA_OPTIONS
 324 Additional Java options to JTReg (`-javaoption`).
 325 
 326 #### VM_OPTIONS
 327 Additional VM options to JTReg (`-vmoption`).
 328 
 329 #### AOT_MODULES
 330 
 331 Generate AOT modules before testing for the specified module, or set of
 332 modules. If multiple modules are specified, they should be separated by space
 333 (or, to help avoid quoting issues, the special value `%20`).
 334 
 335 ### Gtest keywords




   6 built a JDK locally and want to test it. Running common test targets is simple,
   7 and more complex ad-hoc combination of tests is possible. The user interface is
   8 forgiving, and clearly report errors it cannot resolve.
   9 
  10 The main target `test` uses the jdk-image as the tested product. There is
  11 also an alternate target `exploded-test` that uses the exploded image
  12 instead. Not all tests will run successfully on the exploded image, but using
  13 this target can greatly improve rebuild times for certain workflows.
  14 
  15 Previously, `make test` was used to invoke an old system for running tests, and
  16 `make run-test` was used for the new test framework. For backward compatibility
  17 with scripts and muscle memory, `run-test` (and variants like
  18 `exploded-run-test` or `run-test-tier1`) are kept as aliases.
  19 
  20 Some example command-lines:
  21 
  22     $ make test-tier1
  23     $ make test-jdk_lang JTREG="JOBS=8"
  24     $ make test TEST=jdk_lang
  25     $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
  26     $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
  27     $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
  28     $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
  29     $ make exploded-test TEST=tier2
  30 
  31 ### Configuration
  32 
  33 To be able to run JTReg tests, `configure` needs to know where to find the
  34 JTReg test framework. If it is not picked up automatically by configure, use
  35 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
  36 Note that this option should point to the JTReg home, i.e. the top directory,
  37 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
  38 environment variable to point to the JTReg home before running `configure`.)
  39 
  40 To be able to run microbenchmarks, `configure` needs to know where to find
  41 the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
  42 containing the core JMH and transitive dependencies. The recommended dependencies
  43 can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which
  44 `--with-jmh=build/jmh/jars` should work.
  45 
  46 ## Test selection


 163 error, timeout or other problems.
 164 
 165 In case of test failures, `make test` will exit with a non-zero exit value.
 166 
 167 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
 168 where TEST_ID is a path-safe conversion from the fully qualified test
 169 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is
 170 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the
 171 test run.
 172 
 173 Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For
 174 some frameworks, this directory might contain information that is useful in
 175 determining the cause of a failed test.
 176 
 177 ## Test suite control
 178 
 179 It is possible to control various aspects of the test suites using make control
 180 variables.
 181 
 182 These variables use a keyword=value approach to allow multiple values to be
 183 set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTReg
 184 concurrency level to 1 and the timeout factor to 8. This is equivalent to
 185 setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that
 186 the `JTREG` variable is parsed and verified for correctness, so
 187 `JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just
 188 pass unnoticed.
 189 
 190 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
 191 normally eats `;`, the recommended usage is to write the assignment inside
 192 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
 193 as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
 194 
 195 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`.
 196 Also, as a special technique, the string `%20` will be replaced with space for
 197 certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
 198 This can be useful if you have layers of scripts and have trouble getting
 199 proper quoting of command line arguments through.)
 200 
 201 As far as possible, the names of the keywords have been standardized between
 202 test suites.
 203 
 204 ### General keywords (TEST_OPTS)
 205 
 206 Some keywords are valid across different test suites. If you want to run
 207 tests from multiple test suites, or just don't want to care which test suite specific
 208 control variable to use, then you can use the general TEST_OPTS control variable.
 209 
 210 There are also some keywords that applies globally to the test runner system,
 211 not to any specific test suites. These are also available as TEST_OPTS keywords.
 212 
 213 #### JOBS
 214 
 215 Currently only applies to JTReg.


 288 to disable the limits.
 289 
 290 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
 291 
 292 #### KEYWORDS
 293 
 294 JTReg kewords sent to JTReg using `-k`. Please be careful in making sure that
 295 spaces and special characters (like `!`) are properly quoted. To avoid some
 296 issues, the special value `%20` can be used instead of space.
 297 
 298 #### EXTRA_PROBLEM_LISTS
 299 
 300 Use additional problem lists file or files, in addition to the default
 301 ProblemList.txt located at the JTReg test roots.
 302 
 303 If multiple file names are specified, they should be separated by space (or, to
 304 help avoid quoting issues, the special value `%20`).
 305 
 306 The file names should be either absolute, or relative to the JTReg test root of
 307 the tests to be run.








 308 
 309 
 310 #### OPTIONS
 311 Additional options to the JTReg test framework.
 312 
 313 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
 314 
 315 #### JAVA_OPTIONS
 316 Additional Java options to JTReg (`-javaoption`).
 317 
 318 #### VM_OPTIONS
 319 Additional VM options to JTReg (`-vmoption`).
 320 
 321 #### AOT_MODULES
 322 
 323 Generate AOT modules before testing for the specified module, or set of
 324 modules. If multiple modules are specified, they should be separated by space
 325 (or, to help avoid quoting issues, the special value `%20`).
 326 
 327 ### Gtest keywords


< prev index next >