< prev index next >

doc/testing.md

Print this page


   1 % Testing the JDK
   2 
   3 ## Using "make test" (the run-test framework)
   4 
   5 This new way of running tests is developer-centric. It assumes that you have
   6 built a JDK locally and want to test it. Running common test targets is simple,
   7 and more complex ad-hoc combination of tests is possible. The user interface is
   8 forgiving, and clearly report errors it cannot resolve.
   9 
  10 The main target `test` uses the jdk-image as the tested product. There is
  11 also an alternate target `exploded-test` that uses the exploded image
  12 instead. Not all tests will run successfully on the exploded image, but using
  13 this target can greatly improve rebuild times for certain workflows.
  14 
  15 Previously, `make test` was used to invoke an old system for running tests, and
  16 `make run-test` was used for the new test framework. For backward compatibility
  17 with scripts and muscle memory, `run-test` (and variants like
  18 `exploded-run-test` or `run-test-tier1`) are kept as aliases.

  19 
  20 Some example command-lines:
  21 
  22     $ make test-tier1
  23     $ make test-jdk_lang JTREG="JOBS=8"
  24     $ make test TEST=jdk_lang
  25     $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
  26     $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
  27     $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
  28     $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
  29     $ make exploded-test TEST=tier2
  30 
  31 ### Configuration
  32 
  33 To be able to run JTReg tests, `configure` needs to know where to find the
  34 JTReg test framework. If it is not picked up automatically by configure, use
  35 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
  36 Note that this option should point to the JTReg home, i.e. the top directory,
  37 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
  38 environment variable to point to the JTReg home before running `configure`.)
  39 
  40 To be able to run microbenchmarks, `configure` needs to know where to find
  41 the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
  42 containing the core JMH and transitive dependencies. The recommended dependencies
  43 can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which
  44 `--with-jmh=build/jmh/jars` should work.
  45 
  46 ## Test selection
  47 
  48 All functionality is available using the `test` make target. In this use case,
  49 the test or tests to be executed is controlled using the `TEST` variable. To
  50 speed up subsequent test runs with no source code changes, `test-only` can be
  51 used instead, which do not depend on the source and test image build.
  52 
  53 For some common top-level tests, direct make targets have been generated. This
  54 includes all JTReg test groups, the hotspot gtest, and custom tests (if
  55 present). This means that `make test-tier1` is equivalent to `make test
  56 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
  57 test runs, the `test TEST="x"` solution needs to be used.
  58 
  59 The test specifications given in `TEST` is parsed into fully qualified test
  60 descriptors, which clearly and unambigously show which tests will be run. As an
  61 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
  62 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
  63 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can


 184 concurrency level to 1 and the timeout factor to 8. This is equivalent to
 185 setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that
 186 the `JTREG` variable is parsed and verified for correctness, so
 187 `JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just
 188 pass unnoticed.
 189 
 190 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
 191 normally eats `;`, the recommended usage is to write the assignment inside
 192 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
 193 as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
 194 
 195 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`.
 196 Also, as a special technique, the string `%20` will be replaced with space for
 197 certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
 198 This can be useful if you have layers of scripts and have trouble getting
 199 proper quoting of command line arguments through.)
 200 
 201 As far as possible, the names of the keywords have been standardized between
 202 test suites.
 203 
 204 ### General keywords (TEST_OPTS)
 205 
 206 Some keywords are valid across different test suites. If you want to run
 207 tests from multiple test suites, or just don't want to care which test suite specific
 208 control variable to use, then you can use the general TEST_OPTS control variable.
 209 
 210 There are also some keywords that applies globally to the test runner system,
 211 not to any specific test suites. These are also available as TEST_OPTS keywords.
 212 
 213 #### JOBS
 214 
 215 Currently only applies to JTReg.
 216 
 217 #### TIMEOUT_FACTOR
 218 
 219 Currently only applies to JTReg.
 220 
 221 #### VM_OPTIONS
 222 
 223 Applies to JTReg, GTest and Micro.
 224 
 225 #### JAVA_OPTIONS
 226 
 227 Applies to JTReg, GTest and Micro.
 228 
 229 #### AOT_MODULES
 230 
 231 Applies to JTReg and GTest.
 232 
 233 #### JCOV
 234 
 235 This keywords applies globally to the test runner system. If set to `true`, it
 236 enables JCov coverage reporting for all tests run. To be useful, the JDK under
 237 test must be run with a JDK built with JCov instrumentation (`configure
 238 --with-jcov=<path to directory containing lib/jcov.jar>`, `make jcov-image`).
 239 
 240 The simplest way to run tests with JCov coverage report is to use the special
 241 target `jcov-test` instead of `test`, e.g. `make jcov-test TEST=jdk_lang`. This
 242 will make sure the JCov image is built, and that JCov reporting is enabled.
 243 
 244 The JCov report is stored in `build/$BUILD/test-results/jcov-output`.
 245 
 246 Please note that running with JCov reporting can be very memory intensive.
 247 
 248 ### JTReg keywords
 249 
 250 #### JOBS
 251 The test concurrency (`-concurrency`).
 252 
 253 Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
 254 JOBS, except for Hotspot, where the default is *number of CPU cores/2* (for
 255 sparc, if more than 16 cpus, then *number of CPU cores/5*, otherwise *number of
 256 CPU cores/4*), but never more than *memory size in GB/2*.
 257 
 258 #### TIMEOUT_FACTOR
 259 The timeout factor (`-timeoutFactor`).
 260 
 261 Defaults to 4.
 262 
 263 #### TEST_MODE
 264 The test mode (`-agentvm`, `-samevm` or `-othervm`).
 265 
 266 Defaults to `-agentvm`.
 267 
 268 #### ASSERT
 269 Enable asserts (`-ea -esa`, or none).
 270 
 271 Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except
 272 for hotspot.
 273 
 274 #### VERBOSE
 275 The verbosity level (`-verbose`).
 276 
 277 Defaults to `fail,error,summary`.
 278 
 279 #### RETAIN
 280 What test data to retain (`-retain`).
 281 
 282 Defaults to `fail,error`.
 283 
 284 #### MAX_MEM
 285 Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none).
 286 
 287 Limit memory consumption for JTReg test framework and VM under test. Set to 0
 288 to disable the limits.
 289 
 290 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
 291 
 292 #### KEYWORDS
 293 
 294 JTReg kewords sent to JTReg using `-k`. Please be careful in making sure that
 295 spaces and special characters (like `!`) are properly quoted. To avoid some
 296 issues, the special value `%20` can be used instead of space.
 297 
 298 #### EXTRA_PROBLEM_LISTS
 299 
 300 Use additional problem lists file or files, in addition to the default
 301 ProblemList.txt located at the JTReg test roots.
 302 
 303 If multiple file names are specified, they should be separated by space (or, to
 304 help avoid quoting issues, the special value `%20`).
 305 
 306 The file names should be either absolute, or relative to the JTReg test root of
 307 the tests to be run.
 308 
 309 
 310 #### OPTIONS
 311 Additional options to the JTReg test framework.
 312 
 313 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
 314 
 315 #### JAVA_OPTIONS
 316 Additional Java options to JTReg (`-javaoption`).
 317 
 318 #### VM_OPTIONS
 319 Additional VM options to JTReg (`-vmoption`).
 320 
 321 #### AOT_MODULES
 322 
 323 Generate AOT modules before testing for the specified module, or set of
 324 modules. If multiple modules are specified, they should be separated by space
 325 (or, to help avoid quoting issues, the special value `%20`).
 326 
 327 ### Gtest keywords
 328 
 329 #### REPEAT
 330 The number of times to repeat the tests (`--gtest_repeat`).
 331 
 332 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
 333 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
 334 problem.
 335 
 336 #### OPTIONS
 337 Additional options to the Gtest test framework.
 338 
 339 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
 340 
 341 #### AOT_MODULES
 342 
 343 Generate AOT modules before testing for the specified module, or set of
 344 modules. If multiple modules are specified, they should be separated by space
 345 (or, to help avoid quoting issues, the special value `%20`).
 346 
 347 ### Microbenchmark keywords
 348 
 349 #### FORK
 350 Override the number of benchmark forks to spawn. Same as specifying `-f <num>`.
 351 
 352 #### ITER
 353 Number of measurement iterations per fork. Same as specifying `-i <num>`.
 354 
 355 #### TIME
 356 Amount of time to spend in each measurement iteration, in seconds. Same as
 357 specifying `-r <num>`
 358 
 359 #### WARMUP_ITER
 360 Number of warmup iterations to run before the measurement phase in each fork.
 361 Same as specifying `-wi <num>`.
 362 
 363 #### WARMUP_TIME
 364 Amount of time to spend in each warmup iteration. Same as specifying `-w <num>`.
 365 
 366 #### RESULTS_FORMAT
 367 Specify to have the test run save a log of the values. Accepts the same values
 368 as `-rff`, i.e., `text`, `csv`, `scsv`, `json`, or `latex`.
 369 
 370 #### VM_OPTIONS
 371 Additional VM arguments to provide to forked off VMs. Same as `-jvmArgs <args>`
 372 
 373 #### OPTIONS
 374 Additional arguments to send to JMH.
 375 
 376 ## Notes for Specific Tests
 377 
 378 ### Docker Tests
 379 
 380 Docker tests with default parameters may fail on systems with glibc versions not
 381 compatible with the one used in the default docker image (e.g., Oracle Linux 7.6 for x86).
 382 For example, they pass on Ubuntu 16.04 but fail on Ubuntu 18.04 if run like this on x86:
 383 
 384     $ make test TEST="jtreg:test/hotspot/jtreg/containers/docker"
 385 
 386 To run these tests correctly, additional parameters for the correct docker image are
 387 required on Ubuntu 18.04 by using `JAVA_OPTIONS`.
 388 
 389     $ make test TEST="jtreg:test/hotspot/jtreg/containers/docker" JTREG="JAVA_OPTIONS=-Djdk.test.docker.image.name=ubuntu -Djdk.test.docker.image.version=latest"
 390 
 391 ### Non-US locale
 392 
 393 If your locale is non-US, some tests are likely to fail. To work around this you can
 394 set the locale to US. On Unix platforms simply setting `LANG="en_US"` in the
 395 environment before running tests should work. On Windows, setting
 396 `JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US"` helps for most, but not all test cases.
 397 For example:
 398 
 399     $ export LANG="en_US" && make test TEST=...
 400     $ make test JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US" TEST=...
 401 
 402 ---
 403 # Override some definitions in the global css file that are not optimal for
 404 # this document.
 405 header-includes:
 406  - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
 407 ---
   1 % Testing the JDK
   2 
   3 ## Using "make test" (the run-test framework)
   4 
   5 This new way of running tests is developer-centric. It assumes that you have
   6 built a JDK locally and want to test it. Running common test targets is simple,
   7 and more complex ad-hoc combination of tests is possible. The user interface is
   8 forgiving, and clearly report errors it cannot resolve.
   9 
  10 The main target `test` uses the jdk-image as the tested product. There is
  11 also an alternate target `exploded-test` that uses the exploded image
  12 instead. Not all tests will run successfully on the exploded image, but using
  13 this target can greatly improve rebuild times for certain workflows.
  14 
  15 Previously, `make test` was used invoke an old system for running test, and
  16 `make run-test` was used for the new test framework. For backward compatibility
  17 with scripts and muscle memory, `run-test` (and variants like
  18 `exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system
  19 can still be accessed for some time using `cd test && make`.
  20 
  21 Some example command-lines:
  22 
  23     $ make test-tier1
  24     $ make test-jdk_lang JTREG="JOBS=8"
  25     $ make test TEST=jdk_lang
  26     $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
  27     $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
  28     $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
  29     $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
  30     $ make exploded-test TEST=tier2
  31 
  32 ### Configuration
  33 
  34 To be able to run JTReg tests, `configure` needs to know where to find the
  35 JTReg test framework. If it is not picked up automatically by configure, use
  36 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
  37 Note that this option should point to the JTReg home, i.e. the top directory,
  38 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
  39 environment variable to point to the JTReg home before running `configure`.)
  40 
  41 To be able to run microbenchmarks, `configure` needs to know where to find
  42 the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
  43 containing the core JMH and transitive dependencies. The recommended dependencies 
  44 can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which 
  45 `--with-jmh=build/jmh/jars` should work.
  46 
  47 ## Test selection
  48 
  49 All functionality is available using the `test` make target. In this use case,
  50 the test or tests to be executed is controlled using the `TEST` variable. To
  51 speed up subsequent test runs with no source code changes, `test-only` can be
  52 used instead, which do not depend on the source and test image build.
  53 
  54 For some common top-level tests, direct make targets have been generated. This
  55 includes all JTReg test groups, the hotspot gtest, and custom tests (if
  56 present). This means that `make test-tier1` is equivalent to `make test
  57 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
  58 test runs, the `test TEST="x"` solution needs to be used.
  59 
  60 The test specifications given in `TEST` is parsed into fully qualified test
  61 descriptors, which clearly and unambigously show which tests will be run. As an
  62 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
  63 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
  64 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can


 185 concurrency level to 1 and the timeout factor to 8. This is equivalent to
 186 setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that
 187 the `JTREG` variable is parsed and verified for correctness, so
 188 `JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just
 189 pass unnoticed.
 190 
 191 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
 192 normally eats `;`, the recommended usage is to write the assignment inside
 193 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
 194 as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
 195 
 196 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`.
 197 Also, as a special technique, the string `%20` will be replaced with space for
 198 certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
 199 This can be useful if you have layers of scripts and have trouble getting
 200 proper quoting of command line arguments through.)
 201 
 202 As far as possible, the names of the keywords have been standardized between
 203 test suites.
 204 












































 205 ### JTReg keywords
 206 
 207 #### JOBS
 208 The test concurrency (`-concurrency`).
 209 
 210 Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
 211 JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but
 212 never more than 12.

 213 
 214 #### TIMEOUT
 215 The timeout factor (`-timeoutFactor`).
 216 
 217 Defaults to 4.
 218 
 219 #### TEST_MODE
 220 The test mode (`-agentvm`, `-samevm` or `-othervm`).
 221 
 222 Defaults to `-agentvm`.
 223 
 224 #### ASSERT
 225 Enable asserts (`-ea -esa`, or none).
 226 
 227 Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except
 228 for hotspot.
 229 
 230 #### VERBOSE
 231 The verbosity level (`-verbose`).
 232 
 233 Defaults to `fail,error,summary`.
 234 
 235 #### RETAIN
 236 What test data to retain (`-retain`).
 237 
 238 Defaults to `fail,error`.
 239 
 240 #### MAX_MEM
 241 Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none).
 242 
 243 Limit memory consumption for JTReg test framework and VM under test. Set to 0
 244 to disable the limits.
 245 
 246 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
 247 


















 248 #### OPTIONS
 249 Additional options to the JTReg test framework.
 250 
 251 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
 252 
 253 #### JAVA_OPTIONS
 254 Additional Java options to JTReg (`-javaoption`).
 255 
 256 #### VM_OPTIONS
 257 Additional VM options to JTReg (`-vmoption`).
 258 






 259 ### Gtest keywords
 260 
 261 #### REPEAT
 262 The number of times to repeat the tests (`--gtest_repeat`).
 263 
 264 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
 265 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
 266 problem.
 267 
 268 #### OPTIONS
 269 Additional options to the Gtest test framework.
 270 
 271 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
 272 






 273 ### Microbenchmark keywords
 274 
 275 #### FORK
 276 Override the number of benchmark forks to spawn. Same as specifying `-f <num>`.
 277 
 278 #### ITER
 279 Number of measurement iterations per fork. Same as specifying `-i <num>`.
 280 
 281 #### TIME
 282 Amount of time to spend in each measurement iteration, in seconds. Same as
 283 specifying `-r <num>`
 284 
 285 #### WARMUP_ITER
 286 Number of warmup iterations to run before the measurement phase in each fork.
 287 Same as specifying `-wi <num>`.
 288 
 289 #### WARMUP_TIME
 290 Amount of time to spend in each warmup iteration. Same as specifying `-w <num>`.
 291 
 292 #### RESULTS_FORMAT
 293 Specify to have the test run save a log of the values. Accepts the same values
 294 as `-rff`, i.e., `text`, `csv`, `scsv`, `json`, or `latex`.
 295 
 296 #### VM_OPTIONS
 297 Additional VM arguments to provide to forked off VMs. Same as `-jvmArgs <args>`
 298 
 299 #### OPTIONS
 300 Additional arguments to send to JMH.


























 301 
 302 ---
 303 # Override some definitions in the global css file that are not optimal for
 304 # this document.
 305 header-includes:
 306  - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
 307 ---
< prev index next >