--- old/doc/testing.html 2019-07-23 01:20:58.936154598 +0200 +++ new/doc/testing.html 2019-07-23 01:20:58.752154892 +0200 @@ -1,24 +1,19 @@ - + - - - + + + Testing the JDK - - + + -
+

Testing the JDK

Using "make test" (the run-test framework)

This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.

The main target test uses the jdk-image as the tested product. There is also an alternate target exploded-test that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.

-

Previously, make test was used to invoke an old system for running tests, and make run-test was used for the new test framework. For backward compatibility with scripts and muscle memory, run-test (and variants like exploded-run-test or run-test-tier1) are kept as aliases.

+

Previously, make test was used invoke an old system for running test, and make run-test was used for the new test framework. For backward compatibility with scripts and muscle memory, run-test (and variants like exploded-run-test or run-test-tier1) are kept as aliases. The old system can still be accessed for some time using cd test && make.

Some example command-lines:

$ make test-tier1
 $ make test-jdk_lang JTREG="JOBS=8"
@@ -107,29 +97,11 @@
 

To separate multiple keyword=value pairs, use ; (semicolon). Since the shell normally eats ;, the recommended usage is to write the assignment inside qoutes, e.g. JTREG="...;...". This will also make sure spaces are preserved, as in JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug".

(Other ways are possible, e.g. using backslash: JTREG=JOBS=1\;TIMEOUT=8. Also, as a special technique, the string %20 will be replaced with space for certain options, e.g. JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug. This can be useful if you have layers of scripts and have trouble getting proper quoting of command line arguments through.)

As far as possible, the names of the keywords have been standardized between test suites.

-

General keywords (TEST_OPTS)

-

Some keywords are valid across different test suites. If you want to run tests from multiple test suites, or just don't want to care which test suite specific control variable to use, then you can use the general TEST_OPTS control variable.

-

There are also some keywords that applies globally to the test runner system, not to any specific test suites. These are also available as TEST_OPTS keywords.

-

JOBS

-

Currently only applies to JTReg.

-

TIMEOUT_FACTOR

-

Currently only applies to JTReg.

-

VM_OPTIONS

-

Applies to JTReg, GTest and Micro.

-

JAVA_OPTIONS

-

Applies to JTReg, GTest and Micro.

-

AOT_MODULES

-

Applies to JTReg and GTest.

-

JCOV

-

This keywords applies globally to the test runner system. If set to true, it enables JCov coverage reporting for all tests run. To be useful, the JDK under test must be run with a JDK built with JCov instrumentation (configure --with-jcov=<path to directory containing lib/jcov.jar>, make jcov-image).

-

The simplest way to run tests with JCov coverage report is to use the special target jcov-test instead of test, e.g. make jcov-test TEST=jdk_lang. This will make sure the JCov image is built, and that JCov reporting is enabled.

-

The JCov report is stored in build/$BUILD/test-results/jcov-output.

-

Please note that running with JCov reporting can be very memory intensive.

JTReg keywords

-

JOBS

+

JOBS

The test concurrency (-concurrency).

-

Defaults to TEST_JOBS (if set by --with-test-jobs=), otherwise it defaults to JOBS, except for Hotspot, where the default is number of CPU cores/2 (for sparc, if more than 16 cpus, then number of CPU cores/5, otherwise number of CPU cores/4), but never more than memory size in GB/2.

-

TIMEOUT_FACTOR

+

Defaults to TEST_JOBS (if set by --with-test-jobs=), otherwise it defaults to JOBS, except for Hotspot, where the default is number of CPU cores/2, but never more than 12.

+

TIMEOUT

The timeout factor (-timeoutFactor).

Defaults to 4.

TEST_MODE

@@ -148,21 +120,13 @@

Limit memory consumption (-Xmx and -vmoption:-Xmx, or none).

Limit memory consumption for JTReg test framework and VM under test. Set to 0 to disable the limits.

Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).

-

KEYWORDS

-

JTReg kewords sent to JTReg using -k. Please be careful in making sure that spaces and special characters (like !) are properly quoted. To avoid some issues, the special value %20 can be used instead of space.

-

EXTRA_PROBLEM_LISTS

-

Use additional problem lists file or files, in addition to the default ProblemList.txt located at the JTReg test roots.

-

If multiple file names are specified, they should be separated by space (or, to help avoid quoting issues, the special value %20).

-

The file names should be either absolute, or relative to the JTReg test root of the tests to be run.

OPTIONS

Additional options to the JTReg test framework.

Use JTREG="OPTIONS=--help all" to see all available JTReg options.

-

JAVA_OPTIONS

+

JAVA_OPTIONS

Additional Java options to JTReg (-javaoption).

-

VM_OPTIONS

+

VM_OPTIONS

Additional VM options to JTReg (-vmoption).

-

AOT_MODULES

-

Generate AOT modules before testing for the specified module, or set of modules. If multiple modules are specified, they should be separated by space (or, to help avoid quoting issues, the special value %20).

Gtest keywords

REPEAT

The number of times to repeat the tests (--gtest_repeat).

@@ -170,8 +134,6 @@

OPTIONS

Additional options to the Gtest test framework.

Use GTEST="OPTIONS=--help" to see all available Gtest options.

-

AOT_MODULES

-

Generate AOT modules before testing for the specified module, or set of modules. If multiple modules are specified, they should be separated by space (or, to help avoid quoting issues, the special value %20).

Microbenchmark keywords

FORK

Override the number of benchmark forks to spawn. Same as specifying -f <num>.

@@ -185,19 +147,9 @@

Amount of time to spend in each warmup iteration. Same as specifying -w <num>.

RESULTS_FORMAT

Specify to have the test run save a log of the values. Accepts the same values as -rff, i.e., text, csv, scsv, json, or latex.

-

VM_OPTIONS

+

VM_OPTIONS

Additional VM arguments to provide to forked off VMs. Same as -jvmArgs <args>

OPTIONS

Additional arguments to send to JMH.

-

Notes for Specific Tests

-

Docker Tests

-

Docker tests with default parameters may fail on systems with glibc versions not compatible with the one used in the default docker image (e.g., Oracle Linux 7.6 for x86). For example, they pass on Ubuntu 16.04 but fail on Ubuntu 18.04 if run like this on x86:

-
$ make test TEST="jtreg:test/hotspot/jtreg/containers/docker"
-

To run these tests correctly, additional parameters for the correct docker image are required on Ubuntu 18.04 by using JAVA_OPTIONS.

-
$ make test TEST="jtreg:test/hotspot/jtreg/containers/docker" JTREG="JAVA_OPTIONS=-Djdk.test.docker.image.name=ubuntu -Djdk.test.docker.image.version=latest"
-

Non-US locale

-

If your locale is non-US, some tests are likely to fail. To work around this you can set the locale to US. On Unix platforms simply setting LANG="en_US" in the environment before running tests should work. On Windows, setting JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US" helps for most, but not all test cases. For example:

-
$ export LANG="en_US" && make test TEST=...
-$ make test JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US" TEST=...