1 % Testing the JDK
2
3 ## Overview
4
5 The bulk of JDK tests use [jtreg](https://openjdk.org/jtreg/), a regression
6 test framework and test runner built for the JDK's specific needs. Other test
7 frameworks are also used. The different test frameworks can be executed
8 directly, but there is also a set of make targets intended to simplify the
9 interface, and figure out how to run your tests for you.
10
11 ## Running tests locally with `make test`
12
13 This is the easiest way to get started. Assuming you've built the JDK locally,
14 execute:
15
16 $ make test
17
18 This will run a default set of tests against the JDK, and present you with the
19 results. `make test` is part of a family of test-related make targets which
20 simplify running tests, because they invoke the various test frameworks for
21 you. The "make test framework" is simple to start with, but more complex ad-hoc
22 combination of tests is also possible. You can always invoke the test
23 frameworks directly if you want even more control.
24
25 Some example command-lines:
26
27 $ make test-tier1
28 $ make test-jdk_lang JTREG="JOBS=8"
29 $ make test TEST=jdk_lang
30 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
31 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT_FACTOR=8;JAVA_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
32 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
33 $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
34 $ make exploded-test TEST=tier2
35
36 "tier1" and "tier2" refer to tiered testing, see further down. "TEST" is a test
37 selection argument which the make test framework will use to try to find the
38 tests you want. It iterates over the available test frameworks, and if the
39 test isn't present in one, it tries the next one. The main target `test` uses
40 the jdk-image as the tested product. There is also an alternate target
41 `exploded-test` that uses the exploded image instead. Not all tests will run
42 successfully on the exploded image, but using this target can greatly improve
43 rebuild times for certain workflows.
44
45 Previously, `make test` was used to invoke an old system for running tests, and
46 `make run-test` was used for the new test framework. For backward compatibility
47 with scripts and muscle memory, `run-test` and variants like
48 `exploded-run-test` or `run-test-tier1` are kept as aliases.
49
50 ### Configuration
51
52 To be able to run JTReg tests, `configure` needs to know where to find the
53 JTReg test framework. If it is not picked up automatically by configure, use
54 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
55 Note that this option should point to the JTReg home, i.e. the top directory,
56 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
57 environment variable to point to the JTReg home before running `configure`.)
58
59 To be able to run microbenchmarks, `configure` needs to know where to find the
60 JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
61 containing the core JMH and transitive dependencies. The recommended
62 dependencies can be retrieved by running `sh make/devkit/createJMHBundle.sh`,
63 after which `--with-jmh=build/jmh/jars` should work.
64
65 When tests fail or timeout, jtreg runs its failure handler to capture necessary
66 data from the system where the test was run. This data can then be used to
67 analyze the test failures. Collecting this data involves running various
68 commands (which are listed in files residing in
69 `test/failure_handler/src/share/conf`) and some of these commands use `sudo`.
70 If the system's `sudoers` file isn't configured to allow running these
71 commands, then it can result in password being prompted during the failure
72 handler execution. Typically, when running locally, collecting this additional
73 data isn't always necessary. To disable running the failure handler, use
74 `--enable-jtreg-failure-handler=no` when running `configure`. If, however, you
75 want to let the failure handler to run and don't want to be prompted for sudo
76 password, then you can configure your `sudoers` file appropriately. Please read
77 the necessary documentation of your operating system to see how to do that;
78 here we only show one possible way of doing that - edit the
79 `/etc/sudoers.d/sudoers` file to include the following line:
80
81 ```
82 johndoe ALL=(ALL) NOPASSWD: /sbin/dmesg
83 ```
84
85 This line configures `sudo` to _not_ prompt for password for the `/sbin/dmesg`
86 command (this is one of the commands that is listed in the files at
87 `test/failure_handler/src/share/conf`), for the user `johndoe`. Here `johndoe`
88 is the user account under which the jtreg tests are run. Replace the username
89 with a relevant user account of your system.
90
91 ## Test selection
92
93 All functionality is available using the `test` make target. In this use case,
94 the test or tests to be executed is controlled using the `TEST` variable. To
95 speed up subsequent test runs with no source code changes, `test-only` can be
96 used instead, which do not depend on the source and test image build.
97
98 For some common top-level tests, direct make targets have been generated. This
99 includes all JTReg test groups, the hotspot gtest, and custom tests (if
100 present). This means that `make test-tier1` is equivalent to `make test
101 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
102 test runs, the `test TEST="x"` solution needs to be used.
103
104 The test specifications given in `TEST` is parsed into fully qualified test
105 descriptors, which clearly and unambiguously show which tests will be run. As
106 an example, `:tier1` will expand to include all subcomponent test directories
107 that define `tier1`, for example: `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
108 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 ...`. You
109 can always submit a list of fully qualified test descriptors in the `TEST`
110 variable if you want to shortcut the parser.
111
112 ### Common Test Groups
113
114 Ideally, all tests are run for every change but this may not be practical due
115 to the limited testing resources, the scope of the change, etc.
116
117 The source tree currently defines a few common test groups in the relevant
118 `TEST.groups` files. There are test groups that cover a specific component, for
119 example `hotspot_gc`. It is a good idea to look into `TEST.groups` files to get
120 a sense what tests are relevant to a particular JDK component.
121
122 Component-specific tests may miss some unintended consequences of a change, so
123 other tests should also be run. Again, it might be impractical to run all
124 tests, and therefore _tiered_ test groups exist. Tiered test groups are not
125 component-specific, but rather cover the significant parts of the entire JDK.
126
127 Multiple tiers allow balancing test coverage and testing costs. Lower test
128 tiers are supposed to contain the simpler, quicker and more stable tests.
129 Higher tiers are supposed to contain progressively more thorough, slower, and
130 sometimes less stable tests, or the tests that require special configuration.
131
132 Contributors are expected to run the tests for the areas that are changed, and
133 the first N tiers they can afford to run, but at least tier1.
134
135 A brief description of the tiered test groups:
136
137 - `tier1`: This is the most fundamental test tier. Roughly speaking, a failure
138 of a test in this tier has the potential to indicate a problem that would
139 affect many Java programs. Tests in `tier1` include tests of HotSpot, core
140 APIs in the `java.base` module, and the `javac` compiler. Multiple developers
141 run these tests every day. Because of the widespread use, the tests in
142 `tier1` are carefully selected and optimized to run fast, and to run in the
143 most stable manner. As a guideline, nearly all individual tests in `tier1`
144 are expected to run to completion in ten seconds or less when run on common
145 configurations used for development. Long-running tests, even of core
146 functionality, should occur in higher tiers or be covered in other kinds of
147 testing. The test failures in `tier1` are usually followed up on quickly,
148 either with fixes, or adding relevant tests to problem list. GitHub Actions
149 workflows, if enabled, run `tier1` tests.
150
151 - `tier2`: This test group covers even more ground. These contain, among other
152 things, tests that either run for too long to be at `tier1`, or may require
153 special configuration, or tests that are less stable, or cover the broader
154 range of non-core JVM and JDK features/components (for example, XML).
155
156 - `tier3`: This test group includes more stressful tests, the tests for corner
157 cases not covered by previous tiers, plus the tests that require GUIs. As
158 such, this suite should either be run with low concurrency (`TEST_JOBS=1`),
159 or without headful tests(`JTREG_KEYWORDS=\!headful`), or both.
160
161 - `tier4`: This test group includes every other test not covered by previous
162 tiers. It includes, for example, `vmTestbase` suites for Hotspot, which run
163 for many hours even on large machines. It also runs GUI tests, so the same
164 `TEST_JOBS` and `JTREG_KEYWORDS` caveats apply.
165
166 ### JTReg
167
168 JTReg tests can be selected either by picking a JTReg test group, or a
169 selection of files or directories containing JTReg tests. Documentation can be
170 found at [https://openjdk.org/jtreg/](https://openjdk.org/jtreg/), note
171 especially the extensive [FAQ](https://openjdk.org/jtreg/faq.html).
172
173 JTReg test groups can be specified either without a test root, e.g. `:tier1`
174 (or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`,
175 `test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test
176 root can be specified either as an absolute path, or a path relative to the JDK
177 top directory, or the `test` directory. For simplicity, the hotspot JTReg test
178 root, which really is `hotspot/jtreg` can be abbreviated as just `hotspot`.
179
180 When specified without a test root, all matching groups from all test roots
181 will be added. Otherwise, only the group from the specified test root will be
182 added.
183
184 Individual JTReg tests or directories containing JTReg tests can also be
185 specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or
186 `hotspot/jtreg/native_sanity`. Just like for test root selection, you can
187 either specify an absolute path (which can even point to JTReg tests outside
188 the source tree), or a path relative to either the JDK top directory or the
189 `test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as
190 well.
191
192 As long as the test groups or test paths can be uniquely resolved, you do not
193 need to enter the `jtreg:` prefix. If this is not possible, or if you want to
194 use a fully qualified test descriptor, add `jtreg:`, e.g.
195 `jtreg:test/hotspot/jtreg/native_sanity`.
196
197 ### Gtest
198
199 **Note:** To be able to run the Gtest suite, you need to configure your build
200 to be able to find a proper version of the gtest source. For details, see the
201 section **"Running Tests" in the build
202 documentation** ([html](building.html#running-tests), [markdown](building.md#running-tests)).
203
204 Since the Hotspot Gtest suite is so quick, the default is to run all tests.
205 This is specified by just `gtest`, or as a fully qualified test descriptor
206 `gtest:all`.
207
208 If you want, you can single out an individual test or a group of tests, for
209 instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This
210 can be particularly useful if you want to run a shaky test repeatedly.
211
212 For Gtest, there is a separate test suite for each JVM variant. The JVM variant
213 is defined by adding `/<variant>` to the test descriptor, e.g.
214 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
215 variant present (e.g. server, client). So if you only have the server JVM
216 present, then `gtest:all` will be equivalent to `gtest:all/server`.
217
218 ### Microbenchmarks
219
220 Which microbenchmarks to run is selected using a regular expression following
221 the `micro:` test descriptor, e.g., `micro:java.lang.reflect`. This delegates
222 the test selection to JMH, meaning package name, class name and even benchmark
223 method names can be used to select tests.
224
225 Using special characters like `|` in the regular expression is possible, but
226 needs to be escaped multiple times: `micro:ArrayCopy\\\\\|reflect`.
227
228 ### Special tests
229
230 A handful of odd tests that are not covered by any other testing framework are
231 accessible using the `special:` test descriptor. Currently, this includes
232 `failure-handler` and `make`.
233
234 * Failure handler testing is run using `special:failure-handler` or just
235 `failure-handler` as test descriptor.
236
237 * Tests for the build system, including both makefiles and related
238 functionality, is run using `special:make` or just `make` as test
239 descriptor. This is equivalent to `special:make:all`.
240
241 A specific make test can be run by supplying it as argument, e.g.
242 `special:make:idea`. As a special syntax, this can also be expressed as
243 `make-idea`, which allows for command lines as `make test-make-idea`.
244
245 ## Test results and summary
246
247 At the end of the test run, a summary of all tests run will be presented. This
248 will have a consistent look, regardless of what test suites were used. This is
249 a sample summary:
250
251 ==============================
252 Test summary
253 ==============================
254 TEST TOTAL PASS FAIL ERROR
255 >> jtreg:jdk/test:tier1 1867 1865 2 0 <<
256 jtreg:langtools/test:tier1 4711 4711 0 0
257 jtreg:nashorn/test:tier1 133 133 0 0
258 ==============================
259 TEST FAILURE
260
261 Tests where the number of TOTAL tests does not equal the number of PASSed tests
262 will be considered a test failure. These are marked with the `>> ... <<` marker
263 for easy identification.
264
265 The classification of non-passed tests differs a bit between test suites. In
266 the summary, ERROR is used as a catch-all for tests that neither passed nor are
267 classified as failed by the framework. This might indicate test framework
268 error, timeout or other problems.
269
270 In case of test failures, `make test` will exit with a non-zero exit value.
271
272 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
273 where TEST_ID is a path-safe conversion from the fully qualified test
274 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is
275 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the
276 test run.
277
278 Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For
279 some frameworks, this directory might contain information that is useful in
280 determining the cause of a failed test.
281
282 ## Test suite control
283
284 It is possible to control various aspects of the test suites using make control
285 variables.
286
287 These variables use a keyword=value approach to allow multiple values to be
288 set. So, for instance, `JTREG="JOBS=1;TIMEOUT_FACTOR=8"` will set the JTReg
289 concurrency level to 1 and the timeout factor to 8. This is equivalent to
290 setting `JTREG_JOBS=1 JTREG_TIMEOUT_FACTOR=8`, but using the keyword format
291 means that the `JTREG` variable is parsed and verified for correctness, so
292 `JTREG="TMIEOUT_FACTOR=8"` would give an error, while `JTREG_TMIEOUT_FACTOR=8`
293 would just pass unnoticed.
294
295 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
296 normally eats `;`, the recommended usage is to write the assignment inside
297 quotes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
298 as in `JTREG="JAVA_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
299
300 (Other ways are possible, e.g. using backslash:
301 `JTREG=JOBS=1\;TIMEOUT_FACTOR=8`. Also, as a special technique, the string
302 `%20` will be replaced with space for certain options, e.g.
303 `JTREG=JAVA_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. This can be useful if
304 you have layers of scripts and have trouble getting proper quoting of command
305 line arguments through.)
306
307 As far as possible, the names of the keywords have been standardized between
308 test suites.
309
310 ### General keywords (TEST_OPTS)
311
312 Some keywords are valid across different test suites. If you want to run tests
313 from multiple test suites, or just don't want to care which test suite specific
314 control variable to use, then you can use the general TEST_OPTS control
315 variable.
316
317 There are also some keywords that applies globally to the test runner system,
318 not to any specific test suites. These are also available as TEST_OPTS
319 keywords.
320
321 #### JOBS
322
323 Currently only applies to JTReg.
324
325 #### TIMEOUT_FACTOR
326
327 Currently only applies to [JTReg -timeoutFactor](#timeout_factor-1).
328
329 #### JAVA_OPTIONS
330
331 Applies to JTReg, GTest and Micro.
332
333 #### VM_OPTIONS
334
335 Applies to JTReg, GTest and Micro.
336
337 #### JCOV
338
339 This keyword applies globally to the test runner system. If set to `true`, it
340 enables JCov coverage reporting for all tests run. To be useful, the JDK under
341 test must be run with a JDK built with JCov instrumentation (`configure
342 --with-jcov=<path to directory containing lib/jcov.jar>`, `make jcov-image`).
343
344 The simplest way to run tests with JCov coverage report is to use the special
345 target `jcov-test` instead of `test`, e.g. `make jcov-test TEST=jdk_lang`. This
346 will make sure the JCov image is built, and that JCov reporting is enabled.
347
348 To include JCov coverage for just a subset of all modules, you can use the
349 `--with-jcov-modules` arguments to `configure`, e.g.
350 `--with-jcov-modules=jdk.compiler,java.desktop`.
351
352 For more fine-grained control, you can pass arbitrary filters to JCov using
353 `--with-jcov-filters`, and you can specify a specific JDK to instrument
354 using `--with-jcov-input-jdk`.
355
356 The JCov report is stored in `build/$BUILD/test-results/jcov-output/report`.
357
358 Please note that running with JCov reporting can be very memory intensive.
359
360 #### JCOV_DIFF_CHANGESET
361
362 While collecting code coverage with JCov, it is also possible to find coverage
363 for only recently changed code. JCOV_DIFF_CHANGESET specifies a source
364 revision. A textual report will be generated showing coverage of the diff
365 between the specified revision and the repository tip.
366
367 The report is stored in
368 `build/$BUILD/test-results/jcov-output/diff_coverage_report` file.
369
370 #### AOT_JDK
371
372 See [Testing Ahead-of-time optimizations](#testing-ahead-of-time-optimizations).
373
374 ### JTReg keywords
375
376 #### JOBS
377
378 The test concurrency (`-concurrency`).
379
380 Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
381 JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but
382 never more than *memory size in GB/2*.
383
384 #### TIMEOUT_FACTOR
385
386 The `TIMEOUT_FACTOR` is forwarded to JTReg framework itself
387 (`-timeoutFactor`). Also, some test cases that programmatically wait a
388 certain amount of time will apply this factor. If we run in forced
389 compilation mode (`-Xcomp`), the build system will automatically
390 adjust this factor to compensate for less performance. Defaults to 4.
391
392 #### FAILURE_HANDLER_TIMEOUT
393
394 Sets the argument `-timeoutHandlerTimeout` for JTReg. The default value is 0.
395 This is only valid if the failure handler is built.
396
397 #### TEST_THREAD_FACTORY
398
399 Sets the `-testThreadFactory` for JTReg. It should be the fully qualified
400 classname of a class which implements `java.util.concurrent.ThreadFactory`. One
401 such implementation class, named Virtual, is currently part of the JDK build in
402 the `test/jtreg_test_thread_factory/` directory. This class gets compiled
403 during the test image build. The implementation of the Virtual class creates a
404 new virtual thread for executing each test class.
405
406 #### VALUE_CLASS_PLUGIN
407
408 Enables the `ValueClassPlugin` javac plugin when compiling and running JTReg
409 tests. This is a **temporary mode** intended for use while value classes
410 (JEP 401) are a preview feature. The long-term plan is to replace classes
411 annotated with `@jdk.test.lib.valueclass.AsValueClass` with plain
412 `value class` declarations once value classes are finalized.
413
414 In the meantime, this mode allows test sources to compile and run as either
415 value classes or regular identity classes without source-level changes.
416
417 When set to any non-empty value, the following options are appended to every
418 JTReg invocation:
419
420 * `-cpa:<valueClassPlugin.jar>` — appends the plugin JAR to the compile-time
421 classpath (only when the JAR is present in the test image under
422 `jtreg_value_class_plugin/valueClassPlugin.jar`).
423 * `-vmoption:--enable-preview` — enables JVM preview features at runtime.
424 * `-javacoption:-XDaccessInternalAPI` — grants the compiler access to internal
425 APIs required by the plugin.
426 * `-javacoption:--source <version> --enable-preview` — enables preview language
427 features at compile time.
428 * `-javacoption:-Xplugin:ValueClassPlugin` — activates the plugin.
429
430 The plugin scans each compilation unit after parsing and converts any class
431 annotated with `@jdk.test.lib.valueclass.AsValueClass` into a value class by
432 setting the internal `VALUE_CLASS` modifier flag and clearing the
433 `IDENTITY_TYPE` flag. This transformation only takes effect when
434 `--enable-preview` is active; without it the annotation is a no-op and the
435 class compiles as an ordinary identity class, so the same test source can
436 exercise both code paths.
437
438 Example:
439
440 $ make test TEST=jdk_lang JTREG="VALUE_CLASS_PLUGIN=true"
441
442 #### JVMTI_STRESS_AGENT
443
444 Executes JTReg tests with JVM TI stress agent. The stress agent is the part of
445 test library and located in `test/lib/jdk/test/lib/jvmti/libJvmtiStressAgent.cpp`.
446 The value of this argument is set as JVM TI agent options.
447 This mode uses ProblemList-jvmti-stress-agent.txt as an additional exclude list.
448
449 #### TEST_MODE
450
451 The test mode (`agentvm` or `othervm`).
452
453 Defaults to `agentvm`.
454
455 #### ASSERT
456
457 Enable asserts (`-ea -esa`, or none).
458
459 Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except
460 for hotspot.
461
462 #### VERBOSE
463
464 The verbosity level (`-verbose`).
465
466 Defaults to `fail,error,summary`.
467
468 #### RETAIN
469
470 What test data to retain (`-retain`).
471
472 Defaults to `fail,error`.
473
474 #### MAX_MEM
475
476 Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none).
477
478 Limit memory consumption for JTReg test framework and VM under test. Set to 0
479 to disable the limits.
480
481 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
482
483 #### MAX_OUTPUT
484
485 Set the property `javatest.maxOutputSize` for the launcher, to change the
486 default JTReg log limit.
487
488 #### KEYWORDS
489
490 JTReg keywords sent to JTReg using `-k`. Please be careful in making sure that
491 spaces and special characters (like `!`) are properly quoted. To avoid some
492 issues, the special value `%20` can be used instead of space.
493
494 #### EXTRA_PROBLEM_LISTS
495
496 Use additional problem lists file or files, in addition to the default
497 ProblemList.txt located at the JTReg test roots.
498
499 If multiple file names are specified, they should be separated by space (or, to
500 help avoid quoting issues, the special value `%20`).
501
502 The file names should be either absolute, or relative to the JTReg test root of
503 the tests to be run.
504
505 #### RUN_PROBLEM_LISTS
506
507 Use the problem lists to select tests instead of excluding them.
508
509 Set to `true` or `false`. If `true`, JTReg will use `-match:` option, otherwise
510 `-exclude:` will be used. Default is `false`.
511
512 #### OPTIONS
513
514 Additional options to the JTReg test framework.
515
516 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
517
518 #### JAVA_OPTIONS
519
520 Additional Java options for running test classes (sent to JTReg as
521 `-javaoption`).
522
523 #### VM_OPTIONS
524
525 Additional Java options to be used when compiling and running classes (sent to
526 JTReg as `-vmoption`).
527
528 This option is only needed in special circumstances. To pass Java options to
529 your test classes, use `JAVA_OPTIONS`.
530
531 #### LAUNCHER_OPTIONS
532
533 Additional Java options that are sent to the java launcher that starts the
534 JTReg harness.
535
536 #### RETRY_COUNT
537
538 Retry failed tests up to a set number of times, until they pass. This allows to
539 pass the tests with intermittent failures. Defaults to 0.
540
541 #### REPEAT_COUNT
542
543 Repeat the tests up to a set number of times, stopping at first failure. This
544 helps to reproduce intermittent test failures. Defaults to 0.
545
546 #### REPORT
547
548 Use this report style when reporting test results (sent to JTReg as `-report`).
549 Defaults to `files`.
550
551 #### MANUAL
552
553 Set to `true` to execute manual tests only.
554
555 ### Gtest keywords
556
557 #### REPEAT
558
559 The number of times to repeat the tests (`--gtest_repeat`).
560
561 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
562 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
563 problem.
564
565 #### OPTIONS
566
567 Additional options to the Gtest test framework.
568
569 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
570
571 ### Microbenchmark keywords
572
573 #### FORK
574
575 Override the number of benchmark forks to spawn. Same as specifying `-f <num>`.
576
577 #### ITER
578
579 Number of measurement iterations per fork. Same as specifying `-i <num>`.
580
581 #### TIME
582
583 Amount of time to spend in each measurement iteration, in seconds. Same as
584 specifying `-r <num>`
585
586 #### WARMUP_ITER
587
588 Number of warmup iterations to run before the measurement phase in each fork.
589 Same as specifying `-wi <num>`.
590
591 #### WARMUP_TIME
592
593 Amount of time to spend in each warmup iteration. Same as specifying `-w
594 <num>`.
595
596 #### RESULTS_FORMAT
597
598 Specify to have the test run save a log of the values. Accepts the same values
599 as `-rff`, i.e., `text`, `csv`, `scsv`, `json`, or `latex`.
600
601 #### TEST_JDK
602
603 The path to the JDK that will be used to run the benchmarks.
604
605 Defaults to `build/<CONF-NAME>/jdk`.
606
607 #### BENCHMARKS_JAR
608
609 The path to the JAR containing the benchmarks.
610
611 Defaults to `test/micro/benchmarks.jar`.
612
613 #### VM_OPTIONS
614
615 Additional VM arguments to provide to forked off VMs. Same as `-jvmArgs <args>`
616
617 #### OPTIONS
618
619 Additional arguments to send to JMH.
620
621 ## Notes for Specific Tests
622
623 ### Docker Tests
624
625 Docker tests with default parameters may fail on systems with glibc versions
626 not compatible with the one used in the default docker image (e.g., Oracle
627 Linux 7.6 for x86). For example, they pass on Ubuntu 16.04 but fail on Ubuntu
628 18.04 if run like this on x86:
629
630 ```
631 $ make test TEST="jtreg:test/hotspot/jtreg/containers/docker"
632 ```
633
634 To run these tests correctly, additional parameters for the correct docker
635 image are required on Ubuntu 18.04 by using `JAVA_OPTIONS`.
636
637 ```
638 $ make test TEST="jtreg:test/hotspot/jtreg/containers/docker" \
639 JTREG="JAVA_OPTIONS=-Djdk.test.docker.image.name=ubuntu
640 -Djdk.test.docker.image.version=latest"
641 ```
642
643 ### Non-US locale
644
645 If your locale is non-US, some tests are likely to fail. To work around this
646 you can set the locale to US. On Unix platforms simply setting `LANG="en_US"`
647 in the environment before running tests should work. On Windows or macOS,
648 setting `JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US"` helps for
649 most, but not all test cases.
650
651 For example:
652
653 ```
654 $ export LANG="en_US" && make test TEST=...
655 $ make test JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US" TEST=...
656 ```
657
658 ### PKCS11 Tests
659
660 It is highly recommended to use the latest NSS version when running PKCS11
661 tests. Improper NSS version may lead to unexpected failures which are hard to
662 diagnose. For example, sun/security/pkcs11/Secmod/AddTrustedCert.java may fail
663 on Ubuntu 18.04 with the default NSS version in the system. To run these tests
664 correctly, the system property `jdk.test.lib.artifacts.<NAME>` is required on
665 Ubuntu 18.04 to specify the alternative NSS lib directory. The `<NAME>`
666 component should be replaced with the name element of the appropriate
667 `@Artifact` class. (See `test/jdk/sun/security/pkcs11/PKCS11Test.java`)
668
669 For example:
670
671 ```
672 $ make test TEST="jtreg:sun/security/pkcs11/Secmod/AddTrustedCert.java" \
673 JTREG="JAVA_OPTIONS=-Djdk.test.lib.artifacts.nsslib-linux_aarch64=/path/to/NSS-libs"
674 ```
675
676 For more notes about the PKCS11 tests, please refer to
677 test/jdk/sun/security/pkcs11/README.
678
679
680 ### SCTP Tests
681
682 The SCTP tests require the SCTP runtime library, which is often not installed
683 by default in popular Linux distributions. Without this library, the SCTP tests
684 will be skipped. If you want to enable the SCTP tests, you should install the
685 SCTP library before running the tests.
686
687 For distributions using the .deb packaging format and the apt tool
688 (such as Debian, Ubuntu, etc.), try this:
689
690 ```
691 sudo apt install libsctp1
692 sudo modprobe sctp
693 lsmod | grep sctp
694 ```
695
696 For distributions using the .rpm packaging format and the dnf tool
697 (such as Fedora, Red Hat, etc.), try this:
698
699 ```
700 sudo dnf install -y lksctp-tools
701 sudo modprobe sctp
702 lsmod | grep sctp
703 ```
704
705 ### Testing Ahead-of-time Optimizations
706
707 One way to improve test coverage of ahead-of-time (AOT) optimizations in
708 the JDK is to run existing jtreg test cases in a special "AOT_JDK" mode.
709 Example:
710
711 ```
712 $ make test JTREG="AOT_JDK=onestep" \
713 TEST=open/test/hotspot/jtreg/runtime/invokedynamic
714 ```
715
716 In this testing mode, we first perform an AOT training run
717 (see https://openjdk.org/jeps/483) of a special test program
718 ([test/setup_aot/TestSetupAOT.java](../test/setup_aot/TestSetupAOT.java))
719 that accesses about 5,0000 classes in the JDK core libraries.
720 Optimization artifacts for these classes (such as pre-linked
721 lambda expressions, execution profiles, and pre-generated native code)
722 are stored into an AOT cache file, which will be used by all the JVMs
723 launched by the selected jtreg test cases.
724
725 When the jtreg tests call into the core libraries classes that are in
726 the AOT cache, we will be able to test the AOT optimizations that were
727 used on those classes.
728
729 Please note that not all existing jtreg test cases can be executed with
730 the AOT_JDK mode. See
731 [test/hotspot/jtreg/ProblemList-AotJdk.txt](../test/hotspot/jtreg/ProblemList-AotJdk.txt)
732 and [test/jdk/ProblemList-AotJdk.txt](../test/jdk/ProblemList-AotJdk.txt).
733
734 Also, test cases that were written specifically to test AOT, such as the tests
735 under [test/hotspot/jtreg/runtime/cds](../test/hotspot/jtreg/runtime/cds/),
736 cannot be executed with the AOT_JDK mode.
737
738 Valid values for `AOT_JDK` are `onestep` and `twostep`. These control how
739 the AOT cache is generated. See https://openjdk.org/jeps/514 for details.
740 All other values are ignored.
741
742 ### Testing with alternative security providers
743
744 Some security tests use a hardcoded provider for `KeyFactory`, `Cipher`,
745 `KeyPairGenerator`, `KeyGenerator`, `AlgorithmParameterGenerator`,
746 `KeyAgreement`, `Mac`, `MessageDigest`, `SecureRandom`, `Signature`,
747 `AlgorithmParameters`, `Configuration`, `Policy`, or `SecretKeyFactory` objects.
748 Specify the `-Dtest.provider.name=NAME` property to use a different provider for
749 the service(s).
750
751 ### Client UI Tests
752
753 #### System key shortcuts
754
755 Some Client UI tests use key sequences which may be reserved by the operating
756 system. Usually that causes the test failure. So it is highly recommended to
757 disable system key shortcuts prior testing. The steps to access and disable
758 system key shortcuts for various platforms are provided below.
759
760 ##### macOS
761
762 Choose Apple menu; System Preferences, click Keyboard, then click Shortcuts;
763 select or deselect desired shortcut.
764
765 For example,
766 test/jdk/javax/swing/TooltipManager/JMenuItemToolTipKeyBindingsTest/JMenuItemToolTipKeyBindingsTest.java
767 fails on macOS because it uses `CTRL + F1` key sequence to show or hide tooltip
768 message but the key combination is reserved by the operating system. To run the
769 test correctly the default global key shortcut should be disabled using the
770 steps described above, and then deselect "Turn keyboard access on or off"
771 option which is responsible for `CTRL + F1` combination.
772
773 ##### Linux
774
775 Open the Activities overview and start typing Settings; Choose Settings, click
776 Devices, then click Keyboard; set or override desired shortcut.
777
778 ##### Windows
779
780 Type `gpedit` in the Search and then click Edit group policy; navigate to User
781 Configuration -> Administrative Templates -> Windows Components -> File
782 Explorer; in the right-side pane look for "Turn off Windows key hotkeys" and
783 double click on it; enable or disable hotkeys.
784
785 Note: restart is required to make the settings take effect.
786
787 #### Robot API
788
789 Most automated Client UI tests use `Robot` API to control the UI. Usually, the
790 default operating system settings need to be adjusted for Robot to work
791 correctly. The detailed steps how to access and update these settings for
792 different platforms are provided below.
793
794 ##### macOS
795
796 `Robot` is not permitted to control your Mac by default since macOS 10.15. To
797 allow it, choose Apple menu -> System Settings, click Privacy & Security; then
798 click Accessibility and ensure the following apps are allowed to control your
799 computer: *Java* and *Terminal*. If the tests are run from an IDE, the IDE
800 should be granted this permission too.
801
802 ##### Windows
803
804 On Windows if Cygwin terminal is used to run the tests, there is a delay in
805 focus transfer. Usually it causes automated UI test failure. To disable the
806 delay, type `regedit` in the Search and then select Registry Editor; navigate
807 to the following key: `HKEY_CURRENT_USER\Control Panel\Desktop`; make sure the
808 `ForegroundLockTimeout` value is set to 0.
809
810 Additional information about Client UI tests configuration for various
811 operating systems can be obtained at [Automated client GUI testing system set
812 up
813 requirements](https://wiki.openjdk.org/display/ClientLibs/Automated+client+GUI+testing+system+set+up+requirements)
814
815 ## Editing this document
816
817 If you want to contribute changes to this document, edit `doc/testing.md` and
818 then run `make update-build-docs` to generate the same changes in
819 `doc/testing.html`.
820
821 ---
822 # Override some definitions in the global css file that are not optimal for
823 # this document.
824 header-includes:
825 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
826 ---