1 # Demos and Documentation for Leyden-premain Protype
2
3
4 ## Demos
5
6 - [simple](simple) script to demonstrate the "5 step" process of training/production run with leyden-premain prototype
7
8 - [jmh](jmh) run JMH + specjbb2005-jmh-1.25.jar
9
10 - [javac](javac) using javac as a regression test, to compile up to 10000 Java source files
11
12 - [javac_helloworld](javac_helloworld) measures the total elapsed time of `javac HelloWorld.java` using
13 the premain branch vs the JDK mainline.
14 It's a good demonstration of how we can improve start-up time of a complex application.
15
16 - [javac_new_workflow](javac_new_workflow) Example of the new "one step training" workflow
17 (still under development) where you can generate all the Leyden artifacts with a single
18 JVM invocation.
19
20 ## Docs
21
22 - [InvokeDynamic.md](InvokeDynamic.md) CDS optimizations for invokedynamic
23
24 ## Benchmarking Against JDK Mainline
25
26 We have included some demo application for some popular Java application
27 frameworks, so you can compare the performance of Leyden vs the mainline JDK.
28
29 - [helidon-quickstart-se](helidon-quickstart-se): from https://helidon.io/docs/v4/se/guides/quickstart
30 - [micronaut-first-app](micronaut-first-app): from https://guides.micronaut.io/latest/creating-your-first-micronaut-app-maven-java.html
31 - [quarkus-getting-started](quarkus-getting-started): from https://quarkus.io/guides/getting-started
32 - [spring-petclinic](spring-petclinic): from https://github.com/spring-projects/spring-petclinic
33
34 You need:
35
36 - An official build of JDK 21
37 - An up-to-date build of the JDK mainline
38 - The latest Leyden build
39 - Maven (ideally 3.8 or later, as required by some of the demos). Note: if you are behind
40 a firewall, you may need to [set up proxies for Maven](https://maven.apache.org/guides/mini/guide-proxies.html)
41
42 The steps for benchmarking is the same for the above four demos. For example:
43
44 ```
45 $ cd helidon-quickstart-se
46 $ make PREMAIN_HOME=/repos/leyden/build/linux-x64/images/jdk \
47 MAINLINE_HOME=/repos/jdk/build/linux-x64/images/jdk \
48 BLDJDK_HOME=/usr/local/jdk21 \
49 bench
50 run,mainline default,mainline custom static CDS,premain custom static CDS only,premain CDS + AOT
51 1,398,244,144,107
52 2,387,247,142,108
53 3,428,238,143,107
54 4,391,252,142,111
55 5,417,247,141,107
56 6,390,239,139,127
57 7,387,247,145,111
58 8,387,240,147,110
59 9,388,242,147,108
60 10,400,242,167,108
61 Geomean,397.08,243.76,145.52,110.26
62 Stdev,13.55,4.19,7.50,5.73
63 Markdown snippets in mainline_vs_premain.md
64 ```
65
66 The above command runs each configuration 10 times, in an interleaving order. This way
67 the noise of the system (background processes, thermo throttling, etc) is more likely to
68 be spread across the different runs.
69
70 As typical for start-up benchmarking, the numbers are not very steady. You should plot
71 the results (saved in the file mainline_vs_premain.csv) in a spreadsheet to check for noise, etc.
72
73 The "make bench" target also generates GitHub markdown snippets (in the file mainline_vs_premain.md) for creating the
74 graphs below.
75
76 ## Benchmarking Between Two Leyden Builds
77
78 This is useful for Leyden developers to measure the benefits of a particular optimization.
79 The steps are similar to above, but we use the "make compare_premain_builds" target:
80
81 ```
82 $ cd helidon-quickstart-se
83 $ make PM_OLD=/repos/leyden_old/build/linux-x64/images/jdk \
84 PM_NEW=/repos/leyden_new/build/linux-x64/images/jdk \
85 BLDJDK_HOME=/usr/local/jdk21 \
86 compare_premain_builds
87 Old build = /repos/leyden_old/build/linux-x64/images/jdk with options
88 New build = /repos/leyden_new/build/linux-x64/images/jdk with options
89 Run,Old CDS + AOT,New CDS + AOT
90 1,110,109
91 2,131,111
92 3,118,115
93 4,110,108
94 5,117,110
95 6,114,109
96 7,110,109
97 8,118,110
98 9,110,110
99 10,113,114
100 Geomean,114.94,110.48
101 Stdev,6.19,2.16
102 Markdown snippets in compare_premain_builds.md
103 ```
104
105 Please see [lib/Bench.gmk](lib/Bench.gmk) for more details.
106
107 Note: due to the variability of start-up time, the benefit of minor improvements may
108 be difficult to measure.
109
110 ## Preliminary Benchmark Results
111
112 The following charts show the relative start-up performance of the Leyden/Premain branch vs
113 the JDK mainline.
114
115 For example, a number of "premain CDS + AOT : 291" indicates that if the application takes
116 1000 ms to start-up with the JDK mainline, it takes only 291 ms to start up when all the
117 current set of Leyden optimizations for CDS and AOT are enabled.
118
119 The benchmark results are collected with `make bench`in the following directories:
120
121 - helidon-quickstart-se
122 - micronaut-first-app
123 - quarkus-getting-started
124 - spring-petclinic
125
126 The version used in the comparisons:
127
128 - JDK Mainline: https://github.com/openjdk/jdk/commit/70944ca54ad0090c734bb5b3082beb33450c4877
129 - Leyden: https://github.com/openjdk/leyden/commit/9fa972214934d30f67db5fd4d1b8007636ac1428
130
131 The benchmarks were executed on an 8-core Intel i7-10700 CPU @ 2.90GHz with 32GB RAM running Ubuntu 22.04.3 LTS.
132
133 ### Helidon Quick Start (SE) Demo
134
135 ```mermaid
136 gantt
137 title Elapsed time (normalized, smaller is better)
138 todayMarker off
139 dateFormat X
140 axisFormat %s
141
142 mainline default : 0, 1000
143 mainline custom static CDS : 0, 632
144 premain custom static CDS only : 0, 376
145 premain CDS + AOT : 0, 291
146 ```
147
148 ### Micronaut First App Demo
149
150 ```mermaid
151 gantt
152 title Elapsed time (normalized, smaller is better)
153 todayMarker off
154 dateFormat X
155 axisFormat %s
156
157 mainline default : 0, 1000
158 mainline custom static CDS : 0, 558
159 premain custom static CDS only : 0, 410
160 premain CDS + AOT : 0, 353
161 ```
162
163
164 ### Quarkus Getting Started Demo
165
166 ```mermaid
167 gantt
168 title Elapsed time (normalized, smaller is better)
169 todayMarker off
170 dateFormat X
171 axisFormat %s
172
173 mainline default : 0, 1000
174 mainline custom static CDS : 0, 568
175 premain custom static CDS only : 0, 395
176 premain CDS + AOT : 0, 317
177 ```
178
179 ### Spring PetClinic Demo
180
181 ```mermaid
182 gantt
183 title Elapsed time (normalized, smaller is better)
184 todayMarker off
185 dateFormat X
186 axisFormat %s
187
188 mainline default : 0, 1000
189 mainline custom static CDS : 0, 695
190 premain custom static CDS only : 0, 563
191 premain CDS + AOT : 0, 368
192 ```
193
194 ## Regression Testing
195
196 Leyden-specific tests have been added to the following directories in the repo:
197
198 - [test/hotspot/jtreg/runtime/cds/appcds/applications](../runtime/cds/appcds/applications)
199 - [test/hotspot/jtreg/runtime/cds/appcds/indy](../runtime/cds/appcds/indy)
200 - [test/hotspot/jtreg/runtime/cds/appcds/leyden](../runtime/cds/appcds/leyden)
201 - [test/hotspot/jtreg/runtime/cds/appcds/preloadedClasses](../runtime/cds/appcds/preloadedClasses)
202
203 These test cases can be executed using jtreg. Some of the tests (in the applications directories)
204 require binaries to be built separately. Please refer to the script [lib/build-for-jtreg.sh](lib/build-for-jtreg.sh)