1 # Building HAT
2
3 ----
4
5 * [Contents](hat-00.md)
6 * House Keeping
7 * [Project Layout](hat-01-01-project-layout.md)
8 * [Building Babylon](hat-01-02-building-babylon.md)
9 * [Building HAT](hat-01-03-building-hat.md)
10 * [Enabling the CUDA Backend](hat-01-05-building-hat-for-cuda.md)
11 * Programming Model
12 * [Programming Model](hat-03-programming-model.md)
13 * Interface Mapping
14 * [Interface Mapping Overview](hat-04-01-interface-mapping.md)
15 * [Cascade Interface Mapping](hat-04-02-cascade-interface-mapping.md)
16 * Implementation Detail
17 * [Walkthrough Of Accelerator.compute()](hat-accelerator-compute.md)
18 * [How we minimize buffer transfers](hat-minimizing-buffer-transfers.md)
19
20 ---
21
22 # Building HAT with Script
23
24 We initially used maven and cmake to build hat. If you feel more comfortable
25 with maven consider [building with maven and cmake](hat-01-03-building-hat-with-maven.md)
26 but it is possible that maven support will be removed if the `Script` approach takes off.
27
28 ## Dependencies
29
30 Before we start to build HAT we will need `cmake` and `jextract` installed.
31
32 You can download jextract from [here](https://jdk.java.net/jextract/)
33
34 Use `sudo apt` on Linux or `brew install`.
35
36 ```bash
37 sudo apt install cmake
38
39 ```
40
41 ```bash
42 brew install cmake
43 ```
44
45 You will also need a Babylon JDK built (the one we built [here](hat-01-02-building-babylon.md))
46
47 ## Setting your PATH variable
48
49 To build HAT we will need `JAVA_HOME` to point to our prebuilt babylon jdk
50
51 I suggest you also create a `JEXTRACT_HOME` var to point to the location where you placed `JEXTRACT`)
52
53 In my case
54 ```
55 export JEXTRACT_HOME=/Users/me/jextract-22
56 ```
57
58 Make sure also that `cmake` in in your PATH
59
60 ## ./env.bash
61
62 Thankfully just sourcing the top level `env.bash` script should then be able to set up your `PATH` for you.
63
64 It should detect the arch type (AARCH64 or X86_46) and
65 select the correct relative parent dir for your BABYLON_JDK and inject that dir in your `PATH`.
66
67 It should also add jextract to your PATH (based on the value you set above for `JEXTRACT_HOME`)
68
69
70 ```bash
71 cd hat
72 export JEXTRACT_HOME=/Users/me/jextract-22
73 . ./env.bash
74 echo ${JAVA_HOME}
75 /Users/me/github/babylon/hat/../build/macosx-aarch64-server-release/jdk
76 echo ${PATH}
77 /Users/me/github/babylon/hat/../build/macosx-aarch64-server-release/jdk/bin:/Users/me/jextract-22/bin:/usr/local/bin:......
78 ```
79
80 ## Building using bld
81
82 To build hat artifacts (hat jar + backends and examples)
83 ```bash
84 java @hat/bld
85 ```
86
87 This places build artifacts in the `build` and `stages` dirs
88
89 ```bash
90 cd hat
91 . ./env.bash
92 java @hat/bld
93 ls build
94 hat-1.0.jar hat-example-heal-1.0.jar libptx_backend.dylib
95 hat-backend-ffi-cuda-1.0.jar hat-example-mandel-1.0.jar libspirv_backend.dylib
96 hat-backend-ffi-mock-1.0.jar hat-example-squares-1.0.jar mock_info
97 hat-backend-ffi-opencl-1.0.jar hat-example-view-1.0.jar opencl_info
98 hat-backend-ffi-ptx-1.0.jar hat-example-violajones-1.0.jar ptx_info
99 hat-backend-ffi-spirv-1.0.jar libmock_backend.dylib spirv_info
100 hat-example-experiments-1.0.jar libopencl_backend.dylib
101 ls stage
102 opencl_jextracted opengl_jextracted
103 ```
104
105 `bld` relies on cmake to build native code for backends, so if cmake finds OpenCL libs/headers, you will see libopencl_backend (.so or .dylib) in the build dir, if cmake finds CUDA you will see libcuda_backend(.so or .dylib)
106
107 We have another script called `sanity` which will check all .md/.java/.cpp/.h for tabs, lines that end with whitespace
108 or files without appropriate licence headers
109
110 This is run using
111
112 ```bash
113 java @hat/sanity
114 ```
115
116 ## Running an example
117
118 To run a HAT example we can run from the artifacts in `build` dir
119
120 ```bash
121 ${JAVA_HOME}/bin/java \
122 --add-modules jdk.incubator.code --enable-preview --enable-native-access=ALL-UNNAMED \
123 --class-path build/core-1.0.jar:build/hat-backend-ffi-shared-1.0.jar:build/hat-backend-ffi-opencl-1.0.jar:build/hat-example-mandel-1.0.jar \
124 --add-exports=java.base/jdk.internal=ALL-UNNAMED \
125 -Djava.library.path=build\
126 mandel.Main
127 ```
128
129 The `hat/run.java` script can also be used which simply needs the backend
130 name `ffi-opencl|ffi-java|ffi-cuda|ffi-ptx|ffi-mock` and the package name `mandel`
131
132 ```bash
133 java @hat/run ffi-opencl mandel
134 ```
135
136 If you pass `headless` as the first arg
137
138 ```bash
139 java @hat/run headless ffi-opencl mandel
140 ```
141
142 This sets `-Dheadless=true` and passes '--headless' to the example. Some examples can use this to avoid launching UI.
143
144
145 # More Bld info
146 `hat/Script.java` is an evolving set of static methods and types required (so far.. ;) )
147 to be able to build HAT, hat backends and examples via the `bld` script
148
149 We rely on java's ability to launch java source directly (without needing to javac first)
150
151 * [JEP 458: Launch Multi-File Source-Code Program](https://openjdk.org/jeps/458)
152 * [JEP 330: Launch Single-File Source-Code Programs](https://openjdk.org/jeps/330)
153
154 The `hat/bld.java` script (really java source) can be run like this
155
156 ```bash
157 java --add-modules jdk.incubator.code --enable-preview --source 26 hat/bld.java
158 ```
159
160 In our case the magic is under the `hat`subdir
161
162 We also have a handy `hat/XXXX` which allows us to avoid specifying common args `--enable-preview --source 26` eash time we launch a script
163
164 ```
165 hat
166 ├── hat
167 | ├── Script.java
168 | ├── sanity (the args for sanity.java) "--enable-preview --source 26 sanity"
169 | |-- sanity.java (the script)
170 | ├── run (the args for sanity.java) "--enable-preview --source 26 hatrun"
171 | |-- run.java (the script)
172 | ├── bld (the args for bld.java) "--enable-preview --source 26 bld"
173 | ├── bld.java (the script)
174
175 ```
176
177 For example
178 ```bash
179 java @hat/bld
180 ```
181
182 Is just a shortcut for
183 ```bash
184 java --add-modules jdk.incubator.code --enable-preview --source 26 hat/bld.java
185 ```
186
187 # HAT Testing Framework
188
189 ## Local Testing
190
191 For the OpenCL backend:
192
193 ```bash
194 java -cp hat/job.jar hat.java test-suite ffi-opencl
195 ```
196
197 For the CUDA backend:
198
199 ```bash
200 java -cp hat/job.jar hat.java test-suite ffi-cuda
201 ```
202
203 ## Individual tests
204
205 ```bash
206 java -cp hat/job.jar hat.java test-suite ffi-cuda CLASS#method
207 ```
208
209 Passing a particular method to test is optional.
210
211 For example, to test the whole `TestArrays` class:
212
213 ```bash
214 java -cp hat/job.jar hat.java test ffi-opencl hat.test.TestArrays
215
216 HAT Engine Testing Framework
217 Testing Backend: hat.backend.ffi.OpenCLBackend
218
219 Class: hat.test.TestArrays
220 Testing: #testHelloHat ..................... [ok]
221 Testing: #testVectorAddition ..................... [ok]
222 Testing: #testVectorSaxpy ..................... [ok]
223 Testing: #testSmallGrid ..................... [ok]
224
225 passed: 4, failed: 0, unsupported: 0
226 ```
227
228 To test a single method (e.g., `testVectorAddition`):
229
230
231 ```bash
232 java -cp hat/job.jar hat.java test ffi-opencl hat.test.TestArrays#testVectorAddition
233
234 HAT Engine Testing Framework
235 Testing Backend: hat.backend.ffi.OpenCLBackend
236
237 Class: hat.test.TestArrays
238 Testing: #testVectorAddition ..................... [ok]
239
240 passed: 1, failed: 0, unsupported: 0
241 ```
242
243 ## Remote Testing
244
245 HAT provides its own testing framework that can be used to test on remote GPU servers.
246 First, you need to generate and configure the template:
247
248 ```bash
249 bash scripts/remoteTesting.sh --generate-config-file
250 ```
251
252 This flag generates a file in the local directory called `remoteTesting.conf`.
253 We just need to fill the template with the server names, and user names, fork to use, backends to test and the branch to use.
254
255 For instance:
256
257 ```bash
258 # HAT Remote Testing Settings
259 SERVERS=server1 server2
260 REMOTE_USERS=juan juan
261 FORK=git@github.com:my-fork/babylon.git
262
263 #List of Backends to test
264 BACKENDS=ffi-cuda ffi-opencl
265
266 ## Remote path. It assumes all servers use the same path
267 REMOTE_PATH=repos/babylon
268 ## Branch to test
269 BRANCH=code-reflection
270 ```
271
272 This assumes you have the `ssh-keygen` already configured.
273
274 Then, we need to build the project Babylon:
275
276 ```bash
277 bash scripts/remoteTesting.sh --build-babylon
278 ```
279
280 This builds babylon for each of the servers specified. The project is stored in the path specified in `REMOTE_PATH`.
281
282 Once it is finished, you can run the unit-tests on the remote GPU servers as follows:
283
284 ```bash
285 bash scripts/remoteTesting.sh
286 ```
287
288 ----
289 ### HAT runtime environment variable.
290
291 During ffi-backend development we added some useful flags to pass to native code to allow us to trace calls, inject logging, control options.
292
293 These should be considered just development flags.
294
295 At runtime the ffi-backends all communicate from the java side to the native side via a 32 bit 'config' int.
296
297 The Java side class [hat.backend.ffi.Config](https://github.com/openjdk/babylon/blob/code-reflection/hat/backends/ffi/shared/src/main/java/hat/backend/ffi/Config.java)
298
299 Is initialized from an environment variable `HAT` at runtime
300
301 So for example when we launch
302 ```bash
303 java @hat/run ffi-opencl heal
304 ```
305
306 We can pass config info to `HAT` via either `-DHAT=xxx` or via the `HAT` ENV variable.
307
308 So for example to get HAT to dump the text form of the kernel code models.
309
310 ```
311 HAT=INFO,SHOW_KERNEL_MODEL java @hat/run ffi-opencl heal
312 ```
313
314 Or to show generated opencl or cuda code
315 ```
316 HAT=INFO,SHOW_CODE java @hat/run ffi-opencl heal
317 ```
318
319 This is particularly useful for selecting PLATFORM + DEVICE if you have multiple GPU devices.
320
321 Here we select DEVICE 0 on PLATFORM 0 (actually the default)
322 ```
323 HAT=INFO,PLATFORM:0,DEVICE:0,SHOW_CODE ...
324 ```
325 or for DEVICE 1 on PLATFORM 1
326 ```
327 HAT=INFO,PLATFORM:1,DEVICE:1,SHOW_CODE ...
328 ```
329
330 No the platform and device id's are 4 bits. This probably works for development, but we will need a more expressive way of capturing this via the accelerator selection.
331
332 This just allows us to test without code changes.
333
334 To keep he java code and the native code in sync. The `main` method at the end of
335 [hat.backend.ffi.Config](https://github.com/openjdk/babylon/blob/code-reflection/hat/backends/ffi/shared/src/main/java/hat/backend/ffi/Config.java)
336 is actually used to create the C99 header for Cuda and OpenCL ffi-backends.
337
338 So whenever we change the Java config class we should run the main method to generate the header. This is not really robust, (but proved way better than trying to remember for all backends) but you need to know, if you add move config bits to the Java side.
339
340 The Main method uses a variant of the code builders used for C99 style code (CUDA/OpenCL) to generate the config.h header.
341
342 This is how we keep the ffi based backends in sync
343
344 Some more useful `config bits`
345
346 ```
347 HAT=PTX java @hat/run ffi-cuda ....
348 ```
349 Sends PTX generated by our prototype PTX generator to the backend (for CUDA) rather than C99 code.
350
351 ```
352 HAT=INFO java @hat/run ....
353 ```
354
355 Will dump all of the `config bits` from the native side.
356
357 At present this yields
358 ```
359 native minimizeCopies 0
360 native trace 0
361 native profile 0
362 native showCode 0
363 native showKernelModel 0
364 native showComputeModel 0
365 native info 1
366 native traceCopies 0
367 native traceSkippedCopies 0
368 native traceEnqueues 0
369 native traceCalls 0
370 native showWhy 0
371 native showState 0
372 native ptx 0
373 native interpret 0
374 ```
375 Generally the flags all represent single bits (except PLATFORM:n and DEVICE:n) and are set using comma separated uppercase+underscore string forms.
376
377
378 So to experiment with `minimizeCopies` and to `showCode` we set `HAT=MINIMIZE_COPIES,SHOW_CODE`