1 ## State of foreign memory support
  3 **December 2023**
  5 **Maurizio Cimadamore**
  7 A crucial part of any native interop story lies in the ability of accessing off-heap memory efficiently and safely. Java achieves this goal through the Foreign Function & Memory API (FFM API in short), parts of which have been available as an [incubating](https://openjdk.java.net/jeps/11) API since Java [14](https://openjdk.java.net/jeps/370). The FFM API introduces abstractions to allocate and access flat memory regions (whether on- or off-heap), to manage the lifecycle of memory resources and to model native memory addresses.
  9 ### Memory segments and arenas
 11 Memory segments are abstractions which can be used to model contiguous memory regions, located either on-heap (i.e. *heap segments*) or off- the Java heap (i.e. *native segments*). Memory segments provide *strong* spatial, temporal and thread-confinement guarantees which make memory dereference operation *safe* (more on that later), although in most simple cases some properties of memory segments can safely be ignored.
 13 For instance, the following snippet allocates 100 bytes off-heap:
 15 ```java
 16 MemorySegment segment = Arena.global().allocate(100);
 17 ```
 19 The above code allocates a 100-bytes long memory segment, using an *arena*. The FFM API provides several kinds of arena, which can be used to control the lifecycle of the allocated native segments in different ways. In this example, the segment is allocated with the *global* arena. Memory segments allocated with this arena are always *alive* and their backing regions of memory are never deallocated. In other words, we say that the above segment has an *unbounded* lifetime.
 21 > Note: the lifetime of a memory segment is modelled by a *scope* (see `MemorySegment.Scope`). A memory segment can be accessed as long as its associated scope is *alive* (see `Scope::isAlive`). In most cases, the scope of a memory segment is the scope of the arena which allocated that segment. Accessing the scope of a segment can be useful to perform lifetime queries (e.g. asking whether a segment has the same lifetime as that of another segment), creating custom arenas and unsafely assigning new temporal bounds to an existing native memory segments (these topics are explored in more details below).
 23 Most programs, though, require off-heap memory to be deallocated while the program is running, and thus need memory segments with *bounded* lifetimes. The simplest way to obtain a segment with bounded lifetime is to use an *automatic arena*:
 25 ```java
 26 MemorySegment segment = Arena.ofAuto().allocate(100);
 27 ```
 29 Segments allocated with an automatic arena are alive as long as they are determined to be reachable by the garbage collector. In other words, the above snippet creates a native segment whose behavior closely matches that of a `ByteBuffer` allocated with the `allocateDirect` factory.
 31 There are cases, however, where automatic deallocation is not enough: consider the case where a large memory segment is mapped from a file (this is possible using `FileChannel::map`); in this case, an application would probably prefer to release (e.g. `unmap`) the memory associated with this segment in a *deterministic* fashion, to ensure that memory doesn't remain available for longer than it needs to.
 33 A *confined* arena allocates segment featuring a bounded *and* deterministic lifetime. A memory segment allocated with a confined arena is alive from the time when the arena is opened, until the time when the arena is closed (at which point the segments become inaccessible). Multiple segments allocated with the same arena enjoy the *same* bounded lifetime and can safely contain mutual references. For example, this code opens an arena and uses it to allocate several native segments:
 35 ```java
 36 try (Arena arena = Arena.ofConfined()) {
 37     MemorySegment segment1 = arena.allocate(100);
 38     MemorySegment segment2 = arena.allocate(100);
 39     ...
 40     MemorySegment segmentN = arena.allocate(100);
 41 } // all segments are deallocated here
 42 ```
 44 When the arena is closed (above, this is done with the *try-with-resources* construct) the arena is no longer alive, all the segments associated with it are invalidated atomically, and the regions of memory backing the segments are deallocated.
 46 A confined arena's deterministic lifetime comes at a price: only one thread can access the memory segments allocated in a confined arena. If multiple threads need access to a segment, then a *shared* arena can be used (`Arena::ofShared`). The memory segments allocated in a shared arena can be accessed by multiple threads, and any thread (regardless of whether it was involved in access) can close the shared arena to deallocate the segments. The closure will atomically invalidate the segments, though deallocation of the regions of memory backing the segments might not occur immediately: an expensive synchronization operation<a href="#1"><sup>1</sup></a> is needed to detect and cancel pending concurrent access operations on the segments.
 48 In summary, an arena controls *which* threads can access a memory segment and *when*, in order to provide both strong temporal safety and a predictable performance model. The FFM API offers a choice of arenas so that a client can trade off breadth-of-access against timeliness of deallocation.
 50 ### Slicing segments
 52 Memory segments support *slicing* — that is, given a segment, it is possible to create a new segment whose spatial bounds are stricter than that of the original segment:
 54 ```java
 55 MemorySegment segment = Arena.ofAuto().allocate(10);
 56 MemorySegment slice = segment.asSlice(4, 4);
 57 ```
 59 The above code creates a slice that starts at offset 4 and has a length of 4 bytes. Slices have the *same* temporal bounds (i.e. segment scope) as the parent segment. In the above example, the memory associated with the parent segment will not be released as long as there is at least one *reachable* slice derived from that segment.
 61 To process the contents of a memory segment in bulk, a memory segment can be turned into a stream of slices, using the `MemorySegment::elements` method:
 63 ```java
 64 SequenceLayout seq = MemoryLayout.sequenceLayout(1_000_000, JAVA_INT);
 65 SequenceLayout bulk_element = MemoryLayout.sequenceLayout(100, JAVA_INT);
 67 try (Arena arena = Arena.ofShared()) {
 68     MemorySegment segment = arena.allocate(seq);
 69     int sum = segment.elements(bulk_element).parallel()
 70                        .mapToInt(slice -> {
 71                            int res = 0;
 72                            for (int i = 0; i < 100 ; i++) {
 73                                res += slice.getAtIndex(JAVA_INT, i);
 74                            }
 75                            return res;
 76                        }).sum();
 77 }
 78 ```
 80 The `MemorySegment::elements` method takes an element layout and returns a new stream. The stream is built on top of a spliterator instance (see `MemorySegment::spliterator`) which splits the segment into chunks whose size matches that of the provided layout. Here, we want to sum elements in an array which contains a million of elements; now, doing a parallel sum where each computation processes *exactly* one element would be inefficient, so instead we use a *bulk* element layout. The bulk element layout is a sequence layout containing a group of 100 elements — which should make it more amenable to parallel processing. Since we are using `Stream::parallel` to work on disjoint slices in parallel, here we use a *shared* arena, to ensure that the resulting segment can be accessed by multiple threads.
 82 ### Accessing segments
 84 Memory segments can be dereferenced easily, by using *value layouts* (layouts are covered in greater details in the next section). A value layout captures information such as:
 86 - The number of bytes to be dereferenced;
 87 - The alignment constraints of the address at which dereference occurs;
 88 - The endianness with which bytes are stored in said memory region;
 89 - The Java type to be used in the dereference operation (e.g. `int` vs `float`).
 91 For instance, the layout constant `ValueLayout.JAVA_INT` is four bytes wide, has no alignment constraints, uses the native platform endianness (e.g. little-endian on Linux/x64) and is associated with the Java type `int`. The following example reads pairs of 32-bit values (as Java ints) and uses them to construct an array of points:
 93 ```java
 94 record Point(int x, int y);
 95 MemorySegment segment = Arena.ofAuto().allocate(10 * 4 * 2);
 96 Point[] values = new Point[10];
 97 for (int i = 0 ; i < values.length ; i++) {
 98     int x = segment.getAtIndex(JAVA_INT, i * 2);
 99     int y = segment.getAtIndex(JAVA_INT, (i * 2) + 1);
100     values[i] = new Point(x, y);
101 }
102 ```
104 The above snippet allocates a flat array of 80 bytes using an automatic arena. Then, inside the loop, elements in the array are accessed using the `MemorySegment::getAtIndex` method, which accesses `int` elements in a segment at a certain *logical* index (under the hood, the segment offset being accessed is obtained by multiplying the logical index by 4, which is the stride of a Java `int` array). Thus, all coordinates `x` and `y` are collected into instances of a `Point` record.
106 ### Structured access
108 Expressing byte offsets (as in the example above) can lead to code that is hard to read, and very fragile — as memory layout invariants are captured, implicitly, in the constants used to scale offsets. To address this issue, clients can use a `MemoryLayout` to describe the contents of a memory segment *programmatically*. For instance, the layout of the array used in the above example can be expressed using the following code <a href="#2"><sup>2</sup></a>:
110 ```java
111 MemoryLayout points = MemoryLayout.sequenceLayout(10,
112     MemoryLayout.structLayout(
113         JAVA_INT.withName("x"),
114         JAVA_INT.withName("y")
115     )
116 );            
117 ```
119 That is, our layout is a repetition of 10 *struct* elements, each struct element containing two 32-bit values each. Once defined, a memory layout can be queried — for instance we can compute the offset of the `y` coordinate in the 4th element of the `points` array:
121 ```java
122 long y3 = points.byteOffset(PathElement.sequenceElement(3), PathElement.groupElement("y")); // 28
123 ```
125 To specify which nested layout element should be used for the offset calculation we use a *layout path*, a selection expression that navigates the layout, from the *root* layout, down to the leaf layout we wish to select; in this case we need to select the 4th layout element in the sequence, and then select the layout named `y` inside the selected group layout.
127 One of the things that can be derived from a layout is a *memory access var handle*. A memory access var handle is a special kind of var handle which takes a memory segment access coordinate, together with a byte offset — the offset, relative to the segment's base address at which the dereference operation should occur. With memory access var handles we can rewrite our example above as follows:
129 ```java
130 MemorySegment segment = Arena.ofAuto().allocate(points);
131 VarHandle xHandle = points.varHandle(PathElement.sequenceElement(), PathElement.groupElement("x"));
132 VarHandle yHandle = points.varHandle(PathElement.sequenceElement(), PathElement.groupElement("y"));
133 Point[] values = new Point[10];
134 for (int i = 0 ; i < values.length ; i++) {
135     int x = (int)xHandle.get(segment, 0L /* base offset */, (long)i /* index */);
136     int y = (int)yHandle.get(segment, 0L /* base offset */, (long)i /* index */);
137 }
138 ```
140 In the above, `xHandle` and `yHandle` are two var handle instances whose type is `int` and which takes three access coordinates:
142 1. a `MemorySegment` instance; the segment whose memory should be dereferenced
143 2. a *base offset*, which indicates the portions of the memory segment to be accessed; this is typically left to zero (as above), but can be useful when combining memory access var handles (see below);
144 3. a *logical* index, which is used to select the element of the sequence we want to access (as the layout path used to construct these var handles contains one free dimension)
146 In other words, the offset of the access operation can be expressed as follows:
148 ```java
149 offset = baseOffset + (index * JAVA_INT.byteSize());
150 ```
152 Or, equivalently, using the `MemoryLayout::scale` method, as:
154 ```java
155 offset = JAVA_INT.scale(baseOffset, index);
156 ```
158 Note that memory access var handles (as any other var handle) are *strongly* typed; and to get maximum efficiency, it is generally necessary to introduce casts to make sure that the access coordinates match the expected types — in this case we have to cast `i` into a `long`; similarly, since the signature polymorphic method `VarHandle::get` notionally returns `Object` a cast is necessary to force the right return type the var handle operation <a href="#3"><sup>3</sup></a>.
160 In other words, manual offset computation is no longer needed — offsets and strides can in fact be derived from the layout object; note how `yHandle` is able to compute the required offset of the `y` coordinate in the flat array without the need of any error-prone arithmetic computation.
162 ### Combining memory access handles
164 We have seen in the previous sections how memory access var handles dramatically simplify user code when structured access is involved. While deriving memory access var handles from layout is the most convenient option, the FFM API also allows to create such memory access var handles in a standalone fashion, as demonstrated in the following code:
166 ```java
167 VarHandle intHandle = JAVA_INT.varHandle(); // (MS, J) -> I
168 ```
170 The above code creates a memory access var handle which reads/writes `int` values at a certain byte offset in a segment. To create this var handle we have to specify a carrier type — the type we want to use e.g. to extract values from memory, as well as whether any byte swapping should be applied when contents are read from or stored to memory. Additionally, the user might want to impose additional constraints on how memory dereferences should occur; for instance, a client might want to prevent access to misaligned 32 bit values. Of course, all this information can be succinctly derived from the provided value layout (`JAVA_INT` in the above example).
172 The attentive reader might have noted how the var handles obtained from the sequence layout in the previous section can be in fact derived from  the simple memory access var handle we have constructed here. That is, var handles can be adapted and turned into more complex var handles, using var handle *combinators*. Developers familiar with the method handle API know how simpler method handles can be combined into more complex ones using the various combinator methods in the `MethodHandles` class. These methods allow, for instance, to insert (or bind) arguments into a target method handle, filter return values, permute arguments and much more.
174 The FFM API adds a rich set of var handle combinators in the `MethodHandles` class; with these tools, developers can express var handle transformations such as:
176 * mapping a var handle carrier type into a different one, using an embedding/projection method handle pairs
177 * filter one or more var handle access coordinates using unary filters
178 * permute var handle access coordinates
179 * bind concrete access coordinates to an existing var handle
181 Without diving too deep, let's consider how we might want to take a basic memory access handle and turn it into a var handle which dereference a segment at a specific offset (again using the `points` layout defined previously):
183 ```java
184 VarHandle intHandle = JAVA_INT.varHandle(); // (MS, J) -> I
185 long offsetOfY = points.byteOffset(PathElement.sequenceElement(3), PathElement.groupElement("y"));
186 VarHandle valueHandle = MethodHandles.insertCoordinates(intHandle, 1, offsetOfValue); // (MS) -> I
187 ```
189 We have been able to derive, from a basic memory access var handle, a new var handle that dereferences a segment at a given fixed offset. It is easy to see how other, richer, var handles obtained using a memory layout can also be constructed manually using the var handle combinators provided by the FFM API.
191 ### Segment allocators and custom arenas
193 Memory allocation is often a bottleneck when clients use off-heap memory. The FFM API therefore includes a `SegmentAllocator` interface to define operations to allocate and initialize memory segments. As a convenience, the `Arena` interface extends the `SegmentAllocator` interface so that arenas can be used to allocate native segments. In other words, `Arena` is a "one-stop shop" for flexible allocation and timely deallocation of off-heap memory:
195 ```java
196 FileChannel channel = ...
197 try (Arena offHeap = Arena.ofConfined()) {
198     MemorySegment nativeArray   = offHeap.allocateFrom(ValueLayout.JAVA_INT, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9);
199     MemorySegment nativeString  = offHeap.allocateFrom("Hello!");
201     MemorySegment mappedSegment = channel.map(MapMode.READ_WRITE, 0, 1000, arena);
202    ...
203 } // memory released here
204 ```
206 Segment allocators can also be obtained via factories in the `SegmentAllocator` interface. For example, one factory creates a *slicing allocator* that responds to allocation requests by returning memory segments which are part of a previously allocated segment; thus, many requests can be satisfied without physically allocating more memory. The following code obtains a slicing allocator over an existing segment, then uses it to allocate a segment initialized from a Java array:
208 ```java
209 MemorySegment segment = ...
210 SegmentAllocator allocator = SegmentAllocator.slicingAllocator(segment);
211 for (int i = 0 ; i < 10 ; i++) {
212     MemorySegment s = allocator.allocateFrom(JAVA_INT, 1, 2, 3, 4, 5);
213     ...
214 }
215 ```
217 A segment allocator can be used as a building block to create an arena that supports a custom allocation strategy. For example, if many segments share the same bounded lifetime, then an arena could use a slicing allocator to allocate the segments efficiently. This lets clients enjoy both scalable allocation (thanks to slicing) and deterministic deallocation (thanks to the arena).
219 As an example, the following code defines a *slicing arena* that behaves like a confined arena (i.e., single-threaded access), but internally uses a slicing allocator to respond to allocation requests.  When the slicing arena is closed, the underlying confined arena is also closed; this will invalidate all segments allocated with the slicing arena:
221 ```java
222 class SlicingArena {
223      final Arena arena = Arena.ofConfined();
224      final SegmentAllocator slicingAllocator;
226      SlicingArena(long size) {
227          slicingAllocator = SegmentAllocator.slicingAllocator(arena.allocate(size));
228      }
230      public void allocate(long byteSize, long byteAlignment) {
231          return slicingAllocator.allocate(byteSize, byteAlignment);
232      }
234      public MemorySegment.Scope scope() {
235          return arena.scope();
236      }
238      public void close() {
239          return arena.close();
240      }
241 }
242 ```
244 The earlier code which used a slicing allocator directly can now be written more succinctly, as follows:
246 ```java
247 try (Arena slicingArena = new SlicingArena(1000)) {
248      for (int i = 0 ; i < 10 ; i++) {
249          MemorySegment s = arena.allocateFrom(JAVA_INT, 1, 2, 3, 4, 5);
250          ...
251      }
252 } // all memory allocated is released here
253 ```
255 * <a id="1"/>(<sup>1</sup>):<small> Shared arenas rely on VM thread-local handshakes (JEP [312](https://openjdk.java.net/jeps/312)) to implement lock-free, safe, shared memory access; that is, when it comes to memory access, there should be no difference in performance between a shared segment and a confined segment. On the other hand, `Arena::close` might be slower on shared arenas than on confined ones.</small>
256 * <a id="2"/>(<sup>2</sup>):<small> In general, deriving a complete layout from a C `struct` declaration is no trivial matter, and it's one of those areas where tooling can help greatly.</small>
257 * <a id="3"/>(<sup>3</sup>):<small> Clients can enforce stricter type checking when interacting with `VarHandle` instances, by obtaining an *exact* var handle, using the `VarHandle::withInvokeExactBehavior` method.</small>