0
# Core Platform Operations
1

2
The `Platform` class provides the foundation for all unsafe memory operations in Spark, wrapping Java's `sun.misc.Unsafe` for high-performance direct memory access. This class is critical for bypassing Java's safety mechanisms to achieve maximum performance in data processing operations.
3

4
## Core Imports
5

6
```java
7
import org.apache.spark.unsafe.Platform;
8
```
9

10
## Usage Examples
11

12
### Basic Memory Operations
13

14
```java
15
// Allocate 1024 bytes of off-heap memory
16
long address = Platform.allocateMemory(1024);
17

18
// Write data to memory
19
Platform.putLong(null, address, 42L);
20
Platform.putInt(null, address + 8, 100);
21

22
// Read data from memory
23
long longValue = Platform.getLong(null, address);
24
int intValue = Platform.getInt(null, address + 8);
25

26
// Clean up
27
Platform.freeMemory(address);
28
```
29

30
### Array Memory Access
31

32
```java
33
long[] array = new long[10];
34
long baseOffset = Platform.LONG_ARRAY_OFFSET;
35

36
// Write to array using unsafe operations
37
for (int i = 0; i < array.length; i++) {
38
    Platform.putLong(array, baseOffset + (i * 8), i * 10);
39
}
40

41
// Read from array using unsafe operations
42
for (int i = 0; i < array.length; i++) {
43
    long value = Platform.getLong(array, baseOffset + (i * 8));
44
    System.out.println("Index " + i + ": " + value);
45
}
46
```
47

48
### Memory Copying
49

50
```java
51
// Create source and destination buffers
52
long srcAddress = Platform.allocateMemory(1024);
53
long dstAddress = Platform.allocateMemory(1024);
54

55
// Fill source with data
56
Platform.setMemory(srcAddress, (byte) 0xAB, 1024);
57

58
// Copy memory
59
Platform.copyMemory(null, srcAddress, null, dstAddress, 1024);
60

61
// Cleanup
62
Platform.freeMemory(srcAddress);
63
Platform.freeMemory(dstAddress);
64
```
65

66
## API Reference
67

68
### Platform Information
69

70
```java { .api }
71
/**
72
 * Returns true when running JVM supports unaligned memory access.
73
 */
74
public static boolean unaligned()
75
```
76

77
### Array Base Offsets
78

79
```java { .api }
80
public static final int BOOLEAN_ARRAY_OFFSET
81
public static final int BYTE_ARRAY_OFFSET
82
public static final int SHORT_ARRAY_OFFSET
83
public static final int INT_ARRAY_OFFSET
84
public static final int LONG_ARRAY_OFFSET
85
public static final int FLOAT_ARRAY_OFFSET
86
public static final int DOUBLE_ARRAY_OFFSET
87
```
88

89
### Memory Allocation
90

91
```java { .api }
92
/**
93
 * Allocates off-heap memory and returns the address.
94
 */
95
public static long allocateMemory(long size)
96

97
/**
98
 * Frees previously allocated off-heap memory.
99
 */
100
public static void freeMemory(long address)
101

102
/**
103
 * Reallocates memory block to new size, copying existing data.
104
 */
105
public static long reallocateMemory(long address, long oldSize, long newSize)
106

107
/**
108
 * Allocates DirectByteBuffer bypassing JVM MaxDirectMemorySize limit.
109
 */
110
public static java.nio.ByteBuffer allocateDirectBuffer(int size)
111
```
112

113
### Data Access Methods
114

115
#### Integer Operations
116

117
```java { .api }
118
public static int getInt(Object object, long offset)
119
public static void putInt(Object object, long offset, int value)
120
```
121

122
#### Boolean Operations
123

124
```java { .api }
125
public static boolean getBoolean(Object object, long offset)
126
public static void putBoolean(Object object, long offset, boolean value)
127
```
128

129
#### Byte Operations
130

131
```java { .api }
132
public static byte getByte(Object object, long offset)
133
public static void putByte(Object object, long offset, byte value)
134
```
135

136
#### Short Operations
137

138
```java { .api }
139
public static short getShort(Object object, long offset)
140
public static void putShort(Object object, long offset, short value)
141
```
142

143
#### Long Operations
144

145
```java { .api }
146
public static long getLong(Object object, long offset)
147
public static void putLong(Object object, long offset, long value)
148
```
149

150
#### Float Operations
151

152
```java { .api }
153
public static float getFloat(Object object, long offset)
154
public static void putFloat(Object object, long offset, float value)
155
```
156

157
#### Double Operations
158

159
```java { .api }
160
public static double getDouble(Object object, long offset)
161
public static void putDouble(Object object, long offset, double value)
162
```
163

164
#### Object Reference Operations
165

166
```java { .api }
167
/**
168
 * Reads object reference with volatile semantics.
169
 */
170
public static Object getObjectVolatile(Object object, long offset)
171

172
/**
173
 * Writes object reference with volatile semantics.
174
 */
175
public static void putObjectVolatile(Object object, long offset, Object value)
176
```
177

178
### Memory Operations
179

180
```java { .api }
181
/**
182
 * Fills memory with specified byte value.
183
 */
184
public static void setMemory(Object object, long offset, long size, byte value)
185

186
/**
187
 * Fills off-heap memory with specified byte value.
188
 */
189
public static void setMemory(long address, byte value, long size)
190

191
/**
192
 * Copies memory between locations, handling overlapping regions.
193
 */
194
public static void copyMemory(Object src, long srcOffset, Object dst, long dstOffset, long length)
195
```
196

197
### Exception Handling
198

199
```java { .api }
200
/**
201
 * Throws exception bypassing compiler checks for checked exceptions.
202
 */
203
public static void throwException(Throwable t)
204
```
205

206
## Usage Notes
207

208
1. **Memory Management**: Always pair `allocateMemory()` calls with `freeMemory()` to prevent memory leaks.
209

210
2. **Object References**: When accessing object fields, pass the object as the first parameter and use appropriate array base offsets.
211

212
3. **Off-heap Access**: When accessing off-heap memory, pass `null` as the object parameter.
213

214
4. **Alignment**: Use `unaligned()` to check if the platform supports unaligned memory access before performing unaligned operations.
215

216
5. **Thread Safety**: These operations are not inherently thread-safe. Proper synchronization must be implemented at the application level.
217

218
6. **Performance**: These methods bypass Java's bounds checking and type safety for maximum performance. Use with extreme caution.
219

220
## Aligned Offset Utilities
221

222
### UnsafeAlignedOffset Class
223

224
The `UnsafeAlignedOffset` class provides utilities for working with aligned memory offsets, particularly useful for reading and writing size information in memory-efficient data structures.
225

226
```java { .api }
227
public class UnsafeAlignedOffset {
228
    /**
229
     * Returns the aligned offset size in bytes (4 or 8 bytes depending on platform).
230
     * This represents the size of the aligned offset data structure.
231
     */
232
    public static int getUaoSize();
233
    
234
    /**
235
     * Reads a size value from an aligned offset location.
236
     * 
237
     * @param object Base object containing the aligned offset
238
     * @param offset Offset within the base object
239
     * @return Size value stored at the aligned offset
240
     */
241
    public static int getSize(Object object, long offset);
242
    
243
    /**
244
     * Writes a size value to an aligned offset location.
245
     * 
246
     * @param object Base object containing the aligned offset
247
     * @param offset Offset within the base object
248
     * @param value Size value to store
249
     */
250
    public static void putSize(Object object, long offset, int value);
251
}
252
```
253

254
#### Usage Examples
255

256
```java
257
// Working with aligned offsets
258
int uaoSize = UnsafeAlignedOffset.getUaoSize(); // Get platform-specific aligned offset size
259

260
// Allocate memory for data structure with aligned offset header
261
MemoryAllocator allocator = MemoryAllocator.HEAP;
262
MemoryBlock block = allocator.allocate(uaoSize + 1024); // Header + data
263

264
try {
265
    Object baseObject = block.getBaseObject();
266
    long baseOffset = block.getBaseOffset();
267
    
268
    // Write size information to aligned offset header
269
    int dataSize = 1024;
270
    UnsafeAlignedOffset.putSize(baseObject, baseOffset, dataSize);
271
    
272
    // Read size information back
273
    int storedSize = UnsafeAlignedOffset.getSize(baseObject, baseOffset);
274
    assert storedSize == dataSize;
275
    
276
    // Use the remaining space for actual data
277
    long dataOffset = baseOffset + uaoSize;
278
    Platform.setMemory(baseObject, dataOffset, dataSize, (byte) 0xFF);
279
    
280
} finally {
281
    allocator.free(block);
282
}
283
```
284

285
#### Integration with Data Structures
286

287
```java
288
// Example: Variable-length data structure with size header
289
public class VariableLengthData {
290
    private final MemoryBlock memory;
291
    private final Object baseObject;
292
    private final long baseOffset;
293
    private final int headerSize;
294
    
295
    public VariableLengthData(int dataSize) {
296
        this.headerSize = UnsafeAlignedOffset.getUaoSize();
297
        this.memory = MemoryAllocator.HEAP.allocate(headerSize + dataSize);
298
        this.baseObject = memory.getBaseObject();
299
        this.baseOffset = memory.getBaseOffset();
300
        
301
        // Store data size in aligned offset header
302
        UnsafeAlignedOffset.putSize(baseObject, baseOffset, dataSize);
303
    }
304
    
305
    public int getDataSize() {
306
        return UnsafeAlignedOffset.getSize(baseObject, baseOffset);
307
    }
308
    
309
    public long getDataOffset() {
310
        return baseOffset + headerSize;
311
    }
312
    
313
    public void writeData(byte[] data) {
314
        int maxSize = getDataSize();
315
        int writeSize = Math.min(data.length, maxSize);
316
        Platform.copyMemory(
317
            data, Platform.BYTE_ARRAY_OFFSET,
318
            baseObject, getDataOffset(),
319
            writeSize
320
        );
321
    }
322
    
323
    public void close() {
324
        MemoryAllocator.HEAP.free(memory);
325
    }
326
}
327
```
328

329
### Usage Notes for Aligned Offsets
330

331
1. **Platform Dependent**: The aligned offset size varies by platform (typically 4 or 8 bytes).
332

333
2. **Alignment Requirements**: Aligned offsets must be properly aligned in memory for optimal performance.
334

335
3. **Size Limitations**: Size values are stored as 32-bit integers, limiting the maximum representable size.
336

337
4. **Memory Layout**: Aligned offsets are typically used as headers in variable-length data structures.
338

339
5. **Performance**: Direct memory access provides maximum performance for size operations.