or run

tessl search
Log in

Version

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.spark/spark-unsafe_2.13@3.5.x

docs

array-operations.mdbitset-operations.mdbyte-array-utilities.mddata-types-utilities.mdhash-functions.mdindex.mdkv-iterator.mdmemory-management.mdplatform-operations.mdutf8-string-processing.md
tile.json

tessl/maven-org-apache-spark--spark-unsafe_2-13

tessl install tessl/maven-org-apache-spark--spark-unsafe_2-13@3.5.0

Low-level unsafe operations and optimized data structures for Apache Spark's internal memory management and performance-critical operations.

platform-operations.mddocs/

Platform Operations

Core platform utilities for unsafe memory operations, direct memory access, and JVM intrinsics that bypass standard Java safety mechanisms. These operations provide maximum performance for Spark's internal memory management by directly accessing memory addresses and system-level functionality.

Capabilities

Memory Access Operations

Direct typed memory access operations that bypass Java's safety mechanisms for maximum performance. These methods work with both on-heap (object + offset) and off-heap (absolute address) memory locations.

public static int getInt(Object object, long offset);
public static void putInt(Object object, long offset, int value);
public static boolean getBoolean(Object object, long offset);
public static void putBoolean(Object object, long offset, boolean value);
public static byte getByte(Object object, long offset);
public static void putByte(Object object, long offset, byte value);
public static short getShort(Object object, long offset);
public static void putShort(Object object, long offset, short value);
public static long getLong(Object object, long offset);
public static void putLong(Object object, long offset, long value);
public static float getFloat(Object object, long offset);
public static void putFloat(Object object, long offset, float value);
public static double getDouble(Object object, long offset);
public static void putDouble(Object object, long offset, double value);

Usage Examples:

// On-heap memory access (object + offset)
byte[] array = new byte[64];
long offset = Platform.BYTE_ARRAY_OFFSET + 8;
Platform.putLong(array, offset, 12345L);
long value = Platform.getLong(array, offset);

// Off-heap memory access (absolute address)
long address = Platform.allocateMemory(64);
Platform.putLong(null, address, 12345L);
long value = Platform.getLong(null, address);
Platform.freeMemory(address);

Volatile Memory Operations

Memory access operations with volatile semantics for thread-safe operations on shared memory locations.

public static Object getObjectVolatile(Object object, long offset);
public static void putObjectVolatile(Object object, long offset, Object value);

Off-Heap Memory Management

Direct allocation and management of off-heap memory using platform-specific mechanisms. These operations provide control over memory that exists outside the JVM heap.

public static long allocateMemory(long size);
public static void freeMemory(long address);
public static long reallocateMemory(long address, long oldSize, long newSize);
public static ByteBuffer allocateDirectBuffer(int size);

Usage Example:

// Allocate 1KB of off-heap memory
long address = Platform.allocateMemory(1024);

// Use the memory
Platform.putLong(null, address, 42L);
long value = Platform.getLong(null, address);

// Reallocate to 2KB
address = Platform.reallocateMemory(address, 1024, 2048);

// Clean up
Platform.freeMemory(address);

Bulk Memory Operations

High-performance memory operations for copying and filling large memory regions, optimized for bulk data processing scenarios.

public static void setMemory(Object object, long offset, long size, byte value);
public static void setMemory(long address, byte value, long size);
public static void copyMemory(Object src, long srcOffset, Object dst, long dstOffset, long length);

Usage Examples:

// Fill memory region with zeros
byte[] buffer = new byte[1024];
Platform.setMemory(buffer, Platform.BYTE_ARRAY_OFFSET, 1024, (byte)0);

// Copy between arrays
byte[] source = "Hello World".getBytes();
byte[] dest = new byte[source.length];
Platform.copyMemory(source, Platform.BYTE_ARRAY_OFFSET, 
                   dest, Platform.BYTE_ARRAY_OFFSET, source.length);

// Copy from off-heap to on-heap
long offHeapAddr = Platform.allocateMemory(64);
byte[] onHeapArray = new byte[64];
Platform.copyMemory(null, offHeapAddr, 
                   onHeapArray, Platform.BYTE_ARRAY_OFFSET, 64);

Platform Information

Query platform capabilities and characteristics for optimizing memory access patterns and choosing appropriate algorithms.

public static boolean cleanerCreateMethodIsDefined();
public static boolean unaligned();

Usage Example:

// Check if platform supports unaligned memory access
if (Platform.unaligned()) {
    // Use optimized unaligned access algorithms
    processDataUnaligned(data);
} else {
    // Use slower but safer aligned access
    processDataAligned(data);
}

Array Base Offsets

Constants providing the base offset for different array types, used for calculating element positions in typed arrays.

public static final int BOOLEAN_ARRAY_OFFSET;
public static final int BYTE_ARRAY_OFFSET;
public static final int SHORT_ARRAY_OFFSET;
public static final int INT_ARRAY_OFFSET;
public static final int LONG_ARRAY_OFFSET;
public static final int FLOAT_ARRAY_OFFSET;
public static final int DOUBLE_ARRAY_OFFSET;

Usage Example:

// Access third element (index 2) of an int array
int[] array = {10, 20, 30, 40};
long elementOffset = Platform.INT_ARRAY_OFFSET + (2 * 4); // 4 bytes per int
int thirdElement = Platform.getInt(array, elementOffset);

Exception Handling

Bypass checked exception handling mechanisms for performance-critical code paths where exception types are known and handled explicitly.

public static void throwException(Throwable t);

SPARC Platform Alignment

Utilities for handling SPARC platform-specific alignment requirements in uniform record length scenarios.

// UnsafeAlignedOffset class
public static void setUaoSize(int size);
public static int getUaoSize();
public static int getSize(Object object, long offset);
public static void putSize(Object object, long offset, int value);

Usage Example:

// Configure for testing (size must be 0, 4, or 8)
UnsafeAlignedOffset.setUaoSize(8);

// Use aligned size operations
Object buffer = new long[16];
UnsafeAlignedOffset.putSize(buffer, 0, 1024);
int size = UnsafeAlignedOffset.getSize(buffer, 0);