0
# Buffer Management
1

2
Unified buffer abstraction with multiple backing implementations for efficient memory and file-based data handling, providing zero-copy I/O capabilities and automatic resource management.
3

4
## Capabilities
5

6
### Managed Buffer Abstract Base
7

8
Core abstraction for immutable data views with different backing implementations, supporting efficient data transfer and resource management.
9

10
```java { .api }
11
/**
12
 * Immutable view for data in bytes with different backing implementations
13
 * Provides unified interface for file, memory, and Netty-backed data
14
 */
15
public abstract class ManagedBuffer {
16
    /**
17
     * Number of bytes of data in this buffer
18
     * @return Size in bytes
19
     */
20
    public abstract long size();
21
    
22
    /**
23
     * Expose this buffer's data as a NIO ByteBuffer
24
     * Multiple calls may return the same ByteBuffer or different ByteBuffers
25
     * @return ByteBuffer view of the data
26
     * @throws IOException if buffer cannot be converted
27
     */
28
    public abstract ByteBuffer nioByteBuffer() throws IOException;
29
    
30
    /**
31
     * Expose this buffer's data as an InputStream for streaming reads
32
     * @return InputStream for reading the data
33
     * @throws IOException if stream cannot be created
34
     */
35
    public abstract InputStream createInputStream() throws IOException;
36
    
37
    /**
38
     * Increment the reference count of this buffer if applicable
39
     * @return This buffer for method chaining
40
     */
41
    public abstract ManagedBuffer retain();
42
    
43
    /**
44
     * Decrement the reference count and deallocate if zero
45
     * @return This buffer for method chaining
46
     */
47
    public abstract ManagedBuffer release();
48
    
49
    /**
50
     * Convert this buffer to a Netty ByteBuf or FileRegion for efficient transfer
51
     * @return Netty object (ByteBuf for memory, FileRegion for files)
52
     * @throws IOException if conversion fails
53
     */
54
    public abstract Object convertToNetty() throws IOException;
55
}
56
```
57

58
**Usage Examples:**
59

60
```java
61
// Working with a managed buffer
62
ManagedBuffer buffer = // ... obtained from somewhere
63
try {
64
    long size = buffer.size();
65
    System.out.println("Buffer size: " + size + " bytes");
66
    
67
    // Read as ByteBuffer
68
    ByteBuffer bb = buffer.nioByteBuffer();
69
    byte[] data = new byte[bb.remaining()];
70
    bb.get(data);
71
    
72
    // Or read as stream
73
    try (InputStream is = buffer.createInputStream()) {
74
        byte[] streamData = new byte[(int) size];
75
        is.read(streamData);
76
    }
77
    
78
} finally {
79
    // Always release when done
80
    buffer.release();
81
}
82
```
83

84
### File Segment Managed Buffer
85

86
ManagedBuffer implementation backed by a segment of a file, providing efficient file-based data access without loading entire files into memory.
87

88
```java { .api }
89
/**
90
 * ManagedBuffer backed by a segment in a file
91
 * Enables efficient access to portions of large files without loading entire file
92
 */
93
public final class FileSegmentManagedBuffer extends ManagedBuffer {
94
    /**
95
     * Create a buffer backed by a file segment
96
     * @param conf Transport configuration for memory mapping settings
97
     * @param file File to read from
98
     * @param offset Starting offset in the file
99
     * @param length Number of bytes to include from the offset
100
     */
101
    public FileSegmentManagedBuffer(TransportConf conf, File file, long offset, long length);
102
    
103
    /**
104
     * Get the underlying file
105
     * @return File instance
106
     */
107
    public File getFile();
108
    
109
    /**
110
     * Get the starting offset in the file
111
     * @return Offset in bytes
112
     */
113
    public long getOffset();
114
    
115
    /**
116
     * Get the length of the segment
117
     * @return Length in bytes
118
     */
119
    public long getLength();
120
}
121
```
122

123
**Usage Examples:**
124

125
```java
126
import java.io.File;
127

128
// Create a buffer for part of a large file
129
File dataFile = new File("/path/to/large/data.bin");
130
long offset = 1024 * 1024; // Start at 1MB
131
long length = 512 * 1024;  // Read 512KB
132

133
FileSegmentManagedBuffer fileBuffer = new FileSegmentManagedBuffer(
134
    conf, dataFile, offset, length
135
);
136

137
try {
138
    System.out.println("Reading " + fileBuffer.getLength() + " bytes from " + 
139
                      fileBuffer.getFile().getName() + " at offset " + fileBuffer.getOffset());
140
    
141
    // Efficient zero-copy transfer to Netty
142
    Object nettyObject = fileBuffer.convertToNetty();
143
    
144
    // Or read as stream
145
    try (InputStream stream = fileBuffer.createInputStream()) {
146
        // Process stream data
147
    }
148
    
149
} finally {
150
    fileBuffer.release();
151
}
152
```
153

154
### Netty Managed Buffer
155

156
ManagedBuffer implementation backed by a Netty ByteBuf, providing integration with Netty's buffer management and reference counting.
157

158
```java { .api }
159
/**
160
 * ManagedBuffer backed by a Netty ByteBuf
161
 * Integrates with Netty's reference counting and memory management
162
 */
163
public class NettyManagedBuffer extends ManagedBuffer {
164
    /**
165
     * Create a buffer backed by a Netty ByteBuf
166
     * @param buf Netty ByteBuf to wrap
167
     */
168
    public NettyManagedBuffer(ByteBuf buf);
169
}
170
```
171

172
**Usage Examples:**
173

174
```java
175
import io.netty.buffer.ByteBuf;
176
import io.netty.buffer.Unpooled;
177

178
// Create from Netty ByteBuf
179
ByteBuf nettyBuf = Unpooled.copiedBuffer("Hello, World!".getBytes());
180
NettyManagedBuffer buffer = new NettyManagedBuffer(nettyBuf);
181

182
try {
183
    // Use as managed buffer
184
    ByteBuffer nio = buffer.nioByteBuffer();
185
    System.out.println("Data: " + new String(nio.array()));
186
    
187
    // Convert back to Netty (efficient, no copy)
188
    Object nettyObject = buffer.convertToNetty();
189
    
190
} finally {
191
    buffer.release(); // This will release the underlying ByteBuf
192
}
193
```
194

195
### NIO Managed Buffer
196

197
ManagedBuffer implementation backed by a NIO ByteBuffer, providing integration with standard Java NIO buffers.
198

199
```java { .api }
200
/**
201
 * ManagedBuffer backed by a NIO ByteBuffer
202
 * Provides ManagedBuffer interface for standard Java ByteBuffers
203
 */
204
public class NioManagedBuffer extends ManagedBuffer {
205
    /**
206
     * Create a buffer backed by a NIO ByteBuffer
207
     * @param buf NIO ByteBuffer to wrap
208
     */
209
    public NioManagedBuffer(ByteBuffer buf);
210
}
211
```
212

213
**Usage Examples:**
214

215
```java
216
import java.nio.ByteBuffer;
217

218
// Create from NIO ByteBuffer
219
ByteBuffer nioBuf = ByteBuffer.wrap("Hello, NIO!".getBytes());
220
NioManagedBuffer buffer = new NioManagedBuffer(nioBuf);
221

222
try {
223
    // Access data
224
    System.out.println("Buffer size: " + buffer.size());
225
    
226
    // Get as ByteBuffer (may return same instance)
227
    ByteBuffer bb = buffer.nioByteBuffer();
228
    
229
    // Read as stream
230
    try (InputStream stream = buffer.createInputStream()) {
231
        byte[] data = new byte[(int) buffer.size()];
232
        stream.read(data);
233
        System.out.println("Stream data: " + new String(data));
234
    }
235
    
236
} finally {
237
    buffer.release();
238
}
239
```
240

241
## Buffer Usage Patterns
242

243
### Choosing Buffer Type
244

245
Different buffer types are optimized for different use cases:
246

247
```java
248
// For file-based data (large files, zero-copy I/O)
249
FileSegmentManagedBuffer fileBuffer = new FileSegmentManagedBuffer(
250
    conf, file, 0, file.length()
251
);
252

253
// For in-memory data with Netty integration
254
ByteBuf nettyBuf = ctx.alloc().buffer(1024);
255
NettyManagedBuffer nettyBuffer = new NettyManagedBuffer(nettyBuf);
256

257
// For existing NIO ByteBuffer integration
258
ByteBuffer existing = // ... from somewhere
259
NioManagedBuffer nioBuffer = new NioManagedBuffer(existing);
260
```
261

262
### Resource Management
263

264
Always use try-finally or try-with-resources patterns for proper cleanup:
265

266
```java
267
ManagedBuffer buffer = createBuffer();
268
try {
269
    // Use buffer
270
    processBuffer(buffer);
271
} finally {
272
    buffer.release(); // Essential for preventing memory leaks
273
}
274
```
275

276
### Zero-Copy Transfer
277

278
Leverage `convertToNetty()` for efficient network transfer:
279

280
```java
281
ManagedBuffer buffer = new FileSegmentManagedBuffer(conf, file, 0, file.length());
282
try {
283
    // Efficient zero-copy transfer
284
    Object nettyObject = buffer.convertToNetty();
285
    
286
    if (nettyObject instanceof FileRegion) {
287
        // File-based zero-copy
288
        channel.writeAndFlush(nettyObject);
289
    } else if (nettyObject instanceof ByteBuf) {
290
        // Memory-based transfer
291
        channel.writeAndFlush(nettyObject);
292
    }
293
} finally {
294
    buffer.release();
295
}
296
```
297

298
### Buffer Chaining
299

300
For complex data processing pipelines:
301

302
```java
303
public ManagedBuffer processData(ManagedBuffer input) {
304
    try (InputStream stream = input.createInputStream()) {
305
        // Process data
306
        byte[] processed = transformData(stream);
307
        
308
        // Return new buffer with processed data
309
        return new NioManagedBuffer(ByteBuffer.wrap(processed));
310
    } catch (IOException e) {
311
        throw new RuntimeException("Processing failed", e);
312
    }
313
}
314
```