0
# Writers and Formats
1

2
Audit file writers supporting multiple output formats including JSON and ORC, with configurable file management, rotation, and filesystem integration for reliable audit data persistence.
3

4
## Capabilities
5

6
### Ranger Audit Writer Interface
7

8
Primary interface for audit file writers that defines the contract for writing audit events to various storage formats and systems.
9

10
```java { .api }
11
/**
12
 * Interface for audit file writers
13
 */
14
public interface RangerAuditWriter {
15
    /**
16
     * Initialize the audit writer with configuration
17
     * @param props Properties configuration properties
18
     * @param propPrefix String property prefix for configuration
19
     * @param auditProviderName String audit provider name
20
     * @param auditConfigs Map<String,String> additional audit configurations
21
     */
22
    public void init(Properties props, String propPrefix, String auditProviderName, Map<String,String> auditConfigs);
23
    
24
    /**
25
     * Log collection of JSON-formatted audit events
26
     * @param events Collection<String> JSON-formatted events to write
27
     * @return boolean true if successful, false otherwise
28
     * @throws Exception if writing fails
29
     */
30
    public boolean log(Collection<String> events) throws Exception;
31
    
32
    /**
33
     * Log audit events from a file
34
     * @param file File containing audit events to write
35
     * @return boolean true if successful, false otherwise
36
     * @throws Exception if writing fails
37
     */
38
    public boolean logFile(File file) throws Exception;
39
    
40
    /**
41
     * Start the audit writer
42
     */
43
    public void start();
44
    
45
    /**
46
     * Stop the audit writer
47
     */
48
    public void stop();
49
    
50
    /**
51
     * Flush any pending audit events
52
     */
53
    public void flush();
54
}
55
```
56

57
### Abstract Ranger Audit Writer
58

59
Abstract base class for audit writers providing common filesystem functionality and configuration management.
60

61
```java { .api }
62
/**
63
 * Abstract base class for audit writers with filesystem functionality
64
 */
65
public abstract class AbstractRangerAuditWriter implements RangerAuditWriter {
66
    /**
67
     * Initialize writer with configuration and filesystem setup
68
     * @param props Properties configuration properties
69
     * @param basePropertyName String base property name
70
     * @param auditProviderName String provider name
71
     * @param auditConfigs Map<String,String> additional configurations
72
     */
73
    public void init(Properties props, String basePropertyName, String auditProviderName, Map<String,String> auditConfigs);
74
    
75
    /**
76
     * Create Hadoop configuration from properties
77
     * @return Configuration Hadoop configuration object
78
     */
79
    protected Configuration createConfiguration();
80
    
81
    /**
82
     * Create parent directories if they don't exist
83
     * @param path Path directory path to create
84
     * @param fs FileSystem filesystem instance
85
     * @throws IOException if directory creation fails
86
     */
87
    protected void createParents(Path path, FileSystem fs) throws IOException;
88
    
89
    /**
90
     * Flush pending data to filesystem
91
     */
92
    public void flush();
93
    
94
    /**
95
     * Get current file path being written to
96
     * @return String current file path
97
     */
98
    protected String getCurrentFilePath();
99
    
100
    /**
101
     * Check if file rotation is needed
102
     * @return boolean true if rotation needed
103
     */
104
    protected boolean isRotationNeeded();
105
    
106
    /**
107
     * Perform file rotation
108
     */
109
    protected void rotateFile();
110
}
111
```
112

113
### JSON Audit Writer
114

115
JSON audit writer implementation for HDFS with support for file rotation, compression, and configurable output formatting.
116

117
```java { .api }
118
/**
119
 * JSON audit writer implementation for HDFS
120
 */
121
public class RangerJSONAuditWriter extends AbstractRangerAuditWriter {
122
    /**
123
     * Initialize JSON writer with HDFS configuration
124
     * @param props Properties configuration properties
125
     * @param basePropertyName String base property name (e.g., "xasecure.audit.hdfs")
126
     * @param auditProviderName String provider name
127
     * @param auditConfigs Map<String,String> additional configurations
128
     */
129
    public void init(Properties props, String basePropertyName, String auditProviderName, Map<String,String> auditConfigs);
130
    
131
    /**
132
     * Log JSON-formatted audit events to HDFS
133
     * @param events Collection<String> JSON-formatted events
134
     */
135
    public void log(Collection<String> events);
136
    
137
    /**
138
     * Log audit events from file to HDFS
139
     * @param file File containing audit events
140
     */
141
    public void logFile(File file);
142
    
143
    /**
144
     * Start JSON writer and open output stream
145
     */
146
    public void start();
147
    
148
    /**
149
     * Stop JSON writer and close output stream
150
     */
151
    public void stop();
152
    
153
    /**
154
     * Flush JSON data to HDFS
155
     */
156
    public void flush();
157
    
158
    /**
159
     * Get current output stream
160
     * @return FSDataOutputStream current output stream
161
     */
162
    protected FSDataOutputStream getOutputStream();
163
    
164
    /**
165
     * Format audit event as JSON string
166
     * @param event String raw event
167
     * @return String formatted JSON
168
     */
169
    protected String formatAsJSON(String event);
170
}
171
```
172

173
### ORC Audit Writer
174

175
ORC (Optimized Row Columnar) audit writer implementation providing efficient columnar storage format for audit events with schema management and compression.
176

177
```java { .api }
178
/**
179
 * ORC file format audit writer
180
 */
181
public class RangerORCAuditWriter extends AbstractRangerAuditWriter {
182
    /**
183
     * Initialize ORC writer with schema and HDFS configuration
184
     * @param props Properties configuration properties
185
     * @param basePropertyName String base property name
186
     * @param auditProviderName String provider name
187
     * @param auditConfigs Map<String,String> additional configurations
188
     */
189
    public void init(Properties props, String basePropertyName, String auditProviderName, Map<String,String> auditConfigs);
190
    
191
    /**
192
     * Log JSON-formatted audit events as ORC records
193
     * @param events Collection<String> JSON-formatted events to convert and write
194
     */
195
    public void log(Collection<String> events);
196
    
197
    /**
198
     * Log audit events from file as ORC records
199
     * @param file File containing audit events
200
     */
201
    public void logFile(File file);
202
    
203
    /**
204
     * Start ORC writer and initialize schema
205
     */
206
    public void start();
207
    
208
    /**
209
     * Stop ORC writer and close ORC file
210
     */
211
    public void stop();
212
    
213
    /**
214
     * Flush ORC data to HDFS
215
     */
216
    public void flush();
217
    
218
    /**
219
     * Get ORC schema for audit events
220
     * @return TypeDescription ORC schema
221
     */
222
    protected TypeDescription getORCSchema();
223
    
224
    /**
225
     * Convert JSON event to ORC record
226
     * @param jsonEvent String JSON-formatted event
227
     * @param batch VectorizedRowBatch ORC batch to populate
228
     * @param row int row index in batch
229
     */
230
    protected void convertJSONToORC(String jsonEvent, VectorizedRowBatch batch, int row);
231
}
232
```
233

234
### Audit Writer Factory
235

236
Factory class for creating audit writers based on configuration and format requirements.
237

238
```java { .api }
239
/**
240
 * Factory for creating audit file writers
241
 */
242
public class AuditWriterFactory {
243
    /**
244
     * Get singleton instance of the factory
245
     * @return AuditWriterFactory factory instance
246
     */
247
    public static AuditWriterFactory getInstance();
248
    
249
    /**
250
     * Create audit writer for specified format
251
     * @param writerType String writer type ("json", "orc")
252
     * @return RangerAuditWriter writer instance
253
     */
254
    public RangerAuditWriter createWriter(String writerType);
255
    
256
    /**
257
     * Get configured audit writer from properties
258
     * @param props Properties configuration properties
259
     * @param basePropertyName String base property name
260
     * @return RangerAuditWriter configured writer
261
     */
262
    public RangerAuditWriter getAuditWriter(Properties props, String basePropertyName);
263
    
264
    /**
265
     * Register custom writer implementation
266
     * @param writerType String writer type identifier
267
     * @param writerClass Class<? extends RangerAuditWriter> writer implementation class
268
     */
269
    public void registerWriter(String writerType, Class<? extends RangerAuditWriter> writerClass);
270
}
271
```
272

273
### ORC File Utilities
274

275
Utility class for working with ORC files in audit context, providing schema management and file operations.
276

277
```java { .api }
278
/**
279
 * ORC file utilities for audit writing
280
 */
281
public class ORCFileUtil {
282
    /**
283
     * Create standard audit event ORC schema
284
     * @return TypeDescription ORC schema for audit events
285
     */
286
    public static TypeDescription createAuditEventSchema();
287
    
288
    /**
289
     * Convert audit event object to ORC vector batch row
290
     * @param auditEvent AuditEventBase event to convert
291
     * @param batch VectorizedRowBatch target batch
292
     * @param row int row index in batch
293
     */
294
    public static void populateORCBatch(AuditEventBase auditEvent, VectorizedRowBatch batch, int row);
295
    
296
    /**
297
     * Create ORC writer with compression and configuration
298
     * @param path Path output file path
299
     * @param schema TypeDescription ORC schema
300
     * @param conf Configuration Hadoop configuration
301
     * @return Writer ORC writer instance
302
     * @throws IOException if writer creation fails
303
     */
304
    public static Writer createORCWriter(Path path, TypeDescription schema, Configuration conf) throws IOException;
305
    
306
    /**
307
     * Read ORC file and convert to audit events
308
     * @param path Path ORC file path
309
     * @param conf Configuration Hadoop configuration
310
     * @return List<AuditEventBase> audit events from file
311
     * @throws IOException if reading fails
312
     */
313
    public static List<AuditEventBase> readORCAuditEvents(Path path, Configuration conf) throws IOException;
314
}
315
```
316

317
**Usage Examples:**
318

319
```java
320
import org.apache.ranger.audit.utils.*;
321
import org.apache.ranger.audit.model.AuthzAuditEvent;
322

323
// JSON Writer Configuration
324
Properties jsonProps = new Properties();
325
jsonProps.setProperty("xasecure.audit.hdfs.destination.directory", "/ranger/audit");
326
jsonProps.setProperty("xasecure.audit.hdfs.destination.file", "audit.json");
327
jsonProps.setProperty("xasecure.audit.hdfs.destination.flush.interval.seconds", "30");
328
jsonProps.setProperty("xasecure.audit.hdfs.destination.rollover.interval.seconds", "3600");
329

330
RangerJSONAuditWriter jsonWriter = new RangerJSONAuditWriter();
331
jsonWriter.init(jsonProps, "xasecure.audit.hdfs", "hdfs-audit", new HashMap<>());
332
jsonWriter.start();
333

334
// Create and log audit events as JSON
335
AuthzAuditEvent event1 = new AuthzAuditEvent();
336
event1.setUser("alice");
337
event1.setAccessType("read");
338
event1.setResourcePath("/data/file1.txt");
339
event1.setAccessResult(1);
340

341
AuthzAuditEvent event2 = new AuthzAuditEvent();
342
event2.setUser("bob");
343
event2.setAccessType("write");
344
event2.setResourcePath("/data/file2.txt");
345
event2.setAccessResult(0);
346

347
List<String> jsonEvents = Arrays.asList(event1.toJson(), event2.toJson());
348
jsonWriter.log(jsonEvents);
349
jsonWriter.flush();
350

351
// ORC Writer Configuration
352
Properties orcProps = new Properties();
353
orcProps.setProperty("xasecure.audit.hdfs.destination.directory", "/ranger/audit/orc");
354
orcProps.setProperty("xasecure.audit.hdfs.destination.file", "audit.orc");
355
orcProps.setProperty("xasecure.audit.hdfs.destination.compression.type", "SNAPPY");
356

357
RangerORCAuditWriter orcWriter = new RangerORCAuditWriter();
358
orcWriter.init(orcProps, "xasecure.audit.hdfs", "orc-audit", new HashMap<>());
359
orcWriter.start();
360

361
// Log same events in ORC format
362
orcWriter.log(jsonEvents);
363
orcWriter.flush();
364

365
// Using Factory Pattern
366
AuditWriterFactory factory = AuditWriterFactory.getInstance();
367
RangerAuditWriter writer = factory.createWriter("json");
368
writer.init(jsonProps, "xasecure.audit.hdfs", "factory-writer", new HashMap<>());
369

370
// File-based logging
371
File auditFile = new File("/tmp/audit-events.json");
372
jsonWriter.logFile(auditFile);
373

374
// Cleanup
375
jsonWriter.stop();
376
orcWriter.stop();
377
```
378

379
### Configuration Properties
380

381
Key configuration properties for writers and formats:
382

383
**JSON Writer Configuration:**
384
- `xasecure.audit.hdfs.destination.directory`: Output directory path
385
- `xasecure.audit.hdfs.destination.file`: Output filename pattern
386
- `xasecure.audit.hdfs.destination.flush.interval.seconds`: Flush interval
387
- `xasecure.audit.hdfs.destination.rollover.interval.seconds`: File rollover interval
388
- `xasecure.audit.hdfs.destination.open.retry.interval.seconds`: Retry interval for failed opens
389

390
**ORC Writer Configuration:**
391
- `xasecure.audit.hdfs.destination.compression.type`: Compression type (NONE, ZLIB, SNAPPY, LZO)
392
- `xasecure.audit.hdfs.destination.orc.batch.size`: ORC batch size for writing
393
- `xasecure.audit.hdfs.destination.orc.stripe.size`: ORC stripe size
394

395
**Common Writer Configuration:**
396
- `xasecure.audit.hdfs.destination.rollover.size.limit.bytes`: File size limit for rollover
397
- `xasecure.audit.hdfs.config.encoding`: Character encoding for output files
398
- `xasecure.audit.hdfs.config.kerberos.principal`: Kerberos principal for authentication
399
- `xasecure.audit.hdfs.config.kerberos.keytab`: Kerberos keytab file path