0
# CDAP API Common
1

2
Core API classes and utilities for CDAP (Cask Data Application Platform) application development. Provides essential building blocks including data schema definitions, data format abstractions, stream event handling, and comprehensive byte manipulation utilities for big data processing scenarios.
3

4
## Package Information
5

6
- **Package Name**: co.cask.cdap:cdap-api-common
7
- **Package Type**: Maven
8
- **Language**: Java
9
- **Version**: 5.1.2
10
- **Status**: Experimental (marked with @Beta annotation)
11
- **Installation**: Add to Maven dependencies:
12

13
```xml
14
<dependency>
15
    <groupId>co.cask.cdap</groupId>
16
    <artifactId>cdap-api-common</artifactId>
17
    <version>5.1.2</version>
18
</dependency>
19
```
20

21
## Core Imports
22

23
```java
24
import co.cask.cdap.api.data.schema.Schema;
25
import co.cask.cdap.api.data.format.StructuredRecord;
26
import co.cask.cdap.api.common.Bytes;
27
import co.cask.cdap.api.flow.flowlet.StreamEvent;
28
import co.cask.cdap.api.data.format.RecordFormat;
29
import co.cask.cdap.api.stream.StreamEventDecoder;
30
import co.cask.cdap.api.macro.InvalidMacroException;
31
```
32

33
## Basic Usage
34

35
```java
36
import co.cask.cdap.api.data.schema.Schema;
37
import co.cask.cdap.api.data.format.StructuredRecord;
38
import co.cask.cdap.api.common.Bytes;
39
import java.time.LocalDate;
40

41
// Define a schema for structured data
42
Schema personSchema = Schema.recordOf("Person",
43
    Schema.Field.of("name", Schema.of(Schema.Type.STRING)),
44
    Schema.Field.of("age", Schema.of(Schema.Type.INT)),
45
    Schema.Field.of("birthDate", Schema.of(Schema.LogicalType.DATE))
46
);
47

48
// Create structured records
49
StructuredRecord person = StructuredRecord.builder(personSchema)
50
    .set("name", "John Doe")
51
    .set("age", 30)
52
    .setDate("birthDate", LocalDate.of(1993, 5, 15))
53
    .build();
54

55
// Access record data
56
String name = person.get("name");
57
Integer age = person.get("age");
58
LocalDate birthDate = person.getDate("birthDate");
59

60
// Byte manipulation for data processing
61
byte[] nameBytes = Bytes.toBytes("John Doe");
62
String recoveredName = Bytes.toString(nameBytes);
63
long timestamp = System.currentTimeMillis();
64
byte[] timestampBytes = Bytes.toBytes(timestamp);
65
```
66

67
## Architecture
68

69
CDAP API Common is built around several key components:
70

71
- **Schema System**: Type-safe schema definitions supporting primitive types, complex types (arrays, maps, records, unions), and logical types for dates and timestamps
72
- **Structured Records**: Runtime data containers that enforce schema compliance with builder pattern construction
73
- **Stream Processing**: Event-driven data processing with headers, body, and timestamp support
74
- **Byte Utilities**: Comprehensive byte array manipulation optimized for big data scenarios
75
- **Format Abstractions**: Pluggable data format conversion system for various input/output formats
76

77
## Capabilities
78

79
### Data Schema Definition
80

81
Comprehensive schema system supporting primitive types, complex nested structures, logical types for dates/timestamps, and schema compatibility checking. Provides type safety and validation for data processing pipelines.
82

83
```java { .api }
84
// Schema creation methods
85
public static Schema of(Schema.Type type);
86
public static Schema of(Schema.LogicalType logicalType);
87
public static Schema recordOf(String name, Schema.Field... fields);
88
public static Schema arrayOf(Schema componentSchema);
89
public static Schema mapOf(Schema keySchema, Schema valueSchema);
90
public static Schema unionOf(Schema... schemas);
91
public static Schema nullableOf(Schema schema);
92

93
// Schema parsing
94
public static Schema parseJson(String schemaJson) throws IOException;
95
public static Schema parseSQL(String schemaString) throws IOException;
96
```
97

98
[Schema System](./schema-system.md)
99

100
### Structured Data Records
101

102
Type-safe record instances that conform to defined schemas, with builder pattern construction and specialized accessors for date/time logical types. Essential for data pipeline processing and validation.
103

104
```java { .api }
105
public static StructuredRecord.Builder builder(Schema schema);
106

107
// Builder methods
108
public StructuredRecord.Builder set(String fieldName, Object value);
109
public StructuredRecord.Builder setDate(String fieldName, LocalDate date);
110
public StructuredRecord.Builder setTime(String fieldName, LocalTime time);
111
public StructuredRecord.Builder setTimestamp(String fieldName, ZonedDateTime timestamp);
112
public StructuredRecord build();
113

114
// Data access methods  
115
public <T> T get(String fieldName);
116
public LocalDate getDate(String fieldName);
117
public LocalTime getTime(String fieldName);
118
public ZonedDateTime getTimestamp(String fieldName);
119
```
120

121
[Structured Records](./structured-records.md)
122

123
### Stream Event Processing
124

125
Event-driven data processing capabilities for handling streaming data with headers, body content, and timestamps. Supports custom decoders for converting stream events into structured key-value pairs.
126

127
```java { .api }
128
public class StreamEvent extends StreamEventData {
129
    public StreamEvent(Map<String, String> headers, ByteBuffer body, long timestamp);
130
    public long getTimestamp();
131
    public Map<String, String> getHeaders();
132
    public ByteBuffer getBody();
133
}
134

135
public interface StreamEventDecoder<K, V> {
136
    DecodeResult<K, V> decode(StreamEvent event, DecodeResult<K, V> result);
137
}
138
```
139

140
[Stream Processing](./stream-processing.md)
141

142
### Byte Array Utilities
143

144
Comprehensive byte manipulation utilities optimized for big data processing, including conversions between primitive types and byte arrays, array operations, comparisons, and hash computations.
145

146
```java { .api }
147
// Type conversions
148
public static byte[] toBytes(String s);
149
public static byte[] toBytes(long val);
150
public static byte[] toBytes(int val);
151
public static String toString(byte[] b);
152
public static long toLong(byte[] bytes);
153
public static int toInt(byte[] bytes);
154

155
// Array operations
156
public static byte[] add(byte[] a, byte[] b);
157
public static byte[] concat(byte[]... arrays);
158
public static int compareTo(byte[] left, byte[] right);
159
public static boolean equals(byte[] left, byte[] right);
160
public static int hashCode(byte[] b);
161
```
162

163
[Byte Utilities](./byte-utilities.md)
164

165
### Data Format Abstractions
166

167
Pluggable format system for converting data between different representations, with built-in support for common formats and extensible architecture for custom formats.
168

169
```java { .api }
170
public abstract class RecordFormat<FROM, TO> {
171
    public abstract TO read(FROM input) throws UnexpectedFormatException;
172
    public void initialize(FormatSpecification formatSpec);
173
    public Schema getSchema();
174
}
175

176
public class FormatSpecification {
177
    public FormatSpecification(String name, Schema schema, Map<String, String> settings);
178
    public String getName();
179
    public Schema getSchema();
180
    public Map<String, String> getSettings();
181
}
182
```
183

184
[Data Format System](./data-format-system.md)
185

186
## Types
187

188
### Core Schema Types
189

190
```java { .api }
191
public enum Schema.Type {
192
    NULL, BOOLEAN, INT, LONG, FLOAT, DOUBLE, BYTES, STRING,
193
    ENUM, ARRAY, MAP, RECORD, UNION;
194
    
195
    public boolean isSimpleType();
196
}
197

198
public enum Schema.LogicalType {
199
    DATE, TIMESTAMP_MILLIS, TIMESTAMP_MICROS, TIME_MILLIS, TIME_MICROS;
200
    
201
    public String getToken();
202
    public static LogicalType fromToken(String token);
203
}
204

205
public static final class Schema.Field {
206
    public static Field of(String name, Schema schema);
207
    public String getName();
208
    public Schema getSchema();
209
}
210
```
211

212
### Exception Types
213

214
```java { .api }
215
public class UnexpectedFormatException extends RuntimeException {
216
    public UnexpectedFormatException(String message);
217
    public UnexpectedFormatException(String message, Throwable cause);
218
}
219

220
public class UnsupportedTypeException extends Exception {
221
    public UnsupportedTypeException(String message);
222
    public UnsupportedTypeException(String message, Throwable cause);
223
}
224

225
public class InvalidMacroException extends RuntimeException {
226
    public InvalidMacroException(String message);
227
    public InvalidMacroException(String message, Throwable cause);
228
}
229
```
230

231
### Utility Types
232

233
```java { .api }
234
@Beta
235
public @interface Beta {
236
    // Marks experimental APIs
237
}
238

239
public final class SchemaHash {
240
    public SchemaHash(Schema schema);
241
    public byte[] toByteArray();
242
    public String toString(); // Hex representation
243
}
244

245
public final class Formats {
246
    public static final String AVRO = "avro";
247
    public static final String CSV = "csv"; 
248
    public static final String TSV = "tsv";
249
    public static final String TEXT = "text";
250
    public static final String COMBINED_LOG_FORMAT = "clf";
251
    public static final String GROK = "grok";
252
    public static final String SYSLOG = "syslog";
253
}
254
```