0
# Type System Extensions
1

2
Type system extensions provide support for Hive-specific data types including enhanced STRUCT types with field comments and improved type specifications.
3

4
## Capabilities
5

6
### Extended STRUCT Type
7

8
Enhanced STRUCT type specification with field names, types, and comments.
9

10
```java { .api }
11
/**
12
 * STRUCT type specification with field names, types, and comments
13
 * Extends standard row type specification to support Hive-specific STRUCT syntax
14
 */
15
public class ExtendedHiveStructTypeNameSpec extends ExtendedSqlRowTypeNameSpec {
16
    /**
17
     * Creates enhanced STRUCT type specification for Hive
18
     * @param pos Parser position information
19
     * @param fieldNames List of field identifiers
20
     * @param fieldTypes List of field data type specifications
21
     * @param comments List of field comments (can contain null values)
22
     * @throws ParseException if validation fails
23
     */
24
    public ExtendedHiveStructTypeNameSpec(SqlParserPos pos, List<SqlIdentifier> fieldNames,
25
                                         List<SqlDataTypeSpec> fieldTypes, 
26
                                         List<SqlCharStringLiteral> comments) throws ParseException;
27
}
28
```
29

30
**Usage Examples:**
31

32
```java
33
// STRUCT type in table creation
34
String createTableWithStructSql = """
35
    CREATE TABLE user_profile (
36
        user_id BIGINT,
37
        name STRING,
38
        address STRUCT<
39
            street: STRING COMMENT 'Street address',
40
            city: STRING COMMENT 'City name',
41
            state: STRING COMMENT 'State or province',
42
            zip_code: STRING COMMENT 'Postal code',
43
            country: STRING COMMENT 'Country code'
44
        > COMMENT 'User address information',
45
        preferences STRUCT<
46
            language: STRING COMMENT 'Preferred language',
47
            timezone: STRING COMMENT 'User timezone',
48
            notifications: BOOLEAN COMMENT 'Enable notifications'
49
        > COMMENT 'User preferences'
50
    )
51
    """;
52

53
// Nested STRUCT types
54
String nestedStructSql = """
55
    CREATE TABLE order_data (
56
        order_id BIGINT,
57
        customer_info STRUCT<
58
            basic: STRUCT<
59
                name: STRING COMMENT 'Customer name',
60
                email: STRING COMMENT 'Email address'
61
            > COMMENT 'Basic customer information',
62
            address: STRUCT<
63
                billing: STRUCT<
64
                    street: STRING,
65
                    city: STRING,
66
                    country: STRING
67
                > COMMENT 'Billing address',
68
                shipping: STRUCT<
69
                    street: STRING,
70
                    city: STRING,
71
                    country: STRING
72
                > COMMENT 'Shipping address'
73
            > COMMENT 'Customer addresses'
74
        > COMMENT 'Complete customer information'
75
    )
76
    """;
77

78
// Programmatic STRUCT type creation
79
SqlParserPos pos = SqlParserPos.ZERO;
80

81
// Field names
82
List<SqlIdentifier> fieldNames = List.of(
83
    new SqlIdentifier("name", pos),
84
    new SqlIdentifier("age", pos),
85
    new SqlIdentifier("email", pos)
86
);
87

88
// Field types
89
List<SqlDataTypeSpec> fieldTypes = List.of(
90
    new SqlDataTypeSpec(new SqlBasicTypeNameSpec(SqlTypeName.VARCHAR, 255, pos), pos),
91
    new SqlDataTypeSpec(new SqlBasicTypeNameSpec(SqlTypeName.INTEGER, pos), pos),
92
    new SqlDataTypeSpec(new SqlBasicTypeNameSpec(SqlTypeName.VARCHAR, 100, pos), pos)
93
);
94

95
// Field comments
96
List<SqlCharStringLiteral> comments = List.of(
97
    SqlLiteral.createCharString("Full name", pos),
98
    SqlLiteral.createCharString("Age in years", pos),
99
    SqlLiteral.createCharString("Email address", pos)
100
);
101

102
// Create STRUCT type specification
103
ExtendedHiveStructTypeNameSpec structType = new ExtendedHiveStructTypeNameSpec(
104
    pos, fieldNames, fieldTypes, comments
105
);
106
```
107

108
## Advanced Type System Features
109

110
### Complex Type Combinations
111

112
Combine STRUCT with other complex types (ARRAY, MAP):
113

114
```java
115
// STRUCT containing arrays and maps
116
String complexTypeTableSql = """
117
    CREATE TABLE analytics_data (
118
        event_id BIGINT,
119
        user_data STRUCT<
120
            user_id: BIGINT COMMENT 'Unique user identifier',
121
            tags: ARRAY<STRING> COMMENT 'User tags',
122
            attributes: MAP<STRING, STRING> COMMENT 'Dynamic attributes',
123
            preferences: STRUCT<
124
                categories: ARRAY<STRING> COMMENT 'Preferred categories',
125
                settings: MAP<STRING, BOOLEAN> COMMENT 'Feature settings'
126
            > COMMENT 'User preferences'
127
        > COMMENT 'Complete user data structure',
128
        session_data STRUCT<
129
            session_id: STRING COMMENT 'Session identifier',
130
            events: ARRAY<STRUCT<
131
                timestamp: TIMESTAMP COMMENT 'Event timestamp',
132
                event_type: STRING COMMENT 'Type of event',
133
                properties: MAP<STRING, STRING> COMMENT 'Event properties'
134
            >> COMMENT 'Session events'
135
        > COMMENT 'Session information'
136
    )
137
    """;
138
```
139

140
### Type Conversion and Compatibility
141

142
Handle type conversion between Hive and Flink types:
143

144
```java
145
// Type conversion examples
146
String typeConversionSql = """
147
    CREATE TABLE type_conversion_example (
148
        -- Hive TIMESTAMP -> Flink TIMESTAMP
149
        event_time TIMESTAMP COMMENT 'Event timestamp',
150
        
151
        -- Hive BINARY -> Flink BYTES  
152
        data_payload BINARY COMMENT 'Binary data payload',
153
        
154
        -- Complex type with conversions
155
        metadata STRUCT<
156
            created_at: TIMESTAMP COMMENT 'Creation timestamp',
157
            updated_at: TIMESTAMP COMMENT 'Last update timestamp',
158
            checksum: BINARY COMMENT 'Data checksum',
159
            tags: ARRAY<STRING> COMMENT 'Metadata tags'
160
        > COMMENT 'Record metadata'
161
    )
162
    """;
163
```
164

165
### Type Validation and Constraints
166

167
Apply validation to complex types:
168

169
```java
170
// Type validation in table creation
171
String validatedTypesTableSql = """
172
    CREATE TABLE validated_data (
173
        record_id BIGINT NOT NULL,
174
        contact_info STRUCT<
175
            email: STRING COMMENT 'Email address (validated format)',
176
            phone: STRING COMMENT 'Phone number (validated format)',
177
            address: STRUCT<
178
                street: STRING COMMENT 'Street address',
179
                city: STRING COMMENT 'City name', 
180
                postal_code: STRING COMMENT 'Postal code (validated format)'
181
            > COMMENT 'Postal address'
182
        > COMMENT 'Contact information with validation',
183
        
184
        -- Constraints can be applied to complex type fields through CHECK constraints
185
        CONSTRAINT valid_email CHECK (contact_info.email LIKE '%@%.%'),
186
        CONSTRAINT valid_postal CHECK (LENGTH(contact_info.address.postal_code) >= 5)
187
    )
188
    """;
189
```
190

191
## Type System Utilities
192

193
### Type Specification Builders
194

195
Utility methods for building complex type specifications:
196

197
```java
198
public class HiveTypeSpecBuilder {
199
    
200
    /**
201
     * Creates a STRUCT type specification with field comments
202
     */
203
    public static ExtendedHiveStructTypeNameSpec createStructType(
204
            SqlParserPos pos, 
205
            Map<String, String> fieldTypes, 
206
            Map<String, String> fieldComments) throws ParseException {
207
        
208
        List<SqlIdentifier> fieldNames = new ArrayList<>();
209
        List<SqlDataTypeSpec> typeSpecs = new ArrayList<>();
210
        List<SqlCharStringLiteral> comments = new ArrayList<>();
211
        
212
        for (Map.Entry<String, String> entry : fieldTypes.entrySet()) {
213
            fieldNames.add(new SqlIdentifier(entry.getKey(), pos));
214
            
215
            // Parse type specification (simplified - real implementation would be more complex)
216
            SqlTypeNameSpec typeNameSpec = parseTypeString(entry.getValue(), pos);
217
            typeSpecs.add(new SqlDataTypeSpec(typeNameSpec, pos));
218
            
219
            // Add comment if available
220
            String comment = fieldComments.get(entry.getKey());
221
            if (comment != null) {
222
                comments.add(SqlLiteral.createCharString(comment, pos));
223
            } else {
224
                comments.add(null);
225
            }
226
        }
227
        
228
        return new ExtendedHiveStructTypeNameSpec(pos, fieldNames, typeSpecs, comments);
229
    }
230
    
231
    /**
232
     * Creates an ARRAY type containing STRUCT elements
233
     */
234
    public static SqlDataTypeSpec createArrayOfStructType(SqlParserPos pos, 
235
                                                         ExtendedHiveStructTypeNameSpec structSpec) {
236
        ExtendedSqlCollectionTypeNameSpec arraySpec = 
237
            new ExtendedSqlCollectionTypeNameSpec(SqlTypeName.ARRAY, pos, structSpec);
238
        
239
        return new SqlDataTypeSpec(arraySpec, pos);
240
    }
241
    
242
    /**
243
     * Creates a MAP type with STRUCT values
244
     */
245
    public static SqlDataTypeSpec createMapWithStructValuesType(SqlParserPos pos,
246
                                                               SqlTypeNameSpec keyType,
247
                                                               ExtendedHiveStructTypeNameSpec valueStructType) {
248
        SqlMapTypeNameSpec mapSpec = new SqlMapTypeNameSpec(pos, keyType, valueStructType);
249
        
250
        return new SqlDataTypeSpec(mapSpec, pos);
251
    }
252
    
253
    private static SqlTypeNameSpec parseTypeString(String typeString, SqlParserPos pos) {
254
        // Simplified type parsing - real implementation would handle all Hive types
255
        switch (typeString.toUpperCase()) {
256
            case "STRING":
257
                return new SqlBasicTypeNameSpec(SqlTypeName.VARCHAR, pos);
258
            case "BIGINT":
259
                return new SqlBasicTypeNameSpec(SqlTypeName.BIGINT, pos);
260
            case "INT":
261
                return new SqlBasicTypeNameSpec(SqlTypeName.INTEGER, pos);
262
            case "BOOLEAN":
263
                return new SqlBasicTypeNameSpec(SqlTypeName.BOOLEAN, pos);
264
            case "DOUBLE":
265
                return new SqlBasicTypeNameSpec(SqlTypeName.DOUBLE, pos);
266
            case "TIMESTAMP":
267
                return new SqlBasicTypeNameSpec(SqlTypeName.TIMESTAMP, pos);
268
            default:
269
                throw new IllegalArgumentException("Unsupported type: " + typeString);
270
        }
271
    }
272
}
273

274
// Usage examples
275
SqlParserPos pos = SqlParserPos.ZERO;
276

277
// Create user profile struct
278
Map<String, String> userFields = Map.of(
279
    "user_id", "BIGINT",
280
    "username", "STRING", 
281
    "email", "STRING",
282
    "active", "BOOLEAN"
283
);
284

285
Map<String, String> userComments = Map.of(
286
    "user_id", "Unique user identifier",
287
    "username", "Login username",
288
    "email", "User email address",
289
    "active", "Account active status"
290
);
291

292
ExtendedHiveStructTypeNameSpec userProfileType = HiveTypeSpecBuilder.createStructType(
293
    pos, userFields, userComments
294
);
295

296
// Create array of user profiles
297
SqlDataTypeSpec userArrayType = HiveTypeSpecBuilder.createArrayOfStructType(pos, userProfileType);
298

299
// Create map with user profile values
300
SqlTypeNameSpec stringKeyType = new SqlBasicTypeNameSpec(SqlTypeName.VARCHAR, pos);
301
SqlDataTypeSpec userMapType = HiveTypeSpecBuilder.createMapWithStructValuesType(
302
    pos, stringKeyType, userProfileType
303
);
304
```
305

306
### Type Inspection and Metadata
307

308
Extract type information from complex type specifications:
309

310
```java
311
public class HiveTypeInspector {
312
    
313
    /**
314
     * Extracts field information from STRUCT type
315
     */
316
    public static List<FieldInfo> extractStructFields(ExtendedHiveStructTypeNameSpec structType) {
317
        List<FieldInfo> fields = new ArrayList<>();
318
        
319
        // Access field information (implementation would depend on internal structure)
320
        // This is a conceptual example
321
        
322
        return fields;
323
    }
324
    
325
    /**
326
     * Validates STRUCT type compatibility
327
     */
328
    public static boolean areStructTypesCompatible(ExtendedHiveStructTypeNameSpec type1,
329
                                                  ExtendedHiveStructTypeNameSpec type2) {
330
        // Implementation would compare field names, types, and nullability
331
        // Return true if types are assignment-compatible
332
        return true; // Placeholder
333
    }
334
    
335
    /**
336
     * Generates DDL representation of complex type
337
     */
338
    public static String generateTypeDDL(SqlDataTypeSpec typeSpec) {
339
        StringBuilder ddl = new StringBuilder();
340
        
341
        // Generate DDL string representation
342
        // Implementation would handle all complex type combinations
343
        
344
        return ddl.toString();
345
    }
346
    
347
    public static class FieldInfo {
348
        private final String name;
349
        private final String type;
350
        private final String comment;
351
        private final boolean nullable;
352
        
353
        public FieldInfo(String name, String type, String comment, boolean nullable) {
354
            this.name = name;
355
            this.type = type;
356
            this.comment = comment;
357
            this.nullable = nullable;
358
        }
359
        
360
        // Getters
361
        public String getName() { return name; }
362
        public String getType() { return type; }
363
        public String getComment() { return comment; }
364
        public boolean isNullable() { return nullable; }
365
    }
366
}
367
```
368

369
## Integration with Hive Metastore
370

371
### Type Metadata Storage
372

373
Store complex type information in Hive metastore:
374

375
```java
376
// Type metadata is automatically stored in Hive metastore
377
String createTableWithMetadataSql = """
378
    CREATE TABLE complex_data (
379
        record_id BIGINT COMMENT 'Primary key',
380
        data_structure STRUCT<
381
            header: STRUCT<
382
                version: STRING COMMENT 'Data format version',
383
                timestamp: TIMESTAMP COMMENT 'Data creation time'
384
            > COMMENT 'Data header information',
385
            payload: STRUCT<
386
                content: STRING COMMENT 'Main data content',
387
                metadata: MAP<STRING, STRING> COMMENT 'Additional metadata'
388
            > COMMENT 'Data payload'
389
        > COMMENT 'Complete data structure'
390
    )
391
    TBLPROPERTIES (
392
        'type_system_version' = '2.0',
393
        'schema_evolution_enabled' = 'true'
394
    )
395
    """;
396
```
397

398
### Schema Evolution Support
399

400
Handle schema evolution for complex types:
401

402
```java
403
// Schema evolution example
404
String evolveStructTypeSql = """
405
    -- Original table
406
    CREATE TABLE user_events_v1 (
407
        event_id BIGINT,
408
        user_data STRUCT<
409
            user_id: BIGINT,
410
            name: STRING
411
        >
412
    );
413
    
414
    -- Evolved table with additional fields
415
    CREATE TABLE user_events_v2 (
416
        event_id BIGINT,
417
        user_data STRUCT<
418
            user_id: BIGINT,
419
            name: STRING,
420
            email: STRING COMMENT 'Added in v2',
421
            preferences: STRUCT<
422
                language: STRING,
423
                timezone: STRING
424
            > COMMENT 'Added in v2'
425
        >
426
    );
427
    """;
428
```