0
# Apache Spark Tags
1

2
Apache Spark Tags is a Java annotations library that provides annotation interfaces for test categorization and API stability markers within the Apache Spark ecosystem. It includes annotations for organizing Spark's extensive test suites and communicating the maturity and expected evolution of Spark's public APIs.
3

4
The annotations are designed to mark APIs as experimental or intended only for advanced usages by developers, and are used project-wide across Spark, reflected in both Scala and Java documentation.
5

6
## Package Information
7

8
- **Package Name**: spark-tags_2.13
9
- **Package Type**: maven
10
- **Language**: Java
11
- **Group ID**: org.apache.spark
12
- **Artifact ID**: spark-tags_2.13
13
- **Version**: 3.5.6
14
- **Installation**: Add to your Maven `pom.xml`:
15

16
```xml
17
<dependency>
18
    <groupId>org.apache.spark</groupId>
19
    <artifactId>spark-tags_2.13</artifactId>
20
    <version>3.5.6</version>
21
</dependency>
22
```
23

24
For Gradle:
25

26
```groovy
27
implementation 'org.apache.spark:spark-tags_2.13:3.5.6'
28
```
29

30
## Core Imports
31

32
```java
33
import org.apache.spark.annotation.*;
34
import org.apache.spark.tags.*;
35
```
36

37
## Basic Usage
38

39
### API Stability Annotations
40

41
Use these annotations to mark the stability level of your Spark APIs:
42

43
```java
44
import org.apache.spark.annotation.Experimental;
45
import org.apache.spark.annotation.Stable;
46
import org.apache.spark.annotation.DeveloperApi;
47

48
@Experimental
49
public class MyNewFeature {
50
    @Stable
51
    public void stableMethod() {
52
        // Implementation
53
    }
54
    
55
    @DeveloperApi
56
    public void advancedDeveloperMethod() {
57
        // Implementation
58
    }
59
}
60
```
61

62
### Test Category Annotations
63

64
Use these annotations to categorize your tests for selective execution:
65

66
```java
67
import org.apache.spark.tags.SlowHiveTest;
68
import org.apache.spark.tags.DockerTest;
69
import org.apache.spark.tags.ExtendedSQLTest;
70

71
@SlowHiveTest
72
public class MyHiveIntegrationTest {
73
    // Test implementation
74
}
75

76
@DockerTest
77
@Test
78
public void testWithDocker() {
79
    // Test requiring Docker
80
}
81

82
@ExtendedSQLTest  
83
@Test
84
public void testComplexSQL() {
85
    // Extended SQL test
86
}
87
```
88

89
## Architecture
90

91
Apache Spark Tags consists of two main annotation packages:
92

93
- **API Stability Package** (`org.apache.spark.annotation`): Annotations for marking API maturity levels and stability guarantees
94
- **Test Category Package** (`org.apache.spark.tags`): ScalaTest-compatible annotations for categorizing and filtering test execution
95

96
All annotations use runtime retention and support comprehensive targeting of Java language elements.
97

98
## Capabilities
99

100
### API Stability Annotations
101

102
Annotations for communicating API maturity and stability expectations to users and developers.
103

104
#### Experimental APIs
105

106
```java { .api }
107
/**
108
 * An experimental user-facing API.
109
 * Experimental API's might change or be removed in minor versions of Spark, 
110
 * or be adopted as first-class Spark API's.
111
 *
112
 * NOTE: If there exists a Scaladoc comment that immediately precedes this annotation, the first
113
 * line of the comment must be ":: Experimental ::" with no trailing blank line. This is because
114
 * of the known issue that Scaladoc displays only either the annotation or the comment, whichever
115
 * comes first.
116
 */
117
@Documented
118
@Retention(RetentionPolicy.RUNTIME)
119
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
120
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
121
public @interface Experimental {}
122
```
123

124
#### Stable APIs
125

126
```java { .api }
127
/**
128
 * Stable APIs that retain source and binary compatibility within a major release.
129
 * These interfaces can change from one major release to another major release
130
 * (e.g. from 1.0 to 2.0).
131
 */
132
@Documented
133
@Retention(RetentionPolicy.RUNTIME)
134
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
135
  ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
136
public @interface Stable {}
137
```
138

139
#### Evolving APIs
140

141
```java { .api }
142
/**
143
 * APIs that are meant to evolve towards becoming stable APIs, but are not stable APIs yet.
144
 * Evolving interfaces can change from one feature release to another release (i.e. 2.1 to 2.2).
145
 */
146
@Documented
147
@Retention(RetentionPolicy.RUNTIME)
148
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
149
  ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
150
public @interface Evolving {}
151
```
152

153
#### Unstable APIs
154

155
```java { .api }
156
/**
157
 * Unstable APIs, with no guarantee on stability.
158
 * Classes that are unannotated are considered Unstable.
159
 */
160
@Documented
161
@Retention(RetentionPolicy.RUNTIME)
162
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
163
  ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
164
public @interface Unstable {}
165
```
166

167
#### Developer APIs
168

169
```java { .api }
170
/**
171
 * A lower-level, unstable API intended for developers.
172
 * Developer API's might change or be removed in minor versions of Spark.
173
 *
174
 * NOTE: If there exists a Scaladoc comment that immediately precedes this annotation, the first
175
 * line of the comment must be ":: DeveloperApi ::" with no trailing blank line. This is because
176
 * of the known issue that Scaladoc displays only either the annotation or the comment, whichever
177
 * comes first.
178
 */
179
@Documented
180
@Retention(RetentionPolicy.RUNTIME)
181
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
182
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
183
public @interface DeveloperApi {}
184
```
185

186
#### Alpha Components
187

188
```java { .api }
189
/**
190
 * A new component of Spark which may have unstable API's.
191
 *
192
 * NOTE: If there exists a Scaladoc comment that immediately precedes this annotation, the first
193
 * line of the comment must be ":: AlphaComponent ::" with no trailing blank line. This is because
194
 * of the known issue that Scaladoc displays only either the annotation or the comment, whichever
195
 * comes first.
196
 */
197
@Retention(RetentionPolicy.RUNTIME)
198
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
199
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
200
public @interface AlphaComponent {}
201
```
202

203
#### Private APIs
204

205
```java { .api }
206
/**
207
 * A class that is considered private to the internals of Spark -- there is a high-likelihood
208
 * they will be changed in future versions of Spark.
209
 * This should be used only when the standard Scala / Java means of protecting classes are
210
 * insufficient. In particular, Java has no equivalent of private[spark], so we use this annotation
211
 * in its place.
212
 *
213
 * NOTE: If there exists a Scaladoc comment that immediately precedes this annotation, the first
214
 * line of the comment must be ":: Private ::" with no trailing blank line. This is because
215
 * of the known issue that Scaladoc displays only either the annotation or the comment, whichever
216
 * comes first.
217
 */
218
@Documented
219
@Retention(RetentionPolicy.RUNTIME)
220
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
221
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
222
public @interface Private {}
223
```
224

225
### Test Category Annotations
226

227
ScalaTest-compatible annotations for categorizing tests to enable selective test execution during Spark development and CI/CD processes.
228

229
#### Slow Test Categories
230

231
```java { .api }
232
/**
233
 * Tags slow Hive-related tests
234
 */
235
@TagAnnotation
236
@Retention(RetentionPolicy.RUNTIME)
237
@Target({ElementType.METHOD, ElementType.TYPE})
238
public @interface SlowHiveTest {}
239

240
/**
241
 * Tags slow SQL tests  
242
 */
243
@TagAnnotation
244
@Retention(RetentionPolicy.RUNTIME)
245
@Target({ElementType.METHOD, ElementType.TYPE})
246
public @interface SlowSQLTest {}
247
```
248

249
#### Extended Test Categories
250

251
```java { .api }
252
/**
253
 * Tags extended SQL tests
254
 */
255
@TagAnnotation
256
@Retention(RetentionPolicy.RUNTIME)
257
@Target({ElementType.METHOD, ElementType.TYPE})
258
public @interface ExtendedSQLTest {}
259

260
/**
261
 * Tags extended Hive-related tests
262
 */
263
@TagAnnotation
264
@Retention(RetentionPolicy.RUNTIME)
265
@Target({ElementType.METHOD, ElementType.TYPE})
266
public @interface ExtendedHiveTest {}
267

268
/**
269
 * Tags extended LevelDB-related tests
270
 */
271
@TagAnnotation
272
@Retention(RetentionPolicy.RUNTIME)
273
@Target({ElementType.METHOD, ElementType.TYPE})
274
public @interface ExtendedLevelDBTest {}
275

276
/**
277
 * Tags extended YARN-related tests
278
 */
279
@TagAnnotation
280
@Retention(RetentionPolicy.RUNTIME)
281
@Target({ElementType.METHOD, ElementType.TYPE})
282
public @interface ExtendedYarnTest {}
283
```
284

285
#### Infrastructure Test Categories
286

287
```java { .api }
288
/**
289
 * Tags tests that require Docker
290
 */
291
@TagAnnotation
292
@Retention(RetentionPolicy.RUNTIME)
293
@Target({ElementType.METHOD, ElementType.TYPE})
294
public @interface DockerTest {}
295

296
/**
297
 * Tags UI tests that require Chrome browser
298
 */
299
@TagAnnotation
300
@Retention(RetentionPolicy.RUNTIME)
301
@Target({ElementType.METHOD, ElementType.TYPE})
302
public @interface ChromeUITest {}
303
```
304

305
## Types
306

307
### Java Standard Types
308

309
All annotations use standard Java annotation interfaces and elements:
310

311
```java { .api }
312
import java.lang.annotation.Documented;
313
import java.lang.annotation.ElementType;
314
import java.lang.annotation.Retention;
315
import java.lang.annotation.RetentionPolicy;
316
import java.lang.annotation.Target;
317
```
318

319
### ScalaTest Integration
320

321
Test category annotations integrate with ScalaTest:
322

323
```java { .api }
324
import org.scalatest.TagAnnotation;
325
```
326

327
## Usage Patterns
328

329
### API Stability Marking
330

331
Use API stability annotations to communicate the maturity and evolution expectations of your Spark components:
332

333
```java
334
// Mark experimental features that may change
335
@Experimental
336
public class StreamingMLAlgorithm {
337
    
338
    // Mark stable core methods
339
    @Stable
340
    public void fit(Dataset<Row> data) {
341
        // Implementation
342
    }
343
    
344
    // Mark evolving APIs moving toward stability
345
    @Evolving 
346
    public MLModel getModel() {
347
        // Implementation  
348
    }
349
    
350
    // Mark internal developer APIs
351
    @DeveloperApi
352
    public void internalOptimize() {
353
        // Implementation
354
    }
355
}
356
```
357

358
### Test Categorization
359

360
Use test category annotations to organize tests for selective execution:
361

362
```java
363
// Slow tests that should be skipped in fast CI runs
364
@SlowHiveTest
365
public class HiveIntegrationTest {
366
    // Test implementation
367
}
368

369
// Tests requiring external infrastructure
370
@DockerTest 
371
public class KafkaIntegrationTest {
372
    @Test
373
    public void testKafkaConnectivity() {
374
        // Test requiring Docker-based Kafka
375
    }
376
}
377

378
// Extended test suites for comprehensive validation
379
@ExtendedSQLTest
380
public class ComplexSQLTest {
381
    @Test 
382
    public void testComplexJoins() {
383
        // Extended SQL functionality tests
384
    }
385
}
386
```
387

388
### Combining Annotations
389

390
Annotations can be combined as needed:
391

392
```java
393
@Experimental
394
@DeveloperApi
395
public class AdvancedInternalFeature {
396
    // Experimental developer-only functionality
397
}
398

399
@SlowHiveTest
400
@ExtendedHiveTest
401
public class ComprehensiveHiveTest {
402
    // Tests that are both slow and extended
403
}
404
```
405

406
## Error Handling
407

408
These annotations do not throw exceptions - they are purely metadata markers. However, they enable:
409

410
- **Build-time validation**: Tools can validate API usage against stability contracts
411
- **Test filtering**: ScalaTest can include/exclude tests based on tag annotations  
412
- **Documentation generation**: Tools can generate stability documentation from annotations
413
- **Static analysis**: Linters and IDEs can warn about using experimental or private APIs
414

415
## Dependencies
416

417
- **Java Standard Library**: All annotation interfaces and retention policies
418
- **Scala Library**: Runtime dependency for ScalaTest integration (`scala-library` at version specified by `${scala.version}`)
419
- **ScalaTest** (indirect): Test annotations integrate with ScalaTest's tagging system