0
# Spark Tags
1

2
Spark Tags provides Java annotations for marking API stability levels and categorizing test suites within the Apache Spark ecosystem. It offers annotation-based mechanisms to communicate API maturity, stability expectations, and test categorization for better software organization and lifecycle management.
3

4
## Package Information
5

6
- **Package Name**: spark-tags_2.11
7
- **Package Type**: maven
8
- **Language**: Java
9
- **Installation**: Add to Maven `pom.xml`:
10
  ```xml
11
  <dependency>
12
    <groupId>org.apache.spark</groupId>
13
    <artifactId>spark-tags_2.11</artifactId>
14
    <version>2.4.8</version>
15
  </dependency>
16
  ```
17

18
## Core Imports
19

20
```java
21
import org.apache.spark.annotation.Experimental;
22
import org.apache.spark.annotation.DeveloperApi;
23
import org.apache.spark.annotation.Private;
24
import org.apache.spark.annotation.AlphaComponent;
25
import org.apache.spark.annotation.InterfaceStability;
26
```
27

28
For test annotations:
29

30
```java
31
import org.apache.spark.tags.DockerTest;
32
import org.apache.spark.tags.ExtendedHiveTest;
33
import org.apache.spark.tags.ExtendedYarnTest;
34
import org.apache.spark.tags.SlowHiveTest;
35
```
36

37
## Basic Usage
38

39
```java
40
import org.apache.spark.annotation.Experimental;
41
import org.apache.spark.annotation.DeveloperApi;
42
import org.apache.spark.annotation.InterfaceStability;
43

44
// Mark an experimental API
45
@Experimental
46
public class NewFeature {
47
    // Experimental functionality
48
}
49

50
// Mark a developer-focused API
51
@DeveloperApi
52
public class InternalUtils {
53
    // Lower-level API for developers
54
}
55

56
// Mark interface stability
57
@InterfaceStability.Stable
58
public interface PublicApi {
59
    // Stable public interface
60
}
61

62
// Mark test methods
63
@ExtendedHiveTest
64
public class MyHiveIntegrationTest {
65
    // Hive-specific tests
66
}
67
```
68

69
## Capabilities
70

71
### API Stability Annotations
72

73
Annotations for marking the stability and intended audience of APIs within the Spark ecosystem.
74

75
#### Experimental API Marker
76

77
Marks experimental user-facing APIs that may change or be removed in minor versions.
78

79
```java { .api }
80
@Retention(RetentionPolicy.RUNTIME)
81
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, 
82
         ElementType.PARAMETER, ElementType.CONSTRUCTOR, 
83
         ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
84
public @interface Experimental {}
85
```
86

87
**Usage:**
88
- Apply to APIs that are experimental and may change
89
- Indicates APIs might be adopted as first-class or removed in minor versions
90
- Suitable for TYPE, FIELD, METHOD, PARAMETER, CONSTRUCTOR, LOCAL_VARIABLE, PACKAGE
91

92
#### Developer API Marker
93

94
Marks lower-level, unstable APIs intended for developers that may change in minor versions.
95

96
```java { .api }
97
@Retention(RetentionPolicy.RUNTIME)
98
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
99
         ElementType.PARAMETER, ElementType.CONSTRUCTOR,
100
         ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
101
public @interface DeveloperApi {}
102
```
103

104
**Usage:**
105
- Apply to lower-level APIs intended for advanced users
106
- Indicates unstable APIs that may change without notice
107
- Suitable for TYPE, FIELD, METHOD, PARAMETER, CONSTRUCTOR, LOCAL_VARIABLE, PACKAGE
108

109
#### Private API Marker
110

111
Marks classes considered private to Spark internals with high likelihood of change.
112

113
```java { .api }
114
@Retention(RetentionPolicy.RUNTIME)
115
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
116
         ElementType.PARAMETER, ElementType.CONSTRUCTOR,
117
         ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
118
public @interface Private {}
119
```
120

121
**Usage:**
122
- Use when standard Java visibility modifiers are insufficient
123
- Equivalent to Scala's `private[spark]` visibility
124
- Indicates high likelihood of change in future versions
125
- Suitable for TYPE, FIELD, METHOD, PARAMETER, CONSTRUCTOR, LOCAL_VARIABLE, PACKAGE
126

127
#### Alpha Component Marker
128

129
Marks new Spark components that may have unstable APIs.
130

131
```java { .api }
132
@Retention(RetentionPolicy.RUNTIME)
133
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
134
         ElementType.PARAMETER, ElementType.CONSTRUCTOR,
135
         ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
136
public @interface AlphaComponent {}
137
```
138

139
**Usage:**
140
- Apply to new components with potentially unstable APIs
141
- Indicates early-stage components that may undergo significant changes
142
- Suitable for TYPE, FIELD, METHOD, PARAMETER, CONSTRUCTOR, LOCAL_VARIABLE, PACKAGE
143

144
### Interface Stability Annotations
145

146
Nested annotations within the InterfaceStability class for marking interface stability levels.
147

148
```java { .api }
149
public class InterfaceStability {
150
    @Documented
151
    public @interface Stable {}
152
    
153
    @Documented  
154
    public @interface Evolving {}
155
    
156
    @Documented
157
    public @interface Unstable {}
158
}
159
```
160

161
#### Stable Interface Marker
162

163
Marks stable APIs that retain source and binary compatibility within a major release.
164

165
```java { .api }
166
@Documented
167
public @interface Stable {}
168
```
169

170
**Usage:**
171
- Apply to APIs that maintain compatibility within major releases
172
- Changes only allowed between major releases (e.g., 1.0 to 2.0)
173
- Provides strongest stability guarantee
174

175
#### Evolving Interface Marker
176

177
Marks APIs evolving towards stability that may change between feature releases.
178

179
```java { .api }
180
@Documented  
181
public @interface Evolving {}
182
```
183

184
**Usage:**
185
- Apply to APIs intended to become stable but not yet stable
186
- May change between feature releases (e.g., 2.1 to 2.2)
187
- Indicates APIs in transition to stability
188

189
#### Unstable Interface Marker
190

191
Marks unstable APIs with no stability guarantee.
192

193
```java { .api }
194
@Documented
195
public @interface Unstable {}
196
```
197

198
**Usage:**
199
- Apply to APIs with no stability guarantee
200
- Default assumption for unannotated classes
201
- May change at any time without notice
202

203
### Test Categorization Annotations
204

205
ScalaTest tag annotations for categorizing and selectively running test suites based on their requirements.
206

207
#### Docker Test Marker
208

209
ScalaTest tag annotation for tests requiring Docker infrastructure.
210

211
```java { .api }
212
@TagAnnotation
213
@Retention(RetentionPolicy.RUNTIME)
214
@Target({ElementType.METHOD, ElementType.TYPE})
215
public @interface DockerTest {}
216
```
217

218
**Usage:**
219
- Apply to test methods or test classes requiring Docker
220
- Enables selective test execution based on Docker availability
221
- Integrates with ScalaTest tag-based test filtering
222

223
#### Extended Hive Test Marker
224

225
ScalaTest tag annotation for extended Hive integration tests.
226

227
```java { .api }
228
@TagAnnotation
229
@Retention(RetentionPolicy.RUNTIME)
230
@Target({ElementType.METHOD, ElementType.TYPE})
231
public @interface ExtendedHiveTest {}
232
```
233

234
**Usage:**
235
- Apply to comprehensive Hive integration tests
236
- Enables selective execution of extended Hive test suites
237
- Typically used for thorough but time-consuming tests
238

239
#### Extended YARN Test Marker
240

241
ScalaTest tag annotation for extended YARN integration tests.
242

243
```java { .api }
244
@TagAnnotation
245
@Retention(RetentionPolicy.RUNTIME)
246
@Target({ElementType.METHOD, ElementType.TYPE})
247
public @interface ExtendedYarnTest {}
248
```
249

250
**Usage:**
251
- Apply to comprehensive YARN integration tests
252
- Enables selective execution of YARN-specific test suites
253
- Used for tests requiring YARN cluster functionality
254

255
#### Slow Hive Test Marker
256

257
ScalaTest tag annotation for slow-running Hive integration tests.
258

259
```java { .api }
260
@TagAnnotation
261
@Retention(RetentionPolicy.RUNTIME)
262
@Target({ElementType.METHOD, ElementType.TYPE})
263
public @interface SlowHiveTest {}
264
```
265

266
**Usage:**
267
- Apply to slow-running Hive integration tests
268
- Enables exclusion of time-intensive tests during development
269
- Facilitates selective test execution based on time constraints
270

271
## Types
272

273
All annotations use standard Java annotation infrastructure with these common characteristics:
274

275
```java { .api }
276
// Retention policy for API stability annotations
277
RetentionPolicy.RUNTIME
278

279
// Retention policy for test annotations  
280
RetentionPolicy.RUNTIME
281

282
// Common target elements for API annotations
283
ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
284
ElementType.PARAMETER, ElementType.CONSTRUCTOR, 
285
ElementType.LOCAL_VARIABLE, ElementType.PACKAGE
286

287
// Target elements for test annotations
288
ElementType.METHOD, ElementType.TYPE
289
```
290

291
## Error Handling
292

293
These annotations do not throw exceptions. They are processed at compile-time and runtime by annotation processors and testing frameworks. Invalid usage typically results in compilation warnings or test framework configuration issues rather than runtime exceptions.