0
# Apache Spark Tags
1

2
Apache Spark Tags provides annotation utilities for categorizing and marking API components and test suites in Apache Spark. It includes stability annotations to mark API maturity levels and test tags for organizing test execution in Spark's comprehensive testing infrastructure.
3

4
## Package Information
5

6
- **Package Name**: org.apache.spark:spark-tags_2.11
7
- **Package Type**: Maven
8
- **Language**: Scala
9
- **Version**: 2.4.8
10
- **Installation**: Add to Maven pom.xml:
11

12
```xml
13
<dependency>
14
    <groupId>org.apache.spark</groupId>
15
    <artifactId>spark-tags_2.11</artifactId>
16
    <version>2.4.8</version>
17
</dependency>
18
```
19

20
## Core Imports
21

22
Main API annotations:
23
```java
24
import org.apache.spark.annotation.*;
25
```
26

27
Individual API annotations:
28
```java
29
import org.apache.spark.annotation.Experimental;
30
import org.apache.spark.annotation.DeveloperApi;
31
import org.apache.spark.annotation.Private;
32
import org.apache.spark.annotation.AlphaComponent;
33
import org.apache.spark.annotation.InterfaceStability;
34
```
35

36
Scala annotation:
37
```scala
38
import org.apache.spark.annotation.Since;
39
```
40

41
Test tags (test scope only):
42
```java
43
import org.apache.spark.tags.SlowHiveTest;
44
import org.apache.spark.tags.ExtendedHiveTest;
45
import org.apache.spark.tags.ExtendedYarnTest;
46
import org.apache.spark.tags.DockerTest;
47
```
48

49
**Note**: Test tags are located in `src/test` and are not part of the main API.
50

51
## Basic Usage
52

53
### API Stability Annotations
54
```java
55
import org.apache.spark.annotation.Experimental;
56
import org.apache.spark.annotation.DeveloperApi;
57
import org.apache.spark.annotation.InterfaceStability;
58

59
// Mark experimental user-facing API
60
@Experimental
61
public class MySparkComponent {
62
    // Implementation
63
}
64

65
// Mark developer-focused API
66
@DeveloperApi
67
public void internalMethod() {
68
    // Implementation
69
}
70

71
// Mark stable API
72
@InterfaceStability.Stable
73
public interface PublicAPI {
74
    void stableMethod();
75
}
76
```
77

78
### Test Tags
79
```java
80
import org.apache.spark.tags.SlowHiveTest;
81
import org.apache.spark.tags.DockerTest;
82
import org.junit.Test;
83

84
public class MySparkTests {
85
    
86
    @Test
87
    @SlowHiveTest
88
    public void testHiveIntegration() {
89
        // Slow Hive test implementation
90
    }
91
    
92
    @Test
93
    @DockerTest
94
    public void testDockerEnvironment() {
95
        // Docker-based test implementation
96
    }
97
}
98
```
99

100
## Capabilities
101

102
### API Stability Annotations
103

104
These annotations help communicate the stability and intended audience of Spark APIs.
105

106
#### @Experimental
107

108
Marks experimental user-facing APIs that might change or be removed in minor versions.
109

110
```java { .api }
111
@Retention(RetentionPolicy.RUNTIME)
112
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
113
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
114
public @interface Experimental {}
115
```
116

117
**Note**: When used with Scaladoc, the first line of the comment must be ":: Experimental ::" with no trailing blank line.
118

119
#### @DeveloperApi
120

121
Marks lower-level, unstable APIs intended for developers that might change or be removed in minor versions.
122

123
```java { .api }
124
@Retention(RetentionPolicy.RUNTIME)
125
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
126
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
127
public @interface DeveloperApi {}
128
```
129

130
**Note**: When used with Scaladoc, the first line of the comment must be ":: DeveloperApi ::" with no trailing blank line.
131

132
#### @Private
133

134
Marks classes considered private to Spark internals with high likelihood of change, used when standard Java/Scala visibility modifiers are insufficient.
135

136
```java { .api }
137
@Retention(RetentionPolicy.RUNTIME)
138
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
139
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
140
public @interface Private {}
141
```
142

143
**Note**: When used with Scaladoc, the first line of the comment must be ":: Private ::" with no trailing blank line.
144

145
#### @AlphaComponent
146

147
Marks new components of Spark which may have unstable APIs.
148

149
```java { .api }
150
@Retention(RetentionPolicy.RUNTIME)
151
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
152
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
153
public @interface AlphaComponent {}
154
```
155

156
**Note**: When used with Scaladoc, the first line of the comment must be ":: AlphaComponent ::" with no trailing blank line.
157

158
#### InterfaceStability
159

160
Container class providing stability annotations to inform users about API reliability.
161

162
```java { .api }
163
public class InterfaceStability {
164
    
165
    @Documented
166
    public @interface Stable {}
167
    
168
    @Documented
169
    public @interface Evolving {}
170
    
171
    @Documented
172
    public @interface Unstable {}
173
}
174
```
175

176
**@InterfaceStability.Stable**: Marks stable APIs that retain source and binary compatibility within major releases. Can change between major releases but stable within major version.
177

178
**@InterfaceStability.Evolving**: Marks APIs meant to evolve towards becoming stable APIs, but not yet stable. Can change between feature releases (e.g., 2.1 to 2.2).
179

180
**@InterfaceStability.Unstable**: Marks unstable APIs with no guarantee on stability. Unannotated classes are considered unstable.
181

182
#### @Since
183

184
Annotates program elements to specify the Spark version when a feature was first introduced. This is a Spark-internal annotation used for API documentation.
185

186
```scala { .api }
187
private[spark] class Since(version: String) extends StaticAnnotation
188
```
189

190
**Usage Example:**
191
```scala
192
@Since("2.0.0")
193
def newFeature(): Unit = {
194
    // Implementation added in Spark 2.0.0
195
}
196
```
197

198
**Note**: This annotation is used extensively throughout Spark's codebase to track API additions and changes across versions.
199

200
### Test Categorization Tags
201

202
These annotations are ScalaTest TagAnnotations for organizing and selectively running different categories of tests.
203

204
#### @SlowHiveTest
205

206
Tags slow-running Hive-related tests for selective execution.
207

208
```java { .api }
209
@TagAnnotation
210
@Retention(RetentionPolicy.RUNTIME)
211
@Target({ElementType.METHOD, ElementType.TYPE})
212
public @interface SlowHiveTest {}
213
```
214

215
#### @ExtendedHiveTest
216

217
Tags extended Hive-related tests that require additional setup or time.
218

219
```java { .api }
220
@TagAnnotation
221
@Retention(RetentionPolicy.RUNTIME)
222
@Target({ElementType.METHOD, ElementType.TYPE})
223
public @interface ExtendedHiveTest {}
224
```
225

226
#### @ExtendedYarnTest
227

228
Tags extended YARN-related tests for distributed computing scenarios.
229

230
```java { .api }
231
@TagAnnotation
232
@Retention(RetentionPolicy.RUNTIME)
233
@Target({ElementType.METHOD, ElementType.TYPE})
234
public @interface ExtendedYarnTest {}
235
```
236

237
#### @DockerTest
238

239
Tags Docker-based tests that require Docker environment for execution.
240

241
```java { .api }
242
@TagAnnotation
243
@Retention(RetentionPolicy.RUNTIME)
244
@Target({ElementType.METHOD, ElementType.TYPE})
245
public @interface DockerTest {}
246
```
247

248
## Architecture
249

250
The package is organized into two main components:
251

252
1. **org.apache.spark.annotation** (main source: `src/main/java` and `src/main/scala`): API documentation and stability markers
253
   - Used to communicate API maturity and intended audience
254
   - Applied to classes, methods, fields, and other Java elements  
255
   - Helps users understand API stability guarantees
256
   - Contains both Java annotations and the Scala @Since annotation
257

258
2. **org.apache.spark.tags** (test source: `src/test/java`): Test categorization system
259
   - ScalaTest TagAnnotations for test organization only
260
   - Enables selective test execution in CI/CD pipelines
261
   - Supports different test environments (Hive, YARN, Docker)
262
   - **Note**: These are test-scoped annotations, not part of the main API
263

264
## Dependencies
265

266
The module has minimal dependencies:
267

268
- **Scala Library**: Standard Scala runtime library (version 2.11.12)
269

270
**Note**: Despite test tags referencing `@TagAnnotation`, ScalaTest is not a direct dependency of this module. Test environments using these tags need to provide ScalaTest separately.