Apache Spark Tags

Apache Spark Tags provides annotation utilities for categorizing and marking API components and test suites in Apache Spark. It includes stability annotations to mark API maturity levels and test tags for organizing test execution in Spark's comprehensive testing infrastructure.

Package Information

• Package Name: org.apache.spark:spark-tags_2.11
• Package Type: Maven
• Language: Scala
• Version: 2.4.8
• Installation: Add to Maven pom.xml:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-tags_2.11</artifactId>
    <version>2.4.8</version>
</dependency>

Core Imports

Main API annotations:

import org.apache.spark.annotation.*;

Individual API annotations:

import org.apache.spark.annotation.Experimental;
import org.apache.spark.annotation.DeveloperApi;
import org.apache.spark.annotation.Private;
import org.apache.spark.annotation.AlphaComponent;
import org.apache.spark.annotation.InterfaceStability;

Scala annotation:

import org.apache.spark.annotation.Since;

Test tags (test scope only):

import org.apache.spark.tags.SlowHiveTest;
import org.apache.spark.tags.ExtendedHiveTest;
import org.apache.spark.tags.ExtendedYarnTest;
import org.apache.spark.tags.DockerTest;

Note: Test tags are located in src/test and are not part of the main API.

Basic Usage

API Stability Annotations

import org.apache.spark.annotation.Experimental;
import org.apache.spark.annotation.DeveloperApi;
import org.apache.spark.annotation.InterfaceStability;

// Mark experimental user-facing API
@Experimental
public class MySparkComponent {
    // Implementation
}

// Mark developer-focused API
@DeveloperApi
public void internalMethod() {
    // Implementation
}

// Mark stable API
@InterfaceStability.Stable
public interface PublicAPI {
    void stableMethod();
}

Test Tags

import org.apache.spark.tags.SlowHiveTest;
import org.apache.spark.tags.DockerTest;
import org.junit.Test;

public class MySparkTests {
    
    @Test
    @SlowHiveTest
    public void testHiveIntegration() {
        // Slow Hive test implementation
    }
    
    @Test
    @DockerTest
    public void testDockerEnvironment() {
        // Docker-based test implementation
    }
}

Capabilities

API Stability Annotations

These annotations help communicate the stability and intended audience of Spark APIs.

@Experimental

Marks experimental user-facing APIs that might change or be removed in minor versions.

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface Experimental {}

Note: When used with Scaladoc, the first line of the comment must be ":: Experimental ::" with no trailing blank line.

@DeveloperApi

Marks lower-level, unstable APIs intended for developers that might change or be removed in minor versions.

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface DeveloperApi {}

Note: When used with Scaladoc, the first line of the comment must be ":: DeveloperApi ::" with no trailing blank line.

@Private

Marks classes considered private to Spark internals with high likelihood of change, used when standard Java/Scala visibility modifiers are insufficient.

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface Private {}

Note: When used with Scaladoc, the first line of the comment must be ":: Private ::" with no trailing blank line.

@AlphaComponent

Marks new components of Spark which may have unstable APIs.

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
        ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface AlphaComponent {}

Note: When used with Scaladoc, the first line of the comment must be ":: AlphaComponent ::" with no trailing blank line.

InterfaceStability

Container class providing stability annotations to inform users about API reliability.

public class InterfaceStability {
    
    @Documented
    public @interface Stable {}
    
    @Documented
    public @interface Evolving {}
    
    @Documented
    public @interface Unstable {}
}

@InterfaceStability.Stable: Marks stable APIs that retain source and binary compatibility within major releases. Can change between major releases but stable within major version.

@InterfaceStability.Evolving: Marks APIs meant to evolve towards becoming stable APIs, but not yet stable. Can change between feature releases (e.g., 2.1 to 2.2).

@InterfaceStability.Unstable: Marks unstable APIs with no guarantee on stability. Unannotated classes are considered unstable.

@Since

Annotates program elements to specify the Spark version when a feature was first introduced. This is a Spark-internal annotation used for API documentation.

private[spark] class Since(version: String) extends StaticAnnotation

Usage Example:

@Since("2.0.0")
def newFeature(): Unit = {
    // Implementation added in Spark 2.0.0
}

Note: This annotation is used extensively throughout Spark's codebase to track API additions and changes across versions.

Test Categorization Tags

These annotations are ScalaTest TagAnnotations for organizing and selectively running different categories of tests.

@SlowHiveTest

Tags slow-running Hive-related tests for selective execution.

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface SlowHiveTest {}

@ExtendedHiveTest

Tags extended Hive-related tests that require additional setup or time.

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface ExtendedHiveTest {}

@ExtendedYarnTest

Tags extended YARN-related tests for distributed computing scenarios.

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface ExtendedYarnTest {}

@DockerTest

Tags Docker-based tests that require Docker environment for execution.

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface DockerTest {}

Architecture

The package is organized into two main components:

1. org.apache.spark.annotation (main source: src/main/java and src/main/scala): API documentation and stability markers

   • Used to communicate API maturity and intended audience
   • Applied to classes, methods, fields, and other Java elements
   • Helps users understand API stability guarantees
   • Contains both Java annotations and the Scala @Since annotation

2. org.apache.spark.tags (test source: src/test/java): Test categorization system

   • ScalaTest TagAnnotations for test organization only
   • Enables selective test execution in CI/CD pipelines
   • Supports different test environments (Hive, YARN, Docker)
   • Note: These are test-scoped annotations, not part of the main API

Dependencies

The module has minimal dependencies:

• Scala Library: Standard Scala runtime library (version 2.11.12)

Note: Despite test tags referencing @TagAnnotation, ScalaTest is not a direct dependency of this module. Test environments using these tags need to provide ScalaTest separately.