Spark Tags

Spark Tags provides Java annotations for marking API stability levels and categorizing test suites within the Apache Spark ecosystem. It offers annotation-based mechanisms to communicate API maturity, stability expectations, and test categorization for better software organization and lifecycle management.

Package Information

• Package Name: spark-tags_2.11
• Package Type: maven
• Language: Java
• Installation: Add to Maven pom.xml:

  <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-tags_2.11</artifactId>
    <version>2.4.8</version>
  </dependency>

Core Imports

import org.apache.spark.annotation.Experimental;
import org.apache.spark.annotation.DeveloperApi;
import org.apache.spark.annotation.Private;
import org.apache.spark.annotation.AlphaComponent;
import org.apache.spark.annotation.InterfaceStability;

For test annotations:

import org.apache.spark.tags.DockerTest;
import org.apache.spark.tags.ExtendedHiveTest;
import org.apache.spark.tags.ExtendedYarnTest;
import org.apache.spark.tags.SlowHiveTest;

Basic Usage

import org.apache.spark.annotation.Experimental;
import org.apache.spark.annotation.DeveloperApi;
import org.apache.spark.annotation.InterfaceStability;

// Mark an experimental API
@Experimental
public class NewFeature {
    // Experimental functionality
}

// Mark a developer-focused API
@DeveloperApi
public class InternalUtils {
    // Lower-level API for developers
}

// Mark interface stability
@InterfaceStability.Stable
public interface PublicApi {
    // Stable public interface
}

// Mark test methods
@ExtendedHiveTest
public class MyHiveIntegrationTest {
    // Hive-specific tests
}

Capabilities

API Stability Annotations

Annotations for marking the stability and intended audience of APIs within the Spark ecosystem.

Experimental API Marker

Marks experimental user-facing APIs that may change or be removed in minor versions.

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, 
         ElementType.PARAMETER, ElementType.CONSTRUCTOR, 
         ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface Experimental {}

Usage:

• Apply to APIs that are experimental and may change
• Indicates APIs might be adopted as first-class or removed in minor versions
• Suitable for TYPE, FIELD, METHOD, PARAMETER, CONSTRUCTOR, LOCAL_VARIABLE, PACKAGE

Developer API Marker

Marks lower-level, unstable APIs intended for developers that may change in minor versions.

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
         ElementType.PARAMETER, ElementType.CONSTRUCTOR,
         ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface DeveloperApi {}

Usage:

• Apply to lower-level APIs intended for advanced users
• Indicates unstable APIs that may change without notice
• Suitable for TYPE, FIELD, METHOD, PARAMETER, CONSTRUCTOR, LOCAL_VARIABLE, PACKAGE

Private API Marker

Marks classes considered private to Spark internals with high likelihood of change.

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
         ElementType.PARAMETER, ElementType.CONSTRUCTOR,
         ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface Private {}

Usage:

• Use when standard Java visibility modifiers are insufficient
• Equivalent to Scala's private[spark] visibility
• Indicates high likelihood of change in future versions
• Suitable for TYPE, FIELD, METHOD, PARAMETER, CONSTRUCTOR, LOCAL_VARIABLE, PACKAGE

Alpha Component Marker

Marks new Spark components that may have unstable APIs.

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
         ElementType.PARAMETER, ElementType.CONSTRUCTOR,
         ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface AlphaComponent {}

Usage:

• Apply to new components with potentially unstable APIs
• Indicates early-stage components that may undergo significant changes
• Suitable for TYPE, FIELD, METHOD, PARAMETER, CONSTRUCTOR, LOCAL_VARIABLE, PACKAGE

Interface Stability Annotations

Nested annotations within the InterfaceStability class for marking interface stability levels.

public class InterfaceStability {
    @Documented
    public @interface Stable {}
    
    @Documented  
    public @interface Evolving {}
    
    @Documented
    public @interface Unstable {}
}

Stable Interface Marker

Marks stable APIs that retain source and binary compatibility within a major release.

@Documented
public @interface Stable {}

Usage:

• Apply to APIs that maintain compatibility within major releases
• Changes only allowed between major releases (e.g., 1.0 to 2.0)
• Provides strongest stability guarantee

Evolving Interface Marker

Marks APIs evolving towards stability that may change between feature releases.

@Documented  
public @interface Evolving {}

Usage:

• Apply to APIs intended to become stable but not yet stable
• May change between feature releases (e.g., 2.1 to 2.2)
• Indicates APIs in transition to stability

Unstable Interface Marker

Marks unstable APIs with no stability guarantee.

@Documented
public @interface Unstable {}

Usage:

• Apply to APIs with no stability guarantee
• Default assumption for unannotated classes
• May change at any time without notice

Test Categorization Annotations

ScalaTest tag annotations for categorizing and selectively running test suites based on their requirements.

Docker Test Marker

ScalaTest tag annotation for tests requiring Docker infrastructure.

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface DockerTest {}

Usage:

• Apply to test methods or test classes requiring Docker
• Enables selective test execution based on Docker availability
• Integrates with ScalaTest tag-based test filtering

Extended Hive Test Marker

ScalaTest tag annotation for extended Hive integration tests.

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface ExtendedHiveTest {}

Usage:

• Apply to comprehensive Hive integration tests
• Enables selective execution of extended Hive test suites
• Typically used for thorough but time-consuming tests

Extended YARN Test Marker

ScalaTest tag annotation for extended YARN integration tests.

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface ExtendedYarnTest {}

Usage:

• Apply to comprehensive YARN integration tests
• Enables selective execution of YARN-specific test suites
• Used for tests requiring YARN cluster functionality

Slow Hive Test Marker

ScalaTest tag annotation for slow-running Hive integration tests.

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface SlowHiveTest {}

Usage:

• Apply to slow-running Hive integration tests
• Enables exclusion of time-intensive tests during development
• Facilitates selective test execution based on time constraints

Types

All annotations use standard Java annotation infrastructure with these common characteristics:

// Retention policy for API stability annotations
RetentionPolicy.RUNTIME

// Retention policy for test annotations  
RetentionPolicy.RUNTIME

// Common target elements for API annotations
ElementType.TYPE, ElementType.FIELD, ElementType.METHOD,
ElementType.PARAMETER, ElementType.CONSTRUCTOR, 
ElementType.LOCAL_VARIABLE, ElementType.PACKAGE

// Target elements for test annotations
ElementType.METHOD, ElementType.TYPE

Error Handling

These annotations do not throw exceptions. They are processed at compile-time and runtime by annotation processors and testing frameworks. Invalid usage typically results in compilation warnings or test framework configuration issues rather than runtime exceptions.