or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.md
tile.json

tessl/maven-org-apache-spark--spark-tags-2-12

Annotation library for Apache Spark test categorization and API stability markers

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
mavenpkg:maven/org.apache.spark/spark-tags_2.12@3.5.x

To install, run

npx @tessl/cli install tessl/maven-org-apache-spark--spark-tags-2-12@3.5.0

index.mddocs/

Apache Spark Tags

Apache Spark Tags provides Java and Scala annotations for API stability marking and test categorization within the Apache Spark ecosystem. This annotation library enables consistent test organization, clear API stability communication to users, and systematic tracking of feature evolution across Spark releases.

Package Information

  • Package Name: spark-tags_2.12
  • Package Type: maven
  • Group ID: org.apache.spark
  • Language: Java/Scala
  • Installation: Add Maven dependency:
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-tags_2.12</artifactId>
    <version>3.5.6</version>
</dependency>

For Gradle:

implementation 'org.apache.spark:spark-tags_2.12:3.5.6'

Core Imports

// API Stability Annotations
import org.apache.spark.annotation.Stable;
import org.apache.spark.annotation.Unstable;
import org.apache.spark.annotation.Experimental;
import org.apache.spark.annotation.Evolving;
import org.apache.spark.annotation.DeveloperApi;
import org.apache.spark.annotation.Private;
import org.apache.spark.annotation.AlphaComponent;

// Test Category Annotations
import org.apache.spark.tags.ExtendedSQLTest;
import org.apache.spark.tags.DockerTest;
import org.apache.spark.tags.SlowHiveTest;

For Scala:

import org.apache.spark.annotation.Since

Basic Usage

// Mark a stable API
@Stable
public class MyStableAPI {
    @Stable
    public void stableMethod() {
        // Implementation
    }
}

// Mark experimental features
@Experimental
public class NewFeature {
    @DeveloperApi
    public void advancedConfiguration() {
        // Developer-only API
    }
}

// Categorize tests
@ExtendedSQLTest
public class MyExtendedSQLTest {
    @DockerTest
    public void testWithDocker() {
        // Test requiring Docker
    }
}

Architecture

Apache Spark Tags is organized into two main annotation packages:

  • API Stability Package (org.apache.spark.annotation): Annotations for marking API stability and intended audience. This package contains "Spark annotations to mark an API experimental or intended only for advanced usages by developers" and is reflected in Scala and Java docs.
  • Test Category Package (org.apache.spark.tags): ScalaTest tag annotations for test categorization and filtering
  • Version Tracking: Scala-specific annotation for tracking feature introduction versions

Capabilities

API Stability Annotations

Annotations for marking the stability and intended audience of APIs within the Apache Spark ecosystem.

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
         ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface Stable {}

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
         ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface Unstable {}

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
         ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface Experimental {}

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
         ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface Evolving {}

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
         ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface DeveloperApi {}

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
         ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface Private {}

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD, ElementType.PARAMETER,
         ElementType.CONSTRUCTOR, ElementType.LOCAL_VARIABLE, ElementType.PACKAGE})
public @interface AlphaComponent {}

Stability Levels (in order of stability):

  • @Stable: APIs that retain source and binary compatibility within a major release. Can change between major releases (e.g., 1.0 to 2.0).
  • @Evolving: APIs meant to evolve towards becoming stable APIs but not stable yet. Can change between feature releases (e.g., 2.1 to 2.2).
  • @Experimental: Experimental user-facing APIs that might change or be removed in minor versions of Spark, or be adopted as first-class Spark APIs. Note: When used with Scaladoc, the first line of any preceding comment must be ":: Experimental ::" with no trailing blank line due to known Scaladoc display limitations.
  • @Unstable: APIs with no guarantee on stability. This is the default for unannotated classes.

Audience Annotations:

  • @DeveloperApi: Lower-level, unstable APIs intended for developers. Might change or be removed in minor versions.
  • @Private: Classes considered private to Spark internals with high likelihood of change. Used when standard Java/Scala visibility modifiers are insufficient.
  • @AlphaComponent: New components of Spark which may have unstable APIs.

Usage Examples:

// Stable public API
@Stable
public class DataFrameReader {
    @Stable
    public Dataset<Row> json(String path) {
        // Implementation
    }
}

// Experimental feature
@Experimental
public class MLPipeline {
    @Experimental
    public Model fit(Dataset<?> dataset) {
        // Implementation
    }
}

// Developer-only API
@DeveloperApi
public class InternalCacheManager {
    @Private
    void clearInternalCache() {
        // Internal implementation
    }
}

// Alpha component
@AlphaComponent
public class GraphXExperimental {
    @Unstable
    public Graph<VD, ED> processGraph() {
        // Implementation
    }
}

Version Tracking

Scala annotation for tracking when features were introduced in Spark versions.

private[spark] class Since(version: String) extends StaticAnnotation

Usage:

@Since("3.0.0")
def newSparkFeature(): Unit = {
  // Implementation introduced in Spark 3.0.0
}

@Since("2.4.0")
class FeatureFromTwoFour {
  @Since("2.4.5")
  def methodAddedInPatch(): String = "result"
}

The Since annotation is private to the Spark package (private[spark]) and is used internally to track API evolution. Unlike @since JavaDoc tags, this annotation doesn't require explicit JavaDoc and works for overridden methods that inherit documentation from parents. However, it doesn't appear in generated Java API documentation.

Test Category Annotations

ScalaTest tag annotations for categorizing and filtering test execution in the Apache Spark test suite.

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface ExtendedSQLTest {}

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface ExtendedHiveTest {}

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface ExtendedYarnTest {}

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface ExtendedLevelDBTest {}

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface SlowSQLTest {}

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface SlowHiveTest {}

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface DockerTest {}

@TagAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface ChromeUITest {}

All test category annotations are ScalaTest @TagAnnotation interfaces that can be applied to test methods or test classes. They enable selective test execution and organization.

Test Categories:

  • @ExtendedSQLTest: Extended SQL functionality tests
  • @ExtendedHiveTest: Extended Hive integration tests
  • @ExtendedYarnTest: Extended YARN cluster tests
  • @ExtendedLevelDBTest: Extended LevelDB storage tests
  • @SlowSQLTest: SQL tests that take significant time to execute
  • @SlowHiveTest: Hive tests that take significant time to execute
  • @DockerTest: Tests requiring Docker containers
  • @ChromeUITest: UI tests requiring Chrome browser

Usage Examples:

// Categorize a test class
@ExtendedSQLTest
public class ComplexSQLQueryTest {
    @Test
    public void testComplexJoins() {
        // Test implementation
    }
}

// Categorize individual test methods
public class MixedTestSuite {
    @Test
    @SlowSQLTest
    public void testLargeDatasetQuery() {
        // Slow SQL test
    }
    
    @Test
    @DockerTest
    public void testWithExternalDatabase() {
        // Test requiring Docker
    }
    
    @Test
    @ChromeUITest
    public void testWebUI() {
        // UI test requiring Chrome
    }
}

// Multiple annotations
@ExtendedHiveTest
@SlowHiveTest
public class HeavyHiveProcessingTest {
    // Extended and slow Hive tests
}

Test Filtering:

These annotations enable running specific test categories:

# Run only extended SQL tests
mvn test -Dtest.include.tags=org.apache.spark.tags.ExtendedSQLTest

# Exclude slow tests
mvn test -Dtest.exclude.tags=org.apache.spark.tags.SlowSQLTest,org.apache.spark.tags.SlowHiveTest

# Run only Docker tests
mvn test -Dtest.include.tags=org.apache.spark.tags.DockerTest

Type Definitions

All annotations are marker annotations (no parameters) that serve as metadata for the Java/Scala compiler and runtime reflection. The annotations use Java's built-in annotation types:

// Standard annotation imports used by all annotations
import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

// ScalaTest import for test annotations
import org.scalatest.TagAnnotation;

Retention Policies:

  • RetentionPolicy.RUNTIME: Annotations are available at runtime via reflection
  • All annotations in this library use RUNTIME retention

Target Elements:

  • API stability annotations can be applied to: TYPE, FIELD, METHOD, PARAMETER, CONSTRUCTOR, LOCAL_VARIABLE, PACKAGE
  • Test category annotations can be applied to: METHOD, TYPE only

Scala Annotation Features:

import scala.annotation.StaticAnnotation
import scala.annotation.meta._

// Meta-annotations for Scala annotation targets
@param @field @getter @setter @beanGetter @beanSetter
private[spark] class Since(version: String) extends StaticAnnotation

The Since annotation uses Scala's meta-annotation system to apply to multiple target types simultaneously (parameters, fields, getters, setters, and bean accessors).