or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

index.mdprotocol-messages.mdsecurity.mdserver-components.mdshuffle-client.md
tile.json

security.mddocs/

Security and Authentication

The security components provide SASL-based authentication for secure communication between shuffle clients and external shuffle services. This ensures that only authorized clients can access shuffle data.

Secret Management

ShuffleSecretManager

public class ShuffleSecretManager implements SecretKeyHolder {
    public ShuffleSecretManager();
    
    public void registerApp(String appId, String shuffleSecret);
    public void unregisterApp(String appId);
    public String getSecretKey(String appId);
    public String getSaslUser(String appId);
}

Manages SASL secrets used by the external shuffle service for authenticating client connections.

Key Methods:

registerApp

Registers a Spark application with its associated secret key for SASL authentication.

Parameters:

  • appId (String): Spark application identifier
  • shuffleSecret (String): Secret key for SASL authentication

unregisterApp

Removes an application's secret when the application completes or is no longer active.

Parameters:

  • appId (String): Application identifier to unregister

getSecretKey

Retrieves the secret key for a given application. Used during SASL authentication.

Parameters:

  • appId (String): Application identifier

Returns:

  • String: Secret key for the application, or null if not registered

getSaslUser

Gets the SASL username for an application. Returns the standard Spark SASL user.

Parameters:

  • appId (String): Application identifier

Returns:

  • String: SASL username (typically "sparkSaslUser")

SASL Configuration

Client-Side SASL Setup

import org.apache.spark.network.sasl.SecretKeyHolder;
import org.apache.spark.network.shuffle.ExternalShuffleClient;

// Create secret key holder
SecretKeyHolder secretHolder = new SecretKeyHolder() {
    @Override
    public String getSaslUser(String appId) {
        return "sparkSaslUser";
    }
    
    @Override
    public String getSecretKey(String appId) {
        return "my-app-secret-key";
    }
};

// Create client with SASL enabled
ExternalShuffleClient client = new ExternalShuffleClient(
    conf,
    secretHolder,
    true,  // Enable SASL
    true   // Enable SASL encryption
);

Server-Side SASL Setup

The shuffle service uses ShuffleSecretManager to validate client authentication:

// Create secret manager
ShuffleSecretManager secretManager = new ShuffleSecretManager();

// Register application secrets (typically done by Spark driver)
secretManager.registerApp("app-123", "shared-secret-key");

// Use with SASL-enabled transport context
// (Integration with TransportContext and SaslServerBootstrap)

Authentication Flow

SASL Authentication Process:

  1. Client initiates connection to shuffle service
  2. SASL handshake begins using configured mechanism (typically DIGEST-MD5)
  3. Client provides application ID and proves knowledge of secret key
  4. Server validates secret using ShuffleSecretManager
  5. On success, encrypted channel is established (if encryption enabled)
  6. Shuffle requests proceed over authenticated/encrypted connection

Security Features

Authentication

  • SASL-based authentication: Uses industry-standard SASL mechanisms
  • Per-application secrets: Each Spark application has its own secret key
  • Mutual authentication: Both client and server validate each other

Encryption

  • SASL encryption: Optional encryption of data in transit
  • Block data protection: Shuffle block data is encrypted during transfer
  • Metadata protection: Protocol messages are encrypted

Access Control

  • Application isolation: Applications can only access their own shuffle data
  • Executor validation: Blocks can only be fetched by registered executors
  • Secret rotation: Applications can update secrets during runtime

Usage Examples

Basic Authentication Setup

import org.apache.spark.network.sasl.ShuffleSecretManager;

// Server-side: Create and configure secret manager
ShuffleSecretManager secretManager = new ShuffleSecretManager();

// Register applications (typically done by Spark driver)
secretManager.registerApp("spark-app-1", "secret-1");
secretManager.registerApp("spark-app-2", "secret-2");

// Later, validate authentication
String secret = secretManager.getSecretKey("spark-app-1");
if (secret != null) {
    System.out.println("Application is registered");
} else {
    System.out.println("Unknown application");
}

Client Authentication

// Client-side: Implement SecretKeyHolder
class AppSecretHolder implements SecretKeyHolder {
    private final String appSecret;
    
    public AppSecretHolder(String secret) {
        this.appSecret = secret;
    }
    
    @Override
    public String getSaslUser(String appId) {
        return "sparkSaslUser";  // Standard Spark SASL user
    }
    
    @Override
    public String getSecretKey(String appId) {
        return appSecret;
    }
}

// Create authenticated client
AppSecretHolder secretHolder = new AppSecretHolder("my-app-secret");
ExternalShuffleClient client = new ExternalShuffleClient(
    conf,
    secretHolder,
    true,   // SASL enabled
    false   // SASL encryption disabled for performance
);

client.init("spark-app-1");

Full Security Setup

// Maximum security configuration
ExternalShuffleClient secureClient = new ExternalShuffleClient(
    conf,
    secretHolder,
    true,   // Enable SASL authentication
    true    // Enable SASL encryption
);

// This client will:
// 1. Authenticate using SASL DIGEST-MD5
// 2. Encrypt all data in transit
// 3. Validate server identity

Application Lifecycle Management

// Application startup
secretManager.registerApp("new-app", generateSecret());

try {
    // Application runs...
    // Shuffle operations proceed with authentication
    
} finally {
    // Application cleanup
    secretManager.unregisterApp("new-app");
}

Security Best Practices

Secret Management

  • Generate strong secrets: Use cryptographically secure random generators
  • Rotate secrets regularly: Update secrets periodically for enhanced security
  • Secure secret distribution: Use secure channels to distribute secrets to clients
  • Clean up secrets: Remove secrets when applications complete
import java.security.SecureRandom;
import java.util.Base64;

// Generate secure random secret
SecureRandom random = new SecureRandom();
byte[] secretBytes = new byte[32];
random.nextBytes(secretBytes);
String secret = Base64.getEncoder().encodeToString(secretBytes);

secretManager.registerApp("app-id", secret);

Network Security

  • Use encryption in production: Enable SASL encryption for sensitive data
  • Validate certificates: In TLS deployments, validate server certificates
  • Monitor authentication failures: Log and alert on authentication failures

Access Control

  • Principle of least privilege: Only grant access to necessary shuffle data
  • Application isolation: Ensure applications cannot access each other's data
  • Audit access: Log shuffle data access for security auditing

Error Handling

Common security-related errors:

// Authentication failure handling
try {
    client.fetchBlocks(host, port, execId, blockIds, listener);
} catch (SecurityException e) {
    System.err.println("Authentication failed: " + e.getMessage());
    // Handle authentication failure:
    // - Check secret configuration
    // - Verify application registration
    // - Retry with updated credentials
}

// SASL configuration errors
try {
    client.init("app-id");
} catch (IllegalArgumentException e) {
    if (e.getMessage().contains("SASL")) {
        System.err.println("SASL configuration error: " + e.getMessage());
        // Check SASL settings and secret holder configuration
    }
}

Troubleshooting

Common Issues

  1. Authentication Failures

    • Verify secret registration on server
    • Check client secret configuration
    • Ensure application IDs match

  2. SASL Configuration Errors

    • Verify SASL mechanism support
    • Check encryption compatibility
    • Validate secret holder implementation

  3. Network Connectivity

    • Test without SASL first
    • Check firewall rules for SASL ports
    • Verify server SASL configuration

Debug Configuration

// Enable SASL debugging (add to JVM arguments)
// -Djava.security.debug=gssloginconfig,configfile,configparser,logincontext

// Enable Spark network debugging
// spark.network.crypto.enabled=true (for additional encryption)
// spark.authenticate=true (for Spark internal authentication)

Integration with Spark Security

The shuffle service security integrates with Spark's overall security framework:

  • Shared secrets: Uses same secret distribution mechanism as Spark
  • Authentication coordination: Coordinates with Spark's SecurityManager
  • Encryption compatibility: Compatible with Spark's network encryption
  • Access control: Respects Spark's application isolation requirements