0
# Security
1

2
SASL authentication support for secure communication between shuffle clients and external shuffle services, including secret management and user authentication.
3

4
## Capabilities
5

6
### Shuffle Secret Manager
7

8
Manages shuffle secrets for external shuffle service, providing application-level authentication and secret key management.
9

10
```java { .api }
11
/**
12
 * Manages shuffle secrets for external shuffle service
13
 */
14
public class ShuffleSecretManager implements SecretKeyHolder {
15
    /**
16
     * Create shuffle secret manager
17
     */
18
    public ShuffleSecretManager();
19
    
20
    /**
21
     * Register application with string shuffle secret
22
     * @param appId - Application identifier
23
     * @param shuffleSecret - Shuffle secret as string
24
     */
25
    public void registerApp(String appId, String shuffleSecret);
26
    
27
    /**
28
     * Register application with ByteBuffer shuffle secret
29
     * @param appId - Application identifier
30
     * @param shuffleSecret - Shuffle secret as ByteBuffer
31
     */
32
    public void registerApp(String appId, ByteBuffer shuffleSecret);
33
    
34
    /**
35
     * Unregister application and remove its secrets
36
     * @param appId - Application identifier to remove
37
     */
38
    public void unregisterApp(String appId);
39
    
40
    /**
41
     * Get SASL username for application
42
     * @param appId - Application identifier
43
     * @return SASL username for the application
44
     */
45
    public String getSaslUser(String appId);
46
    
47
    /**
48
     * Get secret key for application authentication
49
     * @param appId - Application identifier
50
     * @return Secret key for the application
51
     */
52
    public String getSecretKey(String appId);
53
}
54
```
55

56
**Usage Example:**
57

58
```java
59
import org.apache.spark.network.sasl.ShuffleSecretManager;
60

61
// Create secret manager
62
ShuffleSecretManager secretManager = new ShuffleSecretManager();
63

64
// Register application with secret
65
String appId = "spark-app-123";
66
String shuffleSecret = "my-secure-shuffle-secret-key";
67
secretManager.registerApp(appId, shuffleSecret);
68

69
// Get SASL credentials for authentication
70
String saslUser = secretManager.getSaslUser(appId);
71
String secretKey = secretManager.getSecretKey(appId);
72

73
// Use in external shuffle client
74
ExternalShuffleClient client = new ExternalShuffleClient(
75
    conf, 
76
    secretManager,  // Pass as SecretKeyHolder
77
    true,          // Enable SASL
78
    false          // Disable SASL encryption
79
);
80

81
// Clean up when application finishes
82
secretManager.unregisterApp(appId);
83
```
84

85
## SASL Integration
86

87
### Enabling SASL Authentication
88

89
SASL authentication can be enabled when creating external shuffle clients to provide secure communication channels.
90

91
```java
92
// Configuration for SASL-enabled client
93
TransportConf conf = new TransportConf("shuffle", configProvider);
94
ShuffleSecretManager secretManager = new ShuffleSecretManager();
95

96
// Register application secret
97
secretManager.registerApp("my-app", "secret-key-12345");
98

99
// Create client with SASL enabled
100
ExternalShuffleClient client = new ExternalShuffleClient(
101
    conf,
102
    secretManager,  // Provides secret keys
103
    true,          // SASL authentication enabled
104
    true           // SASL encryption enabled (optional)
105
);
106
```
107

108
### Secret Key Management
109

110
The secret manager maintains a mapping between application IDs and their corresponding authentication secrets.
111

112
```java
113
// Register multiple applications
114
ShuffleSecretManager manager = new ShuffleSecretManager();
115

116
manager.registerApp("app-production", "prod-secret-xyz789");
117
manager.registerApp("app-development", "dev-secret-abc123");
118
manager.registerApp("app-testing", "test-secret-def456");
119

120
// Retrieve secrets for authentication
121
String prodSecret = manager.getSecretKey("app-production");
122
String devSecret = manager.getSecretKey("app-development");
123

124
// Remove application when no longer needed
125
manager.unregisterApp("app-testing");
126
```
127

128
### Security Configuration
129

130
SASL can be configured with different security levels depending on requirements:
131

132
```java
133
/**
134
 * Security configuration options for external shuffle clients
135
 */
136
public class SecurityConfig {
137
    // Basic SASL authentication without encryption
138
    public static ExternalShuffleClient createBasicSecureClient(TransportConf conf, 
139
                                                               SecretKeyHolder secretHolder) {
140
        return new ExternalShuffleClient(conf, secretHolder, true, false);
141
    }
142
    
143
    // Full SASL authentication with encryption
144
    public static ExternalShuffleClient createFullySecureClient(TransportConf conf,
145
                                                               SecretKeyHolder secretHolder) {
146
        return new ExternalShuffleClient(conf, secretHolder, true, true);
147
    }
148
    
149
    // No security (for development/testing)
150
    public static ExternalShuffleClient createInsecureClient(TransportConf conf) {
151
        return new ExternalShuffleClient(conf, null, false, false);
152
    }
153
}
154
```
155

156
## Error Handling
157

158
### Authentication Failures
159

160
When SASL authentication fails, appropriate exceptions are thrown that can be handled by client applications:
161

162
```java
163
try {
164
    ExternalShuffleClient client = new ExternalShuffleClient(conf, secretManager, true, false);
165
    client.init("my-app-id");
166
    
167
    // Attempt to connect and authenticate
168
    client.fetchBlocks(host, port, execId, blockIds, listener);
169
    
170
} catch (SecurityException e) {
171
    System.err.println("SASL authentication failed: " + e.getMessage());
172
    // Handle authentication failure - check secrets, connectivity, etc.
173
    
174
} catch (IOException e) {
175
    System.err.println("Network error during secure connection: " + e.getMessage());
176
    // Handle network connectivity issues
177
}
178
```
179

180
### Secret Management Best Practices
181

182
```java
183
/**
184
 * Example of secure secret management practices
185
 */
186
public class SecureShuffleSetup {
187
    public static void setupSecureShuffleClient(String appId, String secretSource) {
188
        ShuffleSecretManager secretManager = new ShuffleSecretManager();
189
        
190
        try {
191
            // Load secret from secure source (environment, config file, etc.)
192
            String shuffleSecret = loadSecretFromSecureSource(secretSource);
193
            
194
            // Register with secret manager
195
            secretManager.registerApp(appId, shuffleSecret);
196
            
197
            // Create secure client
198
            ExternalShuffleClient client = new ExternalShuffleClient(
199
                conf, secretManager, true, true
200
            );
201
            
202
            // Use client for secure operations...
203
            
204
        } catch (Exception e) {
205
            // Always clean up secrets on error
206
            secretManager.unregisterApp(appId);
207
            throw new RuntimeException("Failed to setup secure shuffle client", e);
208
        }
209
    }
210
    
211
    private static String loadSecretFromSecureSource(String source) {
212
        // Implementation to load secret from secure storage
213
        // Could be environment variable, encrypted config file, key management service, etc.
214
        return System.getenv("SPARK_SHUFFLE_SECRET_" + source);
215
    }
216
}
217
```
218

219
## Types
220

221
### Secret Key Holder Interface
222

223
The `SecretKeyHolder` interface is implemented by `ShuffleSecretManager` to provide secret key lookup functionality:
224

225
```java { .api }
226
/**
227
 * Interface for providing secret keys during SASL authentication
228
 */
229
public interface SecretKeyHolder {
230
    /**
231
     * Get SASL username for the specified application
232
     * @param appId - Application identifier
233
     * @return SASL username
234
     */
235
    String getSaslUser(String appId);
236
    
237
    /**
238
     * Get secret key for the specified application
239
     * @param appId - Application identifier  
240
     * @return Secret key for authentication
241
     */
242
    String getSecretKey(String appId);
243
}
244
```