0
# CDAP Runtime SPI
1

2
CDAP Runtime Service Provider Interface (SPI) library that defines extensible interfaces for runtime provisioning, job management, and infrastructure integration within the CDAP data application platform. This library enables pluggable implementations of cluster provisioning, runtime job lifecycle management, SSH operations, and profile management through standardized interfaces.
3

4
## Package Information
5

6
- **Package Name**: cdap-runtime-spi
7
- **Package Type**: maven
8
- **Language**: Java
9
- **Group ID**: io.cdap.cdap
10
- **Artifact ID**: cdap-runtime-spi
11
- **Installation**: Add to Maven `pom.xml`:
12

13
```xml
14
<dependency>
15
  <groupId>io.cdap.cdap</groupId>
16
  <artifactId>cdap-runtime-spi</artifactId>
17
  <version>6.11.0</version>
18
</dependency>
19
```
20

21
## Core Imports
22

23
```java
24
import io.cdap.cdap.runtime.spi.provisioner.Provisioner;
25
import io.cdap.cdap.runtime.spi.provisioner.ProvisionerContext;
26
import io.cdap.cdap.runtime.spi.runtimejob.RuntimeJobManager;
27
import io.cdap.cdap.runtime.spi.ssh.SSHContext;
28
import io.cdap.cdap.runtime.spi.ssh.SSHKeyPair;
29
import io.cdap.cdap.runtime.spi.ProgramRunInfo;
30
import io.cdap.cdap.runtime.spi.profile.ProfileStatus;
31
import java.security.KeyException;
32
import java.io.IOException;
33
```
34

35
## Basic Usage
36

37
```java
38
// Create program run information
39
ProgramRunInfo runInfo = new ProgramRunInfo.Builder()
40
    .setNamespace("default")
41
    .setApplication("myapp")
42
    .setVersion("1.0.0")
43
    .setProgramType("workflow")
44
    .setProgram("myworkflow")
45
    .setRun("run-123")
46
    .build();
47

48
// Implement a custom provisioner
49
public class MyProvisioner implements Provisioner {
50
    @Override
51
    public ProvisionerSpecification getSpec() {
52
        return new ProvisionerSpecification("my-provisioner", 
53
                                           "My Custom Provisioner",
54
                                           "Custom provisioner for cloud resources");
55
    }
56
    
57
    @Override
58
    public Cluster createCluster(ProvisionerContext context) throws Exception {
59
        // Implementation for creating clusters
60
        return new Cluster(/* cluster details */);
61
    }
62
    
63
    // Additional provisioner methods...
64
}
65
```
66

67
## Architecture
68

69
CDAP Runtime SPI is built around several key architectural patterns:
70

71
- **Service Provider Interface (SPI) Pattern**: Core interfaces (`Provisioner`, `RuntimeJobManager`) define contracts for pluggable implementations
72
- **Context Pattern**: Rich context objects (`ProvisionerContext`, `SSHContext`) provide runtime information and capabilities to implementations
73
- **Builder Pattern**: Complex data objects like `ProgramRunInfo` use fluent builders for construction
74
- **Resource Management**: Interfaces extending `Closeable` ensure proper cleanup of resources like SSH sessions and job managers
75
- **Status Tracking**: Enumerated status types (`ClusterStatus`, `RuntimeJobStatus`) provide clear state management
76
- **Asynchronous Operations**: Support for non-blocking operations with polling strategies and callbacks
77

78
## Capabilities
79

80
### Cluster Provisioning
81

82
Core SPI for creating, managing, and destroying compute clusters across different cloud platforms and environments. Provides standardized interfaces for cluster lifecycle management with pluggable provisioner implementations.
83

84
```java { .api }
85
interface Provisioner {
86
    ProvisionerSpecification getSpec();
87
    void initialize(ProvisionerSystemContext context) throws Exception;
88
    void validateProperties(Map<String, String> properties);
89
    Cluster createCluster(ProvisionerContext context) throws Exception;
90
    ClusterStatus getClusterStatus(ProvisionerContext context, Cluster cluster) throws Exception;
91
    Cluster getClusterDetail(ProvisionerContext context, Cluster cluster) throws Exception;
92
    void initializeCluster(ProvisionerContext context, Cluster cluster) throws Exception;
93
    void deleteClusterWithStatus(ProvisionerContext context, Cluster cluster, 
94
                                ClusterStatus clusterStatus) throws Exception;
95
    PollingStrategy getPollingStrategy(ProvisionerContext context, Cluster cluster);
96
    Capabilities getCapabilities();
97
    Optional<RuntimeJobManager> getRuntimeJobManager(ProvisionerContext context);
98
    String getTotalProcessingCpusLabel(Map<String, String> properties);
99
}
100
```
101

102
[Cluster Provisioning](./provisioner.md)
103

104
### Runtime Job Management
105

106
SPI for launching, monitoring, and managing runtime jobs within provisioned clusters. Handles job lifecycle from submission through completion with status tracking and resource management.
107

108
```java { .api }
109
interface RuntimeJobManager extends Closeable {
110
    void launch(RuntimeJobInfo jobInfo) throws Exception;
111
    Optional<RuntimeJobDetail> getDetail(ProgramRunInfo programRunInfo) throws Exception;
112
    void stop(ProgramRunInfo programRunInfo) throws Exception;
113
    void kill(RuntimeJobDetail runtimeJobDetail) throws Exception;
114
}
115

116
interface RuntimeJob {
117
    void run(RuntimeJobEnvironment environment) throws Exception;
118
    void requestStop();
119
}
120
```
121

122
[Runtime Job Management](./runtime-job.md)
123

124
### SSH Operations
125

126
Comprehensive SSH utilities for secure remote operations including session management, command execution, file transfer, and port forwarding. Essential for cluster initialization and remote job management.
127

128
```java { .api }
129
interface SSHContext {
130
    SSHKeyPair generate(String user) throws Exception;
131
    SSHKeyPair generate(String user, int keySize) throws Exception;
132
    void setSSHKeyPair(SSHKeyPair keyPair);
133
    Optional<SSHKeyPair> getSSHKeyPair();
134
    SSHSession createSSHSession(String host) throws Exception;
135
    SSHSession createSSHSession(SSHKeyPair keyPair, String host) throws Exception;
136
}
137

138
interface SSHSession extends Closeable {
139
    boolean isAlive();
140
    InetSocketAddress getAddress();
141
    String getUsername();
142
    SSHProcess execute(String... commands) throws Exception;
143
    SSHProcess executeAndWait(String... commands) throws Exception;
144
    void copy(Path localFile, String remotePath) throws Exception;
145
    PortForwarding createLocalPortForward(String remoteHost, int remotePort, 
146
                                         int localPort, 
147
                                         PortForwarding.DataConsumer dataConsumer) 
148
                                         throws Exception;
149
    RemotePortForwarding createRemotePortForward(int remotePort, int localPort) 
150
                                                throws Exception;
151
}
152
```
153

154
[SSH Operations](./ssh.md)
155

156
### Core Runtime Types
157

158
Foundation classes and enums that support all SPI operations, including program run information, version handling, compatibility settings, and monitoring configurations.
159

160
```java { .api }
161
class ProgramRunInfo {
162
    String getNamespace();
163
    String getApplication();
164
    String getVersion();
165
    String getProgramType();
166
    String getProgram();
167
    String getRun();
168
    
169
    static Builder builder();
170
    
171
    interface Builder {
172
        Builder setNamespace(String namespace);
173
        Builder setApplication(String application);
174
        Builder setVersion(String version);
175
        Builder setProgramType(String programType);
176
        Builder setProgram(String program);
177
        Builder setRun(String run);
178
        ProgramRunInfo build();
179
    }
180
}
181

182
enum RuntimeMonitorType {
183
    SSH, URL
184
}
185

186
enum SparkCompat {
187
    SPARK2_2_11("spark2_2.11"),
188
    SPARK3_2_12("spark3_2.12");
189
    
190
    String getCompat();
191
}
192
```
193

194
[Core Runtime Types](./core-types.md)
195

196
### Profile Management
197

198
Simple profile status management for runtime configuration profiles, controlling whether profiles can be assigned to programs or deleted from the system.
199

200
```java { .api }
201
enum ProfileStatus {
202
    ENABLED,    // Profile can be assigned to programs, cannot be deleted
203
    DISABLED    // Profile cannot be assigned, can be deleted
204
}
205
```
206

207
[Profile Management](./profile.md)
208

209
## Types
210

211
### Core Data Types
212

213
```java { .api }
214
class Cluster {
215
    String getName();
216
    ClusterStatus getStatus();
217
    List<Node> getNodes();
218
    Map<String, String> getProperties();
219
}
220

221
class Node {
222
    String getId();
223
    Node.Type getType();
224
    String getIpAddress();
225
    long getCreateTime();
226
    Map<String, String> getProperties();
227
    
228
    enum Type {
229
        MASTER, WORKER, UNKNOWN
230
    }
231
}
232

233
interface VersionInfo extends Comparable<Object> {
234
    int getMajor();
235
    int getMinor();
236
    int getFix();
237
    boolean isSnapshot();
238
    long getBuildTime();
239
}
240
```
241

242
### Status Enumerations
243

244
```java { .api }
245
enum ClusterStatus {
246
    CREATING, RUNNING, FAILED, DELETING, NOT_EXISTS, ORPHANED
247
}
248

249
enum RuntimeJobStatus {
250
    STARTING(false), RUNNING(false), STOPPING(false), 
251
    STOPPED(true), COMPLETED(true), FAILED(true), UNKNOWN(false);
252
    
253
    boolean isTerminated();
254
}
255

256
enum LaunchMode {
257
    CLIENT,  // Launch in-process
258
    CLUSTER  // Launch in separate container
259
}
260
```
261

262
### Exception Types
263

264
```java { .api }
265
class RetryableProvisionException extends Exception implements ErrorTagProvider {
266
    Set<String> getErrorTags();
267
}
268

269
class ProgramRunFailureException extends RuntimeException {
270
    // Exception for program runs that completed but failed
271
}
272
```