0
# CDAP Runtime SPI
1
2
CDAP Runtime Service Provider Interface (SPI) library that defines extensible interfaces for runtime provisioning, job management, and infrastructure integration within the CDAP data application platform. This library enables pluggable implementations of cluster provisioning, runtime job lifecycle management, SSH operations, and profile management through standardized interfaces.
3
4
## Package Information
5
6
- **Package Name**: cdap-runtime-spi
7
- **Package Type**: maven
8
- **Language**: Java
9
- **Group ID**: io.cdap.cdap
10
- **Artifact ID**: cdap-runtime-spi
11
- **Installation**: Add to Maven `pom.xml`:
12
13
```xml
14
<dependency>
15
<groupId>io.cdap.cdap</groupId>
16
<artifactId>cdap-runtime-spi</artifactId>
17
<version>6.11.0</version>
18
</dependency>
19
```
20
21
## Core Imports
22
23
```java
24
import io.cdap.cdap.runtime.spi.provisioner.Provisioner;
25
import io.cdap.cdap.runtime.spi.provisioner.ProvisionerContext;
26
import io.cdap.cdap.runtime.spi.runtimejob.RuntimeJobManager;
27
import io.cdap.cdap.runtime.spi.ssh.SSHContext;
28
import io.cdap.cdap.runtime.spi.ssh.SSHKeyPair;
29
import io.cdap.cdap.runtime.spi.ProgramRunInfo;
30
import io.cdap.cdap.runtime.spi.profile.ProfileStatus;
31
import java.security.KeyException;
32
import java.io.IOException;
33
```
34
35
## Basic Usage
36
37
```java
38
// Create program run information
39
ProgramRunInfo runInfo = new ProgramRunInfo.Builder()
40
.setNamespace("default")
41
.setApplication("myapp")
42
.setVersion("1.0.0")
43
.setProgramType("workflow")
44
.setProgram("myworkflow")
45
.setRun("run-123")
46
.build();
47
48
// Implement a custom provisioner
49
public class MyProvisioner implements Provisioner {
50
@Override
51
public ProvisionerSpecification getSpec() {
52
return new ProvisionerSpecification("my-provisioner",
53
"My Custom Provisioner",
54
"Custom provisioner for cloud resources");
55
}
56
57
@Override
58
public Cluster createCluster(ProvisionerContext context) throws Exception {
59
// Implementation for creating clusters
60
return new Cluster(/* cluster details */);
61
}
62
63
// Additional provisioner methods...
64
}
65
```
66
67
## Architecture
68
69
CDAP Runtime SPI is built around several key architectural patterns:
70
71
- **Service Provider Interface (SPI) Pattern**: Core interfaces (`Provisioner`, `RuntimeJobManager`) define contracts for pluggable implementations
72
- **Context Pattern**: Rich context objects (`ProvisionerContext`, `SSHContext`) provide runtime information and capabilities to implementations
73
- **Builder Pattern**: Complex data objects like `ProgramRunInfo` use fluent builders for construction
74
- **Resource Management**: Interfaces extending `Closeable` ensure proper cleanup of resources like SSH sessions and job managers
75
- **Status Tracking**: Enumerated status types (`ClusterStatus`, `RuntimeJobStatus`) provide clear state management
76
- **Asynchronous Operations**: Support for non-blocking operations with polling strategies and callbacks
77
78
## Capabilities
79
80
### Cluster Provisioning
81
82
Core SPI for creating, managing, and destroying compute clusters across different cloud platforms and environments. Provides standardized interfaces for cluster lifecycle management with pluggable provisioner implementations.
83
84
```java { .api }
85
interface Provisioner {
86
ProvisionerSpecification getSpec();
87
void initialize(ProvisionerSystemContext context) throws Exception;
88
void validateProperties(Map<String, String> properties);
89
Cluster createCluster(ProvisionerContext context) throws Exception;
90
ClusterStatus getClusterStatus(ProvisionerContext context, Cluster cluster) throws Exception;
91
Cluster getClusterDetail(ProvisionerContext context, Cluster cluster) throws Exception;
92
void initializeCluster(ProvisionerContext context, Cluster cluster) throws Exception;
93
void deleteClusterWithStatus(ProvisionerContext context, Cluster cluster,
94
ClusterStatus clusterStatus) throws Exception;
95
PollingStrategy getPollingStrategy(ProvisionerContext context, Cluster cluster);
96
Capabilities getCapabilities();
97
Optional<RuntimeJobManager> getRuntimeJobManager(ProvisionerContext context);
98
String getTotalProcessingCpusLabel(Map<String, String> properties);
99
}
100
```
101
102
[Cluster Provisioning](./provisioner.md)
103
104
### Runtime Job Management
105
106
SPI for launching, monitoring, and managing runtime jobs within provisioned clusters. Handles job lifecycle from submission through completion with status tracking and resource management.
107
108
```java { .api }
109
interface RuntimeJobManager extends Closeable {
110
void launch(RuntimeJobInfo jobInfo) throws Exception;
111
Optional<RuntimeJobDetail> getDetail(ProgramRunInfo programRunInfo) throws Exception;
112
void stop(ProgramRunInfo programRunInfo) throws Exception;
113
void kill(RuntimeJobDetail runtimeJobDetail) throws Exception;
114
}
115
116
interface RuntimeJob {
117
void run(RuntimeJobEnvironment environment) throws Exception;
118
void requestStop();
119
}
120
```
121
122
[Runtime Job Management](./runtime-job.md)
123
124
### SSH Operations
125
126
Comprehensive SSH utilities for secure remote operations including session management, command execution, file transfer, and port forwarding. Essential for cluster initialization and remote job management.
127
128
```java { .api }
129
interface SSHContext {
130
SSHKeyPair generate(String user) throws Exception;
131
SSHKeyPair generate(String user, int keySize) throws Exception;
132
void setSSHKeyPair(SSHKeyPair keyPair);
133
Optional<SSHKeyPair> getSSHKeyPair();
134
SSHSession createSSHSession(String host) throws Exception;
135
SSHSession createSSHSession(SSHKeyPair keyPair, String host) throws Exception;
136
}
137
138
interface SSHSession extends Closeable {
139
boolean isAlive();
140
InetSocketAddress getAddress();
141
String getUsername();
142
SSHProcess execute(String... commands) throws Exception;
143
SSHProcess executeAndWait(String... commands) throws Exception;
144
void copy(Path localFile, String remotePath) throws Exception;
145
PortForwarding createLocalPortForward(String remoteHost, int remotePort,
146
int localPort,
147
PortForwarding.DataConsumer dataConsumer)
148
throws Exception;
149
RemotePortForwarding createRemotePortForward(int remotePort, int localPort)
150
throws Exception;
151
}
152
```
153
154
[SSH Operations](./ssh.md)
155
156
### Core Runtime Types
157
158
Foundation classes and enums that support all SPI operations, including program run information, version handling, compatibility settings, and monitoring configurations.
159
160
```java { .api }
161
class ProgramRunInfo {
162
String getNamespace();
163
String getApplication();
164
String getVersion();
165
String getProgramType();
166
String getProgram();
167
String getRun();
168
169
static Builder builder();
170
171
interface Builder {
172
Builder setNamespace(String namespace);
173
Builder setApplication(String application);
174
Builder setVersion(String version);
175
Builder setProgramType(String programType);
176
Builder setProgram(String program);
177
Builder setRun(String run);
178
ProgramRunInfo build();
179
}
180
}
181
182
enum RuntimeMonitorType {
183
SSH, URL
184
}
185
186
enum SparkCompat {
187
SPARK2_2_11("spark2_2.11"),
188
SPARK3_2_12("spark3_2.12");
189
190
String getCompat();
191
}
192
```
193
194
[Core Runtime Types](./core-types.md)
195
196
### Profile Management
197
198
Simple profile status management for runtime configuration profiles, controlling whether profiles can be assigned to programs or deleted from the system.
199
200
```java { .api }
201
enum ProfileStatus {
202
ENABLED, // Profile can be assigned to programs, cannot be deleted
203
DISABLED // Profile cannot be assigned, can be deleted
204
}
205
```
206
207
[Profile Management](./profile.md)
208
209
## Types
210
211
### Core Data Types
212
213
```java { .api }
214
class Cluster {
215
String getName();
216
ClusterStatus getStatus();
217
List<Node> getNodes();
218
Map<String, String> getProperties();
219
}
220
221
class Node {
222
String getId();
223
Node.Type getType();
224
String getIpAddress();
225
long getCreateTime();
226
Map<String, String> getProperties();
227
228
enum Type {
229
MASTER, WORKER, UNKNOWN
230
}
231
}
232
233
interface VersionInfo extends Comparable<Object> {
234
int getMajor();
235
int getMinor();
236
int getFix();
237
boolean isSnapshot();
238
long getBuildTime();
239
}
240
```
241
242
### Status Enumerations
243
244
```java { .api }
245
enum ClusterStatus {
246
CREATING, RUNNING, FAILED, DELETING, NOT_EXISTS, ORPHANED
247
}
248
249
enum RuntimeJobStatus {
250
STARTING(false), RUNNING(false), STOPPING(false),
251
STOPPED(true), COMPLETED(true), FAILED(true), UNKNOWN(false);
252
253
boolean isTerminated();
254
}
255
256
enum LaunchMode {
257
CLIENT, // Launch in-process
258
CLUSTER // Launch in separate container
259
}
260
```
261
262
### Exception Types
263
264
```java { .api }
265
class RetryableProvisionException extends Exception implements ErrorTagProvider {
266
Set<String> getErrorTags();
267
}
268
269
class ProgramRunFailureException extends RuntimeException {
270
// Exception for program runs that completed but failed
271
}
272
```