0
# Transport Context
1

2
The `TransportContext` class is the main entry point for Apache Spark's networking layer. It manages Netty pipeline setup, creates client factories and servers, and handles network configuration. This class provides the foundation for all network communication in Spark.
3

4
## Capabilities
5

6
### TransportContext Creation
7

8
Creates a transport context with the specified configuration and RPC handler.
9

10
```java { .api }
11
/**
12
 * Create a TransportContext with basic configuration
13
 * @param conf - Transport configuration containing network settings
14
 * @param rpcHandler - Handler for processing RPC messages
15
 */
16
public TransportContext(TransportConf conf, RpcHandler rpcHandler);
17

18
/**
19
 * Create a TransportContext with idle connection management
20
 * @param conf - Transport configuration containing network settings
21
 * @param rpcHandler - Handler for processing RPC messages
22
 * @param closeIdleConnections - Whether to close idle connections automatically
23
 */
24
public TransportContext(TransportConf conf, RpcHandler rpcHandler, boolean closeIdleConnections);
25

26
/**
27
 * Create a TransportContext with full configuration options
28
 * @param conf - Transport configuration containing network settings
29
 * @param rpcHandler - Handler for processing RPC messages
30
 * @param closeIdleConnections - Whether to close idle connections automatically
31
 * @param isClientOnly - Whether this context is for client operations only
32
 */
33
public TransportContext(TransportConf conf, RpcHandler rpcHandler, boolean closeIdleConnections, boolean isClientOnly);
34
```
35

36
### Client Factory Creation
37

38
Creates factories for generating transport clients with optional bootstrap customization.
39

40
```java { .api }
41
/**
42
 * Create a client factory with custom bootstrap configurations
43
 * @param bootstraps - List of bootstrap configurations for client initialization
44
 * @return TransportClientFactory for creating transport clients
45
 */
46
public TransportClientFactory createClientFactory(List<TransportClientBootstrap> bootstraps);
47

48
/**
49
 * Create a client factory with default configuration
50
 * @return TransportClientFactory for creating transport clients
51
 */
52
public TransportClientFactory createClientFactory();
53
```
54

55
### Server Creation
56

57
Creates transport servers for handling incoming client connections with flexible configuration options.
58

59
```java { .api }
60
/**
61
 * Create a server bound to a specific port with bootstrap configurations
62
 * @param port - Port number to bind the server to
63
 * @param bootstraps - List of bootstrap configurations for server initialization
64
 * @return TransportServer instance bound to the specified port
65
 */
66
public TransportServer createServer(int port, List<TransportServerBootstrap> bootstraps);
67

68
/**
69
 * Create a server bound to a specific host and port with bootstrap configurations
70
 * @param host - Host address to bind the server to
71
 * @param port - Port number to bind the server to
72
 * @param bootstraps - List of bootstrap configurations for server initialization
73
 * @return TransportServer instance bound to the specified host and port
74
 */
75
public TransportServer createServer(String host, int port, List<TransportServerBootstrap> bootstraps);
76

77
/**
78
 * Create a server with bootstrap configurations, using system-assigned port
79
 * @param bootstraps - List of bootstrap configurations for server initialization
80
 * @return TransportServer instance with system-assigned port
81
 */
82
public TransportServer createServer(List<TransportServerBootstrap> bootstraps);
83

84
/**
85
 * Create a server with default configuration and system-assigned port
86
 * @return TransportServer instance with default configuration
87
 */
88
public TransportServer createServer();
89
```
90

91
### Pipeline Management
92

93
Manages Netty channel pipeline initialization for both client and server channels.
94

95
```java { .api }
96
/**
97
 * Initialize the Netty pipeline for a socket channel with default RPC handler
98
 * @param channel - SocketChannel to initialize pipeline for
99
 * @return TransportChannelHandler for managing the channel
100
 */
101
public TransportChannelHandler initializePipeline(SocketChannel channel);
102

103
/**
104
 * Initialize the Netty pipeline for a socket channel with custom RPC handler
105
 * @param channel - SocketChannel to initialize pipeline for
106
 * @param channelRpcHandler - Custom RPC handler for this specific channel
107
 * @return TransportChannelHandler for managing the channel
108
 */
109
public TransportChannelHandler initializePipeline(SocketChannel channel, RpcHandler channelRpcHandler);
110
```
111

112
### Configuration and Monitoring
113

114
Provides access to configuration settings and connection monitoring capabilities.
115

116
```java { .api }
117
/**
118
 * Get the transport configuration used by this context
119
 * @return TransportConf instance containing all configuration settings
120
 */
121
public TransportConf getConf();
122

123
/**
124
 * Get metrics counter for registered connections
125
 * @return Counter tracking the number of registered connections
126
 */
127
public Counter getRegisteredConnections();
128
```
129

130
### Resource Management
131

132
Properly closes and cleans up all resources associated with the transport context.
133

134
```java { .api }
135
/**
136
 * Close the transport context and release all associated resources
137
 * This includes closing all client factories, servers, and network resources
138
 */
139
public void close();
140
```
141

142
## Usage Examples
143

144
### Basic Transport Context Setup
145

146
```java
147
import org.apache.spark.network.TransportContext;
148
import org.apache.spark.network.server.NoOpRpcHandler;
149
import org.apache.spark.network.util.TransportConf;
150
import org.apache.spark.network.util.MapConfigProvider;
151

152
// Create configuration
153
Map<String, String> config = new HashMap<>();
154
config.put("spark.network.timeout", "120s");
155
config.put("spark.network.io.mode", "NIO");
156
ConfigProvider provider = new MapConfigProvider(config);
157
TransportConf conf = new TransportConf("spark", provider);
158

159
// Create transport context
160
TransportContext context = new TransportContext(conf, new NoOpRpcHandler());
161

162
// Use context to create clients and servers
163
TransportServer server = context.createServer();
164
TransportClientFactory clientFactory = context.createClientFactory();
165

166
// Cleanup
167
clientFactory.close();
168
server.close();
169
context.close();
170
```
171

172
### Advanced Context with Custom Bootstraps
173

174
```java
175
import org.apache.spark.network.TransportContext;
176
import org.apache.spark.network.client.TransportClientBootstrap;
177
import org.apache.spark.network.server.TransportServerBootstrap;
178
import org.apache.spark.network.sasl.SaslClientBootstrap;
179
import org.apache.spark.network.sasl.SaslServerBootstrap;
180

181
// Create context with SASL authentication
182
List<TransportClientBootstrap> clientBootstraps = Arrays.asList(
183
    new SaslClientBootstrap(conf, "myapp", secretKeyHolder)
184
);
185
List<TransportServerBootstrap> serverBootstraps = Arrays.asList(
186
    new SaslServerBootstrap(conf, secretKeyHolder)
187
);
188

189
TransportContext context = new TransportContext(conf, rpcHandler, true, false);
190

191
// Create authenticated client factory and server
192
TransportClientFactory clientFactory = context.createClientFactory(clientBootstraps);
193
TransportServer server = context.createServer(9999, serverBootstraps);
194

195
System.out.println("Server listening on port: " + server.getPort());
196
```
197

198
### Context for Client-Only Operations
199

200
```java
201
// Create a lightweight context for client-only operations
202
TransportContext clientContext = new TransportContext(conf, rpcHandler, true, true);
203
TransportClientFactory factory = clientContext.createClientFactory();
204

205
// Connect to existing servers
206
TransportClient client1 = factory.createClient("server1.example.com", 9999);
207
TransportClient client2 = factory.createClient("server2.example.com", 9999);
208

209
// Use clients for communication
210
client1.sendRpc(message, callback);
211
client2.fetchChunk(streamId, chunkIndex, chunkCallback);
212

213
// Cleanup
214
client1.close();
215
client2.close();
216
factory.close();
217
clientContext.close();
218
```
219

220
## Types
221

222
### Related Classes
223

224
```java { .api }
225
public class TransportClientFactory implements Closeable {
226
    public TransportClient createClient(String remoteHost, int remotePort);
227
    public TransportClient createClient(String remoteHost, int remotePort, int clientId);
228
    public void close();
229
}
230

231
public class TransportServer implements Closeable {
232
    public int getPort();
233
    public MetricSet getAllMetrics();
234
    public Counter getRegisteredConnections();
235
    public void close();
236
}
237

238
public class TransportChannelHandler extends SimpleChannelInboundHandler<Message> {
239
    public TransportClient getClient();
240
}
241
```