0
# Resource Management
1
2
Resource allocation and management for Spark applications including task resources, executor resources, and resource profiles for optimized cluster utilization and performance tuning.
3
4
## Capabilities
5
6
### Resource Profiles
7
8
Define resource requirements for different stages of Spark applications.
9
10
```python { .api }
11
class ResourceProfile:
12
"""Resource profile defining executor and task resource requirements."""
13
14
def __init__(self, executorResources=None, taskResources=None):
15
"""
16
Create ResourceProfile.
17
18
Parameters:
19
- executorResources (dict): Executor resource requirements
20
- taskResources (dict): Task resource requirements
21
"""
22
23
@property
24
def id(self):
25
"""Unique ID of this resource profile."""
26
27
@property
28
def executorResources(self):
29
"""Executor resource requirements."""
30
31
@property
32
def taskResources(self):
33
"""Task resource requirements."""
34
35
class ResourceProfileBuilder:
36
"""Builder for ResourceProfile."""
37
38
def __init__(self):
39
"""Initialize ResourceProfileBuilder."""
40
41
def require(self, resourceRequests):
42
"""
43
Specify resource requirements.
44
45
Parameters:
46
- resourceRequests: Resource requirements
47
48
Returns:
49
ResourceProfileBuilder
50
"""
51
52
def build(self):
53
"""
54
Build ResourceProfile.
55
56
Returns:
57
ResourceProfile
58
"""
59
```
60
61
### Executor Resource Management
62
63
Define resource requirements for Spark executors.
64
65
```python { .api }
66
class ExecutorResourceRequests:
67
"""Executor resource requirements."""
68
69
def __init__(self):
70
"""Initialize ExecutorResourceRequests."""
71
72
def cores(self, amount):
73
"""
74
Specify number of CPU cores.
75
76
Parameters:
77
- amount (int): Number of cores
78
79
Returns:
80
ExecutorResourceRequests
81
"""
82
83
def memory(self, amount):
84
"""
85
Specify memory amount.
86
87
Parameters:
88
- amount (str): Memory amount (e.g., '2g', '1024m')
89
90
Returns:
91
ExecutorResourceRequests
92
"""
93
94
def memoryFraction(self, fraction):
95
"""
96
Specify memory fraction.
97
98
Parameters:
99
- fraction (float): Memory fraction (0.0 to 1.0)
100
101
Returns:
102
ExecutorResourceRequests
103
"""
104
105
def offHeapMemory(self, amount):
106
"""
107
Specify off-heap memory amount.
108
109
Parameters:
110
- amount (str): Off-heap memory amount
111
112
Returns:
113
ExecutorResourceRequests
114
"""
115
116
def memoryOverhead(self, amount):
117
"""
118
Specify memory overhead amount.
119
120
Parameters:
121
- amount (str): Memory overhead amount
122
123
Returns:
124
ExecutorResourceRequests
125
"""
126
127
def pysparkMemory(self, amount):
128
"""
129
Specify PySpark memory amount.
130
131
Parameters:
132
- amount (str): PySpark memory amount
133
134
Returns:
135
ExecutorResourceRequests
136
"""
137
138
def resource(self, resourceName, amount, discoveryScript=None, vendor=None):
139
"""
140
Specify custom resource requirement.
141
142
Parameters:
143
- resourceName (str): Name of the resource
144
- amount (int): Amount of resource
145
- discoveryScript (str): Script to discover resource
146
- vendor (str): Vendor of the resource
147
148
Returns:
149
ExecutorResourceRequests
150
"""
151
152
def build(self):
153
"""
154
Build executor resource requirements.
155
156
Returns:
157
dict: Executor resource requirements
158
"""
159
160
class ExecutorResourceRequest:
161
"""Single executor resource requirement."""
162
163
def __init__(self, resourceName, amount, discoveryScript=None, vendor=None):
164
"""
165
Create ExecutorResourceRequest.
166
167
Parameters:
168
- resourceName (str): Name of the resource
169
- amount (int): Amount of resource
170
- discoveryScript (str): Script to discover resource
171
- vendor (str): Vendor of the resource
172
"""
173
174
@property
175
def resourceName(self):
176
"""Name of the resource."""
177
178
@property
179
def amount(self):
180
"""Amount of resource."""
181
182
@property
183
def discoveryScript(self):
184
"""Discovery script for the resource."""
185
186
@property
187
def vendor(self):
188
"""Vendor of the resource."""
189
```
190
191
### Task Resource Management
192
193
Define resource requirements for individual tasks.
194
195
```python { .api }
196
class TaskResourceRequests:
197
"""Task resource requirements."""
198
199
def __init__(self):
200
"""Initialize TaskResourceRequests."""
201
202
def cpus(self, amount):
203
"""
204
Specify number of CPU cores for task.
205
206
Parameters:
207
- amount (float): Number of CPU cores
208
209
Returns:
210
TaskResourceRequests
211
"""
212
213
def resource(self, resourceName, amount):
214
"""
215
Specify custom resource requirement for task.
216
217
Parameters:
218
- resourceName (str): Name of the resource
219
- amount (float): Amount of resource
220
221
Returns:
222
TaskResourceRequests
223
"""
224
225
def build(self):
226
"""
227
Build task resource requirements.
228
229
Returns:
230
dict: Task resource requirements
231
"""
232
233
class TaskResourceRequest:
234
"""Single task resource requirement."""
235
236
def __init__(self, resourceName, amount):
237
"""
238
Create TaskResourceRequest.
239
240
Parameters:
241
- resourceName (str): Name of the resource
242
- amount (float): Amount of resource
243
"""
244
245
@property
246
def resourceName(self):
247
"""Name of the resource."""
248
249
@property
250
def amount(self):
251
"""Amount of resource."""
252
```
253
254
### Resource Information
255
256
Information about available resources.
257
258
```python { .api }
259
class ResourceInformation:
260
"""Information about a resource."""
261
262
def __init__(self, name, addresses):
263
"""
264
Create ResourceInformation.
265
266
Parameters:
267
- name (str): Resource name
268
- addresses (list): Resource addresses
269
"""
270
271
@property
272
def name(self):
273
"""Name of the resource."""
274
275
@property
276
def addresses(self):
277
"""Addresses of the resource."""
278
```
279
280
### Usage Examples
281
282
```python
283
# Create executor resource requirements
284
executor_req = ExecutorResourceRequests() \
285
.cores(4) \
286
.memory("8g") \
287
.resource("gpu", 1, "/path/to/gpu_discovery.sh")
288
289
# Create task resource requirements
290
task_req = TaskResourceRequests() \
291
.cpus(1.0) \
292
.resource("gpu", 0.5)
293
294
# Build resource profile
295
profile = ResourceProfile(
296
executorResources=executor_req.build(),
297
taskResources=task_req.build()
298
)
299
300
# Use with RDD
301
rdd = sc.parallelize(data)
302
rdd_with_profile = rdd.withResources(profile)
303
result = rdd_with_profile.map(my_gpu_function).collect()
304
```
305
306
## Types
307
308
```python { .api }
309
class ResourceProfile:
310
"""Resource profile for Spark applications."""
311
pass
312
313
class ExecutorResourceRequest:
314
"""Executor resource requirement."""
315
pass
316
317
class TaskResourceRequest:
318
"""Task resource requirement."""
319
pass
320
321
class ResourceInformation:
322
"""Information about available resources."""
323
pass
324
```