0
# Resource Providers
1

2
Parsl resource providers interface with various computing platforms, job schedulers, and cloud services to provision and manage computing resources for parallel execution. Each provider is specialized for specific resource types and management systems.
3

4
## Capabilities
5

6
### Local Provider
7

8
Executes tasks on the local machine using multiple processes. Ideal for development, testing, and small-scale parallel workloads.
9

10
```python { .api }
11
class LocalProvider:
12
    def __init__(self, channel=None, nodes_per_block=1, init_blocks=1,
13
                 min_blocks=0, max_blocks=1, parallelism=1, walltime="00:10:00",
14
                 cmd_timeout=30, launcher=None, move_files=None, worker_init=''):
15
        """
16
        Local execution provider for running tasks on local machine.
17
        
18
        Parameters:
19
        - nodes_per_block: Processes per block (default: 1)
20
        - init_blocks: Initial number of blocks (default: 1)  
21
        - min_blocks: Minimum blocks to maintain (default: 0)
22
        - max_blocks: Maximum blocks allowed (default: 1)
23
        - parallelism: Provider parallelism level (default: 1)
24
        - walltime: Maximum block runtime (default: "00:10:00")
25
        - launcher: Task launcher (default: SimpleLauncher)
26
        - worker_init: Initialization commands for workers
27
        """
28
```
29

30
**Usage Example:**
31

32
```python
33
from parsl.providers import LocalProvider
34
from parsl.executors import HighThroughputExecutor
35

36
local_htex = HighThroughputExecutor(
37
    label='local_parallel',
38
    cores_per_worker=2,
39
    max_workers=4,
40
    provider=LocalProvider(
41
        init_blocks=1,
42
        max_blocks=2,
43
        nodes_per_block=1,
44
        walltime="01:00:00"
45
    )
46
)
47
```
48

49
### SLURM Provider
50

51
Interfaces with SLURM workload manager for job submission and resource management on HPC clusters.
52

53
```python { .api }
54
class SlurmProvider:
55
    def __init__(self, partition=None, account=None, channel=None, nodes_per_block=1,
56
                 cores_per_node=None, mem_per_node=None, init_blocks=1, min_blocks=0,
57
                 max_blocks=1, parallelism=1, walltime="00:10:00", scheduler_options='',
58
                 worker_init='', cmd_timeout=30, launcher=None, move_files=None,
59
                 exclusive=True, qos=None):
60
        """
61
        SLURM execution provider for HPC cluster job submission.
62
        
63
        Parameters:
64
        - partition: SLURM partition name
65
        - account: SLURM account for billing
66
        - nodes_per_block: Nodes per resource block
67
        - cores_per_node: CPU cores per node
68
        - mem_per_node: Memory per node (e.g., '4GB', '1000MB')
69
        - walltime: Job walltime limit (HH:MM:SS format)
70
        - scheduler_options: Additional SLURM directives
71
        - exclusive: Request exclusive node access (default: True)
72
        - qos: Quality of Service specification
73
        - launcher: Job launcher (default: SrunLauncher)
74
        """
75
```
76

77
**Usage Example:**
78

79
```python
80
from parsl.providers import SlurmProvider
81
from parsl.launchers import SrunLauncher
82

83
slurm_provider = SlurmProvider(
84
    partition='compute',
85
    account='my_project',
86
    nodes_per_block=2,
87
    cores_per_node=24,
88
    mem_per_node='100GB',
89
    init_blocks=1,
90
    max_blocks=10,
91
    walltime='04:00:00',
92
    scheduler_options='#SBATCH --constraint=haswell',
93
    launcher=SrunLauncher()
94
)
95
```
96

97
### AWS Provider
98

99
Provisions and manages Amazon EC2 instances for cloud-based parallel computing.
100

101
```python { .api }
102
class AWSProvider:
103
    def __init__(self, image_id, instance_type, region='us-east-1', key_name=None,
104
                 security_groups=None, subnet_id=None, iam_instance_profile_arn=None,
105
                 iam_instance_profile_name=None, state_file=None, spot_max_bid=0,
106
                 nodes_per_block=1, init_blocks=1, min_blocks=0, max_blocks=1,
107
                 parallelism=1, walltime="00:10:00", launcher=None, worker_init=''):
108
        """
109
        AWS EC2 provider for cloud computing resources.
110
        
111
        Parameters:
112
        - image_id: EC2 AMI ID
113
        - instance_type: EC2 instance type (e.g., 't3.medium', 'c5.xlarge')
114
        - region: AWS region (default: 'us-east-1')
115
        - key_name: EC2 key pair name for SSH access
116
        - security_groups: List of security group names
117
        - subnet_id: VPC subnet ID
118
        - spot_max_bid: Maximum bid for spot instances (0 for on-demand)
119
        - state_file: File to store instance state information
120
        """
121
```
122

123
**Usage Example:**
124

125
```python
126
from parsl.providers import AWSProvider
127

128
aws_provider = AWSProvider(
129
    image_id='ami-0abcdef1234567890',
130
    instance_type='c5.2xlarge',
131
    region='us-west-2',
132
    key_name='my-keypair',
133
    security_groups=['parsl-sg'],
134
    spot_max_bid=0.10,  # $0.10/hour max for spot instances
135
    init_blocks=1,
136
    max_blocks=5,
137
    nodes_per_block=1
138
)
139
```
140

141
### Kubernetes Provider
142

143
Manages containerized workloads on Kubernetes clusters with automatic scaling and resource management.
144

145
```python { .api }
146
class KubernetesProvider:
147
    def __init__(self, namespace='default', image=None, nodes_per_block=1,
148
                 init_blocks=1, min_blocks=0, max_blocks=1, max_cpu=1, max_mem="1Gi",
149
                 parallelism=1, worker_init='', pod_name=None, user_id=None,
150
                 group_id=None, run_as_non_root=False, persistent_volumes=None,
151
                 secret=None, incluster_config=True):
152
        """
153
        Kubernetes provider for container orchestration.
154
        
155
        Parameters:
156
        - namespace: Kubernetes namespace (default: 'default')
157
        - image: Container image for workers
158
        - max_cpu: CPU limit per pod (default: 1)
159
        - max_mem: Memory limit per pod (default: "1Gi")
160
        - persistent_volumes: List of persistent volume configurations
161
        - secret: Kubernetes secret for authentication
162
        - incluster_config: Use in-cluster config (default: True)
163
        - run_as_non_root: Run containers as non-root user
164
        """
165
```
166

167
**Usage Example:**
168

169
```python
170
from parsl.providers import KubernetesProvider
171

172
k8s_provider = KubernetesProvider(
173
    namespace='parsl-workflows',
174
    image='python:3.9-slim',
175
    max_cpu=2,
176
    max_mem='4Gi',
177
    init_blocks=1,
178
    max_blocks=10,
179
    persistent_volumes=[{
180
        'name': 'shared-data',
181
        'mount_path': '/data',
182
        'claim_name': 'shared-pvc'
183
    }]
184
)
185
```
186

187
### Google Cloud Provider
188

189
Manages Google Compute Engine instances for cloud-based parallel execution.
190

191
```python { .api }
192
class GoogleCloudProvider:
193
    def __init__(self, project_id, zone, machine_type='n1-standard-1',
194
                 image_id=None, disk_size_gb=10, nodes_per_block=1,
195
                 init_blocks=1, min_blocks=0, max_blocks=1, parallelism=1,
196
                 walltime="00:10:00", launcher=None, worker_init=''):
197
        """
198
        Google Cloud Platform provider for compute resources.
199
        
200
        Parameters:
201
        - project_id: GCP project ID
202
        - zone: GCP zone (e.g., 'us-central1-a')
203
        - machine_type: Instance machine type (e.g., 'n1-standard-4')
204
        - image_id: VM image family or specific image
205
        - disk_size_gb: Boot disk size in GB (default: 10)
206
        """
207
```
208

209
**Usage Example:**
210

211
```python
212
from parsl.providers import GoogleCloudProvider
213

214
gcp_provider = GoogleCloudProvider(
215
    project_id='my-gcp-project',
216
    zone='us-central1-a',
217
    machine_type='n1-standard-4',
218
    image_id='projects/ubuntu-os-cloud/global/images/family/ubuntu-2004-lts',
219
    disk_size_gb=50,
220
    init_blocks=1,
221
    max_blocks=8
222
)
223
```
224

225
### Azure Provider
226

227
Provisions and manages Microsoft Azure virtual machines for cloud computing.
228

229
```python { .api }
230
class AzureProvider:
231
    def __init__(self, vm_size='Standard_D1_v2', region='eastus',
232
                 image=None, nodes_per_block=1, init_blocks=1, min_blocks=0,
233
                 max_blocks=1, parallelism=1, walltime="00:10:00",
234
                 launcher=None, worker_init=''):
235
        """
236
        Microsoft Azure provider for cloud computing resources.
237
        
238
        Parameters:
239
        - vm_size: Azure VM size (e.g., 'Standard_D4_v3')
240
        - region: Azure region (default: 'eastus')
241
        - image: VM image configuration dict
242
        """
243
```
244

245
### HPC Scheduler Providers
246

247
Additional providers for common HPC job schedulers:
248

249
```python { .api }
250
class LSFProvider:
251
    """IBM LSF (Load Sharing Facility) provider."""
252
    def __init__(self, queue=None, account=None, project=None, ...): ...
253

254
class PBSProProvider:
255
    """PBS Professional scheduler provider."""
256
    def __init__(self, queue=None, account=None, nodes_per_block=1, ...): ...
257

258
class TorqueProvider:
259
    """TORQUE resource manager provider."""
260
    def __init__(self, queue=None, nodes_per_block=1, ...): ...
261

262
class CondorProvider:
263
    """HTCondor high-throughput computing provider."""
264
    def __init__(self, nodes_per_block=1, init_blocks=1, ...): ...
265

266
class GridEngineProvider:
267
    """Sun/Oracle Grid Engine provider."""
268
    def __init__(self, queue=None, nodes_per_block=1, ...): ...
269
```
270

271
**HPC Provider Examples:**
272

273
```python
274
from parsl.providers import LSFProvider, PBSProProvider
275
from parsl.launchers import JsrunLauncher, MpiRunLauncher
276

277
# IBM LSF on Summit supercomputer
278
lsf_provider = LSFProvider(
279
    queue='batch',
280
    account='project123',
281
    nodes_per_block=2,
282
    walltime='01:00:00',
283
    launcher=JsrunLauncher()
284
)
285

286
# PBS Pro cluster
287
pbs_provider = PBSProProvider(
288
    queue='normal',
289
    account='allocation123',
290
    nodes_per_block=4,
291
    cores_per_node=28,  
292
    walltime='02:00:00',
293
    launcher=MpiRunLauncher()
294
)
295
```
296

297
## Provider Configuration Patterns
298

299
### Multi-Provider Configuration
300

301
Use multiple providers for different types of workloads:
302

303
```python
304
from parsl.config import Config
305
from parsl.executors import HighThroughputExecutor
306

307
config = Config(executors=[
308
    # Local development and testing
309
    HighThroughputExecutor(
310
        label='local_dev',
311
        max_workers=4,
312
        provider=LocalProvider(max_blocks=1)
313
    ),
314
    
315
    # HPC cluster for compute-intensive tasks
316
    HighThroughputExecutor(
317
        label='hpc_cluster',  
318
        max_workers=100,
319
        provider=SlurmProvider(
320
            partition='compute',
321
            nodes_per_block=4,
322
            max_blocks=25,
323
            walltime='04:00:00'
324
        )
325
    ),
326
    
327
    # Cloud bursting for overflow capacity
328
    HighThroughputExecutor(
329
        label='cloud_burst',
330
        max_workers=50,
331
        provider=AWSProvider(
332
            instance_type='c5.xlarge',
333
            max_blocks=10,
334
            spot_max_bid=0.20
335
        )
336
    )
337
])
338
```
339

340
### Resource Scaling
341

342
Configure automatic resource scaling based on workload:
343

344
```python
345
# Aggressive scaling for burst workloads
346
burst_provider = SlurmProvider(
347
    partition='burst',
348
    init_blocks=0,      # Start with no resources
349
    min_blocks=0,       # Scale down to zero when idle
350
    max_blocks=50,      # Scale up to 50 blocks
351
    parallelism=0.8,    # Submit 80% of pending tasks as blocks
352
    walltime='00:30:00' # Short walltime for responsiveness
353
)
354

355
# Conservative scaling for long-running workflows  
356
steady_provider = SlurmProvider(
357
    partition='normal',
358
    init_blocks=2,      # Always maintain 2 blocks
359
    min_blocks=2,       # Never scale below 2 blocks  
360
    max_blocks=10,      # Conservative maximum
361
    parallelism=0.3,    # Conservative submission rate
362
    walltime='12:00:00' # Long walltime for efficiency
363
)
364
```
365

366
### Provider Error Handling
367

368
Handle provider-specific errors and resource failures:
369

370
```python
371
from parsl.providers.errors import ProviderException, ScaleOutFailed
372

373
try:
374
    parsl.load(config)
375
except ProviderException as e:
376
    print(f"Provider initialization failed: {e}")
377
except ScaleOutFailed as e:
378
    print(f"Resource scaling failed: {e}")
379

380
# Monitor provider status
381
for executor in parsl.dfk().executors.values():
382
    provider = executor.provider
383
    print(f"Provider {provider.label}: {provider.resources}")
384
```