0
# Integrations
1

2
ZenML includes 67 integrations with various ML/MLOps tools, cloud providers, and frameworks. Each integration provides stack component flavors, materializers, steps, and utilities specific to that tool.
3

4
## Integration Categories
5

6
### Cloud Providers
7

8
**AWS Integration**
9
- S3 artifact store
10
- SageMaker orchestrator and step operator
11
- ECR container registry
12
- Secrets Manager integration
13
- Installation: `pip install zenml[aws]`
14

15
**Azure Integration**
16
- Azure Blob Storage artifact store
17
- AzureML orchestrator
18
- ACR container registry
19
- Azure Key Vault secrets
20
- Installation: `pip install zenml[azure]`
21

22
**GCP Integration**
23
- GCS artifact store
24
- Vertex AI orchestrator
25
- GCR container registry
26
- Google Secret Manager
27
- Cloud Build image builder
28
- Installation: `pip install zenml[gcp]`
29

30
### Orchestrators
31

32
**Kubernetes**
33
- Kubernetes orchestrator for running pipelines on K8s clusters
34
- Installation: `pip install zenml[kubernetes]`
35

36
**Kubeflow**
37
- Kubeflow Pipelines orchestrator
38
- Installation: `pip install zenml[kubeflow]`
39

40
**Airflow**
41
- Apache Airflow orchestrator
42
- Installation: `pip install zenml[airflow]`
43

44
**Tekton**
45
- Tekton Pipelines orchestrator
46
- Installation: `pip install zenml[tekton]`
47

48
**Databricks**
49
- Databricks orchestrator for running on Databricks clusters
50
- Installation: `pip install zenml[databricks]`
51

52
**Lightning**
53
- Lightning.ai orchestrator
54
- Installation: `pip install zenml[lightning]`
55

56
**Modal**
57
- Modal orchestrator for serverless execution
58
- Installation: `pip install zenml[modal]`
59

60
**SkyPilot**
61
- SkyPilot orchestrator with variants for AWS, GCP, Azure, Kubernetes, Lambda
62
- Installation: `pip install zenml[skypilot]`
63

64
### Experiment Tracking
65

66
**MLflow**
67
- MLflow tracking server integration
68
- MLflow model registry
69
- MLflow model deployer
70
- Installation: `pip install zenml[mlflow]`
71

72
**Weights & Biases**
73
- W&B experiment tracking
74
- Installation: `pip install zenml[wandb]`
75

76
**Neptune**
77
- Neptune.ai experiment tracking
78
- Installation: `pip install zenml[neptune]`
79

80
**Comet**
81
- Comet.ml experiment tracking
82
- Installation: `pip install zenml[comet]`
83

84
**TensorBoard**
85
- TensorBoard visualization
86
- Installation: `pip install zenml[tensorboard]`
87

88
### ML Frameworks
89

90
**PyTorch**
91
- PyTorch tensor and model materializers
92
- Installation: `pip install zenml[pytorch]`
93

94
**TensorFlow**
95
- TensorFlow tensor and model materializers
96
- Installation: `pip install zenml[tensorflow]`
97

98
**PyTorch Lightning**
99
- Lightning module materializers
100
- Installation: `pip install zenml[pytorch-lightning]`
101

102
**JAX**
103
- JAX array materializers
104
- Installation: `pip install zenml[jax]`
105

106
**Scikit-learn**
107
- Sklearn model materializers
108
- Installation: `pip install zenml[sklearn]`
109

110
**XGBoost**
111
- XGBoost booster and DMatrix materializers
112
- Installation: `pip install zenml[xgboost]`
113

114
**LightGBM**
115
- LightGBM booster and dataset materializers
116
- Installation: `pip install zenml[lightgbm]`
117

118
**HuggingFace**
119
- Transformers model and tokenizer materializers
120
- Datasets integration
121
- Installation: `pip install zenml[huggingface]`
122

123
**LangChain**
124
- LangChain document and chain materializers
125
- Installation: `pip install zenml[langchain]`
126

127
**LlamaIndex**
128
- LlamaIndex materializers
129
- Installation: `pip install zenml[llama-index]`
130

131
### Data Validation
132

133
**Great Expectations**
134
- Data validation and profiling
135
- Installation: `pip install zenml[great-expectations]`
136

137
**Deepchecks**
138
- Model validation and monitoring
139
- Installation: `pip install zenml[deepchecks]`
140

141
**Evidently**
142
- Data and model monitoring
143
- Installation: `pip install zenml[evidently]`
144

145
**Whylogs**
146
- Data logging and profiling
147
- Installation: `pip install zenml[whylogs]`
148

149
### Model Deployment
150

151
**BentoML**
152
- Model serving with BentoML
153
- Installation: `pip install zenml[bentoml]`
154

155
**Seldon Core**
156
- Seldon Core model serving
157
- Installation: `pip install zenml[seldon]`
158

159
**vLLM**
160
- vLLM for LLM serving
161
- Installation: `pip install zenml[vllm]`
162

163
### Annotation Tools
164

165
**Label Studio**
166
- Data annotation with Label Studio
167
- Installation: `pip install zenml[label-studio]`
168

169
**Pigeon**
170
- Simple annotation in notebooks
171
- Installation: `pip install zenml[pigeon]`
172

173
**Argilla**
174
- Data annotation and curation
175
- Installation: `pip install zenml[argilla]`
176

177
### Alerting
178

179
**Slack**
180
- Slack notifications via alerter component
181
- Installation: `pip install zenml[slack]`
182

183
**Discord**
184
- Discord notifications via alerter component
185
- Installation: `pip install zenml[discord]`
186

187
### Feature Stores
188

189
**Feast**
190
- Feast feature store integration
191
- Installation: `pip install zenml[feast]`
192

193
### Visualization
194

195
**Facets**
196
- Facets data visualization
197
- Installation: `pip install zenml[facets]`
198

199
### Code Repositories
200

201
**GitHub**
202
- GitHub integration for code tracking
203
- Installation: `pip install zenml[github]`
204

205
**GitLab**
206
- GitLab integration
207
- Installation: `pip install zenml[gitlab]`
208

209
**Bitbucket**
210
- Bitbucket integration
211
- Installation: `pip install zenml[bitbucket]`
212

213
### Other Tools
214

215
**Spark**
216
- Apache Spark integration
217
- Installation: `pip install zenml[spark]`
218

219
**Kaniko**
220
- Kaniko image builder
221
- Installation: `pip install zenml[kaniko]`
222

223
**NumPy**
224
- NumPy array materializers (core dependency)
225

226
**Pandas**
227
- DataFrame materializers (core dependency)
228

229
**Pillow**
230
- Image materializers
231
- Installation: `pip install zenml[pillow]`
232

233
**PyArrow**
234
- PyArrow table materializers
235
- Installation: `pip install zenml[pyarrow]`
236

237
**SciPy**
238
- SciPy sparse matrix materializers
239
- Installation: `pip install zenml[scipy]`
240

241
**PyCaret**
242
- PyCaret model materializers
243
- Installation: `pip install zenml[pycaret]`
244

245
**Neural Prophet**
246
- Neural Prophet model materializers
247
- Installation: `pip install zenml[neural-prophet]`
248

249
**HyperAI**
250
- HyperAI orchestrator
251
- Installation: `pip install zenml[hyperai]`
252

253
## Usage Examples
254

255
### Installing Integrations
256

257
```bash
258
# Single integration
259
pip install zenml[aws]
260

261
# Multiple integrations
262
pip install zenml[aws,mlflow,pytorch]
263

264
# All integrations
265
pip install zenml[all]
266
```
267

268
### Activating Integration in Code
269

270
```python
271
from zenml.integrations.aws import AWSIntegration
272

273
# Check if integration is installed
274
if AWSIntegration.check_installation():
275
    print("AWS integration available")
276

277
# Integration is auto-activated when components are used
278
```
279

280
### Using AWS S3 Artifact Store
281

282
```python
283
from zenml.client import Client
284

285
client = Client()
286

287
# Create S3 artifact store
288
client.create_stack_component(
289
    name="s3_store",
290
    flavor="s3",
291
    component_type="artifact_store",
292
    configuration={
293
        "path": "s3://my-bucket/artifacts",
294
        "region": "us-east-1"
295
    }
296
)
297
```
298

299
### Using MLflow Experiment Tracker
300

301
```python
302
from zenml import step, pipeline
303
from zenml.client import Client
304

305
client = Client()
306

307
# Create MLflow experiment tracker
308
client.create_stack_component(
309
    name="mlflow_tracker",
310
    flavor="mlflow",
311
    component_type="experiment_tracker",
312
    configuration={
313
        "tracking_uri": "http://localhost:5000"
314
    }
315
)
316

317
# Use in step
318
@step(experiment_tracker="mlflow_tracker")
319
def train_model(data: list) -> dict:
320
    import mlflow
321
    mlflow.log_param("param1", "value1")
322
    mlflow.log_metric("accuracy", 0.95)
323
    return {"model": "trained"}
324
```
325

326
### Using PyTorch with Materializers
327

328
```python
329
from zenml import step
330
import torch
331
import torch.nn as nn
332

333
@step
334
def train_pytorch_model(data: list) -> nn.Module:
335
    """PyTorch models automatically use PyTorch materializer."""
336
    model = nn.Sequential(
337
        nn.Linear(10, 5),
338
        nn.ReLU(),
339
        nn.Linear(5, 1)
340
    )
341
    # Training logic
342
    return model
343

344
@step
345
def evaluate_model(model: nn.Module, test_data: list) -> float:
346
    """Model automatically deserialized."""
347
    # Evaluation logic
348
    return 0.95
349
```
350

351
### Using SageMaker Orchestrator
352

353
```python
354
from zenml.client import Client
355

356
client = Client()
357

358
# Create SageMaker orchestrator
359
client.create_stack_component(
360
    name="sagemaker",
361
    flavor="sagemaker",
362
    component_type="orchestrator",
363
    configuration={
364
        "execution_role": "arn:aws:iam::123456789:role/SageMaker",
365
        "region": "us-east-1",
366
        "instance_type": "ml.m5.large"
367
    }
368
)
369

370
# Create stack with SageMaker
371
client.create_stack(
372
    name="sagemaker_stack",
373
    components={
374
        "orchestrator": "sagemaker",
375
        "artifact_store": "s3_store"
376
    }
377
)
378

379
# Activate and use
380
client.activate_stack("sagemaker_stack")
381
```
382

383
### Using Great Expectations for Data Validation
384

385
```python
386
from zenml import step
387
import pandas as pd
388

389
@step
390
def validate_data(data: pd.DataFrame) -> pd.DataFrame:
391
    """Validate data with Great Expectations."""
392
    from great_expectations.core import ExpectationSuite
393
    from great_expectations.dataset import PandasDataset
394

395
    ge_df = PandasDataset(data)
396

397
    # Define expectations
398
    ge_df.expect_column_to_exist("feature1")
399
    ge_df.expect_column_values_to_not_be_null("feature1")
400
    ge_df.expect_column_values_to_be_between("feature1", 0, 100)
401

402
    # Validate
403
    results = ge_df.validate()
404

405
    if not results.success:
406
        raise ValueError("Data validation failed")
407

408
    return data
409
```
410

411
### Using Slack Alerter
412

413
```python
414
from zenml import pipeline
415
from zenml.hooks import alerter_success_hook, alerter_failure_hook
416
from zenml.client import Client
417

418
client = Client()
419

420
# Create Slack alerter
421
client.create_stack_component(
422
    name="slack_alerter",
423
    flavor="slack",
424
    component_type="alerter",
425
    configuration={
426
        "slack_token": "xoxb-your-token",
427
        "default_slack_channel_id": "C01234567"
428
    }
429
)
430

431
# Use in pipeline
432
@pipeline(
433
    on_success=alerter_success_hook("slack_alerter"),
434
    on_failure=alerter_failure_hook("slack_alerter")
435
)
436
def monitored_pipeline():
437
    # Pipeline definition
438
    pass
439
```
440

441
### Listing Available Integrations
442

443
```python
444
from zenml.integrations.registry import integration_registry
445

446
# List all integrations
447
for name, integration in integration_registry.integrations.items():
448
    print(f"{name}: {integration.NAME}")
449

450
# Check specific integration
451
from zenml.integrations.pytorch import PyTorchIntegration
452

453
print(f"PyTorch Integration: {PyTorchIntegration.NAME}")
454
print(f"Requirements: {PyTorchIntegration.REQUIREMENTS}")
455
```
456

457
## Integration Combinations
458

459
### Full ML Stack Example
460

461
```python
462
from zenml.client import Client
463

464
client = Client()
465

466
# AWS + MLflow + PyTorch stack
467
stack_components = {
468
    "orchestrator": "sagemaker",
469
    "artifact_store": "s3_store",
470
    "container_registry": "ecr",
471
    "experiment_tracker": "mlflow",
472
    "model_registry": "mlflow_registry"
473
}
474

475
client.create_stack(
476
    name="ml_production",
477
    components=stack_components
478
)
479
```
480