0
# Batch Processing
1
2
Submit and manage batch jobs for processing large volumes of requests efficiently. The batch API allows you to send multiple requests in a single operation, which is processed asynchronously and can provide significant cost savings for non-time-sensitive workloads.
3
4
## Capabilities
5
6
### Create Batch Job
7
8
Submit a batch job for processing multiple requests asynchronously.
9
10
```python { .api }
11
def create(
12
input_file_id: str,
13
endpoint: Literal["/v1/chat/completions"],
14
completion_window: Literal["24h"],
15
metadata: Optional[Dict[str, str]] = NOT_GIVEN,
16
extra_headers: Headers | None = None,
17
extra_query: Query | None = None,
18
extra_body: Body | None = None,
19
timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN
20
) -> BatchCreateResponse:
21
"""
22
Create a batch job for processing multiple requests.
23
24
Parameters:
25
- input_file_id: ID of the uploaded file containing batch requests
26
- endpoint: API endpoint to process requests against
27
- completion_window: Time window for batch completion
28
- metadata: Optional metadata to attach to the batch
29
30
Returns:
31
BatchCreateResponse with batch job information and status
32
"""
33
```
34
35
### Retrieve Batch Status
36
37
Get detailed information about a specific batch job by its ID.
38
39
```python { .api }
40
def retrieve(
41
batch_id: str,
42
extra_headers: Headers | None = None,
43
extra_query: Query | None = None,
44
extra_body: Body | None = None,
45
timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN
46
) -> BatchRetrieveResponse:
47
"""
48
Retrieve information about a specific batch job.
49
50
Parameters:
51
- batch_id: ID of the batch job to retrieve
52
53
Returns:
54
BatchRetrieveResponse with detailed batch information and status
55
"""
56
```
57
58
### List Batch Jobs
59
60
Retrieve a list of all batch jobs with their current status.
61
62
```python { .api }
63
def list(
64
after: Optional[str] = NOT_GIVEN,
65
limit: Optional[int] = NOT_GIVEN,
66
extra_headers: Headers | None = None,
67
extra_query: Query | None = None,
68
extra_body: Body | None = None,
69
timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN
70
) -> BatchListResponse:
71
"""
72
List batch jobs.
73
74
Parameters:
75
- after: Cursor for pagination, returns objects after this ID
76
- limit: Number of objects to return (default 20, max 100)
77
78
Returns:
79
BatchListResponse containing list of batch jobs
80
"""
81
```
82
83
### Cancel Batch Job
84
85
Cancel a batch job that is in progress or queued.
86
87
```python { .api }
88
def cancel(
89
batch_id: str,
90
extra_headers: Headers | None = None,
91
extra_query: Query | None = None,
92
extra_body: Body | None = None,
93
timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN
94
) -> BatchCancelResponse:
95
"""
96
Cancel a batch job.
97
98
Parameters:
99
- batch_id: ID of the batch job to cancel
100
101
Returns:
102
BatchCancelResponse with cancellation confirmation
103
"""
104
```
105
106
### Async Batch Operations
107
108
All batch operations have asynchronous counterparts with identical parameters.
109
110
```python { .api }
111
async def create(input_file_id: str, endpoint: str, completion_window: str, **kwargs) -> BatchCreateResponse: ...
112
async def retrieve(batch_id: str, **kwargs) -> BatchRetrieveResponse: ...
113
async def list(**kwargs) -> BatchListResponse: ...
114
async def cancel(batch_id: str, **kwargs) -> BatchCancelResponse: ...
115
```
116
117
## Usage Examples
118
119
### Complete Batch Processing Workflow
120
121
```python
122
from groq import Groq
123
import json
124
import time
125
126
client = Groq()
127
128
# 1. Create batch request data
129
batch_requests = [
130
{
131
"custom_id": "request-1",
132
"method": "POST",
133
"url": "/v1/chat/completions",
134
"body": {
135
"model": "llama3-8b-8192",
136
"messages": [{"role": "user", "content": "What is artificial intelligence?"}],
137
"max_tokens": 100
138
}
139
},
140
{
141
"custom_id": "request-2",
142
"method": "POST",
143
"url": "/v1/chat/completions",
144
"body": {
145
"model": "llama3-8b-8192",
146
"messages": [{"role": "user", "content": "Explain machine learning in simple terms."}],
147
"max_tokens": 100
148
}
149
},
150
{
151
"custom_id": "request-3",
152
"method": "POST",
153
"url": "/v1/chat/completions",
154
"body": {
155
"model": "llama3-8b-8192",
156
"messages": [{"role": "user", "content": "What are the benefits of deep learning?"}],
157
"max_tokens": 100
158
}
159
}
160
]
161
162
# 2. Write requests to JSONL file
163
with open("batch_requests.jsonl", "w") as f:
164
for request in batch_requests:
165
f.write(json.dumps(request) + "\n")
166
167
# 3. Upload the file
168
with open("batch_requests.jsonl", "rb") as file:
169
file_response = client.files.create(
170
file=file,
171
purpose="batch"
172
)
173
174
print(f"File uploaded: {file_response.id}")
175
176
# 4. Create batch job
177
batch = client.batches.create(
178
input_file_id=file_response.id,
179
endpoint="/v1/chat/completions",
180
completion_window="24h",
181
metadata={"description": "AI Q&A batch processing"}
182
)
183
184
print(f"Batch created: {batch.id}")
185
print(f"Status: {batch.status}")
186
187
# 5. Monitor batch progress
188
while True:
189
batch_status = client.batches.retrieve(batch.id)
190
print(f"Batch status: {batch_status.status}")
191
192
if batch_status.status == "completed":
193
print("Batch completed!")
194
print(f"Output file: {batch_status.output_file_id}")
195
break
196
elif batch_status.status == "failed":
197
print("Batch failed!")
198
if batch_status.errors:
199
print(f"Errors: {batch_status.errors}")
200
break
201
elif batch_status.status == "cancelled":
202
print("Batch was cancelled!")
203
break
204
205
time.sleep(30) # Wait 30 seconds before checking again
206
```
207
208
### Create Batch Job
209
210
```python
211
from groq import Groq
212
213
client = Groq()
214
215
# Assuming you have already uploaded a file with batch requests
216
batch = client.batches.create(
217
input_file_id="file-abc123",
218
endpoint="/v1/chat/completions",
219
completion_window="24h",
220
metadata={
221
"project": "customer_support_analysis",
222
"batch_type": "chat_completions"
223
}
224
)
225
226
print(f"Batch ID: {batch.id}")
227
print(f"Status: {batch.status}")
228
print(f"Created at: {batch.created_at}")
229
print(f"Request counts: {batch.request_counts}")
230
```
231
232
### Monitor Batch Status
233
234
```python
235
from groq import Groq
236
237
client = Groq()
238
239
# Get batch information
240
batch = client.batches.retrieve("batch_abc123")
241
242
print(f"Batch ID: {batch.id}")
243
print(f"Status: {batch.status}")
244
print(f"Input file: {batch.input_file_id}")
245
print(f"Endpoint: {batch.endpoint}")
246
print(f"Completion window: {batch.completion_window}")
247
248
# Print request counts
249
if batch.request_counts:
250
print(f"Total requests: {batch.request_counts.total}")
251
print(f"Completed: {batch.request_counts.completed}")
252
print(f"Failed: {batch.request_counts.failed}")
253
254
# Check if completed and get output
255
if batch.status == "completed" and batch.output_file_id:
256
print(f"Output file: {batch.output_file_id}")
257
# You can then download the output file using the files API
258
```
259
260
### List All Batches
261
262
```python
263
from groq import Groq
264
265
client = Groq()
266
267
# List all batches
268
batches = client.batches.list(limit=10)
269
270
print(f"Total batches: {len(batches.data)}")
271
for batch in batches.data:
272
print(f"- {batch.id}: {batch.status}")
273
print(f" Endpoint: {batch.endpoint}")
274
print(f" Created: {batch.created_at}")
275
if batch.request_counts:
276
print(f" Requests: {batch.request_counts.completed}/{batch.request_counts.total}")
277
```
278
279
### Cancel Batch Job
280
281
```python
282
from groq import Groq
283
284
client = Groq()
285
286
# Cancel a batch that's in progress
287
try:
288
result = client.batches.cancel("batch_abc123")
289
print(f"Batch cancelled: {result.id}")
290
print(f"Status: {result.status}")
291
except Exception as e:
292
print(f"Failed to cancel batch: {e}")
293
```
294
295
### Async Usage
296
297
```python
298
import asyncio
299
from groq import AsyncGroq
300
301
async def main():
302
client = AsyncGroq()
303
304
# Create batch asynchronously
305
batch = await client.batches.create(
306
input_file_id="file-abc123",
307
endpoint="/v1/chat/completions",
308
completion_window="24h"
309
)
310
311
print(f"Batch created: {batch.id}")
312
313
# Monitor progress asynchronously
314
while True:
315
batch_status = await client.batches.retrieve(batch.id)
316
if batch_status.status in ["completed", "failed", "cancelled"]:
317
break
318
await asyncio.sleep(30)
319
320
print(f"Final status: {batch_status.status}")
321
322
asyncio.run(main())
323
```
324
325
## Types
326
327
### Request Types
328
329
```python { .api }
330
class BatchCreateParams:
331
input_file_id: str
332
endpoint: Literal["/v1/chat/completions"]
333
completion_window: Literal["24h"]
334
metadata: Optional[Dict[str, str]]
335
```
336
337
### Response Types
338
339
```python { .api }
340
class BatchCreateResponse:
341
id: str
342
object: Literal["batch"]
343
endpoint: str
344
errors: Optional[BatchErrors]
345
input_file_id: str
346
completion_window: str
347
status: Literal["validating", "failed", "in_progress", "finalizing", "completed", "expired", "cancelling", "cancelled"]
348
output_file_id: Optional[str]
349
error_file_id: Optional[str]
350
created_at: int
351
in_progress_at: Optional[int]
352
expires_at: Optional[int]
353
finalizing_at: Optional[int]
354
completed_at: Optional[int]
355
failed_at: Optional[int]
356
expired_at: Optional[int]
357
cancelling_at: Optional[int]
358
cancelled_at: Optional[int]
359
request_counts: Optional[BatchRequestCounts]
360
metadata: Optional[Dict[str, str]]
361
362
class BatchRetrieveResponse:
363
# Same structure as BatchCreateResponse
364
id: str
365
object: Literal["batch"]
366
endpoint: str
367
errors: Optional[BatchErrors]
368
input_file_id: str
369
completion_window: str
370
status: Literal["validating", "failed", "in_progress", "finalizing", "completed", "expired", "cancelling", "cancelled"]
371
output_file_id: Optional[str]
372
error_file_id: Optional[str]
373
created_at: int
374
in_progress_at: Optional[int]
375
expires_at: Optional[int]
376
finalizing_at: Optional[int]
377
completed_at: Optional[int]
378
failed_at: Optional[int]
379
expired_at: Optional[int]
380
cancelling_at: Optional[int]
381
cancelled_at: Optional[int]
382
request_counts: Optional[BatchRequestCounts]
383
metadata: Optional[Dict[str, str]]
384
385
class BatchListResponse:
386
object: Literal["list"]
387
data: List[BatchRetrieveResponse]
388
first_id: Optional[str]
389
last_id: Optional[str]
390
has_more: bool
391
392
class BatchCancelResponse:
393
# Same structure as BatchCreateResponse
394
id: str
395
object: Literal["batch"]
396
endpoint: str
397
errors: Optional[BatchErrors]
398
input_file_id: str
399
completion_window: str
400
status: Literal["validating", "failed", "in_progress", "finalizing", "completed", "expired", "cancelling", "cancelled"]
401
output_file_id: Optional[str]
402
error_file_id: Optional[str]
403
created_at: int
404
in_progress_at: Optional[int]
405
expires_at: Optional[int]
406
finalizing_at: Optional[int]
407
completed_at: Optional[int]
408
failed_at: Optional[int]
409
expired_at: Optional[int]
410
cancelling_at: Optional[int]
411
cancelled_at: Optional[int]
412
request_counts: Optional[BatchRequestCounts]
413
metadata: Optional[Dict[str, str]]
414
```
415
416
### Supporting Types
417
418
```python { .api }
419
class BatchRequestCounts:
420
total: int
421
completed: int
422
failed: int
423
424
class BatchErrors:
425
object: str
426
data: List[BatchError]
427
428
class BatchError:
429
code: str
430
message: str
431
param: Optional[str]
432
line: Optional[int]
433
```
434
435
### Batch Request Format
436
437
Each line in the input JSONL file should follow this format:
438
439
```python { .api }
440
class BatchRequest:
441
custom_id: str # Your unique identifier for the request
442
method: Literal["POST"] # HTTP method
443
url: str # API endpoint (e.g., "/v1/chat/completions")
444
body: Dict[str, Any] # Request parameters for the endpoint
445
```
446
447
### Batch Status Values
448
449
- `validating` - Input file is being validated
450
- `failed` - Batch failed during validation or processing
451
- `in_progress` - Batch is currently being processed
452
- `finalizing` - Batch processing is complete, output file being prepared
453
- `completed` - Batch completed successfully
454
- `expired` - Batch expired before completion
455
- `cancelling` - Batch cancellation in progress
456
- `cancelled` - Batch was cancelled