Groq SDK Patterns

Overview

Production patterns for the groq-sdk package. The Groq SDK mirrors the OpenAI SDK interface (chat.completions.create), so patterns feel familiar but must account for Groq-specific behavior: extreme speed (500+ tok/s), aggressive rate limits on free tier, and unique response metadata like queue_time and completion_time.

The full, copy-paste-ready implementations live in references/ so this file stays a fast map of the workflow. Read the summary here, then drill into the language file you need.

Prerequisites

• groq-sdk (TypeScript) or groq (Python) installed
• GROQ_API_KEY set in the environment
• Understanding of async/await and error handling
• Familiarity with OpenAI SDK patterns (Groq is API-compatible)

Instructions

Build the integration in layers. Each step below is a one-line summary; the full typed implementation is in references/typescript-patterns.md (steps 1–5, 7) and references/python-patterns.md (step 6).

1. Typed client singleton — one shared Groq client with maxRetries and timeout, so the whole app reuses one connection pool and config.
2. Type-safe completion wrapper — return a typed result that surfaces Groq's unique timing fields (queue_time, completion_time, total_time) and a computed tokensPerSec.
3. Streaming with typed events — an AsyncGenerator<string> that yields delta.content tokens.
4. Error handling with Groq error types — branch on Groq.APIError (429, 401, other) and Groq.APIConnectionError; rethrow the unknown.
5. Retry with exponential backoff — honor the retry-after header on 429s, else jittered backoff.
6. Python patterns — sync Groq(), AsyncGroq(), and streaming (see the Python reference).
7. Multi-tenant client factory — cache one client per tenant so API keys stay isolated.

The essential skeleton — a shared singleton every other pattern builds on:

// src/groq/client.ts
import Groq from "groq-sdk";

let _client: Groq | null = null;

export function getGroq(): Groq {
  if (!_client) {
    _client = new Groq({
      apiKey: process.env.GROQ_API_KEY,
      maxRetries: 3,
      timeout: 30_000,
    });
  }
  return _client;
}

Groq differs from OpenAI in a few details (package name, base URL, extra usage timing fields, error class names). The full comparison and error-handling matrix are in references/sdk-differences.md.

Output

Applying these patterns produces:

• A reusable getGroq() client module and, for multi-tenant apps, a getClientForTenant() factory.
• A complete() wrapper returning a typed CompletionResult — content, model, tokens (prompt/completion/total), and timing (queueMs, totalMs, tokensPerSec).
• A safeComplete() variant returning { data, error } so callers never face an uncaught exception.
• Streaming helpers that yield string tokens as they arrive.

Error Handling

• 429 (rate limited): read err.headers["retry-after"] and wait that long before retrying; free tier hits this often.
• 401 (bad key): surface a clear "Check GROQ_API_KEY" message — do not retry.
• APIConnectionError: network issue reaching api.groq.com; retry or fail fast per context.
• Unknown errors: rethrow so they are not silently swallowed.

Full typed handlers: references/typescript-patterns.md (Step 4 and Step 5).

Examples

Non-streaming completion with timing metadata (full code in references/typescript-patterns.md, Step 2):

const result = await complete(
  [{ role: "user", content: "Summarize Groq's speed advantage." }],
  "llama-3.3-70b-versatile"
);
console.log(result.content);
console.log(`${result.timing.tokensPerSec.toFixed(0)} tok/s`);

Streaming tokens to stdout (full code in the TS reference, Step 3):

for await (const token of streamCompletion([{ role: "user", content: "Hello" }])) {
  process.stdout.write(token);
}

Python one-liner (full sync/async/streaming in references/python-patterns.md):

from groq import Groq
client = Groq()
print(client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello"}],
).choices[0].message.content)

Resources

• Groq TypeScript SDK
• Groq API Reference
• Groq Error Codes
• references/typescript-patterns.md — full TS implementations (steps 1–5, 7)
• references/python-patterns.md — full Python implementations (step 6)
• references/sdk-differences.md — OpenAI-vs-Groq comparison and error matrix

Next Steps

Apply these patterns in groq-core-workflow-a for real-world chat completions, then wire safeComplete and withRetry into every call site so rate limits and network errors are handled consistently across the codebase.