Groq Core Workflow B: Audio, Vision & Speech

Overview

Beyond chat completions, Groq offers ultra-fast Whisper transcription (216x real-time), Llama 4 vision, and text-to-speech — all on the same groq-sdk client. This skill covers transcription/translation, vision, TTS, and model benchmarking, with full runnable code in references/implementation.md and worked scripts in references/examples.md.

Prerequisites

• groq-sdk installed, GROQ_API_KEY set (the SDK reads it from the environment automatically)
• For audio: audio files in a supported format
• For vision: image URLs or base64-encoded images

Audio Models

Supported audio formats: flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, webm

Instructions

Each workflow is a single SDK call on the shared groq client. Pick the endpoint for your task, then follow the full walkthrough in references/implementation.md for the complete, copy-pasteable version of each.

1. Transcription — groq.audio.transcriptions.create({ file, model: "whisper-large-v3-turbo", response_format }). Use response_format: "verbose_json" with timestamp_granularities: ["segment"] to get per-segment start/end times.
2. Translation — groq.audio.translations.create({ file, model: "whisper-large-v3" }) transcribes any-language audio directly to English text.
3. Vision — a normal groq.chat.completions.create call where content is an array mixing { type: "text" } and { type: "image_url" } parts. Accepts up to 5 images (URL or data: base64) with meta-llama/llama-4-scout-17b-16e-instruct.
4. Text-to-Speech — groq.audio.speech.create({ model: "playai-tts", input, voice, response_format }), then write Buffer.from(await response.arrayBuffer()) to a file.
5. Benchmarking — loop a prompt across several chat models and time each call to compare latency and tokens/sec (see references/examples.md).

Minimal transcription skeleton:

import Groq from "groq-sdk";
import fs from "fs";

const groq = new Groq();

async function transcribe(filePath: string): Promise<string> {
  const transcription = await groq.audio.transcriptions.create({
    file: fs.createReadStream(filePath),
    model: "whisper-large-v3-turbo",
    response_format: "json",
  });
  return transcription.text;
}

Output

• Transcription/translation: a transcription.text string. With verbose_json, a segments[] array where each segment has start, end, and text.
• Vision: the assistant reply at completion.choices[0].message.content (a natural-language answer about the image(s)).
• Text-to-Speech: an audio response you convert to a Buffer and write to disk (wav, mp3, flac, opus, or aac).
• Benchmarking: one console line per model — latency in ms, throughput in tok/s, and total tokens.

Vision Model Limits

• Maximum 5 images per request
• Supported formats: JPEG, PNG, GIF, WebP
• Images fetched from URL or embedded as base64
• Vision models also support tool use, JSON mode, and streaming

Error Handling

Examples

Complete, runnable scripts live in references/examples.md:

• Python transcription with timestamps — transcribe a local MP3 and print each segment with its start/end time.
• Model benchmarking — run one prompt across llama-3.1-8b-instant, llama-3.3-70b-versatile, and llama-3.3-70b-specdec and print latency + throughput per model.

Quick vision example (analyze one image by URL):

const completion = await groq.chat.completions.create({
  model: "meta-llama/llama-4-scout-17b-16e-instruct",
  messages: [{
    role: "user",
    content: [
      { type: "text", text: "What is in this image?" },
      { type: "image_url", image_url: { url: imageUrl } },
    ],
  }],
  max_tokens: 1024,
});
console.log(completion.choices[0].message.content);

Resources

• Groq Speech-to-Text
• Groq Text-to-Speech
• Groq Vision
• Groq Models

Next Steps

For common errors and troubleshooting patterns across all Groq workflows, see the groq-common-errors skill. For chat completions, streaming, tool use, and JSON mode, see groq-core-workflow-a.