HTTP Replay Package

The httpreplay package provides an API for recording and replaying HTTP traffic from HTTP-based Google API clients. This is useful for creating deterministic, fast tests without requiring network access to real services.

Status: This package is EXPERIMENTAL and is subject to change or removal without notice.

Import

import "cloud.google.com/go/httpreplay"

Overview

The httpreplay package allows you to:

1. Record HTTP interactions between your application and Google API services to a file
2. Replay those recorded interactions in tests without contacting real services

This approach provides several benefits for testing:

• Fast test execution (no network calls)
• Deterministic test results (same responses every time)
• Ability to test against specific API responses
• No need for credentials during replay
• Tests can run offline

The package works by creating a proxy that sits between your HTTP client and the actual service. During recording, the proxy forwards requests to the real service and logs both requests and responses. During replay, the proxy matches incoming requests against the log and returns the recorded responses.

Recording Workflow

To record HTTP interactions:

1. Create a Recorder with NewRecorder
2. Get an HTTP client from the recorder using Client()
3. Use that client to make API calls (which are recorded)
4. Close the recorder to save the recording file

Replay Workflow

To replay recorded interactions:

1. Create a Replayer with NewReplayer
2. Get an HTTP client from the replayer using Client()
3. Make the same API calls (which return recorded responses)
4. Close the replayer

Recorder Type

Type Definition

type Recorder struct {
    // Has unexported fields.
}

A Recorder records HTTP interactions to a file for later replay.

Creating a Recorder

func NewRecorder(filename string, initial []byte) (*Recorder, error)

Creates a recorder that writes to filename. The file will also store initial state that can be retrieved during replay. You must call Close() on the Recorder to ensure all data is written.

Parameters:

• filename: Path to the file where interactions will be recorded
• initial: Optional initial state (e.g., random seed, timestamp) to be saved for replay

Returns: A new Recorder instance or an error if the file cannot be created.

Example:

import (
    "context"
    "log"
    "time"

    "cloud.google.com/go/httpreplay"
    "google.golang.org/api/option"
)

func recordInteractions() {
    // Serialize initial state
    timeNow := time.Now()
    initialBytes, _ := timeNow.MarshalBinary()

    // Create recorder
    rec, err := httpreplay.NewRecorder("recordings/test.replay", initialBytes)
    if err != nil {
        log.Fatal(err)
    }
    defer rec.Close()

    // Get HTTP client for recording
    ctx := context.Background()
    client, err := rec.Client(ctx,
        option.WithCredentialsFile("path/to/key.json"))
    if err != nil {
        log.Fatal(err)
    }

    // Use client to make API calls
    // These calls will be recorded
    resp, err := client.Get("https://example.googleapis.com/api/resource")
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()
}

Recorder Methods

Client

func (r *Recorder) Client(ctx context.Context, opts ...option.ClientOption) (*http.Client, error)

Returns an http.Client to be used for recording. Provide authentication options like option.WithTokenSource or option.WithCredentialsFile as you normally would, or omit them to use Application Default Credentials.

Parameters:

• ctx: Context for client creation
• opts: Client options including authentication credentials

Returns: An HTTP client configured for recording, or an error.

Example:

// With credentials file
client, err := rec.Client(ctx, option.WithCredentialsFile("/path/to/key.json"))

// With ADC (Application Default Credentials)
client, err := rec.Client(ctx)

// With token source
client, err := rec.Client(ctx, option.WithTokenSource(tokenSource))

RemoveRequestHeaders

func (r *Recorder) RemoveRequestHeaders(patterns ...string)

Removes request headers matching patterns from the log and skips matching them during replay. Use this for headers that change between runs (like timestamps) or that should not be logged.

Pattern is taken literally except for *, which matches any sequence of characters.

Example:

rec.RemoveRequestHeaders("Authorization", "User-Agent")
rec.RemoveRequestHeaders("X-*")  // Remove all headers starting with X-

ClearHeaders

func (r *Recorder) ClearHeaders(patterns ...string)

Replaces the value of request and response headers that match any of the patterns with CLEARED, on both recording and replay. Use this when header information is secret or may change from run to run, but you still want to verify that the headers are being sent and received.

Pattern is taken literally except for *, which matches any sequence of characters.

Example:

// Clear sensitive headers but keep them in the log
rec.ClearHeaders("Authorization", "X-API-Key")
rec.ClearHeaders("X-Secret-*")

RemoveQueryParams

func (r *Recorder) RemoveQueryParams(patterns ...string)

Removes URL query parameters matching patterns from the log and skips matching them during replay. Use this for parameters that change between runs.

Pattern is taken literally except for *, which matches any sequence of characters.

Example:

rec.RemoveQueryParams("timestamp", "nonce")
rec.RemoveQueryParams("session_*")

ClearQueryParams

func (r *Recorder) ClearQueryParams(patterns ...string)

Replaces the value of URL query parameters that match any of the patterns with CLEARED, on both recording and replay. Use this when parameter information is secret or may change from run to run, but you still want to verify that it is being sent.

Pattern is taken literally except for *, which matches any sequence of characters.

Example:

rec.ClearQueryParams("api_key", "access_token")

Close

func (r *Recorder) Close() error

Closes the Recorder and saves the log file. You must call this method to ensure all recorded data is written to disk.

Example:

rec, err := httpreplay.NewRecorder("test.replay", nil)
if err != nil {
    log.Fatal(err)
}
defer func() {
    if err := rec.Close(); err != nil {
        log.Printf("Failed to close recorder: %v", err)
    }
}()

Replayer Type

Type Definition

type Replayer struct {
    // Has unexported fields.
}

A Replayer replays previously recorded HTTP interactions from a file.

Creating a Replayer

func NewReplayer(filename string) (*Replayer, error)

Creates a replayer that reads from filename.

Parameters:

• filename: Path to the file containing recorded interactions

Returns: A new Replayer instance or an error if the file cannot be read.

Example:

func replayInteractions() {
    rep, err := httpreplay.NewReplayer("recordings/test.replay")
    if err != nil {
        log.Fatal(err)
    }
    defer rep.Close()

    // Retrieve initial state
    var timeNow time.Time
    if err := timeNow.UnmarshalBinary(rep.Initial()); err != nil {
        log.Fatal(err)
    }

    // Get HTTP client for replay
    ctx := context.Background()
    client, err := rep.Client(ctx)
    if err != nil {
        log.Fatal(err)
    }

    // Use client to make the same API calls
    // These will return recorded responses
    resp, err := client.Get("https://example.googleapis.com/api/resource")
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()
}

Replayer Methods

Client

func (r *Replayer) Client(ctx context.Context) (*http.Client, error)

Returns an HTTP client for replaying. The client does not need to be configured with credentials for authenticating to a server, since it never contacts a real backend.

Parameters:

• ctx: Context for client creation

Returns: An HTTP client configured for replay, or an error.

Example:

client, err := rep.Client(ctx)
if err != nil {
    log.Fatal(err)
}

// Use client to make API calls that will be served from the recording
resp, err := client.Get("https://example.googleapis.com/api/resource")

Initial

func (r *Replayer) Initial() []byte

Returns the initial state saved by the Recorder. Use this to restore random seeds, timestamps, or other values that were used during recording.

Example:

// During recording, a timestamp was saved
rec, _ := httpreplay.NewRecorder("test.replay", []byte("2024-01-15"))
// ... recording happens ...
rec.Close()

// During replay, retrieve the timestamp
rep, _ := httpreplay.NewReplayer("test.replay")
timestamp := string(rep.Initial())  // "2024-01-15"

IgnoreHeader

func (r *Replayer) IgnoreHeader(h string)

Configures the replayer to not use header h when matching requests against recorded interactions. Use this for headers that may differ between recording and replay but should not affect matching.

Example:

rep.IgnoreHeader("User-Agent")
rep.IgnoreHeader("X-Request-ID")

Close

func (r *Replayer) Close() error

Closes the replayer and releases associated resources.

Example:

rep, err := httpreplay.NewReplayer("test.replay")
if err != nil {
    log.Fatal(err)
}
defer rep.Close()

Package-Level Functions

DebugHeaders

func DebugHeaders()

Enables debug mode to help determine whether a header should be ignored. When enabled, if requests have the same method, URL, and body but differ in a header, the first mismatched header is logged. This is useful for debugging replay failures caused by header mismatches.

Example:

// Enable debug logging before creating replayer
httpreplay.DebugHeaders()

rep, _ := httpreplay.NewReplayer("test.replay")
// Now header mismatches will be logged

Supported

func Supported() bool

Reports whether httpreplay is supported in the current version of Go. For Go 1.8 and above, the answer is always true.

Example:

if !httpreplay.Supported() {
    log.Fatal("httpreplay requires Go 1.8 or higher")
}

Complete Example: Testing with Record/Replay

package myapi_test

import (
    "context"
    "flag"
    "io"
    "log"
    "net/http"
    "os"
    "testing"
    "time"

    "cloud.google.com/go/httpreplay"
    "google.golang.org/api/option"
)

var (
    record = flag.Bool("record", false, "record HTTP interactions")
)

func TestAPICall(t *testing.T) {
    ctx := context.Background()
    var client *http.Client
    var err error

    if *record {
        // Recording mode
        timeNow := time.Now()
        initialBytes, _ := timeNow.MarshalBinary()

        rec, err := httpreplay.NewRecorder("testdata/api.replay", initialBytes)
        if err != nil {
            t.Fatal(err)
        }
        defer rec.Close()

        // Configure recorder
        rec.ClearHeaders("Authorization")
        rec.RemoveQueryParams("timestamp")

        client, err = rec.Client(ctx,
            option.WithCredentialsFile(os.Getenv("GOOGLE_APPLICATION_CREDENTIALS")))
        if err != nil {
            t.Fatal(err)
        }
    } else {
        // Replay mode
        rep, err := httpreplay.NewReplayer("testdata/api.replay")
        if err != nil {
            t.Fatal(err)
        }
        defer rep.Close()

        // Restore initial state
        var timeNow time.Time
        if err := timeNow.UnmarshalBinary(rep.Initial()); err != nil {
            t.Fatal(err)
        }

        client, err = rep.Client(ctx)
        if err != nil {
            t.Fatal(err)
        }
    }

    // Make API call (recorded or replayed)
    resp, err := client.Get("https://example.googleapis.com/api/resource")
    if err != nil {
        t.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        t.Errorf("Expected status 200, got %d", resp.StatusCode)
    }

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        t.Fatal(err)
    }

    // Verify response
    if len(body) == 0 {
        t.Error("Expected non-empty response body")
    }
}

To use this test:

# Record interactions (requires credentials and network)
go test -record -v

# Replay interactions (no credentials or network needed)
go test -v

Best Practices

1. Store recordings in version control: Commit replay files to ensure consistent test behavior across environments

2. Use initial state for time-dependent tests: Store timestamps, random seeds, or other initial values that affect test behavior

3. Clear or remove sensitive data: Use ClearHeaders, RemoveRequestHeaders, ClearQueryParams, and RemoveQueryParams to avoid storing secrets in replay files

4. Enable debug headers during development: Use DebugHeaders() to troubleshoot replay matching issues

5. Separate record and replay modes: Use command-line flags or environment variables to control whether tests record or replay

6. Re-record when APIs change: Update recordings when the API you're testing changes to avoid stale test data

7. Test both modes: Periodically run tests in recording mode to ensure your application still works with the real API