or run

tessl search

tessl/golang-cloud-google-com--go--bigquery

tessl install tessl/golang-cloud-google-com--go--bigquery@1.72.0

Google Cloud BigQuery client library providing comprehensive Go APIs for querying, loading data, managing datasets and tables, streaming inserts, and accessing BigQuery's ecosystem of services including Storage, Analytics Hub, Data Transfer, and Migration APIs

Client Setup and Configuration

This document covers creating and configuring BigQuery clients, including authentication, project settings, and advanced client options.

Client Creation

Basic Client Creation

func NewClient(ctx context.Context, projectID string, opts ...option.ClientOption) (*Client, error)

Create a BigQuery client for a specific project:

ctx := context.Background()
client, err := bigquery.NewClient(ctx, "my-project-id")
if err != nil {
    // Handle error
}
defer client.Close()

Auto-Detecting Project ID

Use the DetectProjectID constant to automatically detect the project ID from credentials:

const DetectProjectID = "*detect-project-id*"

client, err := bigquery.NewClient(ctx, bigquery.DetectProjectID)

This extracts the project ID from:

Provided credentials (if they contain project information)
Application Default Credentials (ADC)
Service account JSON key files

Note: JWT-based credentials do not contain project ID and cannot be auto-detected.

Client Type

type Client struct {
    // Location, if set, will be used as the default location for all subsequent
    // dataset creation and job operations. A location specified directly in one of
    // those operations will override this value.
    Location string
}

The Location field sets the default geographic location for dataset and job operations. Common locations include:

"US" - Multi-region in the United States
"EU" - Multi-region in the European Union
"us-central1" - Iowa, USA
"europe-west1" - Belgium
"asia-northeast1" - Tokyo, Japan

Authentication Options

Using Service Account Key File

import "google.golang.org/api/option"

client, err := bigquery.NewClient(ctx, projectID,
    option.WithCredentialsFile("/path/to/service-account-key.json"))

Using Credentials JSON

client, err := bigquery.NewClient(ctx, projectID,
    option.WithCredentialsJSON([]byte(`{...}`)))

Using Access Token

import "golang.org/x/oauth2"

tokenSource := oauth2.StaticTokenSource(&oauth2.Token{
    AccessToken: "your-access-token",
})

client, err := bigquery.NewClient(ctx, projectID,
    option.WithTokenSource(tokenSource))

Using Application Default Credentials (ADC)

// No option needed - this is the default
client, err := bigquery.NewClient(ctx, projectID)

ADC automatically finds credentials in this order:

GOOGLE_APPLICATION_CREDENTIALS environment variable
User credentials from gcloud auth application-default login
GCE/GKE/Cloud Run metadata server

Custom OAuth2 Scopes

const Scope = "https://www.googleapis.com/auth/bigquery"

Specify custom scopes:

client, err := bigquery.NewClient(ctx, projectID,
    option.WithScopes(
        "https://www.googleapis.com/auth/bigquery",
        "https://www.googleapis.com/auth/cloud-platform",
    ))

Client Configuration Options

Setting Default Location

client, err := bigquery.NewClient(ctx, projectID)
if err != nil {
    return err
}
client.Location = "US"

Custom HTTP Client

import "net/http"

httpClient := &http.Client{
    Timeout: 30 * time.Second,
}

client, err := bigquery.NewClient(ctx, projectID,
    option.WithHTTPClient(httpClient))

Custom Endpoint

client, err := bigquery.NewClient(ctx, projectID,
    option.WithEndpoint("https://custom-bigquery-endpoint.example.com"))

Request Options

import "google.golang.org/api/googleapi"

client, err := bigquery.NewClient(ctx, projectID,
    option.WithGRPCDialOption(/* gRPC options */))

User Agent

client, err := bigquery.NewClient(ctx, projectID,
    option.WithUserAgent("my-application/1.0"))

Advanced Client Options

Job Creation Mode

Control how queries are executed for better performance:

func WithDefaultJobCreationMode(mode JobCreationMode) option.ClientOption

type JobCreationMode int

const (
    // JobCreationModeRequired requires a job to be created for all queries
    JobCreationModeRequired JobCreationMode = iota

    // JobCreationModeOptional allows the API to decide whether to create a job
    // This can provide performance improvements for queries with small result sets
    JobCreationModeOptional
)

client, err := bigquery.NewClient(ctx, projectID,
    bigquery.WithDefaultJobCreationMode(bigquery.JobCreationModeOptional))

Storage Read API

Enable the Storage Read API for faster data retrieval:

func (c *Client) EnableStorageReadClient(ctx context.Context, opts ...option.ClientOption) error

err := client.EnableStorageReadClient(ctx)
if err != nil {
    return err
}

Note: Once enabled, pagination methods like PageInfo().Token and RowIterator.StartIndex are not supported. Calling this method twice will return an error.

Client Methods

Closing the Client

func (c *Client) Close() error

Close releases resources held by the client. It should be called when the client is no longer needed:

defer client.Close()

It need not be called at program exit.

Getting Project ID

func (c *Client) Project() string

Returns the project ID used by the client, whether explicitly specified or auto-detected:

projectID := client.Project()

Random Number Generation

For testing or reproducible behavior, seed the internal random number generator used for job and insert IDs:

func Seed(s int64)

bigquery.Seed(12345) // Must be called before creating clients
client, err := bigquery.NewClient(ctx, projectID)

Complete Setup Example

package main

import (
    "context"
    "log"
    "time"

    "cloud.google.com/go/bigquery"
    "google.golang.org/api/option"
)

func setupClient() (*bigquery.Client, error) {
    ctx := context.Background()

    // Create client with custom options
    client, err := bigquery.NewClient(
        ctx,
        "my-project-id",
        option.WithCredentialsFile("/path/to/key.json"),
        bigquery.WithDefaultJobCreationMode(bigquery.JobCreationModeOptional),
    )
    if err != nil {
        return nil, err
    }

    // Set default location
    client.Location = "US"

    // Enable Storage Read API for faster reads
    if err := client.EnableStorageReadClient(ctx); err != nil {
        client.Close()
        return nil, err
    }

    return client, nil
}

func main() {
    client, err := setupClient()
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    // Use client for BigQuery operations
    log.Printf("Connected to project: %s", client.Project())
}

Error Handling

Errors from BigQuery operations often contain detailed information:

type Error struct {
    Location string
    Message  string
    Reason   string
}

func (e Error) Error() string

type MultiError []error

func (m MultiError) Error() string

MultiError contains multiple related errors. This is typically returned when multiple operations fail:

// MultiError is returned by streaming insert operations when multiple rows fail
inserter := table.Inserter()
err := inserter.Put(ctx, rows)
if err != nil {
    if multiErr, ok := err.(bigquery.PutMultiError); ok {
        fmt.Printf("%d row(s) failed to insert\n", len(multiErr))
        for _, rowErr := range multiErr {
            fmt.Printf("Row %d failed: %v\n", rowErr.RowIndex, rowErr.Errors)
        }
    }
}

The MultiError.Error() method returns a formatted string:

With 0 errors: "(0 errors)"
With 1 error: Returns that error's message
With 2 errors: Returns first error plus "(and 1 other error)"
With 3+ errors: Returns first error plus "(and N other errors)"

Use type assertion to access detailed error information:

import (
    "errors"
    "google.golang.org/api/googleapi"
)

_, err := bigquery.NewClient(ctx, projectID)
if err != nil {
    var apiErr *googleapi.Error
    if errors.As(err, &apiErr) {
        log.Printf("API Error: Code=%d, Message=%s", apiErr.Code, apiErr.Message)
    }
    return err
}

Common error codes:

400: Bad request (invalid parameters)
401: Authentication failed
403: Permission denied
404: Resource not found
409: Resource already exists or conflict
500: Internal server error
503: Service unavailable

Quotas and Limits

Be aware of BigQuery quotas and limits:

https://cloud.google.com/bigquery/quotas

Unstructured googleapi.Error responses often indicate quota exceeded. Consider:

Implementing exponential backoff
Reducing request frequency
Requesting quota increases
Using reservations for guaranteed capacity

Version

tessl/golang-cloud-google-com--go--bigquery

client-setup.mddocs/

Client Setup and Configuration

Client Creation

Basic Client Creation

Auto-Detecting Project ID

Client Type

Authentication Options

Using Service Account Key File

Using Credentials JSON

Using Access Token

Using Application Default Credentials (ADC)

Custom OAuth2 Scopes

Client Configuration Options

Setting Default Location

Custom HTTP Client

Custom Endpoint

Request Options

User Agent

Advanced Client Options

Job Creation Mode

Storage Read API

Client Methods

Closing the Client

Getting Project ID

Random Number Generation

Complete Setup Example

Error Handling

Quotas and Limits

Version

tessl/golang-cloud-google-com--go--bigquery

client-setup.md.css-3qkkll{font-size:var(--chakra-font-sizes-sm);font-weight:var(--chakra-font-weights-normal);color:var(--chakra-colors-gray-300);}docs/

Client Setup and Configuration

Client Creation

Basic Client Creation

Auto-Detecting Project ID

Client Type

Authentication Options

Using Service Account Key File

Using Credentials JSON

Using Access Token

Using Application Default Credentials (ADC)

Custom OAuth2 Scopes

Client Configuration Options

Setting Default Location

Custom HTTP Client

Custom Endpoint

Request Options

User Agent

Advanced Client Options

Job Creation Mode

Storage Read API

Client Methods

Closing the Client

Getting Project ID

Random Number Generation

Complete Setup Example

Error Handling

Quotas and Limits

client-setup.mddocs/