Architecture

Back to Index

This document describes the architectural components and design patterns of k8s.io/client-go.

Overview

k8s.io/client-go is organized into layers, from high-level typed clients down to low-level HTTP transport:

┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                         │
│  (Your controllers, operators, CLI tools, applications)      │
└────────────────────┬────────────────────────────────────────┘
                     │
┌────────────────────┴────────────────────────────────────────┐
│              High-Level Client Interfaces                    │
│  • Typed Clientsets (kubernetes.Interface)                   │
│  • Dynamic Client (dynamic.Interface)                        │
│  • Metadata Client (metadata.Interface)                      │
│  • Discovery Client (discovery.Interface)                    │
└────────────────────┬────────────────────────────────────────┘
                     │
┌────────────────────┴────────────────────────────────────────┐
│            Controller Infrastructure                         │
│  • Informers (shared caching & watching)                     │
│  • Listers (cached read operations)                          │
│  • Work Queues (rate-limited event processing)               │
│  • Reflectors (watch & cache synchronization)                │
└────────────────────┬────────────────────────────────────────┘
                     │
┌────────────────────┴────────────────────────────────────────┐
│              REST Client Layer                               │
│  • RESTClient (low-level HTTP operations)                    │
│  • Request Builder (fluent API construction)                 │
│  • Content Negotiation (JSON, Protobuf, CBOR)               │
└────────────────────┬────────────────────────────────────────┘
                     │
┌────────────────────┴────────────────────────────────────────┐
│             Configuration & Auth                             │
│  • rest.Config (connection configuration)                    │
│  • Transport (TLS, auth, rate limiting)                      │
│  • Auth Providers (exec, OIDC, certificates)                 │
└────────────────────┬────────────────────────────────────────┘
                     │
┌────────────────────┴────────────────────────────────────────┐
│                HTTP Transport Layer                          │
│  • http.RoundTripper (Go standard library)                   │
│  • Custom wrappers (retry, metrics, logging)                 │
└──────────────────────────────────────────────────────────────┘
                     │
                     ▼
          Kubernetes API Server

Core Components

1. Configuration (rest.Config)

The rest.Config struct is the foundation for all client creation:

type Config struct {
    // Connection
    Host    string  // API server URL
    APIPath string  // API path prefix

    // Authentication
    BearerToken     string              // Token auth
    Username        string              // Basic auth
    Password        string              // Basic auth
    TLSClientConfig TLSClientConfig     // Certificate auth
    Impersonate     ImpersonationConfig // Impersonation
    AuthProvider    *AuthProviderConfig // Plugin auth
    ExecProvider    *ExecConfig         // External credential provider

    // Rate Limiting
    QPS   float32 // Queries per second
    Burst int     // Burst allowance

    // Transport
    Transport    http.RoundTripper  // Custom transport
    WrapTransport WrapperFunc       // Transport wrapper

    // Timeouts
    Timeout time.Duration
    Dial    func(ctx context.Context, network, address string) (net.Conn, error)
}

Sources:

• rest.InClusterConfig() - From service account in pod
• clientcmd.BuildConfigFromFlags() - From kubeconfig file
• clientcmd.RESTConfigFromKubeConfig() - From kubeconfig bytes
• Manual construction - For custom scenarios

2. Clientsets (Typed Clients)

Purpose: Strongly-typed access to Kubernetes APIs

Structure:

kubernetes.Clientset
├── CoreV1() → CoreV1Interface
│   ├── Pods(namespace) → PodInterface
│   ├── Services(namespace) → ServiceInterface
│   ├── ConfigMaps(namespace) → ConfigMapInterface
│   └── ...
├── AppsV1() → AppsV1Interface
│   ├── Deployments(namespace) → DeploymentInterface
│   ├── StatefulSets(namespace) → StatefulSetInterface
│   └── ...
├── BatchV1() → BatchV1Interface
├── NetworkingV1() → NetworkingV1Interface
└── ... (40+ API groups)

Key Characteristics:

• Compile-time type safety
• Generated from Kubernetes OpenAPI specs
• One interface per resource type
• Supports all CRUD operations + Watch + Patch
• Separate status subresource methods

3. Dynamic Client

Purpose: Untyped access to any Kubernetes resource including CRDs

Architecture:

dynamic.DynamicClient
└── Resource(GVR) → NamespaceableResourceInterface
    ├── Namespace(ns) → ResourceInterface
    │   ├── Create()
    │   ├── Update()
    │   ├── Get()
    │   ├── List()
    │   ├── Delete()
    │   ├── Watch()
    │   ├── Patch()
    │   └── Apply()
    └── (cluster-scoped operations)

Data Format: unstructured.Unstructured (map[string]interface{})

Use Cases:

• Custom Resource Definitions (CRDs)
• Generic tooling (kubectl-like tools)
• Resources unknown at compile time

4. RESTClient

Purpose: Low-level HTTP client for Kubernetes API

Request Builder Pattern:

restClient.
    Get().                              // HTTP method
    Namespace("default").               // Namespace
    Resource("pods").                   // Resource type
    Name("my-pod").                     // Resource name
    VersionedParams(&opts, codec).      // Query parameters
    Timeout(30 * time.Second).          // Request timeout
    Do(ctx).                            // Execute
    Into(&pod)                          // Decode response

Capabilities:

• Fine-grained control over requests
• Custom resource types
• Subresources (status, scale, exec, logs, portforward)
• Streaming operations

5. Informer Architecture

Purpose: Efficient caching and watching of Kubernetes resources

Components:

SharedInformerFactory
└── Resource Informers
    ├── Reflector
    │   ├── ListAndWatch (API calls)
    │   └── Sync to Store
    ├── Store/Indexer
    │   ├── In-memory cache
    │   └── Index functions
    ├── Controller
    │   └── Process Deltas
    └── Event Handlers
        ├── OnAdd
        ├── OnUpdate
        └── OnDelete

Flow:

1. Reflector: Watches API server, maintains ResourceVersion
2. DeltaFIFO: Queue of changes (Added, Modified, Deleted, Sync)
3. Store/Indexer: In-memory cache with indexing
4. Event Handlers: User callbacks for resource changes
5. Work Queue: Optional rate-limited processing queue

Key Features:

• Single watch connection shared across consumers
• Automatic resync at configurable intervals
• Handles watch errors and reconnection
• Provides consistent cache snapshots
• Efficient filtering and indexing

6. Listers

Purpose: Read-only interface to informer caches

Architecture:

PodLister (interface)
├── List(selector) → []*v1.Pod
└── Pods(namespace) → PodNamespaceLister
    ├── List(selector) → []*v1.Pod
    └── Get(name) → *v1.Pod

Benefits:

• No API server load for reads
• Consistent cache reads
• Fast local lookups
• Works across namespaces

7. Work Queues

Purpose: Reliable, rate-limited event processing

Queue Hierarchy:

Interface (basic FIFO)
└── DelayingInterface (delayed items)
    └── RateLimitingInterface (rate limiting with backoff)
        └── TypedRateLimitingQueue[T] (type-safe)

Patterns:

queue := workqueue.NewTypedRateLimitingQueue(
    workqueue.DefaultTypedControllerRateLimiter[string]())

// Add item
queue.Add(key)

// Process items
key, shutdown := queue.Get()
defer queue.Done(key)

// Success - remove from queue
queue.Forget(key)

// Failure - retry with backoff
queue.AddRateLimited(key)

Rate Limiters:

• ItemExponentialFailureRateLimiter: Exponential backoff per item
• ItemFastSlowRateLimiter: Fast retries then slow
• BucketRateLimiter: Token bucket algorithm
• MaxOfRateLimiter: Combines multiple limiters

8. Apply Configurations

Purpose: Server-Side Apply (SSA) support

Design:

• Builder pattern for constructing objects
• All fields are pointers (optional)
• Only specify fields you manage
• Server tracks field ownership per field manager

Architecture:

ApplyConfiguration (interface)
└── Generated per API type
    ├── With<Field>() methods (builders)
    └── Extract<Type>() functions (from existing)

Field Management:

Resource
├── Field A (managed by controller-1)
├── Field B (managed by controller-2)
└── Field C (managed by controller-1)

Multiple controllers can safely manage different fields of the same resource.

Package Organization

Generated Packages

Typed Clients: 110+ packages

• kubernetes/typed/<group>/<version>
• Example: kubernetes/typed/core/v1
• Contains interfaces for each resource type

Informers: 72 packages

• informers/<group>/<version>
• Example: informers/apps/v1
• Shared informer factories per API group

Listers: 52 packages

• listers/<group>/<version>
• Example: listers/core/v1
• Read-only cache interfaces

Apply Configurations: 53 packages

• applyconfigurations/<group>/<version>
• Example: applyconfigurations/apps/v1
• SSA builder types

Core Infrastructure Packages

rest: REST client and configuration kubernetes: Main clientset entry point dynamic: Dynamic (untyped) client discovery: API discovery metadata: Metadata-only client

Controller Infrastructure

tools/cache: Core controller primitives

• Informers, reflectors, stores, indexers
• SharedIndexInformer, Store, Indexer

util/workqueue: Work queue implementations

• Rate-limiting, delaying, typed queues

Utilities

tools/clientcmd: Kubeconfig loading tools/leaderelection: Leader election tools/record: Event recording tools/remotecommand: Pod exec tools/portforward: Port forwarding tools/pager: Large list pagination transport: HTTP transport customization util/retry: Retry utilities util/flowcontrol: Rate limiting util/cert: Certificate utilities

Design Patterns

1. Interface Segregation

All clients expose interfaces, not concrete types:

// Interface (not concrete Clientset)
type Interface interface {
    CoreV1() corev1.CoreV1Interface
    AppsV1() appsv1.AppsV1Interface
    // ...
}

Benefits:

• Easy mocking for tests
• Dependency injection
• Fake implementations for testing

2. Namespace Scoping

Resources are scoped to namespaces explicitly:

// Namespace-scoped
clientset.CoreV1().Pods("default").Get(...)

// Cluster-scoped
clientset.CoreV1().Nodes().Get(...)

// All namespaces
clientset.CoreV1().Pods("").List(...)

3. Builder Pattern

Fluent API for request construction:

restClient.
    Post().
    Resource("pods").
    Namespace("default").
    Body(pod).
    Do(ctx)

4. Shared Informers

Multiple controllers share single watch connection:

factory := informers.NewSharedInformerFactory(clientset, resyncPeriod)

// Multiple consumers of same informer
podInformer := factory.Core().V1().Pods()
controller1.usePodInformer(podInformer)
controller2.usePodInformer(podInformer)

// Single watch connection to API server
factory.Start(stopCh)

5. Typed vs Untyped

Typed (preferred when possible):

• Compile-time safety
• IDE autocomplete
• Clear API contracts

Untyped (when necessary):

• Custom resources
• Generic tooling
• Unknown resource types

6. Event Handlers + Work Queues

Decouple event notification from processing:

// Event Handler: Add to queue
informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
    AddFunc: func(obj interface{}) {
        key, _ := cache.MetaNamespaceKeyFunc(obj)
        queue.Add(key)  // Fast, non-blocking
    },
})

// Worker: Process from queue
func worker() {
    for processNextItem() {
    }
}

func processNextItem() bool {
    key, quit := queue.Get()
    if quit {
        return false
    }
    defer queue.Done(key)

    // Heavy processing here
    err := syncHandler(key)
    handleErr(err, key)
    return true
}

Benefits:

• Non-blocking event handlers
• Rate limiting and backoff
• Batch processing
• Graceful shutdown

Threading Model

Informers

• Reflector goroutine: Watches API server, updates DeltaFIFO
• Controller goroutine: Processes DeltaFIFO, updates Store, calls handlers
• User goroutines: Event handlers run synchronously in controller goroutine

Important: Event handlers should be fast and non-blocking. Use work queues for heavy processing.

Work Queues

• Producer goroutines: Add items (event handlers, reconcilers)
• Worker goroutines: Process items (Get, Done, Forget)
• Queue internals: Thread-safe, uses mutexes and condition variables

Caching Strategy

Informer Cache

Cache Invalidation: None - relies on watches for updates Resync: Periodic full list to detect missed events Consistency: Eventually consistent with API server

Cache Miss Handling:

• Informer doesn't have "cache miss"
• Either object is in cache or it doesn't exist (from informer's view)
• If object might have been created recently, query API server directly

Best Practices

1. Use Listers for Reads: Don't call Get/List on clientset repeatedly
2. Use Clientset for Writes: Always write through API server
3. Check Cache Sync: Wait for HasSynced() before using cache
4. Handle Stale Reads: Cache might be seconds behind API server

Resource Version Tracking

Purpose: Optimistic concurrency control

Flow:

1. Client lists/watches with ResourceVersion=0
2. API server returns current state + ResourceVersion
3. Client stores ResourceVersion
4. Client watches from ResourceVersion (gets incremental updates)
5. On update, client provides ResourceVersion
6. API server checks if ResourceVersion matches
7. If mismatch: Conflict error (resource was modified)

Informer Behavior:

• Stores latest ResourceVersion
• On watch error, resumes from last ResourceVersion
• On reconnect, uses latest ResourceVersion

Performance Considerations

API Server Load

Don't:

• Poll with Get/List in loops
• Create separate watches for each object
• Ignore rate limiting

Do:

• Use informers for watching
• Use listers for repeated reads
• Configure appropriate QPS/Burst
• Use label/field selectors

Memory Usage

Informer Memory:

• Stores all objects of watched type
• Cluster with 10,000 pods ≈ 50-100MB per pod informer
• Use namespace filtering when possible
• Use field selectors to reduce cache size

Work Queue Memory:

• Stores keys (strings), not full objects
• Bounded by queue length
• Use rate limiting to prevent unbounded growth

Watch Efficiency

Shared Informers:

• Single watch connection per resource type
• Shared across all consumers
• Dramatically reduces API server load

Watch Semantics:

• Long-lived HTTP connections
• Server push model (efficient)
• Automatic reconnection on errors

Testing Strategies

Fake Clientset

import "k8s.io/client-go/kubernetes/fake"

clientset := fake.NewSimpleClientset(initialObjects...)

Provides:

• In-memory implementation
• No API server needed
• Immediate operations (no async)
• Reactor pattern for custom behavior

Fake Informers

import "k8s.io/client-go/informers/fake"

// Not commonly used - prefer using real informers with fake clientset

Integration Testing

Use real clientset with test cluster (kind, k3d, etc.)

Security Architecture

Authentication Methods

1. Certificates: Mutual TLS with client certificates
2. Bearer Tokens: Static tokens or service account tokens
3. Basic Auth: Username/password (deprecated)
4. Exec Plugins: External credential providers (kubectl plugins)
5. OIDC: OpenID Connect tokens
6. Auth Providers: Plugin system (Azure, GCP, etc.)

Transport Security

TLS Configuration:

• Server certificate validation
• Client certificate authentication
• Custom CA certificates
• Insecure skip verify (testing only)

Rate Limiting:

• Client-side: QPS and Burst limits
• Prevents overwhelming API server
• Protects against accidental DoS

Impersonation

config.Impersonate = rest.ImpersonationConfig{
    UserName: "system:serviceaccount:default:my-sa",
    Groups:   []string{"system:authenticated"},
}

Used for testing RBAC policies without multiple credentials.

Extensibility Points

1. Custom Transport

config.WrapTransport = func(rt http.RoundTripper) http.RoundTripper {
    return &customRoundTripper{
        base: rt,
        // Add logging, metrics, tracing, etc.
    }
}

2. Custom Codec

config.ContentConfig = rest.ContentConfig{
    ContentType:  "application/json",
    GroupVersion: &schema.GroupVersion{...},
    Negotiator:   runtime.ClientNegotiator{...},
}

3. Custom Auth Provider

rest.RegisterAuthProviderPlugin("custom", func(...) (rest.AuthProvider, error) {
    // Custom authentication logic
})

4. Custom Rate Limiter

config.RateLimiter = flowcontrol.NewTokenBucketRateLimiter(qps, burst)

5. Custom Indexers

indexers := cache.Indexers{
    "byOwner": func(obj interface{}) ([]string, error) {
        // Custom indexing logic
    },
}

informer := cache.NewSharedIndexInformer(..., indexers)

Comparison with Other Clients

client-go vs kubectl

kubectl:

• CLI tool
• Uses client-go internally
• Higher-level abstractions (apply, rollout, etc.)
• Human-friendly output

client-go:

• Library for Go programs
• Lower-level API access
• Programmatic control
• Building block for tools

client-go vs controller-runtime

controller-runtime (builds on client-go):

• Higher-level controller framework
• Opinionated patterns
• Used by Kubebuilder/Operator SDK
• Manager, reconcilers, webhooks

client-go:

• Lower-level primitives
• More flexibility
• Requires more boilerplate
• Direct API access

Version Compatibility

Kubernetes Version Skew:

• client-go version should match cluster minor version
• ±1 minor version tolerance
• Example: client-go v0.35.0 works with Kubernetes 1.34, 1.35, 1.36

Go Version:

• Requires Go 1.22+ for v0.35.0
• Uses generics (introduced in Go 1.18)

Breaking Changes:

• Major version bumps (v1.x to v2.x)
• API deprecations follow Kubernetes deprecation policy
• Beta APIs may change between versions