tessl/golang-github-com-rs-zerolog

Zero Allocation JSON Logger - High-performance structured logging library for Go

Overview

Eval results

Files

Diode Writer

Name: tessl/golang-github-com-rs-zerolog
Author: tessl

The diode subpackage provides a thread-safe, lock-free, non-blocking writer implementation using a ring buffer (diode pattern). This is ideal for high-throughput applications where logging should never block the main execution path.

Package Imports

import (
    "time"
    "github.com/rs/zerolog/diode"
)

Overview

A diode writer wraps any io.Writer and makes writes non-blocking by using a lock-free ring buffer. When the buffer is full, messages are dropped rather than blocking. This ensures logging never slows down your application.

Key features:

Non-blocking writes - Never blocks the caller
Lock-free - Uses atomic operations, no mutex contention
Thread-safe - Safe for concurrent use
Lossy - Drops messages when buffer is full (by design)
Zero allocation - Pooled buffers for efficiency

Writer Type

type Writer struct {
    // unexported fields
}

Creating a Diode Writer

// Create non-blocking writer with ring buffer
func NewWriter(w io.Writer, size int, pollInterval time.Duration, f Alerter) Writer

Parameters:

w io.Writer - Underlying writer (e.g., os.File, network socket)
size int - Ring buffer size (number of messages to buffer)
pollInterval time.Duration - Polling interval (0 for waiter mode)
f Alerter - Callback for dropped messages (nil to ignore)

Returns: Configured diode writer

Example:

file, _ := os.Create("app.log")

wr := diode.NewWriter(file, 1000, 10*time.Millisecond, func(missed int) {
    fmt.Printf("Dropped %d log messages\n", missed)
})
defer wr.Close()

logger := zerolog.New(wr).With().Timestamp().Logger()

Alerter Callback

// Callback for dropped messages
type Alerter func(missed int)

The alerter is called when messages are dropped due to a full buffer. It receives the count of dropped messages.

Example:

// Log dropped messages to stderr
alerter := func(missed int) {
    fmt.Fprintf(os.Stderr, "WARNING: Dropped %d log messages\n", missed)
}

wr := diode.NewWriter(file, 1000, 10*time.Millisecond, alerter)

Example with metrics:

alerter := func(missed int) {
    metrics.DroppedLogs.Add(float64(missed))
    fmt.Fprintf(os.Stderr, "Dropped %d logs\n", missed)
}

wr := diode.NewWriter(file, 1000, 10*time.Millisecond, alerter)

Nil alerter (silent drops):

// Silently drop messages without notification
wr := diode.NewWriter(file, 1000, 10*time.Millisecond, nil)

Writer Methods

// Write implements io.Writer (non-blocking)
func (dw Writer) Write(p []byte) (n int, err error)

// Close stops polling and closes underlying writer if io.Closer
func (dw Writer) Close() error

Example:

wr := diode.NewWriter(file, 1000, 10*time.Millisecond, nil)
defer wr.Close()

logger := zerolog.New(wr)
logger.Info().Msg("message")  // Non-blocking write

Polling Modes

The diode writer has two modes for reading from the ring buffer:

Poller Mode (pollInterval > 0)

Polls the buffer at regular intervals. Better for steady, high-throughput logging.

// Poll every 10 milliseconds
wr := diode.NewWriter(file, 1000, 10*time.Millisecond, nil)

Characteristics:

Polls at fixed interval regardless of data
Lower CPU usage for sporadic logs
Slight latency (up to pollInterval)
Good for steady throughput

Waiter Mode (pollInterval = 0)

Blocks waiting for data in the buffer. Better for low-latency logging.

// Wait for data (no polling)
wr := diode.NewWriter(file, 1000, 0, nil)

Characteristics:

Wakes immediately when data available
Lower latency
Slightly higher CPU usage
Good for real-time logging

Usage Examples

Basic Usage

package main

import (
    "os"
    "time"

    "github.com/rs/zerolog"
    "github.com/rs/zerolog/diode"
)

func main() {
    file, err := os.Create("app.log")
    if err != nil {
        panic(err)
    }

    // Non-blocking writer with 1000 message buffer
    wr := diode.NewWriter(file, 1000, 10*time.Millisecond, func(missed int) {
        fmt.Fprintf(os.Stderr, "Dropped %d messages\n", missed)
    })
    defer wr.Close()

    logger := zerolog.New(wr).With().Timestamp().Logger()

    // Log without blocking
    for i := 0; i < 10000; i++ {
        logger.Info().Int("i", i).Msg("iteration")
    }
}

High-Throughput Logging

// Large buffer for high throughput
wr := diode.NewWriter(file, 10000, 100*time.Millisecond, func(missed int) {
    // Track drops with metrics
    droppedLogsCounter.Add(float64(missed))
})
defer wr.Close()

logger := zerolog.New(wr)

// Heavy logging workload
for i := 0; i < 1000000; i++ {
    logger.Debug().Int("iteration", i).Msg("processing")
}

Low-Latency Logging

// Waiter mode for low latency
wr := diode.NewWriter(file, 500, 0, func(missed int) {
    fmt.Fprintf(os.Stderr, "Dropped %d messages\n", missed)
})
defer wr.Close()

logger := zerolog.New(wr)

// Logs written with minimal latency
logger.Info().Msg("low latency message")

Multiple Writers

// Combine diode writer with other writers
file, _ := os.Create("app.log")
diodeWriter := diode.NewWriter(file, 1000, 10*time.Millisecond, nil)
defer diodeWriter.Close()

// Write to both console (blocking) and file (non-blocking)
multi := zerolog.MultiLevelWriter(
    os.Stdout,
    diodeWriter,
)

logger := zerolog.New(multi)

With Monitoring

var droppedTotal int64

alerter := func(missed int) {
    total := atomic.AddInt64(&droppedTotal, int64(missed))

    // Log to stderr
    fmt.Fprintf(os.Stderr, "Dropped %d messages (total: %d)\n", missed, total)

    // Send to monitoring
    metrics.LogDrops.Add(float64(missed))

    // Alert if too many drops
    if missed > 100 {
        sendAlert("High log drop rate: %d", missed)
    }
}

wr := diode.NewWriter(file, 1000, 10*time.Millisecond, alerter)
defer wr.Close()

Network Writer

// Non-blocking network logging
conn, err := net.Dial("tcp", "logserver:514")
if err != nil {
    panic(err)
}

wr := diode.NewWriter(conn, 5000, 50*time.Millisecond, func(missed int) {
    fmt.Fprintf(os.Stderr, "Network congestion: dropped %d logs\n", missed)
})
defer wr.Close()

logger := zerolog.New(wr)

Buffer Sizing

Choose buffer size based on your logging rate and acceptable loss:

Small buffer (100-500):

Low memory footprint
Suitable for low to moderate logging rates
Higher chance of drops during bursts

wr := diode.NewWriter(file, 500, 10*time.Millisecond, alerter)

Medium buffer (1000-5000):

Balanced memory and drop resistance
Suitable for moderate to high logging rates
Handles most burst scenarios

wr := diode.NewWriter(file, 1000, 10*time.Millisecond, alerter)

Large buffer (10000+):

Higher memory footprint
Suitable for very high logging rates
Excellent burst handling

wr := diode.NewWriter(file, 10000, 100*time.Millisecond, alerter)

Performance Tuning

Poll Interval

Short interval (1-10ms):

Lower latency
More frequent polling (higher CPU)
Better for real-time logging

wr := diode.NewWriter(file, 1000, 5*time.Millisecond, nil)

Medium interval (10-100ms):

Balanced latency and CPU usage
Good for most applications

wr := diode.NewWriter(file, 1000, 50*time.Millisecond, nil)

Long interval (100-1000ms):

Higher latency
Lower CPU usage
Good for batch-style logging

wr := diode.NewWriter(file, 1000, 500*time.Millisecond, nil)

Waiter Mode (Zero Latency)

// No polling delay, wake on data
wr := diode.NewWriter(file, 1000, 0, nil)

Best Practices

1. Always Close the Writer

Closing flushes remaining messages and stops the poller:

wr := diode.NewWriter(file, 1000, 10*time.Millisecond, nil)
defer wr.Close()

2. Monitor Dropped Messages

Always provide an alerter to track drops:

// Good - monitor drops
wr := diode.NewWriter(file, 1000, 10*time.Millisecond, func(missed int) {
    metrics.DroppedLogs.Add(float64(missed))
})

// Avoid - silent drops
wr := diode.NewWriter(file, 1000, 10*time.Millisecond, nil)

3. Size Buffer Appropriately

Size buffer for peak load plus headroom:

// Calculate buffer size
peakLogsPerSecond := 10000
bufferSeconds := 2  // Handle 2 seconds of peak load
bufferSize := peakLogsPerSecond * bufferSeconds

wr := diode.NewWriter(file, bufferSize, 10*time.Millisecond, alerter)

4. Use for Hot Paths Only

Only use diode writer where non-blocking is critical:

// Good - hot path needs non-blocking
func handleRequest(w http.ResponseWriter, r *http.Request) {
    logger.Info().Msg("request received")  // Non-blocking
    // ... critical code ...
}

// Avoid - application startup doesn't need non-blocking
func main() {
    logger := zerolog.New(os.Stdout)  // Simple writer is fine
    logger.Info().Msg("starting up")
}

5. Test Under Load

Test with realistic load to tune buffer size:

func TestLoggingUnderLoad(t *testing.T) {
    var dropped int64
    alerter := func(missed int) {
        atomic.AddInt64(&dropped, int64(missed))
    }

    wr := diode.NewWriter(io.Discard, 1000, 10*time.Millisecond, alerter)
    defer wr.Close()

    logger := zerolog.New(wr)

    // Generate load
    for i := 0; i < 100000; i++ {
        logger.Info().Int("i", i).Msg("test")
    }

    wr.Close()

    droppedCount := atomic.LoadInt64(&dropped)
    if droppedCount > 0 {
        t.Logf("Dropped %d messages, consider increasing buffer", droppedCount)
    }
}

6. Don't Use for Critical Logs

Diode writer is lossy by design. Don't use for critical logs:

// Good - audit logs use blocking writer
auditLogger := zerolog.New(auditFile)  // Blocking, reliable

// Regular logs use non-blocking writer
appLogger := zerolog.New(diodeWriter)  // Non-blocking, lossy

When to Use Diode Writer

Use diode writer when:

Logging is in hot path / critical section
High throughput is required (10k+ logs/sec)
Can tolerate occasional message loss
Want to prevent logging from impacting performance
Have lock contention issues with standard writer

Don't use diode writer when:

All logs must be guaranteed to be written
Logging rate is low to moderate
Audit trail or compliance logging
Development/debugging (harder to debug with drops)
Memory is very constrained

Comparison with SyncWriter

Feature	DiodeWriter	SyncWriter
Blocking	Non-blocking	Blocking
Locking	Lock-free	Mutex-locked
Reliability	Lossy	Lossless
Throughput	Very high	High
Latency	Lowest	Low
Use case	Hot paths	General use

Example:

// SyncWriter - blocks but reliable
syncWr := zerolog.SyncWriter(file)
logger := zerolog.New(syncWr)

// DiodeWriter - never blocks but lossy
diodeWr := diode.NewWriter(file, 1000, 10*time.Millisecond, nil)
logger := zerolog.New(diodeWr)

Thread Safety

Diode writer is fully thread-safe and lock-free. Multiple goroutines can safely write concurrently without coordination:

wr := diode.NewWriter(file, 1000, 10*time.Millisecond, nil)
logger := zerolog.New(wr)

// Safe concurrent access
for i := 0; i < 100; i++ {
    go func(id int) {
        logger.Info().Int("worker", id).Msg("processing")
    }(i)
}

Memory Management

Diode writer uses buffer pooling to minimize allocations:

Buffers up to 64 KiB are pooled
Larger buffers are not pooled (GC'd)
Ring buffer size is fixed at creation
No dynamic allocation during writes

Memory usage:

Memory = (buffer_size × average_log_size) + overhead

Example:

// Buffer: 1000 messages
// Avg message size: 200 bytes
// Memory: ~200KB + overhead
wr := diode.NewWriter(file, 1000, 10*time.Millisecond, nil)

Error Handling

Diode writer always returns success from Write(), even if the buffer is full:

n, err := diodeWriter.Write(data)
// n == len(data), err == nil (always)
// Message may be dropped if buffer full

Actual write errors on the underlying writer are not propagated to the caller. Monitor the underlying writer separately if needed.

tessl/golang-github-com-rs-zerolog

diode-writer.mddocs/

Diode Writer

Package Imports

Overview

Writer Type

Creating a Diode Writer

Alerter Callback

Writer Methods

Polling Modes

Poller Mode (pollInterval > 0)

Waiter Mode (pollInterval = 0)

Usage Examples

Basic Usage

High-Throughput Logging

Low-Latency Logging

Multiple Writers

With Monitoring

Network Writer

Buffer Sizing

Performance Tuning

Poll Interval

Waiter Mode (Zero Latency)

Best Practices

1. Always Close the Writer

2. Monitor Dropped Messages

3. Size Buffer Appropriately

4. Use for Hot Paths Only

5. Test Under Load

6. Don't Use for Critical Logs

When to Use Diode Writer

Comparison with SyncWriter

Thread Safety

Memory Management

Error Handling

See Also

tessl/golang-github-com-rs-zerolog

diode-writer.md.css-3qkkll{font-size:var(--chakra-font-sizes-sm);font-weight:var(--chakra-font-weights-normal);color:var(--chakra-colors-gray-300);}docs/

Diode Writer

Package Imports

Overview

Writer Type

Creating a Diode Writer

Alerter Callback

Writer Methods

Polling Modes

Poller Mode (pollInterval > 0)

Waiter Mode (pollInterval = 0)

Usage Examples

Basic Usage

High-Throughput Logging

Low-Latency Logging

Multiple Writers

With Monitoring

Network Writer

Buffer Sizing

Performance Tuning

Poll Interval

Waiter Mode (Zero Latency)

Best Practices

1. Always Close the Writer

2. Monitor Dropped Messages

3. Size Buffer Appropriately

4. Use for Hot Paths Only

5. Test Under Load

6. Don't Use for Critical Logs

When to Use Diode Writer

Comparison with SyncWriter

Thread Safety

Memory Management

Error Handling

See Also

diode-writer.mddocs/