Memory-Mapped File I/O (mmap)

The mmap package provides efficient memory-mapped file access for Go applications. Memory mapping allows you to access file contents as if they were in memory, enabling fast random access and concurrent reading without loading the entire file into RAM.

Package Information

• Package Name: golang.org/x/exp/mmap
• Package Type: Go module (golang.org/x)
• Language: Go
• Installation: go get golang.org/x/exp/mmap

Core Imports

import "golang.org/x/exp/mmap"

Basic Usage

package main

import (
	"fmt"
	"log"
	"golang.org/x/exp/mmap"
)

func main() {
	// Open and memory-map a file
	reader, err := mmap.Open("data.bin")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	// Get file size
	size := reader.Len()
	fmt.Printf("File size: %d bytes\n", size)

	// Access individual bytes
	firstByte := reader.At(0)
	fmt.Printf("First byte: 0x%02x\n", firstByte)

	// Read data from offset
	buf := make([]byte, 10)
	n, err := reader.ReadAt(buf, 0)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Read %d bytes: %v\n", n, buf[:n])
}

Capabilities

File Mapping

Opens and memory-maps a file for reading.

func Open(filename string) (*ReaderAt, error)

Opens the named file and returns a ReaderAt that can be used to access the file's contents through a memory-mapped interface. The file is memory-mapped for reading, allowing efficient random access. An error is returned if the file cannot be opened or memory-mapped.

Sequential and Random Reading

Implements the standard io.ReaderAt interface for reading from arbitrary offsets in the memory-mapped file.

func (r *ReaderAt) ReadAt(p []byte, off int64) (int, error)

Reads bytes from the memory-mapped file starting at the specified offset. It implements the io.ReaderAt interface and can be safely called concurrently from multiple goroutines. Returns the number of bytes read (which may be less than the length of p) and any error encountered.

Direct Byte Access

Provides low-level access to individual bytes in the memory-mapped file.

func (r *ReaderAt) At(i int) byte

Returns the byte at the specified index i in the memory-mapped file. This provides direct, efficient access to individual bytes without allocating buffers.

File Size Query

Returns the total size of the memory-mapped file.

func (r *ReaderAt) Len() int

Returns the length in bytes of the underlying memory-mapped file. This allows you to determine the bounds of the file without external system calls.

Resource Cleanup

Closes the memory-mapped file and releases associated resources.

func (r *ReaderAt) Close() error

Closes the ReaderAt and releases the memory mapping. After Close is called, subsequent calls to ReadAt, At, or Len will fail. It is not safe to call Close concurrently with any reading method; callers must ensure synchronization.

Types

type ReaderAt struct {
	// Has unexported fields.
}

ReaderAt represents a memory-mapped file opened for reading. It provides efficient random access to file contents through memory mapping.

Like any io.ReaderAt, clients can execute parallel ReadAt calls, but it is not safe to call Close and reading methods concurrently. All concurrent ReadAt calls are allowed, but you must coordinate with Close() to ensure thread-safe access.

Methods:

• Open(filename string) (*ReaderAt, error) - Factory function to create a new ReaderAt
• ReadAt(p []byte, off int64) (int, error) - Read bytes from offset (implements io.ReaderAt)
• At(i int) byte - Get individual byte at index
• Len() int - Get file size in bytes
• Close() error - Release resources

Usage Examples

Example 1: Reading a Binary File

package main

import (
	"encoding/binary"
	"fmt"
	"log"
	"golang.org/x/exp/mmap"
)

func main() {
	// Open a binary file
	reader, err := mmap.Open("data.bin")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	// Read a 32-bit integer from offset 0
	buf := make([]byte, 4)
	reader.ReadAt(buf, 0)
	value := binary.LittleEndian.Uint32(buf)
	fmt.Printf("Value at offset 0: %d\n", value)
}

Example 2: Efficient Large File Processing

package main

import (
	"fmt"
	"log"
	"golang.org/x/exp/mmap"
)

func main() {
	// Open a large file efficiently
	reader, err := mmap.Open("large_file.dat")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	// Process file without loading it all into memory
	fileSize := reader.Len()
	chunkSize := 4096

	for offset := int64(0); offset < int64(fileSize); offset += int64(chunkSize) {
		end := offset + int64(chunkSize)
		if end > int64(fileSize) {
			end = int64(fileSize)
		}

		buf := make([]byte, end-offset)
		n, err := reader.ReadAt(buf, offset)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Printf("Processed chunk at %d: %d bytes\n", offset, n)
	}
}

Example 3: Concurrent Reading

package main

import (
	"fmt"
	"log"
	"sync"
	"golang.org/x/exp/mmap"
)

func main() {
	reader, err := mmap.Open("data.bin")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	// Multiple goroutines can safely read concurrently
	var wg sync.WaitGroup
	numWorkers := 4

	for i := 0; i < numWorkers; i++ {
		wg.Add(1)
		go func(workerID int) {
			defer wg.Done()

			// Each worker reads from a different offset
			offset := int64(workerID * 100)
			buf := make([]byte, 100)

			n, err := reader.ReadAt(buf, offset)
			if err != nil {
				log.Printf("Worker %d error: %v\n", workerID, err)
				return
			}
			fmt.Printf("Worker %d read %d bytes from offset %d\n", workerID, n, offset)
		}(i)
	}

	wg.Wait()
}

Example 4: Random Access Pattern

package main

import (
	"fmt"
	"log"
	"golang.org/x/exp/mmap"
)

func main() {
	reader, err := mmap.Open("data.bin")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	// Access bytes at specific positions
	positions := []int64{0, 100, 500, 1000}

	for _, pos := range positions {
		buf := make([]byte, 8)
		n, err := reader.ReadAt(buf, pos)
		if err != nil {
			log.Printf("Error at offset %d: %v\n", pos, err)
			continue
		}
		fmt.Printf("Read %d bytes from offset %d\n", n, pos)
	}
}

Example 5: Direct Byte Access for Scanning

package main

import (
	"fmt"
	"log"
	"golang.org/x/exp/mmap"
)

func main() {
	reader, err := mmap.Open("data.bin")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	// Find pattern in file using direct byte access
	pattern := byte(0xFF)
	fileSize := reader.Len()

	for i := 0; i < fileSize; i++ {
		if reader.At(i) == pattern {
			fmt.Printf("Found pattern at offset %d\n", i)
		}
	}
}

Performance Considerations

1. Memory Efficiency: Memory-mapped files don't load the entire file into memory. Only accessed pages are loaded by the OS.

2. Concurrent Reading: Multiple goroutines can safely call ReadAt concurrently, making it ideal for parallel file processing.

3. Random Access: Unlike buffered I/O, you can efficiently seek to arbitrary offsets without sequential reading overhead.

4. Platform Specific: The underlying mmap syscall behavior may vary across operating systems (Linux, macOS, Windows).

5. Resource Management: Always defer Close() to ensure resources are properly released.

Error Handling

Common errors when using the mmap package:

• File Not Found: Open returns an error if the file doesn't exist or can't be accessed
• Permission Denied: Occurs when the process lacks read permissions for the file
• Invalid Offset: ReadAt or At with out-of-bounds offsets may return errors or zero values
• Concurrent Access: Calling Close while reading methods are executing is unsafe and may cause panics

Thread Safety

• ReadAt calls are safe for concurrent execution from multiple goroutines
• Close is NOT safe to call concurrently with any reading method
• You must synchronize access to ensure Close is not called while other operations are in progress
• The At method is also safe for concurrent reads but unsafe with Close