Large Dataset Report Generator

A utility that processes large datasets from a database efficiently and generates summary reports without loading all records into memory at once.

Capabilities

Process records in configurable batches

• Given a database with 1000 user records, processing them in batches of 100 returns exactly 10 batch callbacks @test
• Processing an empty result set (0 records) triggers no batch callbacks and returns no error @test

Handle batch processing errors

• When a batch processing callback returns an error, the iteration stops and the error is propagated @test
• When a batch processing callback returns nil, the iteration continues to the next batch @test

Aggregate data across batches

• Processing user records in batches correctly counts the total number of active users across all batches @test
• Processing order records in batches correctly sums the total revenue across all batches @test

Implementation

@generates

API

package processor

import (
	"gorm.io/gorm"
)

// UserRecord represents a user in the database
type UserRecord struct {
	ID       uint   `gorm:"primaryKey"`
	Name     string
	Email    string
	IsActive bool
	Age      int
}

// OrderRecord represents an order in the database
type OrderRecord struct {
	ID       uint `gorm:"primaryKey"`
	UserID   uint
	Amount   float64
	Status   string
}

// ProcessUserBatches processes user records in batches and executes a callback for each batch.
// Returns the total number of records processed and any error encountered.
func ProcessUserBatches(db *gorm.DB, batchSize int, callback func(users []UserRecord) error) (int, error)

// CountActiveUsers counts the total number of active users by processing records in batches.
func CountActiveUsers(db *gorm.DB, batchSize int) (int, error)

// CalculateTotalRevenue calculates the total revenue from completed orders by processing in batches.
func CalculateTotalRevenue(db *gorm.DB, batchSize int) (float64, error)

Dependencies { .dependencies }

gorm { .dependency }

Provides ORM functionality for database operations including batch processing.

@satisfied-by