CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/npm-minio

S3 Compatible Cloud Storage client for JavaScript/TypeScript

—

Pending
Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

SecuritybySnyk

Pending

The risk profile of this skill

Overview
Eval results
Files

notifications.mddocs/

Notifications

This document covers the MinIO notification system, including bucket notification configuration, event types, polling mechanisms, and notification target setup for real-time event processing.

Notification Overview

MinIO supports bucket notifications that trigger when specific events occur on objects. Notifications can be sent to various targets like SNS topics, SQS queues, or Lambda functions, and can also be consumed via polling.

Event Types

Standard S3 Events

// Import event type constants
import { 
  ObjectCreatedAll,
  ObjectCreatedPut,
  ObjectCreatedPost, 
  ObjectCreatedCopy,
  ObjectCreatedCompleteMultipartUpload,
  ObjectRemovedAll,
  ObjectRemovedDelete,
  ObjectRemovedDeleteMarkerCreated,
  ObjectReducedRedundancyLostObject
} from 'minio'

// Event type values
ObjectCreatedAll                      = 's3:ObjectCreated:*'
ObjectCreatedPut                      = 's3:ObjectCreated:Put'
ObjectCreatedPost                     = 's3:ObjectCreated:Post'
ObjectCreatedCopy                     = 's3:ObjectCreated:Copy'
ObjectCreatedCompleteMultipartUpload  = 's3:ObjectCreated:CompleteMultipartUpload'
ObjectRemovedAll                      = 's3:ObjectRemoved:*'
ObjectRemovedDelete                   = 's3:ObjectRemoved:Delete'
ObjectRemovedDeleteMarkerCreated      = 's3:ObjectRemoved:DeleteMarkerCreated'
ObjectReducedRedundancyLostObject     = 's3:ReducedRedundancyLostObject'

Event Type Usage

// Listen for all object creation events
const events = [ObjectCreatedAll]

// Listen for specific events
const specificEvents = [
  ObjectCreatedPut,
  ObjectCreatedPost,
  ObjectRemovedDelete
]

// Listen for all events (creation and deletion)
const allEvents = [ObjectCreatedAll, ObjectRemovedAll]

Notification Configuration

NotificationConfig Class

import { NotificationConfig } from 'minio'

class NotificationConfig {
  // Methods
  add(target: TargetConfig): void  // Add notification target
}

Base TargetConfig Class

abstract class TargetConfig {
  // Methods
  setId(id: unknown): void                    // Set target identifier
  addEvent(newevent: Event): void            // Add event type to listen for
  addFilterSuffix(suffix: string): void      // Add object suffix filter
  addFilterPrefix(prefix: string): void      // Add object prefix filter
}

Target Types

TopicConfig (SNS)

import { TopicConfig } from 'minio'

class TopicConfig extends TargetConfig {
  constructor(arn: string)  // SNS topic ARN
}

QueueConfig (SQS)

import { QueueConfig } from 'minio'

class QueueConfig extends TargetConfig {
  constructor(arn: string)  // SQS queue ARN
}

CloudFunctionConfig (Lambda)

import { CloudFunctionConfig } from 'minio'

class CloudFunctionConfig extends TargetConfig {
  constructor(arn: string)  // Lambda function ARN
}

Set Bucket Notification

client.setBucketNotification(bucketName, config, callback)

// Parameters
bucketName: string           // Bucket name
config: object              // Notification configuration object
callback: function          // Callback function (err) => void

// Note: This method uses callback pattern only

Get Bucket Notification

client.getBucketNotification(bucketName, callback)

// Parameters  
bucketName: string          // Bucket name
callback: function         // Callback function (err, config) => void

Remove All Notifications

client.removeAllBucketNotification(bucketName, callback)

// Parameters
bucketName: string         // Bucket name  
callback: function        // Callback function (err) => void

Notification Configuration Examples

import { 
  NotificationConfig,
  TopicConfig, 
  QueueConfig,
  CloudFunctionConfig,
  ObjectCreatedAll,
  ObjectRemovedAll 
} from 'minio'

// Configure SNS topic notification
const topicConfig = new TopicConfig('arn:aws:sns:us-east-1:123456789012:my-topic')
topicConfig.addEvent(ObjectCreatedAll)
topicConfig.addFilterPrefix('photos/')
topicConfig.addFilterSuffix('.jpg')

const notificationConfig = new NotificationConfig()
notificationConfig.add(topicConfig)

// Set bucket notification
client.setBucketNotification('my-bucket', notificationConfig, (err) => {
  if (err) throw err
  console.log('Notification configured successfully')
})

// Configure multiple notification targets
const snsConfig = new TopicConfig('arn:aws:sns:us-east-1:123456789012:uploads-topic')
snsConfig.addEvent(ObjectCreatedPut)
snsConfig.addEvent(ObjectCreatedPost)
snsConfig.addFilterPrefix('uploads/')

const sqsConfig = new QueueConfig('arn:aws:sqs:us-east-1:123456789012:deletions-queue')  
sqsConfig.addEvent(ObjectRemovedDelete)

const lambdaConfig = new CloudFunctionConfig('arn:aws:lambda:us-east-1:123456789012:function:processImage')
lambdaConfig.addEvent(ObjectCreatedAll)
lambdaConfig.addFilterSuffix('.jpg')
lambdaConfig.addFilterSuffix('.png')

const multiTargetConfig = new NotificationConfig()
multiTargetConfig.add(snsConfig)
multiTargetConfig.add(sqsConfig) 
multiTargetConfig.add(lambdaConfig)

client.setBucketNotification('multi-notifications', multiTargetConfig, (err) => {
  if (err) throw err
  console.log('Multiple notifications configured')
})

// Get current notification configuration
client.getBucketNotification('my-bucket', (err, config) => {
  if (err) throw err
  console.log('Current notification config:', JSON.stringify(config, null, 2))
})

// Remove all notifications
client.removeAllBucketNotification('my-bucket', (err) => {
  if (err) throw err
  console.log('All notifications removed')
})

Notification Polling

NotificationPoller Class

import { NotificationPoller } from 'minio'

class NotificationPoller extends EventEmitter {
  constructor(
    client: TypedClient,          // MinIO client instance
    bucketName: string,           // Bucket to monitor
    prefix: string,               // Object prefix filter
    suffix: string,               // Object suffix filter  
    events: NotificationEvent[]   // Events to listen for
  )
  
  // Methods
  start(): void                   // Start polling for notifications
  stop(): void                   // Stop polling for notifications
  
  // Events (extends EventEmitter)
  on('notification', listener: (notification) => void): this
  on('error', listener: (error) => void): this
}

Listen for Notifications

const notificationStream = client.listenBucketNotification(bucketName, prefix, suffix, events)

// Parameters
bucketName: string    // Bucket name
prefix: string       // Object prefix filter (empty string for all)
suffix: string       // Object suffix filter (empty string for all)  
events: string[]     // Array of event types to listen for

// Returns: EventEmitter

Polling Examples

// Listen for all object creation events
const notificationStream = client.listenBucketNotification(
  'my-bucket',
  '',      // No prefix filter (all objects)
  '',      // No suffix filter (all objects)
  [ObjectCreatedAll]
)

notificationStream.on('notification', (record) => {
  console.log('New object created:', record.s3.object.key)
  console.log('Event name:', record.eventName)
  console.log('Bucket:', record.s3.bucket.name)
  console.log('Object size:', record.s3.object.size)
})

notificationStream.on('error', (err) => {
  console.error('Notification error:', err)
})

// Listen for photo uploads only
const photoNotifications = client.listenBucketNotification(
  'photos',
  'uploads/',           // Only objects with 'uploads/' prefix
  '.jpg',              // Only .jpg files
  [ObjectCreatedPut, ObjectCreatedPost]
)

photoNotifications.on('notification', (record) => {
  const objectKey = record.s3.object.key
  const bucketName = record.s3.bucket.name
  
  console.log(`New photo uploaded: ${objectKey}`)
  
  // Process the uploaded photo
  processNewPhoto(bucketName, objectKey)
})

// Listen for deletion events
const deletionNotifications = client.listenBucketNotification(
  'documents',
  'important/',        // Monitor important documents
  '',                 // All file types
  [ObjectRemovedDelete, ObjectRemovedDeleteMarkerCreated]
)

deletionNotifications.on('notification', (record) => {
  console.log('Document deleted:', record.s3.object.key)
  
  // Log deletion for audit
  auditLog.record({
    event: 'document_deleted',
    object: record.s3.object.key,
    timestamp: record.eventTime,
    source: record.eventSource
  })
})

// Advanced notification handling with filtering
async function processNewPhoto(bucketName, objectKey) {
  try {
    // Get object metadata
    const stat = await client.statObject(bucketName, objectKey)
    
    // Process only if file is reasonable size
    if (stat.size > 10 * 1024 * 1024) { // > 10MB
      console.log('Photo too large, skipping processing')
      return
    }
    
    // Download and process photo
    const photoStream = await client.getObject(bucketName, objectKey)
    // ... image processing logic
    
  } catch (error) {
    console.error('Failed to process photo:', error)
  }
}

Advanced Polling Patterns

// Multiple concurrent listeners
function setupMultipleListeners(bucketName) {
  // Listen for uploads
  const uploadListener = client.listenBucketNotification(
    bucketName, '', '', [ObjectCreatedAll]
  )
  
  // Listen for deletions
  const deleteListener = client.listenBucketNotification(
    bucketName, '', '', [ObjectRemovedAll]
  )
  
  // Separate handlers for different event types
  uploadListener.on('notification', handleUpload)
  deleteListener.on('notification', handleDeletion)
  
  // Common error handling
  [uploadListener, deleteListener].forEach(listener => {
    listener.on('error', (err) => {
      console.error('Notification listener error:', err)
      
      // Implement retry logic
      setTimeout(() => {
        console.log('Restarting notification listener...')
        // Restart listeners if needed
      }, 5000)
    })
  })
  
  return { uploadListener, deleteListener }
}

function handleUpload(record) {
  const { eventName, s3: { bucket, object } } = record
  
  console.log(`Upload event: ${eventName}`)
  console.log(`Bucket: ${bucket.name}, Object: ${object.key}`)
  
  // Route to appropriate handler based on object type
  if (object.key.endsWith('.jpg') || object.key.endsWith('.png')) {
    processImage(bucket.name, object.key)
  } else if (object.key.endsWith('.pdf')) {
    processDocument(bucket.name, object.key)
  } else if (object.key.endsWith('.csv')) {
    processData(bucket.name, object.key)
  }
}

function handleDeletion(record) {
  const { eventName, s3: { bucket, object } } = record
  
  console.log(`Deletion event: ${eventName}`)
  
  // Clean up related resources
  cleanupRelatedData(bucket.name, object.key)
  
  // Update search index
  updateSearchIndex('remove', bucket.name, object.key)
}

// Graceful shutdown
process.on('SIGINT', () => {
  console.log('Shutting down notification listeners...')
  
  // Stop all listeners
  Object.values(listeners).forEach(listener => {
    if (listener && listener.stop) {
      listener.stop()
    }
  })
  
  process.exit(0)
})

ARN Building Utility

import { buildARN } from 'minio'

const arn = buildARN(partition, service, region, accountId, resource)

// Parameters
partition: string   // AWS partition ('aws', 'aws-cn', 'aws-us-gov')
service: string    // AWS service ('sns', 'sqs', 'lambda')
region: string     // AWS region ('us-east-1', 'eu-west-1', etc.)
accountId: string  // AWS account ID
resource: string   // Resource identifier

// Returns: string - Complete ARN

ARN Examples

// Build SNS topic ARN
const snsArn = buildARN('aws', 'sns', 'us-east-1', '123456789012', 'my-topic')
console.log(snsArn) // arn:aws:sns:us-east-1:123456789012:my-topic

// Build SQS queue ARN  
const sqsArn = buildARN('aws', 'sqs', 'us-west-2', '123456789012', 'my-queue')
console.log(sqsArn) // arn:aws:sqs:us-west-2:123456789012:my-queue

// Build Lambda function ARN
const lambdaArn = buildARN('aws', 'lambda', 'eu-west-1', '123456789012', 'function:myFunction')
console.log(lambdaArn) // arn:aws:lambda:eu-west-1:123456789012:function:myFunction

// Use built ARNs in notification config
const config = new NotificationConfig()

const topic = new TopicConfig(buildARN('aws', 'sns', 'us-east-1', '123456789012', 'uploads'))
topic.addEvent(ObjectCreatedAll)

const queue = new QueueConfig(buildARN('aws', 'sqs', 'us-east-1', '123456789012', 'processing'))
queue.addEvent(ObjectCreatedPut)

config.add(topic)
config.add(queue)

Notification Record Format

When notifications are received, they follow the S3 notification record format:

// Example notification record structure
const notificationRecord = {
  eventVersion: '2.1',
  eventSource: 'minio:s3',
  eventTime: '2023-01-01T12:00:00.000Z',
  eventName: 's3:ObjectCreated:Put',
  userIdentity: {
    principalId: 'minio'
  },
  requestParameters: {
    principalId: 'minio',
    region: 'us-east-1',
    sourceIPAddress: '192.168.1.100'
  },
  responseElements: {
    'x-amz-request-id': '17C5B45A35D8CB22',
    'x-minio-deployment-id': '9b777c1a-3b3d-4f3c-b1c3-8f5b3a1e9c2d'
  },
  s3: {
    s3SchemaVersion: '1.0',
    configurationId: 'Config',
    bucket: {
      name: 'my-bucket',
      ownerIdentity: {
        principalId: 'minio'
      },
      arn: 'arn:aws:s3:::my-bucket'
    },
    object: {
      key: 'photos/vacation.jpg',
      size: 2048576,
      eTag: 'd41d8cd98f00b204e9800998ecf8427e',
      contentType: 'image/jpeg',
      userMetadata: {
        'x-amz-meta-uploader': 'mobile-app'
      },
      sequencer: '17C5B45A35D8CB23'
    }
  }
}

Error Handling

import { S3Error } from 'minio'

// Handle notification configuration errors
client.setBucketNotification('my-bucket', config, (err) => {
  if (err) {
    if (err instanceof S3Error) {
      switch (err.code) {
        case 'InvalidArgument':
          console.error('Invalid notification configuration:', err.message)
          break
        case 'UnsupportedNotification':
          console.error('Notification type not supported:', err.message)
          break
        default:
          console.error('S3 notification error:', err.code, err.message)
      }
    } else {
      console.error('Notification error:', err.message)
    }
  }
})

// Handle polling errors
const listener = client.listenBucketNotification('bucket', '', '', [ObjectCreatedAll])

listener.on('error', (err) => {
  console.error('Notification polling error:', err)
  
  // Implement exponential backoff retry
  const retryDelay = Math.min(1000 * Math.pow(2, retryCount), 30000)
  
  setTimeout(() => {
    console.log('Retrying notification listener...')
    // Restart listener
  }, retryDelay)
})

Best Practices

1. Event Filtering

  • Use prefix and suffix filters to reduce notification volume
  • Filter events at the source rather than in handlers
  • Consider object naming strategies that support filtering
  • Monitor notification volume and costs

2. Reliability

  • Implement error handling and retry logic
  • Use dead letter queues for failed notifications
  • Monitor notification delivery rates
  • Set appropriate timeouts for processing

3. Performance

  • Process notifications asynchronously
  • Use appropriate batching for high-volume events
  • Consider fan-out patterns for multiple consumers
  • Monitor processing latency and throughput

4. Security

  • Use IAM policies to restrict notification access
  • Validate notification sources
  • Encrypt sensitive notification data
  • Audit notification configuration changes

5. Monitoring

  • Track notification delivery and processing metrics
  • Set up alerts for notification failures
  • Monitor resource usage of notification handlers
  • Log notification events for troubleshooting

Next: Types and Errors - Complete reference for all error classes, types, and interfaces

docs

advanced-objects.md

bucket-operations.md

client-setup.md

index.md

notifications.md

object-operations.md

presigned-operations.md

types-and-errors.md

tile.json