CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/npm-cacache

Fast, fault-tolerant, cross-platform, disk-based, data-agnostic, content-addressable cache.

90

1.16x
Overview
Eval results
Files

verification.mddocs/

Verification and Maintenance

Cache integrity verification and maintenance operations including garbage collection, index rebuilding, and corruption detection.

Capabilities

Verify Cache Integrity

Performs comprehensive cache verification and maintenance, including garbage collection, index rebuilding, and integrity checking.

/**
 * Verifies and repairs cache integrity
 * @param {string} cache - Path to cache directory
 * @param {object} opts - Options object
 * @param {number} [opts.concurrency=20] - Number of concurrent operations
 * @param {object} [opts.log] - Logger object with silly() method
 * @param {function} [opts.filter] - Function to filter entries during verification
 * @returns {Promise<VerifyStats>} Promise resolving to verification statistics
 */
function verify(cache, opts);

Usage Examples:

const cacache = require('cacache');

// Basic verification
const stats = await cacache.verify('./cache');
console.log('Verification completed:', stats);
console.log(`Runtime: ${stats.runTime.total}ms`);
console.log(`Verified content: ${stats.verifiedContent} files`);
console.log(`Reclaimed space: ${stats.reclaimedSize} bytes`);

// Verification with custom concurrency
const stats = await cacache.verify('./cache', {
  concurrency: 10  // Reduce concurrency for limited systems
});

// Verification with logging
const stats = await cacache.verify('./cache', {
  log: {
    silly: (prefix, message, ...args) => {
      console.log(`[${prefix}] ${message}`, ...args);
    }
  }
});

// Verification with entry filtering
const stats = await cacache.verify('./cache', {
  filter: (entry) => {
    // Only verify entries newer than 1 week
    const weekAgo = Date.now() - (7 * 24 * 60 * 60 * 1000);
    return entry.time > weekAgo;
  }
});

console.log('Filtered verification results:', stats);

Get Last Verification Time

Returns the timestamp of the last cache verification run.

/**
 * Gets timestamp of last verification run
 * @param {string} cache - Path to cache directory
 * @returns {Promise<Date>} Promise resolving to Date object of last verification
 */
function verify.lastRun(cache);

Usage Examples:

// Check when cache was last verified
try {
  const lastRun = await cacache.verify.lastRun('./cache');
  console.log('Cache last verified:', lastRun);
  
  const hoursSinceVerification = (Date.now() - lastRun.getTime()) / (1000 * 60 * 60);
  console.log(`Hours since last verification: ${hoursSinceVerification.toFixed(1)}`);
  
  // Run verification if it's been more than 24 hours
  if (hoursSinceVerification > 24) {
    console.log('Running cache verification...');
    await cacache.verify('./cache');
  }
} catch (error) {
  if (error.code === 'ENOENT') {
    console.log('Cache has never been verified');
    await cacache.verify('./cache');
  } else {
    console.error('Error checking verification time:', error);
  }
}

// Conditional verification based on age
async function verifyIfNeeded(cache, maxAgeHours = 24) {
  try {
    const lastRun = await cacache.verify.lastRun(cache);
    const ageHours = (Date.now() - lastRun.getTime()) / (1000 * 60 * 60);
    
    if (ageHours > maxAgeHours) {
      console.log(`Verification needed (${ageHours.toFixed(1)} hours old)`);
      return await cacache.verify(cache);
    } else {
      console.log(`Verification not needed (${ageHours.toFixed(1)} hours old)`);
      return null;
    }
  } catch (error) {
    console.log('No previous verification found, running now');
    return await cacache.verify(cache);
  }
}

const stats = await verifyIfNeeded('./cache', 12);

Verification Process

The verification process consists of several steps:

  1. Mark Start Time - Records verification start timestamp
  2. Fix Permissions - Ensures cache directory exists and has correct permissions
  3. Garbage Collection - Removes unreferenced content and verifies integrity
  4. Rebuild Index - Reconstructs index from valid content
  5. Clean Temporary Files - Removes temporary files from cache directory
  6. Write Verification File - Records completion timestamp
  7. Mark End Time - Records verification end timestamp

Verification Statistics

The verify() function returns detailed statistics about the verification process:

interface VerifyStats {
  startTime: Date;              // When verification started
  endTime: Date;                // When verification completed
  runTime: {                    // Runtime breakdown by step
    total: number;              // Total verification time (ms)
    markStartTime: number;      // Time for start marking (ms)
    fixPerms: number;           // Time for permission fixing (ms)
    garbageCollect: number;     // Time for garbage collection (ms)
    rebuildIndex: number;       // Time for index rebuild (ms)
    cleanTmp: number;           // Time for temp cleanup (ms)
    writeVerifile: number;      // Time for verification file write (ms)
    markEndTime: number;        // Time for end marking (ms)
  };
  verifiedContent: number;      // Number of content files verified
  reclaimedCount: number;       // Number of files removed
  reclaimedSize: number;        // Bytes of space reclaimed
  badContentCount: number;      // Number of corrupted files found
  keptSize: number;             // Bytes of valid data kept
  missingContent: number;       // Index entries without content
  rejectedEntries: number;      // Index entries filtered out
  totalEntries: number;         // Total index entries processed
}

Advanced Verification Patterns

Scheduled Verification

Set up automatic cache verification:

// Simple scheduled verification
async function scheduleVerification(cache, intervalHours = 24) {
  const verify = async () => {
    try {
      console.log('Starting scheduled cache verification...');
      const stats = await cacache.verify(cache);
      console.log(`Verification completed in ${stats.runTime.total}ms`);
      console.log(`Reclaimed ${stats.reclaimedSize} bytes from ${stats.reclaimedCount} files`);
    } catch (error) {
      console.error('Scheduled verification failed:', error);
    }
  };
  
  // Run immediately and then on interval
  await verify();
  setInterval(verify, intervalHours * 60 * 60 * 1000);
}

// Start scheduled verification every 12 hours
await scheduleVerification('./cache', 12);

Conditional Verification

Run verification based on cache conditions:

// Verify based on cache size
async function verifyIfCacheTooLarge(cache, maxSizeBytes) {
  const entries = await cacache.ls(cache);
  const totalSize = Object.values(entries).reduce((sum, entry) => sum + entry.size, 0);
  
  if (totalSize > maxSizeBytes) {
    console.log(`Cache size (${totalSize}) exceeds limit (${maxSizeBytes}), verifying...`);
    return await cacache.verify(cache);
  }
  
  return null;
}

// Verify based on error count
async function verifyIfErrors(cache, maxErrors = 5) {
  let errorCount = 0;
  
  // Track errors during normal operations
  const originalConsoleError = console.error;
  console.error = (...args) => {
    if (args[0] && args[0].includes && args[0].includes('cache')) {
      errorCount++;
    }
    originalConsoleError(...args);
  };
  
  if (errorCount >= maxErrors) {
    console.log(`Error count (${errorCount}) exceeds threshold, verifying cache...`);
    return await cacache.verify(cache);
  }
  
  return null;
}

Custom Verification Filters

Filter entries during verification based on specific criteria:

// Verify only specific entry types
async function verifyByType(cache, allowedTypes) {
  return await cacache.verify(cache, {
    filter: (entry) => {
      const entryType = entry.metadata?.type;
      return allowedTypes.includes(entryType);
    }
  });
}

// Verify only recent entries
async function verifyRecentEntries(cache, maxAgeHours = 24) {
  const cutoffTime = Date.now() - (maxAgeHours * 60 * 60 * 1000);
  
  return await cacache.verify(cache, {
    filter: (entry) => entry.time > cutoffTime
  });
}

// Verify entries by size
async function verifyBySize(cache, minSize = 0, maxSize = Infinity) {
  return await cacache.verify(cache, {
    filter: (entry) => entry.size >= minSize && entry.size <= maxSize
  });
}

// Usage examples
const stats1 = await verifyByType('./cache', ['image', 'document']);
const stats2 = await verifyRecentEntries('./cache', 48);
const stats3 = await verifyBySize('./cache', 1024, 1024 * 1024); // 1KB - 1MB

Verification Reporting

Create detailed verification reports:

async function generateVerificationReport(cache) {
  const startTime = Date.now();
  console.log('Starting comprehensive cache verification...');
  
  // Get initial cache state
  const initialEntries = await cacache.ls(cache);
  const initialStats = {
    entryCount: Object.keys(initialEntries).length,
    totalSize: Object.values(initialEntries).reduce((sum, entry) => sum + entry.size, 0)
  };
  
  console.log(`Initial state: ${initialStats.entryCount} entries, ${initialStats.totalSize} bytes`);
  
  // Run verification with detailed logging
  const verifyStats = await cacache.verify(cache, {
    log: {
      silly: (prefix, message, ...args) => {
        console.log(`[${new Date().toISOString()}] [${prefix}] ${message}`);
      }
    }
  });
  
  // Get final cache state
  const finalEntries = await cacache.ls(cache);
  const finalStats = {
    entryCount: Object.keys(finalEntries).length,
    totalSize: Object.values(finalEntries).reduce((sum, entry) => sum + entry.size, 0)
  };
  
  // Generate report
  const report = {
    timestamp: new Date().toISOString(),
    duration: Date.now() - startTime,
    initialState: initialStats,
    finalState: finalStats,
    changes: {
      entriesRemoved: initialStats.entryCount - finalStats.entryCount,
      spaceReclaimed: initialStats.totalSize - finalStats.totalSize
    },
    verificationStats: verifyStats
  };
  
  console.log('Verification Report:', JSON.stringify(report, null, 2));
  return report;
}

const report = await generateVerificationReport('./cache');

Health Check

Implement cache health monitoring:

async function checkCacheHealth(cache) {
  const health = {
    status: 'healthy',
    issues: [],
    recommendations: [],
    stats: {}
  };
  
  try {
    // Check if cache directory exists
    const fs = require('fs');
    await fs.promises.access(cache);
    
    // Get cache statistics
    const entries = await cacache.ls(cache);
    health.stats.entryCount = Object.keys(entries).length;
    health.stats.totalSize = Object.values(entries).reduce((sum, entry) => sum + entry.size, 0);
    
    // Check last verification time
    try {
      const lastVerification = await cacache.verify.lastRun(cache);
      const hoursSinceVerification = (Date.now() - lastVerification.getTime()) / (1000 * 60 * 60);
      health.stats.hoursSinceVerification = hoursSinceVerification;
      
      if (hoursSinceVerification > 72) {
        health.status = 'warning';
        health.issues.push('Cache has not been verified in over 72 hours');
        health.recommendations.push('Run cacache.verify() to ensure cache integrity');
      }
    } catch (error) {
      health.status = 'warning';
      health.issues.push('Cache has never been verified');
      health.recommendations.push('Run initial cache verification');
    }
    
    // Check for unusually large cache
    if (health.stats.totalSize > 1024 * 1024 * 1024) { // > 1GB
      health.status = 'warning';
      health.issues.push('Cache size is unusually large');
      health.recommendations.push('Consider running verification to reclaim space');
    }
    
    // Check for too many entries
    if (health.stats.entryCount > 100000) {
      health.status = 'warning';
      health.issues.push('Cache has unusually many entries');
      health.recommendations.push('Consider cache cleanup or partitioning');
    }
    
  } catch (error) {
    health.status = 'error';
    health.issues.push(`Cache inaccessible: ${error.message}`);
    health.recommendations.push('Check cache directory permissions and disk space');
  }
  
  return health;
}

const health = await checkCacheHealth('./cache');
console.log('Cache Health:', health);

if (health.status !== 'healthy') {
  console.log('Issues found:', health.issues);
  console.log('Recommendations:', health.recommendations);
}

Performance Considerations

Verification Frequency

  • Run verification during application idle time
  • More frequent verification for critical caches
  • Consider cache size and I/O performance when scheduling

Concurrency Settings

// Adjust concurrency based on system resources
const stats = await cacache.verify(cache, {
  concurrency: process.env.NODE_ENV === 'production' ? 5 : 20
});

Filtered Verification

  • Use filters to verify only relevant entries
  • Skip verification of large, stable content when possible
  • Filter by age, size, or metadata for targeted verification

Error Handling

Verification operations may encounter various errors:

try {
  const stats = await cacache.verify('./cache');
  console.log('Verification successful:', stats);
} catch (error) {
  switch (error.code) {
    case 'ENOENT':
      console.error('Cache directory does not exist');
      break;
    case 'EACCES':
      console.error('Permission denied accessing cache');
      break;
    case 'ENOSPC':
      console.error('Insufficient disk space for verification');
      break;
    default:
      console.error('Verification failed:', error);
  }
}

Important Notes

Verification Impact

  • Verification is I/O intensive and may impact application performance
  • Content files are read and verified during garbage collection
  • Index files are rebuilt, which requires temporary disk space

Data Safety

  • Verification never removes valid data
  • Corrupted content is removed only after integrity check failure
  • Index entries are preserved unless content is completely missing

Concurrency

  • Verification is safe to run while other cache operations are in progress
  • Multiple verification processes should not run simultaneously on the same cache
  • Use appropriate concurrency limits based on system resources

Install with Tessl CLI

npx tessl i tessl/npm-cacache

docs

index-management.md

index.md

listing.md

reading.md

removal.md

utilities.md

verification.md

writing.md

tile.json