tessl/npm-csstools--css-tokenizer

Specification-compliant CSS tokenizer following W3C CSS Syntax Level 3 for parsing CSS into tokens

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Overview

Eval results

Files

Error Handling

Name: tessl/npm-csstools--css-tokenizer
Author: tessl

Parse error classes and error handling mechanisms for tokenization errors with detailed source position information.

Capabilities

Parse Error Classes

Error classes for handling tokenization parsing errors with source position tracking.

/**
 * Base parse error with source position and parser state
 */
class ParseError {
  /** Starting position of the error in the source CSS */
  sourceStart: number;
  /** Ending position of the error in the source CSS */
  sourceEnd: number;
  /** Parser state information at time of error */
  parserState: Array<string>;
}

/**
 * Parse error that includes the associated problematic token
 */
class ParseErrorWithToken extends ParseError {
  /** The token that caused or is associated with the error */
  token: CSSToken;
}

/**
 * Standard error message constants for common parsing issues
 */
const ParseErrorMessage: {
  readonly UnexpectedNewLineInString: string;
  readonly UnexpectedEOFInString: string;
  readonly UnexpectedEOFInComment: string;
  readonly UnexpectedEOFInURL: string;
  readonly UnexpectedEOFInEscapedCodePoint: string;
  readonly UnexpectedCharacterInURL: string;
  readonly InvalidEscapeSequenceInURL: string;
  readonly InvalidEscapeSequenceAfterBackslash: string;
};

Error Message Constants

Predefined error message constants for common parsing scenarios.

/**
 * Standard error message constants for common parsing issues
 */
const ParseErrorMessage: {
  readonly UnexpectedNewLineInString: string;
  readonly UnexpectedEOFInString: string;
  readonly UnexpectedEOFInComment: string;
  readonly UnexpectedEOFInURL: string;
  readonly UnexpectedEOFInEscapedCodePoint: string;
  readonly UnexpectedCharacterInURL: string;
  readonly InvalidEscapeSequenceInURL: string;
  readonly InvalidEscapeSequenceAfterBackslash: string;
};

Available Error Messages:

// Actual error message values from the specification
ParseErrorMessage.UnexpectedNewLineInString = 'Unexpected newline while consuming a string token.';
ParseErrorMessage.UnexpectedEOFInString = 'Unexpected EOF while consuming a string token.';
ParseErrorMessage.UnexpectedEOFInComment = 'Unexpected EOF while consuming a comment.';
ParseErrorMessage.UnexpectedEOFInURL = 'Unexpected EOF while consuming a url token.';
ParseErrorMessage.UnexpectedEOFInEscapedCodePoint = 'Unexpected EOF while consuming an escaped code point.';
ParseErrorMessage.UnexpectedCharacterInURL = 'Unexpected character while consuming a url token.';
ParseErrorMessage.InvalidEscapeSequenceInURL = 'Invalid escape sequence while consuming a url token.';
ParseErrorMessage.InvalidEscapeSequenceAfterBackslash = 'Invalid escape sequence after "\\"';

Error Handling Patterns

Basic Error Handling

Handle parse errors during tokenization with callback functions.

import { tokenize, tokenizer, ParseError } from "@csstools/css-tokenizer";

// Basic error handling with tokenize
const tokens = tokenize({
  css: "invalid CSS with 'unclosed string and @bad-syntax"
}, {
  onParseError: (error: ParseError) => {
    console.error(`Parse error at positions ${error.sourceStart}-${error.sourceEnd}`);
    console.error('Parser state:', error.parserState);
  }
});

// Error handling with streaming tokenizer
const t = tokenizer({
  css: "body { color: 'unclosed string; }"
}, {
  onParseError: (error: ParseError) => {
    console.error('Tokenization error occurred:', error);
  }
});

while (!t.endOfFile()) {
  const token = t.nextToken();
  // Process tokens, errors are handled by callback
}

Advanced Error Handling

Collect and analyze errors for comprehensive error reporting.

import { 
  tokenize, 
  ParseError, 
  ParseErrorWithToken, 
  ParseErrorMessage 
} from "@csstools/css-tokenizer";

interface ErrorReport {
  errors: ParseError[];
  tokens: CSSToken[];
  hasErrors: boolean;
}

function tokenizeWithErrorCollection(css: string): ErrorReport {
  const errors: ParseError[] = [];
  
  const tokens = tokenize({ css }, {
    onParseError: (error: ParseError) => {
      errors.push(error);
    }
  });
  
  return {
    errors,
    tokens,
    hasErrors: errors.length > 0
  };
}

// Usage
const result = tokenizeWithErrorCollection(`
  .valid-class { color: red; }
  .invalid { content: 'unclosed string; }
  @media screen and invalid-syntax { }
`);

if (result.hasErrors) {
  console.log(`Found ${result.errors.length} parse errors:`);
  result.errors.forEach((error, index) => {
    console.log(`Error ${index + 1}:`);
    console.log(`  Position: ${error.sourceStart}-${error.sourceEnd}`);
    console.log(`  Parser state: ${error.parserState.join(' → ')}`);
    
    if (error instanceof ParseErrorWithToken) {
      console.log(`  Associated token: ${error.token[0]}`);
    }
  });
}

Error Recovery and Token Types

The tokenizer continues processing after errors, generating appropriate error tokens.

import { 
  tokenize, 
  isTokenBadString, 
  isTokenBadURL, 
  stringify 
} from "@csstools/css-tokenizer";

function analyzeTokenizationErrors(css: string) {
  const errors: ParseError[] = [];
  
  const tokens = tokenize({ css }, {
    onParseError: (error) => errors.push(error)
  });
  
  // Find error tokens generated by the tokenizer
  const badStringTokens = tokens.filter(isTokenBadString);
  const badURLTokens = tokens.filter(isTokenBadURL);
  
  console.log(`Bad string tokens: ${badStringTokens.length}`);
  console.log(`Bad URL tokens: ${badURLTokens.length}`);
  console.log(`Parse errors reported: ${errors.length}`);
  
  // The tokenizer continues despite errors
  console.log('Tokenization result:', stringify(...tokens));
}

// Example with various error conditions
analyzeTokenizationErrors(`
  .class1 { content: 'unclosed string; }
  .class2 { background: url(malformed url); }
  .class3 { color: red; } /* This works fine */
`);

Source Position Mapping

Use error source positions to provide detailed feedback.

import { tokenize, ParseError } from "@csstools/css-tokenizer";

function createDetailedErrorReport(css: string) {
  const lines = css.split('\n');
  const errors: Array<ParseError & { line: number; column: number; excerpt: string }> = [];
  
  const tokens = tokenize({ css }, {
    onParseError: (error: ParseError) => {
      // Calculate line and column from source position
      let line = 1;
      let column = 1;
      let position = 0;
      
      for (let i = 0; i < error.sourceStart && i < css.length; i++) {
        if (css[i] === '\n') {
          line++;
          column = 1;
        } else {
          column++;
        }
        position++;
      }
      
      // Get context around the error
      const errorLine = lines[line - 1] || '';
      const excerpt = errorLine.substring(Math.max(0, column - 20), column + 20);
      
      errors.push({
        ...error,
        line,
        column,
        excerpt
      });
    }
  });
  
  return { tokens, errors };
}

// Usage
const css = `
.header {
  content: 'unclosed string;
  color: red;
}
`;

const { tokens, errors } = createDetailedErrorReport(css);

errors.forEach(error => {
  console.log(`Error at line ${error.line}, column ${error.column}:`);
  console.log(`Context: "${error.excerpt}"`);
  console.log(`Parser state: ${error.parserState.join(' → ')}`);
});

Error Types and Recovery

The CSS tokenizer follows the W3C specification for error handling:

Bad String Tokens: Generated for unclosed strings
Bad URL Tokens: Generated for malformed URLs
Continued Processing: Tokenization continues after errors
Error Callbacks: Optional error reporting without stopping tokenization

Best Practices

Always Handle Errors: Provide onParseError callback for production code
Log Context: Use source positions to provide meaningful error messages
Continue Processing: Don't stop tokenization due to errors - the tokenizer recovers
Validate Input: Consider pre-validation for critical CSS processing
Error Reporting: Collect errors for user feedback in CSS tools and editors

import { tokenize, ParseError } from "@csstools/css-tokenizer";

// Production-ready error handling
function robustCSSTokenization(css: string) {
  const errors: ParseError[] = [];
  
  try {
    const tokens = tokenize({ css }, {
      onParseError: (error: ParseError) => {
        // Log for debugging
        console.warn('CSS parse error:', {
          position: `${error.sourceStart}-${error.sourceEnd}`,
          state: error.parserState
        });
        
        errors.push(error);
      }
    });
    
    return {
      success: true,
      tokens,
      errors,
      hasErrors: errors.length > 0
    };
  } catch (fatalError) {
    return {
      success: false,
      tokens: [],
      errors,
      fatalError
    };
  }
}

Install with Tessl CLI

npx tessl i tessl/npm-csstools--css-tokenizer

docs

tessl/npm-csstools--css-tokenizer

error-handling.md.css-3qkkll{font-size:var(--chakra-font-sizes-sm);font-weight:var(--chakra-font-weights-normal);color:var(--chakra-colors-gray-300);}docs/

Error Handling

Capabilities

Parse Error Classes

Error Message Constants

Error Handling Patterns

Basic Error Handling

Advanced Error Handling

Error Recovery and Token Types

Source Position Mapping

Error Types and Recovery

Best Practices

error-handling.mddocs/