Nearley

Nearley is a comprehensive JavaScript parsing toolkit that implements the Earley parsing algorithm, enabling developers to parse any context-free grammar including left-recursive ones that challenge other parser generators. It offers a powerful domain-specific language for grammar definition, an efficient streaming parser with comprehensive error handling and ambiguity resolution, and extensive tooling including railroad diagram generation, grammar testing, and fuzzing capabilities.

Package Information

• Package Name: nearley
• Package Type: npm
• Language: JavaScript
• Installation: npm install nearley

Core Imports

const nearley = require("nearley");
const { Parser, Grammar, Rule } = nearley;

For ES6 modules:

import * as nearley from "nearley";
import { Parser, Grammar, Rule } from "nearley";

Basic Usage

const nearley = require("nearley");

// Use a pre-compiled grammar (generated with nearleyc)
const grammar = nearley.Grammar.fromCompiled(require("./my-grammar.js"));

// Create a parser instance
const parser = new nearley.Parser(grammar);

// Parse input text
try {
  parser.feed("your input text here");
  console.log("Parse results:", parser.results);
} catch (parseError) {
  console.error("Parse error:", parseError.message);
}

Architecture

Nearley is built around several key components:

• Parser Engine: Core Parser class implementing the Earley parsing algorithm with streaming support
• Grammar System: Grammar and Rule classes for representing compiled grammars and production rules
• Compilation Pipeline: Tools to compile .ne grammar files into executable JavaScript
• Lexer Integration: Support for various lexers including the built-in StreamLexer and external lexers like moo
• Tooling Ecosystem: CLI tools for compilation, testing, diagram generation, and text generation
• Error Handling: Comprehensive error reporting with state stack traces and position information

Capabilities

Core Parsing

Main parsing functionality using the Earley algorithm for parsing any context-free grammar. Supports streaming input, error recovery, and ambiguous grammars.

class Parser {
  constructor(rules, start, options);
  constructor(grammar, options);
  
  feed(chunk: string): Parser;
  save(): Column;
  restore(column: Column): void;
  finish(): any[];
}

static Parser.fail: object;

Core Parsing

Grammar Management

Grammar representation and compilation system for working with context-free grammars and production rules.

class Grammar {
  constructor(rules: Rule[], start?: string);
  static fromCompiled(rules: object, start?: string): Grammar;
}

class Rule {
  constructor(name: string, symbols: any[], postprocess?: Function);
  toString(withCursorAt?: number): string;
}

Grammar Management

Stream Processing

Node.js stream integration for parsing large inputs or continuous data streams.

class StreamWrapper extends Writable {
  constructor(parser: Parser);
}

Stream Processing

Text Generation

Random text generation from grammars for testing, fuzzing, and example generation.

function Unparse(grammar: Grammar, start: string, depth?: number): string;

Text Generation

CLI Tools

Command-line tools for grammar compilation, testing, railroad diagram generation, and text generation.

• nearleyc - Compile .ne grammar files to JavaScript
• nearley-test - Test compiled grammars with input
• nearley-railroad - Generate railroad diagrams from grammars
• nearley-unparse - Generate random text from grammars

CLI Tools

Built-in Resources

Default Lexer

class StreamLexer {
  constructor();
  reset(data: string, state?: object): void;
  next(): {value: string} | undefined;
  save(): {line: number, col: number};
  formatError(token: object, message: string): string;
}

Built-in Grammars

Nearley includes several built-in grammar modules that can be included in your own grammars:

• builtin/number.ne - Number parsing patterns (unsigned_int, int, decimal, percentage, jsonfloat)
• builtin/string.ne - String parsing patterns with escape sequences
• builtin/whitespace.ne - Whitespace handling patterns
• builtin/postprocessors.ne - Common postprocessing functions (id, nuller, joiner, etc.)
• builtin/cow.ne - Simple example grammar for matching "MOO", "MOOO", etc.

Usage in Grammar Files:

@include "builtin/number.ne"
@include "builtin/whitespace.ne"

# Your grammar rules can now use built-in rules
expr -> number _:* "+" _:* number {%
    function(d) { return d[0] + d[4]; }
%}

Importing in JavaScript:

// Built-in grammars are typically included at compile time
// For runtime access to compiled built-ins, they must be pre-compiled:
const numberGrammar = require("nearley/builtin/number.js"); // if compiled

Types

interface ParserOptions {
  keepHistory?: boolean;
  lexer?: object;
}

interface State {
  rule: Rule;
  dot: number;
  reference: number;
  data: any[];
  wantedBy: State[];
  isComplete: boolean;
}

interface Column {
  grammar: Grammar;
  index: number;
  states: State[];
  wants: object;
  scannable: State[];
  completed: object;
}