tessl/npm-yargs-parser

The mighty option parser used by yargs for parsing command-line arguments with extensive configuration options.

Overview

Eval results

Files

Argument String Tokenization

Name: tessl/npm-yargs-parser
Author: tessl

Low-level utility function for tokenizing argument strings into arrays of individual arguments. This function handles quoted strings, escape sequences, and proper parsing of shell-like argument syntax.

Import

Node.js (ESM):

import { tokenizeArgString } from "yargs-parser/build/lib/tokenize-arg-string.js";

Node.js (CommonJS):

// Not directly available in CommonJS - tokenizeArgString is internal
// Use the main parser for argument processing instead

Browser/Deno:

// tokenizeArgString is not exposed in browser or Deno builds
// This is an internal utility primarily for Node.js environments

API

/**
 * Take an un-split argv string and tokenize it into an array of arguments
 * @param argString - String or array to tokenize 
 * @returns Array of tokenized argument strings
 */
function tokenizeArgString(argString: string | any[]): string[];

Usage Examples

Basic String Tokenization

import { tokenizeArgString } from "yargs-parser/build/lib/tokenize-arg-string.js";

// Simple argument string
const args1 = tokenizeArgString("--foo bar --baz");
console.log(args1);
// Output: ["--foo", "bar", "--baz"]

// With quoted arguments
const args2 = tokenizeArgString('--name "John Doe" --age 30');
console.log(args2);
// Output: ["--name", "John Doe", "--age", "30"]

Handling Quoted Strings

// Single quotes
const args1 = tokenizeArgString("--title 'My Great App' --debug");
console.log(args1);
// Output: ["--title", "My Great App", "--debug"]

// Double quotes with spaces
const args2 = tokenizeArgString('--message "Hello world from app"');
console.log(args2);
// Output: ["--message", "Hello world from app"]

// Mixed quotes
const args3 = tokenizeArgString(`--config '{"theme": "dark"}' --verbose`);
console.log(args3);
// Output: ["--config", '{"theme": "dark"}', "--verbose"]

Array Input Handling

// If array is provided, converts all elements to strings
const args1 = tokenizeArgString(["--port", 3000, "--debug", true]);
console.log(args1);
// Output: ["--port", "3000", "--debug", "true"]

// String array passes through unchanged
const args2 = tokenizeArgString(["--name", "Alice", "--age", "25"]);
console.log(args2);
// Output: ["--name", "Alice", "--age", "25"]

Complex Argument Parsing

// Multiple spaces and complex quoting
const complexArgs = tokenizeArgString(
  `--input "file with spaces.txt"   --output 'result file.json'  --verbose`
);
console.log(complexArgs);
// Output: ["--input", "file with spaces.txt", "--output", "result file.json", "--verbose"]

// Empty quotes
const emptyArgs = tokenizeArgString(`--name "" --value ''`);
console.log(emptyArgs);
// Output: ["--name", "", "--value", ""]

Behavior Details

String Processing

Splits on spaces unless inside quotes
Handles both single (') and double (") quotes
Preserves spaces within quoted strings
Removes surrounding quotes from quoted arguments
Trims whitespace from the input string

Array Processing

If input is already an array, converts all elements to strings
Non-string elements are converted using string concatenation (+ '')
Returns a new array, doesn't modify the input

Quote Handling

Opening quote must match closing quote (single with single, double with double)
Unclosed quotes will include the quote character in the result
Nested quotes are not supported - the first matching quote closes the string

This function is primarily used internally by yargs-parser but is exposed for advanced use cases where you need to manually tokenize argument strings before parsing.

Install with Tessl CLI