0
# Argument String Tokenization
1

2
Low-level utility function for tokenizing argument strings into arrays of individual arguments. This function handles quoted strings, escape sequences, and proper parsing of shell-like argument syntax.
3

4
## Import
5

6
**Node.js (ESM):**
7
```typescript
8
import { tokenizeArgString } from "yargs-parser/build/lib/tokenize-arg-string.js";
9
```
10

11
**Node.js (CommonJS):**
12
```javascript
13
// Not directly available in CommonJS - tokenizeArgString is internal
14
// Use the main parser for argument processing instead
15
```
16

17
**Browser/Deno:**
18
```typescript
19
// tokenizeArgString is not exposed in browser or Deno builds
20
// This is an internal utility primarily for Node.js environments
21
```
22

23
## API
24

25
```typescript { .api }
26
/**
27
 * Take an un-split argv string and tokenize it into an array of arguments
28
 * @param argString - String or array to tokenize 
29
 * @returns Array of tokenized argument strings
30
 */
31
function tokenizeArgString(argString: string | any[]): string[];
32
```
33

34
## Usage Examples
35

36
### Basic String Tokenization
37

38
```typescript
39
import { tokenizeArgString } from "yargs-parser/build/lib/tokenize-arg-string.js";
40

41
// Simple argument string
42
const args1 = tokenizeArgString("--foo bar --baz");
43
console.log(args1);
44
// Output: ["--foo", "bar", "--baz"]
45

46
// With quoted arguments
47
const args2 = tokenizeArgString('--name "John Doe" --age 30');
48
console.log(args2);
49
// Output: ["--name", "John Doe", "--age", "30"]
50
```
51

52
### Handling Quoted Strings
53

54
```typescript
55
// Single quotes
56
const args1 = tokenizeArgString("--title 'My Great App' --debug");
57
console.log(args1);
58
// Output: ["--title", "My Great App", "--debug"]
59

60
// Double quotes with spaces
61
const args2 = tokenizeArgString('--message "Hello world from app"');
62
console.log(args2);
63
// Output: ["--message", "Hello world from app"]
64

65
// Mixed quotes
66
const args3 = tokenizeArgString(`--config '{"theme": "dark"}' --verbose`);
67
console.log(args3);
68
// Output: ["--config", '{"theme": "dark"}', "--verbose"]
69
```
70

71
### Array Input Handling
72

73
```typescript
74
// If array is provided, converts all elements to strings
75
const args1 = tokenizeArgString(["--port", 3000, "--debug", true]);
76
console.log(args1);
77
// Output: ["--port", "3000", "--debug", "true"]
78

79
// String array passes through unchanged
80
const args2 = tokenizeArgString(["--name", "Alice", "--age", "25"]);
81
console.log(args2);
82
// Output: ["--name", "Alice", "--age", "25"]
83
```
84

85
### Complex Argument Parsing
86

87
```typescript
88
// Multiple spaces and complex quoting
89
const complexArgs = tokenizeArgString(
90
  `--input "file with spaces.txt"   --output 'result file.json'  --verbose`
91
);
92
console.log(complexArgs);
93
// Output: ["--input", "file with spaces.txt", "--output", "result file.json", "--verbose"]
94

95
// Empty quotes
96
const emptyArgs = tokenizeArgString(`--name "" --value ''`);
97
console.log(emptyArgs);
98
// Output: ["--name", "", "--value", ""]
99
```
100

101
## Behavior Details
102

103
### String Processing
104
- Splits on spaces unless inside quotes
105
- Handles both single (`'`) and double (`"`) quotes
106
- Preserves spaces within quoted strings
107
- Removes surrounding quotes from quoted arguments
108
- Trims whitespace from the input string
109

110
### Array Processing
111
- If input is already an array, converts all elements to strings
112
- Non-string elements are converted using string concatenation (`+ ''`)
113
- Returns a new array, doesn't modify the input
114

115
### Quote Handling
116
- Opening quote must match closing quote (single with single, double with double)
117
- Unclosed quotes will include the quote character in the result
118
- Nested quotes are not supported - the first matching quote closes the string
119

120
This function is primarily used internally by yargs-parser but is exposed for advanced use cases where you need to manually tokenize argument strings before parsing.