0
# Testing Framework
1

2
Comprehensive testing infrastructure for Tree-sitter parsers including corpus testing, syntax highlighting validation, tag generation testing, query verification, and fuzzing. The framework ensures parser correctness and robustness across different input scenarios.
3

4
## Capabilities
5

6
### Corpus Testing
7

8
Run unit tests for Tree-sitter parsers using corpus-based testing with expected parse tree outputs and assertion syntax.
9

10
```bash { .api }
11
tree-sitter test [options]  # Alias: t
12
```
13

14
**Options:**
15
- `--include <REGEX>, -i`: Only run tests matching regex pattern
16
- `--exclude <REGEX>, -e`: Skip tests matching regex pattern  
17
- `--file-name <FILE>`: Only run tests from specific corpus file
18
- `--update, -u`: Update syntax trees with current parser output
19
- `--debug, -d`: Show parsing debug log during tests
20
- `--debug-build, -0`: Compile parser in debug mode for testing
21
- `--debug-graph, -D`: Generate log.html with debug graphs
22
- `--wasm`: Use WebAssembly parsers instead of native libraries
23
- `--open-log`: Open log.html in default browser (requires --debug-graph)
24
- `--config-path <PATH>`: Use alternative config.json file
25
- `--show-fields`: Force display of fields in test diffs
26
- `--stat <LEVEL>`: Show parsing statistics (all, outliers-and-total, total-only)
27
- `--rebuild, -r`: Force rebuild parser before testing
28
- `--overview-only`: Show only pass-fail overview tree
29

30
**Test Discovery:**
31
Tests are automatically discovered in these locations:
32
- `test/corpus/`: Corpus tests (primary test suite)
33
- `test/highlight/`: Syntax highlighting tests
34
- `test/tags/`: Tag generation tests  
35
- `test/<query-name>/`: Tests for custom queries in `queries/<query-name>.scm`
36

37
**Example:**
38
```bash
39
# Run all tests
40
tree-sitter test
41

42
# Run tests matching pattern
43
tree-sitter test --include "function.*declaration"
44

45
# Update test expectations
46
tree-sitter test --update
47

48
# Run with debug output
49
tree-sitter test --debug --debug-graph --open-log
50

51
# Test with WebAssembly parser
52
tree-sitter test --wasm
53

54
# Show parsing statistics
55
tree-sitter test --stat all
56

57
# Test specific corpus file
58
tree-sitter test --file-name expressions.txt
59
```
60

61
### Corpus Test Format
62

63
Corpus tests use a specific format with test names, input code, and expected parse trees:
64

65
```
66
================================================================================
67
Function Declaration
68
================================================================================
69

70
function hello() {
71
  return "world";
72
}
73

74
--------------------------------------------------------------------------------
75

76
(source_file
77
  (function_declaration
78
    name: (identifier)
79
    parameters: (formal_parameters)
80
    body: (statement_block
81
      (return_statement
82
        (string (string_fragment))))))
83
```
84

85
**Test Attributes:**
86
Tests can include attributes to control behavior:
87

88
```
89
================================================================================
90
Function Declaration
91
:skip
92
:platform(linux)
93
:error
94
:language(javascript,typescript)
95
================================================================================
96
```
97

98
- `:skip`: Skip this test
99
- `:platform(os)`: Only run on specific platform
100
- `:error`: Expect parsing to fail
101
- `:language(lang1,lang2)`: Test with specific languages
102

103
### Fuzzing
104

105
Perform fuzz testing to find parser crashes, infinite loops, and other robustness issues through randomized input mutations.
106

107
```bash { .api }
108
tree-sitter fuzz [options]  # Alias: f
109
```
110

111
**Options:**
112
- `--skip <TESTS>, -s`: List of test names to skip during fuzzing
113
- `--subdir <DIR>`: Subdirectory within grammar project to fuzz
114
- `--edits <COUNT>`: Maximum edits per fuzz iteration (default: varies)
115
- `--iterations <COUNT>`: Number of fuzzing iterations per test (default: varies)
116
- `--include <REGEX>, -i`: Only fuzz tests matching regex
117
- `--exclude <REGEX>, -e`: Skip tests matching regex
118
- `--log-graphs`: Enable graph logging during fuzzing
119
- `--log, -l`: Enable detailed parser logging
120
- `--rebuild, -r`: Force rebuild parser before fuzzing
121

122
**Fuzzing Process:**
123
1. Load corpus tests as seed inputs
124
2. Apply random edits (insertions, deletions, modifications)
125
3. Parse mutated inputs looking for crashes/hangs
126
4. Report any issues found with minimal reproduction cases
127

128
**Example:**
129
```bash
130
# Basic fuzzing
131
tree-sitter fuzz
132

133
# Fuzz with more iterations
134
tree-sitter fuzz --iterations 1000 --edits 50
135

136
# Fuzz specific test patterns
137
tree-sitter fuzz --include "expression.*" --exclude ".*comment.*"
138

139
# Enable detailed logging
140
tree-sitter fuzz --log --log-graphs
141

142
# Skip problematic tests
143
tree-sitter fuzz --skip "test1,test2,test3"
144
```
145

146
## Test Types
147

148
### Syntax Highlighting Tests
149

150
Validate syntax highlighting query results against expected highlight captures:
151

152
**Test Structure:**
153
```
154
test/highlight/
155
├── basic.txt          # Test input files
156
├── functions.txt
157
└── ...
158
```
159

160
**Assertion Format:**
161
Uses special comment syntax to mark expected highlights:
162
```javascript
163
function hello() {
164
//       ^^^^^ keyword
165
  return "world";
166
//^^^^^^ keyword
167
//       ^^^^^^^ string
168
}
169
```
170

171
### Tag Generation Tests
172

173
Test ctags-style tag generation for code navigation:
174

175
**Test Structure:**
176
```
177
test/tags/
178
├── functions.txt      # Test input files  
179
├── classes.txt
180
└── ...
181
```
182

183
**Expected Output:**
184
Tags with positions and metadata for language elements like functions, classes, variables.
185

186
### Query Tests
187

188
Validate custom Tree-sitter queries beyond highlights and tags:
189

190
**Test Organization:**
191
```
192
queries/
193
├── custom-query.scm   # Query definition
194
test/custom-query/     # Corresponding tests
195
├── input1.txt
196
├── input2.txt
197
└── ...
198
```
199

200
**Query Validation:**
201
- Syntax validation of query files
202
- Execution tests with expected results
203
- Performance regression detection
204

205
## Advanced Testing Features
206

207
### Statistical Analysis
208

209
Track parsing performance and success rates:
210

211
```bash
212
# Show all statistics
213
tree-sitter test --stat all
214

215
# Show outliers and totals only
216
tree-sitter test --stat outliers-and-total
217

218
# Show total statistics only
219
tree-sitter test --stat total-only
220
```
221

222
**Tracked Metrics:**
223
- Parse success rate
224
- Parsing speed (bytes/second)
225
- Memory usage patterns
226
- Error recovery effectiveness
227

228
### Visual Debugging
229

230
Generate visual representations of parse trees and debugging information:
231

232
```bash
233
# Generate debug graphs
234
tree-sitter test --debug-graph
235

236
# Automatically open in browser
237
tree-sitter test --debug-graph --open-log
238
```
239

240
**Debug Output:**
241
- `log.html`: Interactive parse tree visualization
242
- State transition graphs
243
- Parsing decision trees
244
- Error location highlights
245

246
### Test Maintenance
247

248
Keep test expectations current with parser changes:
249

250
```bash
251
# Update all test expectations
252
tree-sitter test --update
253

254
# Update specific test pattern
255
tree-sitter test --include "function.*" --update
256

257
# Show field differences in failing tests
258
tree-sitter test --show-fields
259
```
260

261
**Update Process:**
262
- Runs parser on test inputs
263
- Captures current parse tree output
264
- Updates corpus files with new expectations
265
- Preserves test structure and metadata
266

267
## Continuous Integration
268

269
### Automated Testing
270

271
Recommended CI workflow:
272
1. Build parser in both debug and release modes
273
2. Run full test suite including fuzzing
274
3. Validate all query files
275
4. Check for performance regressions
276
5. Test WASM compilation if applicable
277

278
### Platform Testing
279

280
Test across platforms and configurations:
281

282
```bash
283
# Test with different build modes
284
tree-sitter test --debug-build
285
tree-sitter test  # release build
286

287
# Test WebAssembly builds
288
tree-sitter test --wasm
289

290
# Force rebuilds to catch build issues
291
tree-sitter test --rebuild
292
```
293

294
## Error Analysis
295

296
### Test Failure Diagnosis
297

298
Common test failure patterns and solutions:
299

300
**Parse Tree Mismatches:**
301
- Grammar rule changes affecting structure
302
- Precedence/associativity modifications
303
- Lexer token recognition changes
304

305
**Highlight Assertion Failures:**
306
- Query pattern updates needed
307
- Scope assignment changes
308
- Theme configuration issues
309

310
**Performance Regressions:**
311
- Algorithm complexity increases
312
- Memory allocation changes
313
- Infinite loop introduction
314

315
### Debug Strategies
316

317
Effective debugging approaches:
318

319
1. **Isolate failures**: Use `--include` to focus on specific tests
320
2. **Enable logging**: Add `--debug` and `--debug-graph` for visibility
321
3. **Compare outputs**: Use `--show-fields` to see detailed differences
322
4. **Incremental updates**: Update tests gradually with `--update`
323
5. **Visual inspection**: Use browser visualization with `--open-log`