0
# Parsing and Token Management
1

2
Low-level parsing capabilities for Velocity template syntax with comprehensive error handling. These components provide the foundation for AST generation and are primarily used internally by the PMD framework, though they can be accessed for advanced use cases.
3

4
## Capabilities
5

6
### Template Parser
7

8
Core parser for converting Velocity template source code into Abstract Syntax Trees.
9

10
```java { .api }
11
/**
12
 * Parser adapter for Velocity templates within PMD framework.
13
 * Converts Velocity template source code into AST nodes for analysis.
14
 * 
15
 * @deprecated Marked as internal API - use VmHandler.getParser() instead
16
 * @InternalApi This class is intended for internal PMD use
17
 */
18
@Deprecated
19
@InternalApi
20
public class VmParser extends AbstractParser {
21
    
22
    /**
23
     * Creates a token manager for the provided source reader.
24
     * Token manager handles lexical analysis of Velocity syntax.
25
     * @param source Reader containing Velocity template source code
26
     * @return VmTokenManager instance for tokenizing the source
27
     */
28
    public TokenManager createTokenManager(Reader source);
29
    
30
    /**
31
     * Parses Velocity template source into an AST.
32
     * Main entry point for converting template text into analyzable structure.
33
     * @param fileName Name or identifier of the template being parsed
34
     * @param source Reader containing the template source code
35
     * @return Root Node of the parsed AST (specifically a VmNode implementation)
36
     * @throws ParseException If source contains syntax errors
37
     */
38
    public Node parse(String fileName, Reader source) throws ParseException;
39
    
40
    /**
41
     * Returns suppression map for PMD rule violations.
42
     * Maps line numbers to suppression comments for rule exclusions.
43
     * @return Map of line numbers to suppression strings
44
     */
45
    public Map<Integer, String> getSuppressMap();
46
}
47
```
48

49
### Token Manager
50

51
Manages lexical analysis and token generation for Velocity template parsing.
52

53
```java { .api }
54
/**
55
 * Token manager for Velocity template lexical analysis.
56
 * Handles tokenization of Velocity syntax elements including directives,
57
 * references, literals, and operators.
58
 * 
59
 * @deprecated Marked as internal API - use via VmParser instead
60
 * @InternalApi This class is intended for internal PMD use
61
 */
62
@Deprecated
63
@InternalApi
64
public class VmTokenManager implements TokenManager {
65
    
66
    /**
67
     * Returns the next token from the input stream.
68
     * Advances the tokenizer position and returns the next syntactic element.
69
     * @return Token representing the next syntactic element
70
     * @throws TokenMgrError If tokenization fails due to invalid syntax
71
     */
72
    public Token getNextToken();
73
    
74
    /**
75
     * Sets the filename for error reporting and debugging.
76
     * Used to provide context in error messages and violation reports.
77
     * @param fileName Name or path of the template file being processed
78
     */
79
    public void setFileName(String fileName);
80
}
81
```
82

83
## Exception Handling
84

85
### Template Parse Exceptions
86

87
Specialized exception for template-specific parsing errors with location information.
88

89
```java { .api }
90
/**
91
 * Exception thrown when Velocity template parsing fails.
92
 * Provides detailed location information and context for syntax errors.
93
 * 
94
 * @deprecated Marked for potential removal - use standard ParseException
95
 */
96
@Deprecated
97
public class TemplateParseException extends ParseException {
98
    
99
    /**
100
     * Creates exception with basic error message.
101
     * @param message Descriptive error message
102
     */
103
    public TemplateParseException(String message);
104
    
105
    /**
106
     * Creates exception with full location context.
107
     * @param message Descriptive error message
108
     * @param templateName Name of template where error occurred
109
     * @param lineNumber Line number of error (1-based)
110
     * @param columnNumber Column number of error (1-based)
111
     */
112
    public TemplateParseException(String message, String templateName, 
113
                                 int lineNumber, int columnNumber);
114
    
115
    /**
116
     * Gets the name of the template where the error occurred.
117
     * @return Template name or identifier
118
     */
119
    public String getTemplateName();
120
    
121
    /**
122
     * Gets the line number where the error occurred.
123
     * @return Line number (1-based)
124
     */
125
    public int getLineNumber();
126
    
127
    /**
128
     * Gets the column number where the error occurred.
129
     * @return Column number (1-based)
130
     */
131
    public int getColumnNumber();
132
}
133
```
134

135
### Macro Parse Exceptions
136

137
Specialized exceptions for macro-related parsing errors.
138

139
```java { .api }
140
/**
141
 * Exception thrown when macro parsing fails in Velocity templates.
142
 * Handles errors specific to macro definition and invocation syntax.
143
 */
144
public class MacroParseException extends ParseException {
145
    // Extends ParseException with macro-specific error handling
146
    // Used for macro definition syntax errors and invocation problems
147
}
148
```
149

150
### Token Manager Errors
151

152
Error handling for lexical analysis failures.
153

154
```java { .api }
155
/**
156
 * Error class for token manager failures during lexical analysis.
157
 * Thrown when tokenizer encounters invalid character sequences or
158
 * unexpected input that cannot be tokenized according to Velocity grammar.
159
 */
160
public class TokenMgrError {
161
    // Handles lexical analysis errors
162
    // Provides information about invalid character sequences
163
    // Used internally by JavaCC-generated token manager
164
}
165
```
166

167
## Generated Parser Components
168

169
### Visitor Interface
170

171
Generated visitor interface providing visit methods for all AST node types.
172

173
```java { .api }
174
/**
175
 * Visitor interface generated from JavaCC grammar.
176
 * Defines visit methods for all Velocity AST node types.
177
 * Generated during build process from VmParser.jjt grammar file.
178
 */
179
public interface VmParserVisitor {
180
    // Visit method for each AST node type
181
    Object visit(VmNode node, Object data);
182
    Object visit(ASTprocess node, Object data);
183
    Object visit(ASTReference node, Object data);
184
    Object visit(ASTDirective node, Object data);
185
    // ... Complete set of visit methods for all 40+ node types
186
}
187
```
188

189
### Parser Constants
190

191
Generated constants for AST node type identification.
192

193
```java { .api }
194
/**
195
 * Constants for AST node types generated from JavaCC grammar.
196
 * Provides integer constants for identifying different node types.
197
 */
198
public interface VmParserTreeConstants {
199
    // Integer constants for each AST node type
200
    int JJTPROCESS = 0;
201
    int JJTREFERENCE = 1;
202
    int JJTDIRECTIVE = 2;
203
    // ... Constants for all AST node types
204
    
205
    // String names corresponding to node type constants
206
    String[] jjtNodeName = {
207
        "process",
208
        "Reference", 
209
        "Directive",
210
        // ... Names for all node types
211
    };
212
}
213
```
214

215
## Advanced Usage Examples
216

217
### Custom Parser Configuration
218

219
```java
220
import java.io.StringReader;
221
import net.sourceforge.pmd.lang.ParserOptions;
222

223
// Access parser through VmHandler (recommended approach)
224
VmHandler handler = new VmHandler();
225
ParserOptions options = new ParserOptions();
226

227
// Configure parser options if needed
228
// options.setSuppressMarker("NOPMD");
229

230
Parser parser = handler.getParser(options);
231

232
// Parse template content
233
String templateContent = "#set($name = 'World')\nHello $name!";
234
StringReader reader = new StringReader(templateContent);
235

236
try {
237
    VmNode rootNode = (VmNode) parser.parse("example.vm", reader);
238
    // Process the AST
239
    processAST(rootNode);
240
} catch (ParseException e) {
241
    handleParseError(e);
242
}
243
```
244

245
### Error Handling and Recovery
246

247
```java
248
import net.sourceforge.pmd.lang.ast.ParseException;
249

250
public class TemplateParser {
251
    
252
    public void parseTemplate(String templatePath, Reader source) {
253
        try {
254
            VmParser parser = new VmParser(); // Direct usage (deprecated)
255
            VmNode ast = parser.parse(templatePath, source);
256
            
257
            // Successful parsing
258
            analyzeAST(ast);
259
            
260
        } catch (TemplateParseException e) {
261
            // Handle template-specific errors
262
            System.err.printf("Template error in %s at line %d, column %d: %s%n",
263
                            e.getTemplateName(), e.getLineNumber(), 
264
                            e.getColumnNumber(), e.getMessage());
265
            
266
        } catch (MacroParseException e) {
267
            // Handle macro-specific errors
268
            System.err.println("Macro parsing error: " + e.getMessage());
269
            
270
        } catch (ParseException e) {
271
            // Handle general parsing errors
272
            System.err.println("Parse error: " + e.getMessage());
273
            
274
        } catch (TokenMgrError e) {
275
            // Handle tokenization errors
276
            System.err.println("Tokenization error: " + e.getMessage());
277
        }
278
    }
279
    
280
    private void analyzeAST(VmNode ast) {
281
        // Custom AST analysis logic
282
        ast.jjtAccept(new CustomAnalyzer(), null);
283
    }
284
}
285
```
286

287
### Token Stream Analysis
288

289
```java
290
import java.io.StringReader;
291

292
public class TokenAnalyzer {
293
    
294
    public void analyzeTokens(String templateContent) {
295
        try {
296
            VmTokenManager tokenManager = new VmTokenManager(
297
                new VelocityCharStream(new StringReader(templateContent))
298
            );
299
            
300
            tokenManager.setFileName("analysis.vm");
301
            
302
            Token token;
303
            while ((token = tokenManager.getNextToken()).kind != 0) { // EOF
304
                System.out.printf("Token: %s, Type: %d, Line: %d, Column: %d%n",
305
                                token.image, token.kind, 
306
                                token.beginLine, token.beginColumn);
307
                
308
                // Analyze specific token types
309
                analyzeToken(token);
310
            }
311
            
312
        } catch (TokenMgrError e) {
313
            System.err.println("Token analysis failed: " + e.getMessage());
314
        }
315
    }
316
    
317
    private void analyzeToken(Token token) {
318
        // Custom token analysis logic
319
        switch (token.kind) {
320
            case VmParserConstants.REFERENCE:
321
                System.out.println("Found reference: " + token.image);
322
                break;
323
            case VmParserConstants.DIRECTIVE:
324
                System.out.println("Found directive: " + token.image);
325
                break;
326
            // Handle other token types
327
        }
328
    }
329
}
330
```
331

332
### AST Dumping and Debugging
333

334
```java
335
import java.io.StringWriter;
336

337
public class ASTDebugger {
338
    
339
    public void dumpAST(VmNode rootNode) {
340
        StringWriter output = new StringWriter();
341
        
342
        // Use VmHandler's dump facility (recommended)
343
        VmHandler handler = new VmHandler();
344
        VisitorStarter dumper = handler.getDumpFacade(output, "  ", true);
345
        
346
        // Start AST dump
347
        rootNode.jjtAccept(dumper, null);
348
        
349
        // Print results
350
        System.out.println("AST Structure:");
351
        System.out.println(output.toString());
352
    }
353
    
354
    public void customDump(VmNode node, int depth) {
355
        // Custom AST dumping logic
356
        String indent = "  ".repeat(depth);
357
        System.out.printf("%s%s (line %d, col %d)%n", 
358
                        indent, node.getClass().getSimpleName(),
359
                        node.getLine(), node.getColumn());
360
        
361
        // Recursively dump children
362
        for (VmNode child : node.children()) {
363
            customDump(child, depth + 1);
364
        }
365
    }
366
}
367
```
368

369
## Character Stream Utilities
370

371
### Velocity Character Stream
372

373
Specialized character stream implementation for Velocity parser.
374

375
```java { .api }
376
/**
377
 * Character stream implementation optimized for Velocity template parsing.
378
 * Provides efficient character-by-character access with position tracking.
379
 */
380
public class VelocityCharStream {
381
    
382
    /**
383
     * Creates character stream from Reader.
384
     * @param reader Source reader for template content
385
     */
386
    public VelocityCharStream(Reader reader);
387
    
388
    // Character stream methods for parser integration
389
    // Implementation details are internal to parsing framework
390
}
391
```
392

393
**Usage Example:**
394

395
```java
396
import java.io.StringReader;
397

398
String template = "#if($user)\n  Hello $user.name!\n#end";
399
VelocityCharStream stream = new VelocityCharStream(new StringReader(template));
400

401
// Stream is typically used internally by VmTokenManager
402
VmTokenManager tokenManager = new VmTokenManager(stream);
403
```