0
# Token Navigation
1

2
Functions and methods for navigating and searching through tokenized source code, finding specific tokens by position, type, or content. These methods work with the enhanced Token objects created by ASTTokens.
3

4
## Capabilities
5

6
### Token Access by Position
7

8
Methods for retrieving tokens based on character offsets or line/column positions.
9

10
```python { .api }
11
class ASTTokens:
12
    def get_token_from_offset(self, offset) -> Token:
13
        """
14
        Get token at specific character offset in source.
15
        
16
        Parameters:
17
        - offset (int): Character offset in source text
18
        
19
        Returns:
20
        Token: Token at the specified offset
21
        """
22
        
23
    def get_token(self, lineno, col_offset) -> Token:
24
        """
25
        Get token at line and column position.
26
        
27
        Parameters:
28
        - lineno (int): Line number (1-based)
29
        - col_offset (int): Column offset (0-based, unicode)
30
        
31
        Returns:
32
        Token: Token at the specified position
33
        """
34
        
35
    def get_token_from_utf8(self, lineno, col_offset) -> Token:
36
        """
37
        Get token at line and column position with UTF8 offsets.
38
        
39
        Parameters:
40
        - lineno (int): Line number (1-based)
41
        - col_offset (int): Column offset (0-based, UTF8 bytes)
42
        
43
        Returns:
44
        Token: Token at the specified position
45
        """
46
```
47

48
#### Usage Example
49

50
```python
51
import asttokens
52

53
source = "def hello():\n    print('world')"
54
atok = asttokens.ASTTokens(source, parse=True)
55

56
# Get token at character offset
57
token = atok.get_token_from_offset(0)
58
print(token.string)  # 'def'
59

60
# Get token at line/column
61
token = atok.get_token(1, 4)  # Line 1, column 4
62
print(token.string)  # 'hello'
63

64
# Get token at second line
65
token = atok.get_token(2, 4)  # Line 2, column 4  
66
print(token.string)  # 'print'
67
```
68

69
### Sequential Token Navigation
70

71
Methods for moving through tokens sequentially in forward or backward direction.
72

73
```python { .api }
74
class ASTTokens:
75
    def next_token(self, tok, include_extra=False) -> Token:
76
        """
77
        Get the next token after the given token.
78
        
79
        Parameters:
80
        - tok (Token): Current token
81
        - include_extra (bool): Include non-coding tokens (comments, newlines)
82
        
83
        Returns:
84
        Token: Next token in sequence
85
        """
86
        
87
    def prev_token(self, tok, include_extra=False) -> Token:
88
        """
89
        Get the previous token before the given token.
90
        
91
        Parameters:
92
        - tok (Token): Current token
93
        - include_extra (bool): Include non-coding tokens (comments, newlines)
94
        
95
        Returns:
96
        Token: Previous token in sequence
97
        """
98
```
99

100
#### Usage Example
101

102
```python
103
import asttokens
104

105
source = "x = 42  # comment"
106
atok = asttokens.ASTTokens(source, parse=True)
107

108
# Start with first token
109
token = atok.get_token_from_offset(0)
110
print(token.string)  # 'x'
111

112
# Navigate forward
113
token = atok.next_token(token)
114
print(token.string)  # '='
115

116
token = atok.next_token(token)
117
print(token.string)  # '42'
118

119
# Include comments and extra tokens
120
token = atok.next_token(token, include_extra=True)
121
print(token.string)  # '# comment'
122

123
# Navigate backward
124
token = atok.prev_token(token)
125
print(token.string)  # '42'
126
```
127

128
### Token Search
129

130
Methods for finding specific tokens by type, content, or other criteria.
131

132
```python { .api }
133
class ASTTokens:
134
    def find_token(self, start_token, tok_type, tok_str=None, reverse=False) -> Token:
135
        """
136
        Find token by type and optionally by string content.
137
        
138
        Parameters:
139
        - start_token (Token): Token to start search from
140
        - tok_type (int): Token type to search for (from token module)
141
        - tok_str (str, optional): Specific token string to match
142
        - reverse (bool): Search backwards if True
143
        
144
        Returns:
145
        Token: First matching token found
146
        
147
        Raises:
148
        ValueError: If no matching token is found
149
        """
150
```
151

152
#### Usage Example
153

154
```python
155
import asttokens
156
import token
157

158
source = "def func(a, b): return a + b"
159
atok = asttokens.ASTTokens(source, parse=True)
160

161
# Start from beginning
162
start = atok.get_token_from_offset(0)
163

164
# Find opening parenthesis
165
paren = atok.find_token(start, token.OP, '(')
166
print(paren.string)  # '('
167

168
# Find specific operator
169
plus = atok.find_token(start, token.OP, '+')
170
print(plus.string)  # '+'
171

172
# Find any name token
173
name = atok.find_token(start, token.NAME)
174
print(name.string)  # 'def'
175

176
# Search backwards from end
177
end = atok.tokens[-1]
178
last_name = atok.find_token(end, token.NAME, reverse=True)
179
print(last_name.string)  # 'b'
180
```
181

182
### Token Range Iteration
183

184
Methods for working with ranges of tokens, useful for processing all tokens within an AST node.
185

186
```python { .api }
187
class ASTTokens:
188
    def token_range(self, first_token, last_token, include_extra=False) -> Iterator[Token]:
189
        """
190
        Yield all tokens in range from first_token to last_token (inclusive).
191
        
192
        Parameters:
193
        - first_token (Token): Starting token
194
        - last_token (Token): Ending token
195
        - include_extra (bool): Include non-coding tokens
196
        
197
        Yields:
198
        Token: Each token in the range
199
        """
200
        
201
    def get_tokens(self, node, include_extra=False) -> Iterator[Token]:
202
        """
203
        Yield all tokens for an AST node.
204
        
205
        Parameters:
206
        - node (ast.AST): AST node to get tokens for
207
        - include_extra (bool): Include non-coding tokens
208
        
209
        Yields:
210
        Token: Each token that belongs to the node
211
        """
212
```
213

214
#### Usage Example
215

216
```python
217
import asttokens
218
import ast
219

220
source = "result = func(x, y)  # calculation"
221
atok = asttokens.ASTTokens(source, parse=True)
222

223
# Get all tokens for assignment statement
224
assign_node = atok.tree.body[0]
225
tokens = list(atok.get_tokens(assign_node))
226
print([t.string for t in tokens])  # ['result', '=', 'func', '(', 'x', ',', 'y', ')']
227

228
# Include comments and whitespace
229
tokens_extra = list(atok.get_tokens(assign_node, include_extra=True))
230
print([t.string for t in tokens_extra])  # Includes '# calculation'
231

232
# Manual range iteration
233
first = assign_node.first_token
234
last = assign_node.last_token
235
tokens_range = list(atok.token_range(first, last))
236
print([t.string for t in tokens_range])  # Same as get_tokens without extra
237
```