0
# Token Operations and Local Tokenizer
1

2
Count tokens and compute detailed token information for content, with support for local tokenization without API calls. Token counting helps estimate costs, manage context limits, and optimize prompts.
3

4
## Capabilities
5

6
### Count Tokens
7

8
Count the number of tokens in content using the API.
9

10
```python { .api }
11
def count_tokens(
12
    *,
13
    model: str,
14
    contents: Union[str, list[Content], Content],
15
    config: Optional[CountTokensConfig] = None
16
) -> CountTokensResponse:
17
    """
18
    Count tokens in content via API.
19

20
    Parameters:
21
        model (str): Model identifier (e.g., 'gemini-2.0-flash').
22
        contents (Union[str, list[Content], Content]): Content to count tokens for.
23
        config (CountTokensConfig, optional): Configuration including:
24
            - generation_config: Generation config to count with
25
            - system_instruction: System instruction to include
26
            - tools: Tools to include
27

28
    Returns:
29
        CountTokensResponse: Token count including prompt tokens.
30

31
    Raises:
32
        ClientError: For client errors (4xx status codes)
33
        ServerError: For server errors (5xx status codes)
34
    """
35
    ...
36

37
async def count_tokens(
38
    *,
39
    model: str,
40
    contents: Union[str, list[Content], Content],
41
    config: Optional[CountTokensConfig] = None
42
) -> CountTokensResponse:
43
    """Async version of count_tokens."""
44
    ...
45
```
46

47
**Usage Example:**
48

49
```python
50
from google.genai import Client
51

52
client = Client(api_key='YOUR_API_KEY')
53

54
# Count tokens in text
55
response = client.models.count_tokens(
56
    model='gemini-2.0-flash',
57
    contents='How many tokens is this sentence?'
58
)
59

60
print(f"Total tokens: {response.total_tokens}")
61

62
# Count tokens with context
63
from google.genai.types import CountTokensConfig, Content, Part
64

65
config = CountTokensConfig(
66
    system_instruction='You are a helpful assistant.'
67
)
68

69
response = client.models.count_tokens(
70
    model='gemini-2.0-flash',
71
    contents=[
72
        Content(role='user', parts=[Part(text='Hello')]),
73
        Content(role='model', parts=[Part(text='Hi there!')]),
74
        Content(role='user', parts=[Part(text='How are you?')])
75
    ],
76
    config=config
77
)
78

79
print(f"Total tokens including context: {response.total_tokens}")
80
```
81

82
### Compute Tokens
83

84
Compute detailed token information including token IDs via API.
85

86
```python { .api }
87
def compute_tokens(
88
    *,
89
    model: str,
90
    contents: Union[str, list[Content], Content],
91
    config: Optional[ComputeTokensConfig] = None
92
) -> ComputeTokensResponse:
93
    """
94
    Compute detailed token information via API.
95

96
    Parameters:
97
        model (str): Model identifier.
98
        contents (Union[str, list[Content], Content]): Content to tokenize.
99
        config (ComputeTokensConfig, optional): Configuration.
100

101
    Returns:
102
        ComputeTokensResponse: Detailed token information including token IDs.
103

104
    Raises:
105
        ClientError: For client errors (4xx status codes)
106
        ServerError: For server errors (5xx status codes)
107
    """
108
    ...
109

110
async def compute_tokens(
111
    *,
112
    model: str,
113
    contents: Union[str, list[Content], Content],
114
    config: Optional[ComputeTokensConfig] = None
115
) -> ComputeTokensResponse:
116
    """Async version of compute_tokens."""
117
    ...
118
```
119

120
**Usage Example:**
121

122
```python
123
from google.genai import Client
124

125
client = Client(api_key='YOUR_API_KEY')
126

127
response = client.models.compute_tokens(
128
    model='gemini-2.0-flash',
129
    contents='Hello world'
130
)
131

132
print(f"Total tokens: {response.total_tokens}")
133

134
for token_info in response.tokens_info:
135
    for token in token_info.tokens:
136
        print(f"Token: '{token.token}' (ID: {token.token_id})")
137
```
138

139
### Local Tokenizer
140

141
Perform token counting locally without API calls (Experimental). The local tokenizer downloads model tokenizer data once and performs tokenization locally, useful for offline scenarios or reducing API calls.
142

143
```python { .api }
144
class LocalTokenizer:
145
    """Local tokenizer for text-only token counting (Experimental)."""
146

147
    def __init__(self, model_name: str):
148
        """
149
        Initialize local tokenizer for a model.
150

151
        Parameters:
152
            model_name (str): Model name (e.g., 'gemini-2.0-flash').
153
                Downloads tokenizer data on first use.
154

155
        Raises:
156
            ValueError: If model doesn't support local tokenization
157
        """
158
        ...
159

160
    def count_tokens(
161
        self,
162
        contents: Union[str, list[Content], Content],
163
        *,
164
        config: Optional[CountTokensConfig] = None
165
    ) -> CountTokensResult:
166
        """
167
        Count tokens locally (text-only).
168

169
        Parameters:
170
            contents (Union[str, list[Content], Content]): Content to count.
171
                Only text parts are counted; images/audio are ignored.
172
            config (CountTokensConfig, optional): Configuration.
173

174
        Returns:
175
            CountTokensResult: Token count. Note: May differ slightly from API
176
                count due to local approximation.
177
        """
178
        ...
179

180
    def compute_tokens(
181
        self,
182
        contents: Union[str, list[Content], Content]
183
    ) -> ComputeTokensResult:
184
        """
185
        Compute detailed token information locally.
186

187
        Parameters:
188
            contents (Union[str, list[Content], Content]): Content to tokenize.
189

190
        Returns:
191
            ComputeTokensResult: Detailed token information with IDs.
192
        """
193
        ...
194
```
195

196
**Usage Example:**
197

198
```python
199
from google.genai.local_tokenizer import LocalTokenizer
200

201
# Initialize tokenizer (downloads data on first use)
202
tokenizer = LocalTokenizer('gemini-2.0-flash')
203

204
# Count tokens locally
205
result = tokenizer.count_tokens('How many tokens is this?')
206
print(f"Total tokens (local): {result.total_tokens}")
207

208
# Compute detailed tokens
209
result = tokenizer.compute_tokens('Hello world')
210
for token_info in result.tokens_info:
211
    for token in token_info.tokens:
212
        print(f"Token: '{token.token}' (ID: {token.token_id})")
213
```
214

215
## Types
216

217
```python { .api }
218
from typing import Optional, Union, List
219

220
# Configuration types
221
class CountTokensConfig:
222
    """
223
    Configuration for token counting.
224

225
    Attributes:
226
        generation_config (GenerationConfig, optional): Generation config to include.
227
        system_instruction (Union[str, Content], optional): System instruction to include.
228
        tools (list[Tool], optional): Tools to include in count.
229
    """
230
    generation_config: Optional[GenerationConfig] = None
231
    system_instruction: Optional[Union[str, Content]] = None
232
    tools: Optional[list[Tool]] = None
233

234
class ComputeTokensConfig:
235
    """Configuration for computing tokens."""
236
    pass
237

238
# Response types
239
class CountTokensResponse:
240
    """
241
    Response from count_tokens API.
242

243
    Attributes:
244
        total_tokens (int): Total number of tokens.
245
        total_billable_characters (int, optional): Billable characters count (for embeddings).
246
    """
247
    total_tokens: int
248
    total_billable_characters: Optional[int] = None
249

250
class ComputeTokensResponse:
251
    """
252
    Response from compute_tokens API.
253

254
    Attributes:
255
        tokens_info (list[TokensInfo]): Detailed token information per content part.
256
        total_tokens (int): Total number of tokens.
257
    """
258
    tokens_info: list[TokensInfo]
259
    total_tokens: int
260

261
class TokensInfo:
262
    """
263
    Token information for content.
264

265
    Attributes:
266
        tokens (list[Token]): Individual tokens.
267
        role (str, optional): Content role.
268
    """
269
    tokens: list[Token]
270
    role: Optional[str] = None
271

272
class Token:
273
    """
274
    Individual token.
275

276
    Attributes:
277
        token (str): Token string.
278
        token_id (int): Token ID in model vocabulary.
279
    """
280
    token: str
281
    token_id: int
282

283
# Local tokenizer result types
284
class CountTokensResult:
285
    """
286
    Result from local token counting.
287

288
    Attributes:
289
        total_tokens (int): Total number of tokens.
290
    """
291
    total_tokens: int
292

293
class ComputeTokensResult:
294
    """
295
    Result from local token computation.
296

297
    Attributes:
298
        tokens_info (list[TokensInfo]): Detailed token information.
299
        total_tokens (int): Total number of tokens.
300
    """
301
    tokens_info: list[TokensInfo]
302
    total_tokens: int
303

304
# Core types (shared)
305
class Content:
306
    """Content container."""
307
    parts: list[Part]
308
    role: Optional[str] = None
309

310
class Part:
311
    """Content part."""
312
    text: Optional[str] = None
313
    inline_data: Optional[Blob] = None
314

315
class Blob:
316
    """Binary data."""
317
    mime_type: str
318
    data: bytes
319

320
class GenerationConfig:
321
    """Generation configuration."""
322
    pass
323

324
class Tool:
325
    """Tool definition."""
326
    pass
327
```
328