0
# Parsing System
1
2
Dual parser implementations for Sphinx and docutils environments, with factory functions for creating configured markdown-it-py parsers and comprehensive directive/option parsing support.
3
4
## Capabilities
5
6
### Sphinx Parser
7
8
MyST parser implementation for Sphinx documentation system, providing seamless integration with Sphinx's parsing pipeline and configuration system.
9
10
```python { .api }
11
class MystParser:
12
"""
13
Main Sphinx parser for MyST Markdown files.
14
15
Attributes:
16
supported: Tuple of supported file extensions
17
"""
18
supported: tuple[str, ...] = ("md", "markdown", "myst")
19
20
def parse(self, inputstring: str, document) -> None:
21
"""
22
Parse MyST markdown source to populate docutils document.
23
24
Args:
25
inputstring: MyST markdown source text
26
document: Docutils document node to populate
27
"""
28
```
29
30
Usage example:
31
32
```python
33
from myst_parser.parsers.sphinx_ import MystParser
34
from docutils.utils import new_document
35
from docutils.frontend import OptionParser
36
from docutils.parsers.rst import Parser as RstParser
37
38
# Create Sphinx parser
39
parser = MystParser()
40
41
# Check supported extensions
42
print(parser.supported) # ("md", "markdown", "myst")
43
44
# Create document for parsing
45
settings = OptionParser(components=(RstParser,)).get_default_values()
46
document = new_document('<rst-doc>', settings=settings)
47
48
# Parse MyST content
49
source = """
50
# MyST Document
51
52
This is MyST markdown with {emphasis}`substitution syntax`.
53
54
:::{note}
55
This is a directive.
56
:::
57
"""
58
59
parser.parse(source, document)
60
print(document.pformat())
61
```
62
63
### Docutils Parser
64
65
Pure docutils parser implementation for MyST markdown, enabling use of MyST outside of Sphinx environments with full docutils compatibility.
66
67
```python { .api }
68
class Parser:
69
"""
70
Docutils parser for MyST markdown.
71
72
Provides MyST parsing capabilities in pure docutils environments
73
without Sphinx dependencies.
74
"""
75
76
def parse(self, inputstring: str, document) -> None:
77
"""
78
Parse MyST markdown source to populate docutils document.
79
80
Args:
81
inputstring: MyST markdown source text
82
document: Docutils document node to populate
83
"""
84
85
class Unset:
86
"""Sentinel class for unset settings."""
87
88
# Module constant
89
DOCUTILS_UNSET: Unset
90
```
91
92
Usage example:
93
94
```python
95
from myst_parser.parsers.docutils_ import Parser
96
from docutils.core import publish_doctree, publish_parts
97
98
# Create docutils parser
99
parser = Parser()
100
101
# Parse to document tree
102
source = """
103
# MyST Document
104
105
- [x] Completed task
106
- [ ] Pending task
107
108
:::{admonition} Note
109
This is an admonition.
110
:::
111
"""
112
113
# Method 1: Direct parsing
114
document = publish_doctree(source, parser=parser)
115
print(document.pformat())
116
117
# Method 2: Publish with specific output
118
html_output = publish_parts(
119
source,
120
parser=parser,
121
writer_name='html5'
122
)
123
print(html_output['html_body'])
124
```
125
126
### Markdown Parser Factory
127
128
Factory function for creating configured markdown-it-py parsers with MyST extensions and custom renderers, providing the core parsing engine.
129
130
```python { .api }
131
def create_md_parser(config: MdParserConfig, renderer) -> MarkdownIt:
132
"""
133
Create markdown parser with MyST configuration.
134
135
Args:
136
config: MdParserConfig instance with parsing options
137
renderer: Renderer instance for token processing
138
139
Returns:
140
Configured MarkdownIt parser instance with MyST extensions
141
"""
142
```
143
144
Usage example:
145
146
```python
147
from myst_parser.parsers.mdit import create_md_parser
148
from myst_parser.config.main import MdParserConfig
149
from myst_parser.mdit_to_docutils.base import DocutilsRenderer, make_document
150
151
# Create configuration
152
config = MdParserConfig(
153
enable_extensions={"tasklist", "deflist", "substitution"},
154
heading_anchors=2,
155
footnote_sort=True
156
)
157
158
# Create document and renderer
159
document = make_document("example.md")
160
renderer = DocutilsRenderer(document)
161
162
# Create configured parser
163
md_parser = create_md_parser(config, renderer)
164
165
# Parse content
166
source = """
167
# Document Title
168
169
Term
170
: Definition here
171
172
- [x] Completed
173
- [ ] Pending
174
175
{emphasis}`Substitution text`
176
"""
177
178
# Render to docutils
179
result = md_parser.render(source)
180
print(document.pformat())
181
```
182
183
### Directive Parsing
184
185
Comprehensive directive parsing system for MyST directives, providing validation, argument processing, and warning generation.
186
187
```python { .api }
188
class ParseWarnings:
189
"""
190
Dataclass for parsing warnings.
191
192
Attributes:
193
warnings: List of warning messages
194
errors: List of error messages
195
"""
196
warnings: list[str] = field(default_factory=list)
197
errors: list[str] = field(default_factory=list)
198
199
class DirectiveParsingResult:
200
"""
201
Dataclass for directive parsing results.
202
203
Attributes:
204
arguments: Parsed directive arguments
205
options: Parsed directive options
206
content: Directive content lines
207
warnings: Parsing warnings and errors
208
"""
209
arguments: list[str] = field(default_factory=list)
210
options: dict[str, any] = field(default_factory=dict)
211
content: list[str] = field(default_factory=list)
212
warnings: ParseWarnings = field(default_factory=ParseWarnings)
213
214
def parse_directive_text(
215
directive_class,
216
first_line: str,
217
content: list[str],
218
validate_options: bool = True,
219
**kwargs
220
) -> DirectiveParsingResult:
221
"""
222
Parse and validate directive text.
223
224
Args:
225
directive_class: Docutils directive class
226
first_line: First line containing directive name and arguments
227
content: List of directive content lines
228
validate_options: Whether to validate directive options
229
**kwargs: Additional parsing options
230
231
Returns:
232
DirectiveParsingResult with parsed components and warnings
233
234
Raises:
235
MarkupError: When directive parsing fails critically
236
"""
237
```
238
239
Usage example:
240
241
```python
242
from myst_parser.parsers.directives import parse_directive_text, DirectiveParsingResult
243
from docutils.parsers.rst.directives import admonitions
244
245
# Parse note directive
246
result = parse_directive_text(
247
admonitions.Note,
248
"note Important Information",
249
["This is important content.", "", "With multiple paragraphs."],
250
validate_options=True
251
)
252
253
print(result.arguments) # ["Important Information"]
254
print(result.content) # ["This is important content.", "", "With multiple paragraphs."]
255
print(result.warnings.warnings) # Any parsing warnings
256
257
# Parse directive with options
258
result = parse_directive_text(
259
admonitions.Note,
260
"note",
261
[":class: custom-note", ":name: my-note", "", "Content here"],
262
validate_options=True
263
)
264
265
print(result.options) # {"class": ["custom-note"], "name": "my-note"}
266
```
267
268
### Option String Parsing
269
270
Advanced option string parsing for directive options, supporting complex option formats with proper tokenization and validation.
271
272
```python { .api }
273
class Position:
274
"""Line/column position tracking."""
275
line: int
276
column: int
277
278
class StreamBuffer:
279
"""Buffer for parsing option strings."""
280
281
def __init__(self, data: str): ...
282
def read(self, n: int = -1) -> str: ...
283
def peek(self, n: int = 1) -> str: ...
284
285
class Token:
286
"""Base token class."""
287
value: str
288
position: Position
289
290
class KeyToken(Token):
291
"""Token representing option key."""
292
293
class ValueToken(Token):
294
"""Token representing option value."""
295
296
class ColonToken(Token):
297
"""Token representing colon separator."""
298
299
class TokenizeError(Exception):
300
"""Exception raised during tokenization."""
301
302
class State:
303
"""Parser state management."""
304
305
def __init__(self): ...
306
def push_state(self, new_state: str): ...
307
def pop_state(self) -> str: ...
308
309
def options_to_items(options_string: str) -> list[tuple[str, str]]:
310
"""
311
Parse options string to key-value items.
312
313
Args:
314
options_string: Directive options as string
315
316
Returns:
317
List of (key, value) tuples
318
319
Raises:
320
TokenizeError: When tokenization fails
321
"""
322
```
323
324
Usage example:
325
326
```python
327
from myst_parser.parsers.options import options_to_items, TokenizeError
328
329
# Parse simple options
330
options = ":class: custom-class\n:name: my-element"
331
items = options_to_items(options)
332
print(items) # [("class", "custom-class"), ("name", "my-element")]
333
334
# Parse complex options with multiple values
335
options = """
336
:class: class1 class2 class3
337
:style: color: red; font-weight: bold
338
:data-attr: complex-value
339
"""
340
341
try:
342
items = options_to_items(options)
343
for key, value in items:
344
print(f"{key}: {value}")
345
except TokenizeError as e:
346
print(f"Parsing error: {e}")
347
```
348
349
## Types
350
351
Parsing-related type definitions:
352
353
```python { .api }
354
class ParseWarnings:
355
"""Container for directive parsing warnings and errors."""
356
warnings: list[str]
357
errors: list[str]
358
359
class DirectiveParsingResult:
360
"""Results from directive parsing operation."""
361
arguments: list[str]
362
options: dict[str, any]
363
content: list[str]
364
warnings: ParseWarnings
365
366
class Position:
367
"""Position tracking for parsing operations."""
368
line: int
369
column: int
370
371
class TokenizeError(Exception):
372
"""Exception raised during option string tokenization."""
373
374
class Unset:
375
"""Sentinel class for unset configuration values."""
376
377
# Re-exported from docutils
378
MarkupError = docutils.parsers.rst.directives.MarkupError
379
```