0
# ASDF - Advanced Scientific Data Format
1

2
ASDF is a next-generation interchange format for scientific data. This Python implementation provides hierarchical and human-readable metadata in YAML format, efficient binary array storage with memory mapping and flexible compression, content validation using JSON Schema, and native support for most Python data types with an extensible API for custom objects.
3

4
## Package Information
5

6
- **Package Name**: asdf
7
- **Language**: Python
8
- **Installation**: `pip install asdf`
9
- **Requires Python**: >=3.9
10

11
## Core Imports
12

13
```python
14
import asdf
15
```
16

17
Common patterns:
18

19
```python
20
from asdf import AsdfFile, open, load, dump
21
```
22

23
## Basic Usage
24

25
```python
26
import asdf
27
import numpy as np
28

29
# Create some data
30
sequence = np.arange(100)
31
squares = sequence**2
32
random = np.random.random(100)
33

34
# Store the data in an arbitrarily nested dictionary
35
tree = {
36
    "foo": 42,
37
    "name": "Monty",
38
    "sequence": sequence,
39
    "powers": {"squares": squares},
40
    "random": random,
41
}
42

43
# Create and write ASDF file
44
af = asdf.AsdfFile(tree)
45
af.write_to("example.asdf")
46

47
# Read the file back
48
with asdf.open("example.asdf") as af:
49
    print(af.tree["foo"])  # 42
50
    print(af.tree["sequence"])  # numpy array
51
```
52

53
## Architecture
54

55
ASDF follows a layered architecture:
56

57
- **File Format**: YAML header with binary array blocks
58
- **AsdfFile**: Main file object managing tree data and metadata
59
- **Extensions**: Plugin system for custom types and validation
60
- **Block Manager**: Handles binary array storage and compression
61
- **Schema Validation**: JSON Schema-based validation system
62
- **Tree Structure**: Hierarchical data organization with references
63

64
## Capabilities
65

66
### File Operations
67

68
Core file handling functionality for creating, reading, writing, and managing ASDF files. Includes both high-level convenience functions and low-level file management.
69

70
```python { .api }
71
class AsdfFile:
72
    def __init__(self, tree=None, uri=None, extensions=None, version=None, 
73
                 ignore_unrecognized_tag=False, memmap=False, lazy_load=True, 
74
                 custom_schema=None): ...
75

76
def open(fd, uri=None, mode=None, validate_checksums=False, extensions=None, 
77
         ignore_unrecognized_tag=False, _force_raw_types=False, memmap=False, 
78
         lazy_tree=NotSet, lazy_load=True, custom_schema=None, 
79
         strict_extension_check=False, ignore_missing_extensions=False): ...
80

81
def load(fp, *, uri=None, validate_checksums=False, extensions=None, 
82
         custom_schema=None): ...
83

84
def dump(tree, fp, *, version=None, extensions=None, all_array_storage=NotSet, 
85
         all_array_compression=NotSet, compression_kwargs=NotSet, pad_blocks=False, 
86
         custom_schema=None): ...
87
```
88

89
[File Operations](./file-operations.md)
90

91
### Data Serialization
92

93
High-level functions for serializing and deserializing Python objects to/from ASDF format, providing convenient string-based operations alongside file-based operations.
94

95
```python { .api }
96
def loads(asdf_string, *, uri=None, validate_checksums=False, extensions=None, 
97
          custom_schema=None): ...
98

99
def dumps(tree, *, version=None, extensions=None, all_array_storage=NotSet, 
100
          all_array_compression=NotSet, compression_kwargs=NotSet, pad_blocks=False, 
101
          custom_schema=None) -> str: ...
102
```
103

104
[Data Serialization](./data-serialization.md)
105

106
### Configuration Management
107

108
Global and context-specific configuration for controlling ASDF behavior including validation, version defaults, array handling, and I/O settings.
109

110
```python { .api }
111
def get_config(): ...
112
def config_context(): ...
113

114
class AsdfConfig:
115
    validate_on_read: bool
116
    default_version: str
117
    io_block_size: int
118
    array_inline_threshold: int
119
    all_array_storage: str
120
    all_array_compression: str
121
    all_array_compression_kwargs: dict
122
    legacy_fill_schema_defaults: bool
123
    default_array_save_base: bool
124
    convert_unknown_ndarray_subclasses: bool
125
    lazy_tree: bool
126
    resource_mappings: list  # read-only
127
    resource_manager: object  # read-only
128
    extensions: list  # read-only
129
```
130

131
[Configuration Management](./configuration.md)
132

133
### Core Data Types
134

135
Built-in ASDF data types for handling specialized scientific data including large integers, external array references, and streaming arrays.
136

137
```python { .api }
138
class IntegerType:
139
    def __init__(self, value, storage_type="internal"): ...
140

141
class ExternalArrayReference:
142
    def __init__(self, fileuri, target, dtype, shape): ...
143

144
class Stream:
145
    def __init__(self, shape, dtype, strides=None): ...
146
```
147

148
[Core Data Types](./core-data-types.md)
149

150
### Extension System
151

152
Plugin architecture for extending ASDF with custom types, validators, compressors, and tags. Enables integration with domain-specific libraries and custom data formats.
153

154
```python { .api }
155
class Extension:
156
    extension_uri: str
157
    tags: List[str]
158
    converters: List[Converter]
159
    compressors: List[Compressor]
160
    validators: List[Validator]
161

162
class Converter:
163
    def can_convert(self, obj): ...
164
    def convert(self, obj, **kwargs): ...
165

166
class ExtensionManager:
167
    def get_extensions(self): ...
168
    def get_converter_for_type(self, typ): ...
169
```
170

171
[Extension System](./extension-system.md)
172

173
### Utilities and Helpers
174

175
Utility functions for inspecting ASDF files, testing extensions, and working with ASDF data structures programmatically.
176

177
```python { .api }
178
def info(node_or_path, max_rows=24, max_cols=120, show_values=True): ...
179
```
180

181
[Utilities and Helpers](./utilities.md)
182

183
## Types
184

185
```python { .api }
186
class ValidationError(Exception):
187
    """Raised when ASDF validation fails."""
188

189
class AsdfObject(dict):
190
    """Dict-like container for ASDF objects with special YAML serialization."""
191

192
class Software(dict):
193
    """Metadata about software used to create ASDF file."""
194

195
class HistoryEntry(dict):
196
    """Single entry in ASDF file history."""
197

198
__version__: str
199
    """Current version of the ASDF library."""
200
```