0
# HDMF
1

2
The Hierarchical Data Modeling Framework (HDMF) is a Python package for working with hierarchical data. It provides APIs for specifying data models, reading and writing data to different storage backends (including HDF5 and Zarr), and representing data with Python objects. HDMF serves as the foundational technology for neuroscience data standards like NWB (Neurodata Without Borders) and provides comprehensive infrastructure for creating, validating, and managing complex scientific datasets.
3

4
## Package Information
5

6
- **Package Name**: hdmf
7
- **Language**: Python
8
- **Installation**: `pip install hdmf`
9
- **Documentation**: https://hdmf.readthedocs.io
10

11
## Core Imports
12

13
```python
14
import hdmf
15
```
16

17
Common imports for working with containers and data:
18

19
```python
20
from hdmf import Container, Data, HERDManager
21
from hdmf import docval, getargs
22
from hdmf import HDMFDataset
23
```
24

25
For HDF5 I/O operations:
26

27
```python
28
from hdmf.backends.hdf5 import HDF5IO, H5DataIO
29
```
30

31
For common data structures:
32

33
```python
34
from hdmf.common import DynamicTable, VectorData, VectorIndex
35
```
36

37
For specifications and validation:
38

39
```python
40
from hdmf.spec import GroupSpec, DatasetSpec, SpecCatalog
41
from hdmf.validate import ValidatorMap
42
```
43

44
For data utilities:
45

46
```python
47
from hdmf.data_utils import DataChunkIterator, DataIO
48
```
49

50
## Basic Usage
51

52
```python
53
import hdmf
54
from hdmf import Container, Data, docval
55
from hdmf.backends.hdf5 import HDF5IO
56
import numpy as np
57

58
# Create a simple data container
59
data_array = np.random.randn(100, 50)
60
data_container = Data(name='my_data', data=data_array)
61

62
# Create a container to hold the data
63
container = Container(name='my_container')
64
container.set_data_io(data_container)
65

66
# Write to HDF5 file
67
with HDF5IO('example.h5', mode='w') as io:
68
    io.write(container)
69

70
# Read back from HDF5 file
71
with HDF5IO('example.h5', mode='r') as io:
72
    read_container = io.read()
73
    print(f"Container name: {read_container.name}")
74
    print(f"Data shape: {read_container.data.shape}")
75
```
76

77
## Architecture
78

79
HDMF follows a specification-driven architecture with several key components:
80

81
- **Container System**: Hierarchical containers (`Container`, `Data`) that organize and hold data with metadata
82
- **Specification System**: Schema definitions that describe data structure and validation rules  
83
- **Build System**: Converts between container objects and storage builders for different backends
84
- **I/O Backends**: Pluggable storage backends (HDF5, Zarr) for reading/writing data
85
- **Validation System**: Comprehensive validation against specifications and schemas
86
- **Type System**: Dynamic type registration and validation with ontology support
87

88
This design enables HDMF to serve as both a standalone framework and the foundation for domain-specific standards like NWB, providing strong typing, metadata preservation, and cross-platform compatibility.
89

90
## Capabilities
91

92
### Container System
93

94
Core container classes for organizing hierarchical data structures with metadata, parent-child relationships, and data management capabilities.
95

96
```python { .api }
97
class Container:
98
    def __init__(self, name: str): ...
99
    def set_data_io(self, data_io): ...
100
    def get_ancestor(self, neurodata_type: str = None) -> 'AbstractContainer': ...
101

102
class Data(Container):
103
    def __init__(self, name: str, data): ...
104
    def append(self, arg): ...
105
    def extend(self, arg): ...
106
    def get(self): ...
107

108
class HERDManager:
109
    def __init__(self): ...
110
    def link_resources(self, container: Container, resources: dict): ...
111
    def get_linked_resources(self, container: Container) -> dict: ...
112
```
113

114
[Container System](./containers.md)
115

116
### Utilities and Validation
117

118
Decorators and utilities for parameter validation, argument handling, and type checking throughout the HDMF ecosystem.
119

120
```python { .api }
121
def docval(*args, **kwargs):
122
    """Decorator for parameter validation and documentation."""
123

124
def getargs(arg_names, kwargs: dict):
125
    """Retrieve specified arguments from dictionary."""
126

127
def check_type(value, type_, name: str = None) -> bool:
128
    """Check if value matches expected type."""
129

130
def is_ragged(data) -> bool:
131
    """Test if array-like data is ragged."""
132
```
133

134
[Utilities](./utils.md)
135

136
### I/O Backends  
137

138
Reading and writing data to different storage formats with comprehensive backend support for HDF5, Zarr, and extensible I/O system.
139

140
```python { .api }
141
class HDF5IO:
142
    def __init__(self, path: str, mode: str = 'r', **kwargs): ...
143
    def write(self, container, **kwargs): ...
144
    def read(self, **kwargs) -> Container: ...
145
    def close(self): ...
146

147
class H5DataIO:
148
    def __init__(self, data, **kwargs): ...
149
    @property
150
    def data(self): ...
151
    @property
152
    def io_settings(self) -> dict: ...
153
```
154

155
[I/O Backends](./io-backends.md)
156

157
### Specification System
158

159
Schema definition and management for data models, including namespace catalogs, specification readers/writers, and validation rules.
160

161
```python { .api }
162
class SpecCatalog:
163
    def __init__(self): ...
164
    def register_spec(self, spec, source_file: str = None): ...
165
    def get_spec(self, neurodata_type: str) -> 'BaseStorageSpec': ...
166

167
class GroupSpec:
168
    def __init__(self, doc: str, name: str = None, **kwargs): ...
169

170
class DatasetSpec:
171
    def __init__(self, doc: str, name: str = None, **kwargs): ...
172
```
173

174
[Specification System](./specification.md)
175

176
### Build System
177

178
Converting containers to storage representations and managing type mappings between specifications and Python classes.
179

180
```python { .api }
181
class BuildManager:
182
    def __init__(self, type_map: 'TypeMap'): ...
183
    def build(self, container, source: str = None, **kwargs) -> 'Builder': ...
184

185
class TypeMap:
186
    def __init__(self, namespaces: 'NamespaceCatalog'): ...
187
    def register_container_type(self, namespace: str, data_type: str, container_cls): ...
188
```
189

190
[Build System](./build-system.md)
191

192
### Common Data Structures
193

194
Pre-built data structures for scientific data including dynamic tables, vector data, sparse matrices, and multi-container systems.
195

196
```python { .api }
197
class DynamicTable(Container):
198
    def __init__(self, name: str, description: str, **kwargs): ...
199
    def add_row(self, **kwargs): ...
200
    def to_dataframe(self): ...
201

202
class VectorData(Data):
203
    def __init__(self, name: str, description: str, data, **kwargs): ...
204

205
class CSRMatrix(Container):
206
    def __init__(self, data, indices, indptr, shape: tuple, **kwargs): ...
207
```
208

209
[Common Data Structures](./common-data.md)
210

211
### Query System
212

213
Querying and filtering capabilities for datasets and containers with reference resolution and advanced data access patterns.
214

215
```python { .api }
216
class HDMFDataset:
217
    def __getitem__(self, key): ...
218
    def append(self, data): ...
219

220
class ContainerResolver:
221
    def __init__(self, type_map: 'TypeMap', container: Container): ...
222
```
223

224
[Query System](./query.md)
225

226
### Term Sets and Ontologies  
227

228
Integration with ontologies and controlled vocabularies through term sets, type configuration, and semantic validation.
229

230
```python { .api }
231
class TermSet:
232
    def __init__(self, term_schema_path: str = None, **kwargs): ...
233
    def validate(self, value): ...
234

235
class TermSetWrapper:
236
    def __init__(self, value, field: str, termset: TermSet, **kwargs): ...
237

238
class TypeConfigurator:
239
    @staticmethod
240
    def get_config(): ...
241
    @staticmethod
242
    def load_type_config(config_path: str): ...
243
```
244

245
[Term Sets](./term-sets.md)
246

247
### Validation System
248

249
Comprehensive validation of data against specifications with detailed error reporting and schema compliance checking.
250

251
```python { .api }
252
class ValidatorMap:
253
    def __init__(self): ...
254
    def register_validator(self, neurodata_type: str, validator): ...
255

256
class Validator:
257
    def __init__(self, spec): ...
258
    def validate(self, builder): ...
259
```
260

261
[Validation](./validation.md)
262

263
### Data Utilities
264

265
Essential utilities for handling large datasets, chunk iterators, and I/O configurations with efficient memory management and streaming operations.
266

267
```python { .api }
268
class DataChunkIterator:
269
    def __init__(self, data, **kwargs): ...
270
    def __next__(self): ...
271

272
class DataIO:
273
    def __init__(self, data, **kwargs): ...
274

275
def append_data(data, new_data): ...
276
def extend_data(data, extension_data): ...
277
```
278

279
[Data Utilities](./data-utils.md)
280

281
### Testing Utilities
282

283
Test case classes and utilities for testing HDMF extensions and applications with support for HDF5 round-trip testing.
284

285
```python { .api }
286
class TestCase:
287
    def setUp(self): ...
288
    def tearDown(self): ...
289

290
class H5RoundTripMixin:
291
    def test_roundtrip(self): ...
292

293
def remove_test_file(filename: str): ...
294
```
295

296
Testing utilities are available from `hdmf.testing` for building test suites.