0
# Low-Level File Access
1

2
High-performance, memory-efficient access to electrophysiology files through Neo's RawIO interface. This low-level API provides direct access to signal chunks, header information, and metadata without creating full Neo objects, enabling efficient processing of large datasets and real-time applications.
3

4
## Capabilities
5

6
### RawIO Class Detection
7

8
Functions for automatically detecting and instantiating appropriate RawIO classes.
9

10
```python { .api }
11
def get_rawio(filename):
12
    """
13
    Auto-detect file format and return appropriate RawIO class instance.
14
    
15
    Parameters:
16
    - filename (str): Path to file or directory
17
    
18
    Returns:
19
    RawIO instance for the detected file format
20
    
21
    Raises:
22
    IOError: If format cannot be determined or no compatible RawIO found
23
    """
24
```
25

26
### Core RawIO Interface
27

28
Base interface shared by all RawIO classes for consistent low-level data access.
29

30
```python { .api }
31
class BaseRawIO:
32
    """Base class defining the RawIO interface for all file formats."""
33
    
34
    def parse_header(self):
35
        """Parse file header and extract metadata."""
36
        
37
    def get_signal_size(self, block_index, seg_index, channel_indexes=None):
38
        """
39
        Get the size of analog signal data.
40
        
41
        Parameters:
42
        - block_index (int): Block index in file
43
        - seg_index (int): Segment index within block  
44
        - channel_indexes (list, optional): Specific channels
45
        
46
        Returns:
47
        int: Number of samples in the signal
48
        """
49
        
50
    def get_analogsignal_chunk(self, block_index, seg_index, 
51
                              i_start=None, i_stop=None, 
52
                              channel_indexes=None):
53
        """
54
        Get analog signal data chunk.
55
        
56
        Parameters:
57
        - block_index (int): Block index
58
        - seg_index (int): Segment index
59
        - i_start (int, optional): Start sample index
60
        - i_stop (int, optional): Stop sample index  
61
        - channel_indexes (list, optional): Channel indices to read
62
        
63
        Returns:
64
        numpy.ndarray: Signal data chunk (samples x channels)
65
        """
66
        
67
    def get_spike_timestamps(self, block_index, seg_index, unit_index):
68
        """
69
        Get spike timestamps for a specific unit.
70
        
71
        Parameters:
72
        - block_index (int): Block index
73
        - seg_index (int): Segment index
74
        - unit_index (int): Unit/channel index
75
        
76
        Returns:
77
        numpy.ndarray: Spike timestamps
78
        """
79
        
80
    def get_spike_raw_waveforms(self, block_index, seg_index, unit_index):
81
        """
82
        Get raw spike waveform data.
83
        
84
        Parameters:
85
        - block_index (int): Block index  
86
        - seg_index (int): Segment index
87
        - unit_index (int): Unit index
88
        
89
        Returns:
90
        numpy.ndarray: Waveform data (spikes x samples x channels)
91
        """
92
```
93

94
### Hardware-Specific RawIO Classes
95

96
Low-level access for major hardware manufacturer formats.
97

98
```python { .api }
99
class AxonRawIO:
100
    """
101
    Low-level Axon file format access (.abf).
102
    
103
    Provides efficient access to Molecular Devices pCLAMP and AxoScope
104
    files with direct header parsing and signal chunk retrieval.
105
    """
106
    def __init__(self, filename): ...
107
    def parse_header(self): ...
108
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
109
    
110
    # Axon-specific properties
111
    header: dict                    # Parsed ABF header information
112
    sampling_rate: float           # Sampling rate in Hz
113
    protocol_info: dict            # Protocol and stimulus information
114

115
class BlackrockRawIO:
116
    """Low-level Blackrock Microsystems file access (.nev, .ns1-.ns6)."""
117
    def __init__(self, filename): ...
118
    def parse_header(self): ...
119
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
120
    def get_spike_timestamps(self, block_index, seg_index, unit_index): ...
121

122
class PlexonRawIO:
123
    """Low-level Plexon file format access (.plx)."""
124
    def __init__(self, filename): ...
125
    def parse_header(self): ...
126
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
127
    def get_spike_timestamps(self, block_index, seg_index, unit_index): ...
128
    def get_spike_raw_waveforms(self, block_index, seg_index, unit_index): ...
129

130
class Plexon2RawIO:
131
    """Low-level Plexon2 file format access (.pl2)."""
132
    def __init__(self, filename): ...
133
    def parse_header(self): ...
134
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
135

136
class IntanRawIO:
137
    """Low-level Intan Technologies file access (.rhd, .rhs)."""
138
    def __init__(self, filename): ...
139
    def parse_header(self): ...
140
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
141

142
class NeuralynxRawIO:
143
    """Low-level Neuralynx file format access (.ncs, .nev, .nse, .ntt)."""
144
    def __init__(self, dirname): ...
145
    def parse_header(self): ...
146
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
147
    def get_spike_timestamps(self, block_index, seg_index, unit_index): ...
148

149
class TdtRawIO:
150
    """Low-level Tucker-Davis Technologies file access (.tev, .tsq, .tnt)."""
151
    def __init__(self, dirname): ...
152
    def parse_header(self): ...
153
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
154

155
class Spike2RawIO:
156
    """Low-level Cambridge Electronic Design Spike2 access (.smr, .son)."""
157
    def __init__(self, filename): ...
158
    def parse_header(self): ...
159
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
160
```
161

162
### Multi-Electrode Array RawIO
163

164
Specialized low-level access for high-channel-count MEA and probe recordings.
165

166
```python { .api }
167
class BiocamRawIO:
168
    """Low-level 3Brain Biocam file access (.brw, .bxr)."""
169
    def __init__(self, filename): ...
170
    def parse_header(self): ...
171
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
172

173
class MaxwellRawIO:
174
    """Low-level Maxwell Biosystems file access (.raw.h5)."""
175
    def __init__(self, filename): ...
176
    def parse_header(self): ...
177
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
178

179
class NeuroNexusRawIO:
180
    """Low-level NeuroNexus probe file access (.xdaq)."""
181
    def __init__(self, filename): ...
182
    def parse_header(self): ...
183
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
184

185
class SpikeGadgetsRawIO:
186
    """Low-level SpikeGadgets file access (.rec, .mda)."""
187
    def __init__(self, filename): ...
188
    def parse_header(self): ...
189
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
190

191
class RawMCSRawIO:
192
    """Low-level Multi Channel Systems raw file access (.raw)."""
193
    def __init__(self, filename): ...
194
    def parse_header(self): ...
195
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
196
```
197

198
### Open Source Format RawIO
199

200
Low-level access for open source and analysis software formats.
201

202
```python { .api }
203
class OpenEphysRawIO:
204
    """Low-level Open Ephys GUI file access (.continuous, .events, .spikes)."""
205
    def __init__(self, dirname): ...
206
    def parse_header(self): ...
207
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
208

209
class OpenEphysBinaryRawIO:
210
    """Low-level Open Ephys binary file access (.dat)."""
211
    def __init__(self, dirname): ...
212
    def parse_header(self): ...
213
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
214

215
class PhyRawIO:
216
    """Low-level Phy spike sorting file access (params.py, cluster_info.tsv)."""
217
    def __init__(self, dirname): ...
218
    def parse_header(self): ...
219
    def get_spike_timestamps(self, block_index, seg_index, unit_index): ...
220
    def get_spike_raw_waveforms(self, block_index, seg_index, unit_index): ...
221

222
class SpikeGLXRawIO:
223
    """Low-level SpikeGLX acquisition file access (.bin, .meta)."""
224
    def __init__(self, dirname): ...
225
    def parse_header(self): ...
226
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
227
```
228

229
### Standard Format RawIO
230

231
Low-level access for industry-standard and research formats.
232

233
```python { .api }
234
class EDFRawIO:
235
    """Low-level European Data Format access (.edf, .bdf)."""
236
    def __init__(self, filename): ...
237
    def parse_header(self): ...
238
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
239

240
class NIXRawIO:
241
    """Low-level NIX format access (.nix)."""
242
    def __init__(self, filename): ...
243
    def parse_header(self): ...
244
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
245

246
class RawBinarySignalRawIO:
247
    """
248
    Low-level raw binary signal file access (.raw, .bin).
249
    
250
    Flexible access to custom binary formats with user-specified
251
    data layout parameters.
252
    """
253
    def __init__(self, filename, dtype='float32', sampling_rate=1.0, 
254
                 nb_channel=1, signal_gain=1.0, signal_offset=0.0): ...
255
    def parse_header(self): ...
256
    def get_analogsignal_chunk(self, block_index, seg_index, i_start, i_stop, channel_indexes): ...
257
```
258

259
## Usage Examples
260

261
### Basic RawIO Operations
262

263
```python
264
import neo.rawio
265
import numpy as np
266

267
# Auto-detect and create RawIO instance
268
rawio = neo.rawio.get_rawio('recording.abf')
269

270
# Parse file header
271
rawio.parse_header()
272

273
# Examine file structure
274
print(f"Number of blocks: {rawio.block_count()}")
275
print(f"Number of segments: {rawio.segment_count(0)}")
276
print(f"Number of channels: {rawio.signal_channels_count()}")
277
print(f"Sampling rate: {rawio.get_signal_sampling_rate()}")
278

279
# Get channel information
280
signal_channels = rawio.header['signal_channels']
281
for i, ch in enumerate(signal_channels):
282
    print(f"Channel {i}: {ch['name']} ({ch['units']})")
283
```
284

285
### Efficient Data Access
286

287
```python
288
# Get signal chunk without loading entire file
289
chunk = rawio.get_analogsignal_chunk(
290
    block_index=0,
291
    seg_index=0,
292
    i_start=1000,      # Start at sample 1000
293
    i_stop=2000,       # End at sample 2000  
294
    channel_indexes=[0, 1, 2]  # Only channels 0, 1, 2
295
)
296

297
print(f"Chunk shape: {chunk.shape}")  # (1000 samples, 3 channels)
298
print(f"Data type: {chunk.dtype}")
299
print(f"Memory usage: {chunk.nbytes} bytes")
300

301
# Stream processing of large files
302
chunk_size = 10000  # Process 10k samples at a time
303
total_samples = rawio.get_signal_size(0, 0)
304
n_chunks = total_samples // chunk_size
305

306
processed_data = []
307
for i in range(n_chunks):
308
    i_start = i * chunk_size  
309
    i_stop = min((i + 1) * chunk_size, total_samples)
310
    
311
    chunk = rawio.get_analogsignal_chunk(0, 0, i_start, i_stop)
312
    
313
    # Process chunk (e.g., filter, downsample, etc.)
314
    processed_chunk = np.mean(chunk, axis=1)  # Example: channel average
315
    processed_data.append(processed_chunk)
316

317
final_result = np.concatenate(processed_data)
318
print(f"Processed {len(final_result)} samples")
319
```
320

321
### Spike Data Access
322

323
```python
324
# Access spike data (for formats that support it)
325
if rawio.spike_channels_count() > 0:
326
    spike_channels = rawio.header['spike_channels']
327
    
328
    for unit_index in range(len(spike_channels)):
329
        # Get spike timestamps
330
        spike_times = rawio.get_spike_timestamps(
331
            block_index=0,
332
            seg_index=0, 
333
            unit_index=unit_index
334
        )
335
        
336
        print(f"Unit {unit_index}: {len(spike_times)} spikes")
337
        
338
        # Get spike waveforms if available
339
        if hasattr(rawio, 'get_spike_raw_waveforms'):
340
            waveforms = rawio.get_spike_raw_waveforms(0, 0, unit_index)
341
            if waveforms is not None:
342
                print(f"  Waveform shape: {waveforms.shape}")
343
```
344

345
### Memory-Efficient Analysis
346

347
```python
348
def analyze_signal_statistics(rawio, block_index=0, seg_index=0):
349
    """Compute signal statistics without loading entire file."""
350
    
351
    total_samples = rawio.get_signal_size(block_index, seg_index)
352
    n_channels = rawio.signal_channels_count()
353
    chunk_size = 50000  # Adjust based on available memory
354
    
355
    # Initialize accumulators
356
    running_sum = np.zeros(n_channels)
357
    running_sum_squares = np.zeros(n_channels)
358
    sample_count = 0
359
    
360
    # Process file in chunks
361
    for i_start in range(0, total_samples, chunk_size):
362
        i_stop = min(i_start + chunk_size, total_samples)
363
        
364
        chunk = rawio.get_analogsignal_chunk(
365
            block_index, seg_index, i_start, i_stop
366
        )
367
        
368
        # Update statistics
369
        running_sum += np.sum(chunk, axis=0)
370
        running_sum_squares += np.sum(chunk**2, axis=0)
371
        sample_count += chunk.shape[0]
372
    
373
    # Compute final statistics
374
    means = running_sum / sample_count
375
    variances = (running_sum_squares / sample_count) - means**2
376
    stds = np.sqrt(variances)
377
    
378
    return means, stds
379

380
# Use the function
381
means, stds = analyze_signal_statistics(rawio)
382
print(f"Channel means: {means}")
383
print(f"Channel standard deviations: {stds}")
384
```
385

386
### Real-Time Processing Simulation
387

388
```python
389
def simulate_realtime_processing(rawio, chunk_duration=0.1):
390
    """Simulate real-time processing by reading time-based chunks."""
391
    
392
    sampling_rate = rawio.get_signal_sampling_rate()
393
    chunk_samples = int(chunk_duration * sampling_rate)
394
    total_samples = rawio.get_signal_size(0, 0)
395
    
396
    print(f"Simulating real-time processing:")
397
    print(f"  Chunk duration: {chunk_duration}s")
398
    print(f"  Chunk size: {chunk_samples} samples")
399
    print(f"  Total duration: {total_samples / sampling_rate:.2f}s")
400
    
401
    for i_start in range(0, total_samples, chunk_samples):
402
        i_stop = min(i_start + chunk_samples, total_samples)
403
        
404
        # Read chunk  
405
        chunk = rawio.get_analogsignal_chunk(0, 0, i_start, i_stop)
406
        
407
        # Simulate processing time
408
        import time
409
        processing_start = time.time()
410
        
411
        # Example processing: detect threshold crossings
412
        threshold = 3 * np.std(chunk, axis=0)
413
        crossings = np.sum(np.abs(chunk) > threshold, axis=0)
414
        
415
        processing_time = time.time() - processing_start
416
        chunk_time = i_start / sampling_rate
417
        
418
        print(f"  t={chunk_time:.1f}s: {crossings.sum()} crossings, "
419
              f"processed in {processing_time*1000:.1f}ms")
420

421
simulate_realtime_processing(rawio, chunk_duration=1.0)
422
```
423

424
### Format-Specific Operations
425

426
```python
427
# Axon-specific operations
428
if isinstance(rawio, neo.rawio.AxonRawIO):
429
    print("Axon ABF file detected")
430
    print(f"Protocol name: {rawio.header.get('protocol', 'Unknown')}")
431
    
432
    # Access ABF-specific metadata
433
    if hasattr(rawio, '_axon_info'):
434
        axon_info = rawio._axon_info
435
        print(f"Episode count: {axon_info.get('lEpisodesPerRun', 'Unknown')}")
436
        print(f"Sample interval: {axon_info.get('fADCSampleInterval', 'Unknown')} μs")
437

438
# Blackrock-specific operations  
439
elif isinstance(rawio, neo.rawio.BlackrockRawIO):
440
    print("Blackrock file detected")
441
    
442
    # Access NEV event data if available
443
    if rawio.event_channels_count() > 0:
444
        events = rawio.get_event_timestamps(0, 0, 0)
445
        print(f"Found {len(events)} digital events")
446

447
# SpikeGLX-specific operations
448
elif isinstance(rawio, neo.rawio.SpikeGLXRawIO):
449
    print("SpikeGLX file detected")
450
    
451
    # Access probe geometry information
452
    if hasattr(rawio, '_geometry'):
453
        geometry = rawio._geometry
454
        print(f"Probe has {len(geometry)} recording sites")
455
```
456

457
### Error Handling and Validation
458

459
```python
460
try:
461
    rawio = neo.rawio.get_rawio('data_file.ns5')
462
    rawio.parse_header()
463
    
464
except Exception as e:
465
    print(f"Failed to open file: {e}")
466
    
467
# Validate file integrity
468
try:
469
    # Test reading small chunk
470
    test_chunk = rawio.get_analogsignal_chunk(0, 0, 0, 100)
471
    print(f"File validation successful: {test_chunk.shape}")
472
    
473
except Exception as e:
474
    print(f"File validation failed: {e}")
475

476
# Check for required capabilities
477
if rawio.signal_channels_count() == 0:
478
    print("Warning: No analog signal channels found")
479
    
480
if rawio.spike_channels_count() == 0:
481
    print("Warning: No spike channels found") 
482
    
483
# Memory usage estimation
484
total_samples = rawio.get_signal_size(0, 0)
485
n_channels = rawio.signal_channels_count()
486
bytes_per_sample = 4  # Assuming float32
487
total_memory = total_samples * n_channels * bytes_per_sample
488
print(f"Full file would require {total_memory / 1e9:.2f} GB memory")
489
```
490

491
## Performance Considerations
492

493
### Memory Management
494
- RawIO avoids creating full Neo objects, reducing memory usage by 5-10x
495
- Use chunked processing for files larger than available RAM
496
- Optimal chunk sizes: 10k-100k samples depending on channel count
497

498
### I/O Efficiency  
499
- Channel selection reduces disk I/O and memory allocation
500
- Sequential access patterns are faster than random access
501
- Consider file system caching for repeated access to same regions
502

503
### Format-Specific Optimizations
504
- **ABF files**: Use lazy loading for multi-episode files
505
- **Blackrock files**: Leverage separate .ns files for different sampling rates
506
- **SpikeGLX**: Take advantage of memory-mapped file access
507
- **Binary formats**: Specify exact data types to avoid unnecessary conversions
508

509
## Types
510

511
```python { .api }
512
# RawIO class types
513
RawIOClass = type[BaseRawIO]           # Base RawIO class type
514
RawIOInstance = BaseRawIO              # RawIO instance type
515

516
# Index and size types
517
BlockIndex = int                       # Block index in file
518
SegmentIndex = int                     # Segment index within block  
519
ChannelIndex = int                     # Channel index
520
SampleIndex = int                      # Sample index
521
ChannelIndexes = list[int] | slice     # Channel selection
522
SampleRange = tuple[int, int]          # Sample start and stop indices
523

524
# Data chunk types
525
SignalChunk = np.ndarray              # Signal data chunk (samples x channels)
526
SpikeTimestamps = np.ndarray          # Spike timestamp array
527
SpikeWaveforms = np.ndarray           # Spike waveform data (spikes x samples x channels)
528
EventData = np.ndarray                # Event timestamp and code data
529

530
# Header and metadata types
531
HeaderDict = dict[str, Any]           # Parsed header information
532
ChannelInfo = dict[str, Any]          # Channel metadata
533
SamplingRate = float                  # Sampling rate in Hz
534
SignalUnits = str                     # Physical units of signal
535

536
# File format types
537
FileName = str                        # File path string
538
DirectoryName = str                   # Directory path string (for multi-file formats)
539
FileExtension = str                   # File extension string
540
```