0
# Data Manipulation Utilities
1

2
Utility functions for extracting, filtering, and manipulating electrophysiology data objects within Neo's hierarchical structure. These tools provide high-level operations for common analysis workflows, data organization, and dataset management.
3

4
## Capabilities
5

6
### Event and Epoch Extraction
7

8
Functions for finding and extracting specific events and epochs based on properties and criteria.
9

10
```python { .api }
11
def get_events(container, **properties):
12
    """
13
    Extract events matching specified criteria from Neo containers.
14
    
15
    Parameters:
16
    - container: Neo container (Block, Segment, or Group)
17
    - **properties: Event properties to match (name, labels, times, etc.)
18
    
19
    Returns:
20
    list: Event objects matching the specified criteria
21
    
22
    Examples:
23
    - get_events(segment, name='stimulus')
24
    - get_events(block, labels=['start', 'stop'])
25
    - get_events(segment, times=GreaterThan(1.0*pq.s))
26
    """
27

28
def get_epochs(container, **properties):
29
    """
30
    Extract epochs matching specified criteria from Neo containers.
31
    
32
    Parameters:
33
    - container: Neo container (Block, Segment, or Group)  
34
    - **properties: Epoch properties to match (name, labels, times, durations, etc.)
35
    
36
    Returns:
37
    list: Epoch objects matching the specified criteria
38
    
39
    Examples:
40
    - get_epochs(segment, name='stimulus_period')
41
    - get_epochs(block, labels=['baseline', 'response'])
42
    - get_epochs(segment, durations=GreaterThan(0.5*pq.s))
43
    """
44

45
def add_epoch(container, epoch):
46
    """
47
    Add epoch annotations to Neo containers.
48
    
49
    Parameters:
50
    - container: Neo container (Block, Segment, or Group)
51
    - epoch: Epoch object to add
52
    
53
    The epoch is added to all appropriate containers in the hierarchy.
54
    """
55

56
def match_events(container1, container2, **matching_criteria):
57
    """
58
    Match events across different Neo containers based on specified criteria.
59
    
60
    Parameters:
61
    - container1, container2: Neo containers to compare
62
    - **matching_criteria: Criteria for matching events
63
    
64
    Returns:
65
    list: Pairs of matched events from the two containers
66
    
67
    Useful for aligning events across different recording sessions
68
    or comparing events before and after processing.
69
    """
70
```
71

72
### Data Segmentation and Slicing
73

74
Functions for temporal segmentation and extraction of data based on epochs and time windows.
75

76
```python { .api }
77
def cut_block_by_epochs(block, epochs):
78
    """
79
    Create new Block containing data segmented by epoch boundaries.
80
    
81
    Parameters:
82
    - block: Neo Block object containing data to segment
83
    - epochs: Epoch objects or list of epochs defining boundaries
84
    
85
    Returns:
86
    Block: New Block with segments corresponding to each epoch
87
    
88
    Each epoch becomes a new Segment containing all data objects
89
    that fall within the epoch's time boundaries.
90
    """
91

92
def cut_segment_by_epoch(segment, epoch):
93
    """
94
    Create new Segment containing data within specified epoch boundaries.
95
    
96
    Parameters:
97
    - segment: Neo Segment object to slice
98
    - epoch: Single Epoch object defining time boundaries
99
    
100
    Returns:
101
    Segment: New Segment containing only data within epoch timespan
102
    
103
    All data objects (AnalogSignal, SpikeTrain, Event, etc.) are
104
    temporally sliced to fit within the epoch boundaries.
105
    """
106
```
107

108
### Data Compatibility and Validation
109

110
Functions for checking data compatibility and validation across different Neo operations.
111

112
```python { .api }
113
def is_block_rawio_compatible(block):
114
    """
115
    Check if Block is compatible with RawIO interface requirements.
116
    
117
    Parameters:
118
    - block: Neo Block object to validate
119
    
120
    Returns:
121
    bool: True if Block structure is compatible with RawIO operations
122
    
123
    Validates that the Block structure, sampling rates, channel
124
    organizations, and data types are consistent with RawIO
125
    expectations for high-performance file access.
126
    """
127
```
128

129
### Dataset Management
130

131
Functions for downloading and managing public electrophysiology datasets for testing and examples.
132

133
```python { .api }
134
def download_dataset(name, data_home=None):
135
    """
136
    Download public electrophysiology datasets for testing and examples.
137
    
138
    Parameters:
139
    - name (str): Name of dataset to download
140
    - data_home (str, optional): Directory to store datasets
141
    
142
    Returns:
143
    str: Path to downloaded dataset directory
144
    
145
    Available datasets include example recordings from various
146
    hardware systems and file formats for testing I/O capabilities
147
    and exploring Neo functionality.
148
    """
149

150
def get_local_testing_data_folder():
151
    """
152
    Get path to local testing data folder containing example files.
153
    
154
    Returns:
155
    str: Path to directory containing test data files
156
    
157
    Returns the location where Neo stores small example files
158
    for testing different I/O classes and data structures.
159
    """
160
```
161

162
## Usage Examples
163

164
### Event and Epoch Operations
165

166
```python
167
import neo
168
from neo.utils import get_events, get_epochs, cut_segment_by_epoch
169
from neo.core.filters import Equals, GreaterThan
170
import quantities as pq
171

172
# Load data
173
io = neo.io.get_io('experiment.abf')
174
block = io.read_block()
175
segment = block.segments[0]
176

177
# Extract specific events
178
stimulus_events = get_events(segment, name='stimulus_onset')
179
response_events = get_events(segment, labels=['button_press', 'lever_pull'])
180

181
# Extract events by timing criteria  
182
late_events = get_events(segment, times=GreaterThan(10.0*pq.s))
183

184
# Extract epochs
185
baseline_epochs = get_epochs(segment, name='baseline')
186
long_epochs = get_epochs(segment, durations=GreaterThan(2.0*pq.s))
187

188
# Segment data by epochs
189
if baseline_epochs:
190
    baseline_segment = cut_segment_by_epoch(segment, baseline_epochs[0])
191
    print(f"Baseline segment duration: {baseline_segment.duration}")
192
    print(f"Number of spike trains: {len(baseline_segment.spiketrains)}")
193
    print(f"Number of analog signals: {len(baseline_segment.analogsignals)}")
194
```
195

196
### Data Segmentation Workflow
197

198
```python
199
from neo.utils import cut_block_by_epochs, add_epoch
200
import numpy as np
201

202
# Create trial epochs for segmentation
203
trial_starts = np.array([0, 30, 60, 90]) * pq.s
204
trial_duration = 25 * pq.s
205

206
trial_epochs = []
207
for i, start in enumerate(trial_starts):
208
    epoch = neo.Epoch(
209
        times=[start],
210
        durations=[trial_duration], 
211
        labels=[f'trial_{i+1}']
212
    )
213
    trial_epochs.append(epoch)
214
    add_epoch(segment, epoch)
215

216
# Segment entire block by trials
217
segmented_block = cut_block_by_epochs(block, trial_epochs)
218

219
print(f"Original block: {len(block.segments)} segments")
220
print(f"Segmented block: {len(segmented_block.segments)} segments")
221

222
# Process each trial segment
223
for i, trial_segment in enumerate(segmented_block.segments):
224
    print(f"Trial {i+1}:")
225
    print(f"  Duration: {trial_segment.duration}")
226
    print(f"  Spike trains: {len(trial_segment.spiketrains)}")
227
    print(f"  Events: {len(trial_segment.events)}")
228
```
229

230
### Event Matching Across Sessions
231

232
```python
233
from neo.utils import match_events
234

235
# Load pre and post-treatment sessions
236
pre_io = neo.io.get_io('pre_treatment.abf')
237
post_io = neo.io.get_io('post_treatment.abf')
238

239
pre_block = pre_io.read_block()
240
post_block = post_io.read_block()
241

242
# Match stimulus events across sessions
243
matched_stimuli = match_events(
244
    pre_block.segments[0], 
245
    post_block.segments[0],
246
    name=Equals('stimulus'),
247
    tolerance=0.1*pq.s  # Allow 100ms timing difference
248
)
249

250
print(f"Found {len(matched_stimuli)} matched stimulus events")
251

252
# Compare responses to matched stimuli
253
for pre_event, post_event in matched_stimuli:
254
    pre_time = pre_event.times[0]
255
    post_time = post_event.times[0]
256
    time_diff = post_time - pre_time  
257
    print(f"Stimulus at {pre_time}: time shift = {time_diff}")
258
```
259

260
### Data Validation and Compatibility
261

262
```python
263
from neo.utils import is_block_rawio_compatible
264

265
# Check if data is suitable for high-performance access
266
if is_block_rawio_compatible(block):
267
    print("Block is compatible with RawIO - can use high-performance access")
268
    
269
    # Use RawIO for efficient data access
270
    import neo.rawio
271
    rawio = neo.rawio.get_rawio('data_file.ns5')
272
    rawio.parse_header()
273
    
274
    # Access signal chunks efficiently
275
    chunk = rawio.get_analogsignal_chunk(
276
        block_index=0,
277
        seg_index=0, 
278
        i_start=1000,
279
        i_stop=2000,
280
        channel_indexes=[0, 1, 2]
281
    )
282
else:
283
    print("Block requires standard I/O access")
284
    # Use regular Neo I/O methods
285
```
286

287
### Dataset Management
288

289
```python
290
from neo.utils import download_dataset, get_local_testing_data_folder
291

292
# Download example datasets
293
dataset_path = download_dataset('example_abf_files')
294
print(f"Downloaded dataset to: {dataset_path}")
295

296
# Get local test data location
297
test_data_folder = get_local_testing_data_folder()
298
print(f"Test data available at: {test_data_folder}")
299

300
# Use test data for exploring formats
301
import os
302
test_files = os.listdir(test_data_folder)
303
abf_files = [f for f in test_files if f.endswith('.abf')]
304

305
if abf_files:
306
    test_file = os.path.join(test_data_folder, abf_files[0])
307
    io = neo.io.AxonIO(test_file)
308
    block = io.read_block()
309
    print(f"Test file loaded: {len(block.segments)} segments")
310
```
311

312
### Advanced Filtering and Processing
313

314
```python
315
from neo.core.filters import Equals, InRange, GreaterThan
316
import quantities as pq
317

318
# Complex event filtering
319
movement_events = get_events(
320
    segment,
321
    name=Equals('movement'),
322
    times=InRange(5.0*pq.s, 15.0*pq.s),  # Between 5-15 seconds
323
    annotations={'movement_type': 'reach'}  # Custom annotation matching
324
)
325

326
# Multi-criteria epoch extraction
327
response_periods = get_epochs(
328
    segment,
329
    labels=['response_start', 'response_end'],
330
    durations=GreaterThan(0.5*pq.s),
331
    annotations={'condition': 'experimental'}
332
)
333

334
# Batch processing across multiple segments
335
all_segments = []
336
for original_segment in block.segments:
337
    for epoch in response_periods:
338
        if epoch in original_segment.epochs:
339
            processed_segment = cut_segment_by_epoch(original_segment, epoch)
340
            processed_segment.name = f"{original_segment.name}_{epoch.labels[0]}"
341
            all_segments.append(processed_segment)
342

343
print(f"Created {len(all_segments)} processed segments")
344
```
345

346
### Data Organization and Cleanup
347

348
```python
349
# Reorganize data by extracting specific time windows
350
def extract_peri_event_data(segment, events, window=(-1.0, 2.0)):
351
    """Extract data windows around events."""
352
    peri_event_segments = []
353
    
354
    for i, event in enumerate(events):
355
        event_time = event.times[0]
356
        start_time = event_time + window[0]*pq.s
357
        end_time = event_time + window[1]*pq.s
358
        
359
        # Create epoch for this time window
360
        peri_epoch = neo.Epoch(
361
            times=[start_time],
362
            durations=[end_time - start_time],
363
            labels=[f'peri_event_{i}']
364
        )
365
        
366
        # Extract data for this window
367
        peri_segment = cut_segment_by_epoch(segment, peri_epoch)
368
        peri_segment.name = f"event_{i}_window"
369
        peri_event_segments.append(peri_segment)
370
    
371
    return peri_event_segments
372

373
# Extract peri-stimulus data
374
stimulus_events = get_events(segment, name='stimulus')
375
peri_stim_segments = extract_peri_event_data(
376
    segment, 
377
    stimulus_events, 
378
    window=(-0.5, 1.5)  # 500ms before to 1.5s after stimulus
379
)
380

381
print(f"Extracted {len(peri_stim_segments)} peri-stimulus segments")
382
```
383

384
## Types
385

386
```python { .api }
387
# Container types for utilities
388
Container = Block | Segment | Group     # Any Neo container type
389
ContainerList = list[Container]         # List of containers
390

391
# Event and epoch types
392
EventList = list[Event]                 # List of Event objects  
393
EpochList = list[Epoch]                 # List of Epoch objects
394
EventPair = tuple[Event, Event]         # Matched event pair
395
MatchedEvents = list[EventPair]         # List of matched event pairs
396

397
# Property matching types
398
PropertyDict = dict[str, Any]           # Property name to value mapping
399
FilterCriteria = dict[str, FilterCondition]  # Filtering criteria
400
MatchingCriteria = dict[str, Any]       # Event matching criteria
401

402
# Time and boundary types
403
TimeWindow = tuple[pq.Quantity, pq.Quantity]  # Start and end times
404
TimeBoundaries = list[TimeWindow]             # Multiple time windows
405
EpochBoundaries = list[Epoch]                 # Epoch-based boundaries
406

407
# Dataset management types
408
DatasetName = str                       # Name of available dataset
409
DatasetPath = str                       # Path to dataset directory
410
TestDataPath = str                      # Path to test data folder
411

412
# Validation types
413
CompatibilityResult = bool              # Compatibility check result
414
ValidationError = Exception             # Validation error type
415
```