0
# Filtering
1

2
Advanced filtering capabilities for selective processing of Eliot tasks based on various criteria. Filters work on the original Eliot message dictionaries before task parsing, enabling efficient selection using JMESPath queries, UUID matching, timestamp ranges, and logical combinations.
3

4
## Imports
5

6
```python
7
from datetime import datetime
8
from iso8601 import parse_date
9
```
10

11
## Capabilities
12

13
### JMESPath Query Filtering
14

15
Create filter functions using JMESPath query expressions to select tasks based on any field or condition in the original Eliot message structure.
16

17
```python { .api }
18
def filter_by_jmespath(query):
19
    """
20
    Produce a function for filtering a task by a JMESPath query expression.
21

22
    Parameters:
23
    - query: str - JMESPath query string used as a predicate
24

25
    Returns:
26
    Callable[[dict], bool] - Filter function that returns True if task matches query
27
    """
28
```
29

30
**Usage Examples:**
31

32
```python
33
from eliottree import filter_by_jmespath
34

35
# Filter tasks containing a 'uri' field
36
uri_filter = filter_by_jmespath('uri')
37

38
# Filter by specific action type
39
http_filter = filter_by_jmespath('action_type == `"http_client:request"`')
40

41
# Filter by HTTP status codes
42
error_filter = filter_by_jmespath('contains(`[401, 500]`, status)')
43

44
# Complex filtering with multiple conditions
45
critical_filter = filter_by_jmespath(
46
    'http_status == `401` && uri && contains(uri, `"/criticalEndpoint"`)'
47
)
48

49
# Apply filter to messages
50
filtered_tasks = [task for task in tasks if uri_filter(task)]
51
```
52

53
### UUID-Based Filtering
54

55
Filter entire task trees by their unique identifier, useful for isolating specific request flows or debugging particular operations.
56

57
```python { .api }
58
def filter_by_uuid(task_uuid):
59
    """
60
    Produce a function for filtering tasks by their UUID.
61
    
62
    Implementation note: This is implemented as a JMESPath wrapper that creates
63
    a query equivalent to: task_uuid == `<uuid_value>`
64

65
    Parameters:
66
    - task_uuid: str - UUID string to match against task_uuid field
67

68
    Returns:
69
    Callable[[dict], bool] - Filter function for UUID matching
70
    """
71
```
72

73
**Usage Example:**
74

75
```python
76
from eliottree import filter_by_uuid
77

78
# Filter by specific task UUID
79
uuid_filter = filter_by_uuid("f3a32bb3-ea6b-457c-aa99-08a3d0491ab4")
80

81
# Apply filter
82
specific_tasks = [task for task in tasks if uuid_filter(task)]
83
```
84

85
### Timestamp-Based Filtering
86

87
Filter tasks based on their occurrence time using start and end date boundaries, enabling time-range analysis of log data.
88

89
```python { .api }
90
def filter_by_start_date(start_date):
91
    """
92
    Produce a function for filtering by task timestamps after (or on) a certain date and time.
93

94
    Parameters:
95
    - start_date: datetime - Tasks must occur at or after this datetime
96

97
    Returns:
98
    Callable[[dict], bool] - Filter function for start date filtering
99
    """
100

101
def filter_by_end_date(end_date):
102
    """
103
    Produce a function for filtering by task timestamps before a certain date and time.
104

105
    Parameters:
106
    - end_date: datetime - Tasks must occur before this datetime
107

108
    Returns:
109
    Callable[[dict], bool] - Filter function for end date filtering
110
    """
111
```
112

113
**Usage Examples:**
114

115
```python
116
from datetime import datetime
117
from iso8601 import parse_date
118
from eliottree import filter_by_start_date, filter_by_end_date
119

120
# Filter tasks after a specific time
121
start_time = parse_date("2015-03-03T04:28:00Z")
122
after_filter = filter_by_start_date(start_time)
123

124
# Filter tasks before a specific time
125
end_time = parse_date("2015-03-03T04:30:00Z")
126
before_filter = filter_by_end_date(end_time)
127

128
# Apply filters
129
recent_tasks = [task for task in tasks if after_filter(task)]
130
time_range_tasks = [task for task in tasks 
131
                   if after_filter(task) and before_filter(task)]
132
```
133

134
### Logical Filter Combination
135

136
Combine multiple filter functions using logical AND operations to create sophisticated filtering criteria.
137

138
```python { .api }
139
def combine_filters_and(*filters):
140
    """
141
    Combine several filters together in a logical-AND fashion.
142

143
    Parameters:
144
    - *filters: Variable number of filter functions
145

146
    Returns:
147
    Callable[[Any], bool] - Combined filter function that returns True only 
148
                           if all input filters return True
149
    """
150
```
151

152
**Usage Examples:**
153

154
```python
155
from eliottree import (
156
    filter_by_jmespath, filter_by_start_date, filter_by_end_date,
157
    combine_filters_and
158
)
159

160
# Combine multiple filters
161
error_filter = filter_by_jmespath('http_status == `401`')
162
uri_filter = filter_by_jmespath('uri && contains(uri, `"/api"`)')
163
time_filter = filter_by_start_date(parse_date("2015-03-03T04:00:00Z"))
164

165
# Create combined filter
166
combined_filter = combine_filters_and(error_filter, uri_filter, time_filter)
167

168
# Apply combined filter
169
filtered_tasks = [task for task in tasks if combined_filter(task)]
170
```
171

172
## Complete Filtering Pipeline
173

174
```python
175
import json
176
from datetime import datetime
177
from iso8601 import parse_date
178
from eliottree import (
179
    tasks_from_iterable, render_tasks, get_theme,
180
    filter_by_jmespath, filter_by_start_date, combine_filters_and
181
)
182

183
def process_filtered_logs(log_file, start_time_str, action_type):
184
    """Process logs with multiple filtering criteria."""
185
    
186
    # Load messages
187
    with open(log_file, 'r') as f:
188
        messages = [json.loads(line) for line in f]
189
    
190
    # Create filters
191
    time_filter = filter_by_start_date(parse_date(start_time_str))
192
    action_filter = filter_by_jmespath(f'action_type == `"{action_type}"`')
193
    
194
    # Combine filters
195
    combined_filter = combine_filters_and(time_filter, action_filter)
196
    
197
    # Apply filtering
198
    filtered_messages = [msg for msg in messages if combined_filter(msg)]
199
    
200
    # Process and render
201
    tasks = tasks_from_iterable(filtered_messages)
202
    theme = get_theme(dark_background=True)
203
    render_tasks(sys.stdout.write, tasks, theme=theme)
204

205
# Usage
206
process_filtered_logs(
207
    'eliot.log', 
208
    '2015-03-03T04:28:00Z', 
209
    'http_client:request'
210
)
211
```
212

213
## Filter Function Composition
214

215
Filters can be composed and reused for different analysis scenarios:
216

217
```python
218
# Define reusable filters
219
def create_error_filters():
220
    return {
221
        'http_errors': filter_by_jmespath('status >= `400`'),
222
        'timeout_errors': filter_by_jmespath('contains(error_type || `""`, `"timeout"`)'),
223
        'auth_errors': filter_by_jmespath('status == `401` || status == `403`'),
224
    }
225

226
def create_time_range_filter(start_str, end_str):
227
    start_filter = filter_by_start_date(parse_date(start_str))
228
    end_filter = filter_by_end_date(parse_date(end_str))
229
    return combine_filters_and(start_filter, end_filter)
230

231
# Usage
232
error_filters = create_error_filters()
233
time_range = create_time_range_filter('2015-03-03T04:00:00Z', '2015-03-03T05:00:00Z')
234

235
# Combine for specific analysis
236
auth_errors_in_range = combine_filters_and(
237
    error_filters['auth_errors'], 
238
    time_range
239
)
240
```
241

242
## JMESPath Query Patterns
243

244
Common query patterns for Eliot log analysis:
245

246
```python
247
# Existence checks
248
filter_by_jmespath('uri')                    # Has URI field
249
filter_by_jmespath('error_message')          # Has error message
250

251
# Exact matches
252
filter_by_jmespath('action_type == `"http_request"`')
253
filter_by_jmespath('status == `200`')
254

255
# Numeric comparisons
256
filter_by_jmespath('status >= `400`')        # HTTP errors
257
filter_by_jmespath('duration > `5.0`')      # Slow operations
258

259
# String operations
260
filter_by_jmespath('contains(uri, `"/api/"`)')
261
filter_by_jmespath('starts_with(action_type, `"db:"`)')
262

263
# Array operations
264
filter_by_jmespath('contains(`[401, 403, 500]`, status)')
265
filter_by_jmespath('length(task_level) > `2`')
266

267
# Complex conditions
268
filter_by_jmespath('status >= `400` && contains(uri, `"/critical"`)')
269
```