0
# Tools and Utilities
1
2
Helper functions and classes for common task patterns, uptodate checkers, development utilities, and IPython integration for enhanced dodo file functionality.
3
4
## Capabilities
5
6
### File System Utilities
7
8
Helper functions for common file system operations within tasks.
9
10
```python { .api }
11
def create_folder(dir_path):
12
"""
13
Create a folder in the given path if it doesn't exist yet.
14
15
Uses os.makedirs with exist_ok=True to safely create directory hierarchies
16
without raising errors if directories already exist.
17
18
Args:
19
dir_path (str): Path to directory to create
20
"""
21
```
22
23
### Task Presentation
24
25
Functions for customizing how tasks are displayed and titled in output.
26
27
```python { .api }
28
def title_with_actions(task):
29
"""
30
Return task name with task actions for display.
31
32
Formats task display to show both the task name and its actions,
33
or shows group task information if no actions are present.
34
35
Args:
36
task: Task object with name, actions, and task_dep attributes
37
38
Returns:
39
str: Formatted string showing task name and actions
40
"""
41
```
42
43
### Uptodate Checkers
44
45
Classes and functions for determining when tasks need to be re-executed based on various conditions.
46
47
```python { .api }
48
def run_once(task, values):
49
"""
50
Execute task just once (uptodate checker).
51
52
Used when user manually manages a dependency and wants the task
53
to run only on first execution, regardless of file changes.
54
55
Args:
56
task: Task object to configure
57
values (dict): Saved values from previous executions
58
59
Returns:
60
bool: True if task should be considered up-to-date
61
"""
62
```
63
64
```python { .api }
65
class config_changed:
66
"""
67
Check if passed configuration was modified (uptodate checker).
68
69
Monitors configuration data (strings or dictionaries) and determines
70
if the configuration has changed since the last successful run.
71
"""
72
73
def __init__(self, config, encoder=None):
74
"""
75
Initialize configuration checker.
76
77
Args:
78
config (str|dict): Configuration data to monitor
79
encoder (json.JSONEncoder, optional): Custom JSON encoder for non-default values
80
"""
81
82
def __call__(self, task, values):
83
"""
84
Check if configuration is unchanged.
85
86
Args:
87
task: Task object being checked
88
values (dict): Saved values from previous executions
89
90
Returns:
91
bool: True if configuration is unchanged (task up-to-date)
92
"""
93
94
def configure_task(self, task):
95
"""Configure task to save configuration digest"""
96
```
97
98
```python { .api }
99
class timeout:
100
"""
101
Add timeout to task (uptodate checker).
102
103
Task is considered not up-to-date if more time has elapsed since
104
last execution than the specified timeout limit.
105
"""
106
107
def __init__(self, timeout_limit):
108
"""
109
Initialize timeout checker.
110
111
Args:
112
timeout_limit (datetime.timedelta|int): Timeout in seconds or timedelta object
113
114
Raises:
115
Exception: If timeout_limit is not timedelta or int
116
"""
117
118
def __call__(self, task, values):
119
"""
120
Check if task has timed out.
121
122
Args:
123
task: Task object being checked
124
values (dict): Saved values from previous executions
125
126
Returns:
127
bool: True if task has not timed out (still up-to-date)
128
"""
129
```
130
131
```python { .api }
132
class check_timestamp_unchanged:
133
"""
134
Check if timestamp of a file/directory is unchanged since last run (uptodate checker).
135
136
Monitors file timestamps and considers task up-to-date if the specified
137
timestamp has not changed since last successful execution.
138
"""
139
140
def __init__(self, file_name, time='mtime', cmp_op=operator.eq):
141
"""
142
Initialize timestamp checker.
143
144
Args:
145
file_name (str): Path to file/directory to monitor
146
time (str): Timestamp type - 'atime'/'access', 'ctime'/'status', 'mtime'/'modify'
147
cmp_op (callable): Comparison function (prev_time, current_time) -> bool
148
149
Raises:
150
ValueError: If invalid time value is specified
151
"""
152
153
def __call__(self, task, values):
154
"""
155
Check if file timestamp is unchanged.
156
157
Args:
158
task: Task object being checked
159
values (dict): Saved values from previous executions
160
161
Returns:
162
bool: True if timestamp is unchanged (task up-to-date)
163
164
Raises:
165
OSError: If cannot stat the specified file
166
"""
167
```
168
169
### Development and Debugging
170
171
Utilities for debugging and development workflow enhancement.
172
173
```python { .api }
174
def set_trace():
175
"""
176
Start debugger, ensuring stdout shows pdb output.
177
178
Sets up Python debugger (pdb) with proper stdin/stdout configuration
179
for debugging doit tasks. Output is not restored after debugging.
180
181
Note: This is a development utility and output streams are not restored.
182
"""
183
```
184
185
### IPython Integration
186
187
Functions for integrating doit with IPython/Jupyter notebooks for interactive development.
188
189
```python { .api }
190
def load_ipython_extension(ip=None):
191
"""
192
Define a %doit magic function for IPython that discovers and executes tasks
193
from interactive variables (global namespace).
194
195
Creates a magic function that allows running doit commands directly within
196
IPython sessions, using the current namespace as the source for task definitions.
197
198
Args:
199
ip (IPython instance, optional): IPython instance, auto-detected if None
200
201
Raises:
202
ImportError: If not running within an IPython environment
203
204
Usage in IPython:
205
%load_ext doit
206
%doit list # List tasks from current namespace
207
%doit # Run tasks from current namespace
208
%doit --help # Show help
209
"""
210
```
211
212
```python { .api }
213
register_doit_as_IPython_magic = load_ipython_extension
214
# Alternative alias for registering IPython extension
215
```
216
217
### Backward Compatibility
218
219
Legacy imports maintained for backward compatibility.
220
221
```python { .api }
222
from .task import result_dep # Imported for backward compatibility
223
```
224
225
### Usage Examples
226
227
#### Configuration Change Detection
228
229
```python
230
from doit.tools import config_changed
231
import json
232
233
def task_deploy():
234
"""Deploy with configuration change detection"""
235
236
config = {
237
'server': 'production.example.com',
238
'port': 443,
239
'ssl': True,
240
'workers': 4
241
}
242
243
return {
244
'actions': ['deploy.sh --config deployment.json'],
245
'uptodate': [config_changed(config)],
246
'verbosity': 2
247
}
248
249
# Task will only run if config dictionary changes
250
```
251
252
#### Timeout-Based Execution
253
254
```python
255
from doit.tools import timeout
256
import datetime
257
258
def task_backup():
259
"""Run backup every 6 hours"""
260
return {
261
'actions': ['backup_script.sh'],
262
'uptodate': [timeout(datetime.timedelta(hours=6))],
263
'verbosity': 2
264
}
265
266
# Task runs again if more than 6 hours have passed
267
```
268
269
#### File Timestamp Monitoring
270
271
```python
272
from doit.tools import check_timestamp_unchanged
273
274
def task_process_config():
275
"""Process configuration file"""
276
return {
277
'actions': ['process_config.py config.yaml'],
278
'uptodate': [check_timestamp_unchanged('config.yaml')],
279
'targets': ['processed_config.json']
280
}
281
282
# Task runs only if config.yaml timestamp changes
283
```
284
285
#### IPython Magic Usage
286
287
```python
288
# In IPython/Jupyter notebook cell:
289
290
def task_analyze():
291
"""Analyze data in current notebook"""
292
return {
293
'actions': ['python analyze_data.py'],
294
'file_dep': ['data.csv']
295
}
296
297
def task_plot():
298
"""Generate plots"""
299
return {
300
'actions': ['python generate_plots.py'],
301
'file_dep': ['processed_data.json'],
302
'targets': ['plots.png']
303
}
304
305
# Load the extension
306
%load_ext doit
307
308
# List available tasks from notebook namespace
309
%doit list
310
311
# Run tasks
312
%doit analyze
313
%doit plot
314
```
315
316
#### Custom Development Workflow
317
318
```python
319
from doit.tools import create_folder, set_trace, config_changed
320
321
def task_setup_dev():
322
"""Setup development environment"""
323
324
dev_config = {
325
'debug': True,
326
'log_level': 'DEBUG',
327
'db_name': 'myapp_dev'
328
}
329
330
def setup_action():
331
create_folder('logs')
332
create_folder('tmp')
333
create_folder('data/dev')
334
335
# Uncomment for debugging
336
# set_trace()
337
338
print("Development environment ready")
339
340
return {
341
'actions': [setup_action],
342
'uptodate': [config_changed(dev_config)],
343
'verbosity': 2
344
}
345
```