0
# Etils
1

2
Etils (eclectic utils) is a comprehensive collection of Python utilities designed for machine learning and scientific computing workflows. The package is architected as a collection of independent, self-contained submodules that can be imported individually to avoid unnecessary dependencies.
3

4
## Package Information
5

6
- **Package Name**: etils
7
- **Language**: Python
8
- **Installation**: `pip install etils` or `pip install etils[array_types,epath,epy]` (selective modules)
9

10
## Core Imports
11

12
Etils follows a modular import pattern where each submodule is imported individually:
13

14
```python
15
from etils import epath  # Path utils
16
from etils import etree  # Tree utils
17
from etils import enp    # NumPy utils
18
from etils import ecolab # Colab utils
19
from etils import array_types # Type annotations
20
from etils import edc    # Dataclass utils
21
from etils import epy    # Python utils
22
from etils import eapp   # Absl app utils
23
from etils import etqdm  # TQDM utils
24
from etils import exm    # XManager utils
25
from etils import lazy_imports # Lazy import utils
26
```
27

28
## Basic Usage
29

30
```python
31
# Path operations with cloud storage support
32
from etils import epath
33
path = epath.Path('gs://my-bucket/data.txt')
34
path.write_text('Hello, world!')
35
content = path.read_text()
36

37
# Tree operations compatible with ML frameworks
38
from etils import etree
39
data = {'a': [1, 2], 'b': {'c': 3}}
40
mapped = etree.py.map(lambda x: x * 2, data)
41
# Result: {'a': [2, 4], 'b': {'c': 6}}
42

43
# Enhanced NumPy utilities
44
from etils import enp
45
import numpy as np
46
arrays = [np.array([1, 2, 3]), np.array([4, 5, 6])]
47
normalized = enp.check_and_normalize_arrays(arrays)
48

49
# Colab-specific utilities
50
from etils import ecolab
51
ecolab.auto_display(['item1', 'item2'])  # Enhanced display in Colab
52

53
# Type annotations for ML arrays
54
from etils import array_types
55
def process_data(data: array_types.FloatArray) -> array_types.IntArray:
56
    return data.astype(int)
57
```
58

59
## Architecture
60

61
Etils is designed around **independent submodules** with minimal cross-dependencies:
62

63
- **Modular Design**: Each submodule can be imported separately
64
- **Optional Dependencies**: Dependencies loaded only when needed
65
- **Cloud Integration**: Native support for gs://, s3:// through epath
66
- **ML Framework Compatibility**: Works with TensorFlow, JAX, PyTorch, NumPy
67
- **Development Environment**: Enhanced support for Jupyter/Colab workflows
68

69
## Capabilities
70

71
### Path Operations (epath)
72

73
Pathlib-compatible API that extends standard file operations to cloud storage systems including Google Cloud Storage (gs://), AWS S3 (s3://), and other remote filesystems.
74

75
```python { .api }
76
class Path:
77
    def __init__(self, path: str | PathLike) -> None: ...
78
    def read_text(self, encoding: str = 'utf-8') -> str: ...
79
    def write_text(self, data: str, encoding: str = 'utf-8') -> int: ...
80
    def exists(self) -> bool: ...
81
    def mkdir(self, parents: bool = False, exist_ok: bool = False) -> None: ...
82
    def glob(self, pattern: str) -> Iterator[Path]: ...
83

84
def register_path_cls(cls: type[Path]) -> None: ...
85
def resource_path(package: str, resource: str) -> Path: ...
86
```
87

88
[Path Operations](./path-operations.md)
89

90
### Tree Manipulation (etree)
91

92
Universal tree manipulation utilities compatible with TensorFlow nest, JAX tree_utils, DeepMind tree, and pure Python data structures.
93

94
```python { .api }
95
# Core API objects
96
jax: TreeAPI  # JAX tree operations
97
nest: TreeAPI  # TensorFlow nest operations  
98
tree: TreeAPI  # DeepMind tree operations
99
py: TreeAPI    # Pure Python operations
100

101
# Core functions (via py API)
102
def map(fn: Callable, tree: Tree) -> Tree: ...
103
def parallel_map(fn: Callable, tree: Tree) -> Tree: ...
104
def unzip(tree: Tree) -> Tree: ...
105
def stack(tree: Tree) -> Tree: ...
106
```
107

108
[Tree Manipulation](./tree-manipulation.md)
109

110
### NumPy Utilities (enp)
111

112
Enhanced NumPy utilities providing array specifications, compatibility layers, mathematical operations, and geometry utilities for scientific computing.
113

114
```python { .api }
115
class ArraySpec:
116
    def __init__(self, shape: tuple, dtype: np.dtype) -> None: ...
117
    
118
def check_and_normalize_arrays(*arrays) -> list[np.ndarray]: ...
119
def is_array_str(arr: np.ndarray) -> bool: ...
120
def flatten(arr: np.ndarray, pattern: str) -> np.ndarray: ...
121
def angle_between(v1: np.ndarray, v2: np.ndarray) -> float: ...
122
```
123

124
[NumPy Utilities](./numpy-utilities.md)
125

126
### Google Colab Integration (ecolab)
127

128
Utilities specifically designed for Google Colab environments including enhanced display functions, code inspection, HTML rendering, and Python-JavaScript communication.
129

130
```python { .api }
131
def auto_display(obj: Any) -> None: ...
132
def collapse(content: str, title: str = 'Details') -> None: ...
133
def inspect(obj: Any) -> None: ...
134
def highlight_html(code: str, language: str = 'python') -> str: ...
135
```
136

137
[Colab Integration](./colab-integration.md)
138

139
### Array Type Annotations (array_types)
140

141
Comprehensive type annotations for NumPy, JAX, TensorFlow, and PyTorch arrays with specific precision types for type-safe ML code development.
142

143
```python { .api }
144
# Core array types
145
ArrayLike = Union[np.ndarray, list, tuple]
146
Array = np.ndarray
147
FloatArray = np.ndarray  # Float arrays
148
IntArray = np.ndarray    # Integer arrays
149
BoolArray = np.ndarray   # Boolean arrays
150

151
# Precision-specific types
152
f32 = np.ndarray  # 32-bit float
153
f64 = np.ndarray  # 64-bit float
154
i32 = np.ndarray  # 32-bit int
155
ui32 = np.ndarray # 32-bit uint
156
```
157

158
[Array Types](./array-types.md)
159

160
### Dataclass Enhancements (edc)
161

162
Enhanced dataclass functionality with automatic type casting, context management, and improved representation for robust data structures.
163

164
```python { .api }
165
class AutoCast:
166
    def __init__(self, cast_fn: Callable) -> None: ...
167

168
def dataclass(cls: type) -> type: ...
169
def field(**kwargs) -> Any: ...
170
def repr(obj: Any) -> str: ...
171
```
172

173
[Dataclass Enhancements](./dataclass-enhancements.md)
174

175
### Python Utilities (epy)
176

177
Collection of general-purpose Python utilities including environment detection, iteration helpers, text processing, error handling, and language feature enhancements.
178

179
```python { .api }
180
def is_notebook() -> bool: ...
181
def is_test() -> bool: ...
182
def groupby(iterable: Iterable, key: Callable) -> dict: ...
183
def zip_dict(*dicts: dict) -> dict: ...
184
def lazy_imports(**modules) -> Any: ...
185
```
186

187
[Python Utilities](./python-utilities.md)
188

189
### Application Framework (eapp)
190

191
Absl flags and application utilities for building command-line applications with dataclass-based flag parsing and enhanced logging.
192

193
```python { .api }
194
def make_flags_parser(dataclass_cls: type) -> Callable: ...
195
def better_logging(level: str = 'INFO') -> None: ...
196
```
197

198
[Application Framework](./application-framework.md)
199

200
### Progress Bars (etqdm)
201

202
Enhanced TQDM progress bars with smart defaults and improved integration for iterative operations.
203

204
```python { .api }
205
def tqdm(iterable: Optional[Iterable] = None, **kwargs) -> Any: ...
206
```
207

208
### XManager Integration (exm)
209

210
Google XManager experiment management utilities for distributed machine learning workflows and experiment tracking.
211

212
```python { .api }
213
def current_experiment() -> Any: ...
214
def current_work_unit() -> Any: ...
215
def is_running_under_xmanager() -> bool: ...
216
def add_experiment_artifact(name: str, path: str) -> None: ...
217
def add_work_unit_artifact(name: str, path: str) -> None: ...
218
def curr_job_name() -> str: ...
219
def url_to_python_only_logs() -> str: ...
220
def set_citc_source(source: str) -> None: ...
221
```
222

223
### Lazy Import Management (lazy_imports)
224

225
Utilities for managing lazy imports and module loading to optimize startup time and memory usage.
226

227
```python { .api }
228
def print_current_imports() -> None: ...
229
def __dir__() -> list[str]: ...
230
LAZY_MODULES: dict[str, Any]
231
```
232

233
## Version Information
234

235
```python { .api }
236
__version__: str  # Current version: "1.13.0"
237
```