0
# NumExpr
1
2
A fast numerical expression evaluator for NumPy that accelerates array operations through optimized C++ implementations and multi-threading capabilities. NumExpr parses mathematical expressions into custom op-codes executed by an integrated virtual machine, avoiding intermediate memory allocations and enabling substantial performance improvements over pure NumPy operations.
3
4
## Package Information
5
6
- **Package Name**: numexpr
7
- **Package Type**: pypi
8
- **Language**: Python (with C++ extensions)
9
- **Installation**: `pip install numexpr`
10
11
## Core Imports
12
13
```python
14
import numexpr as ne
15
```
16
17
For specific functions:
18
19
```python
20
from numexpr import evaluate, re_evaluate, NumExpr, set_num_threads, get_num_threads
21
```
22
23
## Basic Usage
24
25
```python
26
import numpy as np
27
import numexpr as ne
28
29
# Create sample arrays
30
a = np.arange(1e6) # Large arrays show better performance gains
31
b = np.arange(1e6)
32
33
# Simple expression evaluation
34
result = ne.evaluate("a + 1")
35
36
# Complex expressions with multiple operations
37
result = ne.evaluate("a * b - 4.1 * a > 2.5 * b")
38
39
# Using variables from local scope
40
x = 2.5
41
y = 4.1
42
result = ne.evaluate("a * b - y * a > x * b")
43
44
# With explicit variable dictionary
45
result = ne.evaluate("a * b - y * a > x * b",
46
local_dict={'a': a, 'b': b, 'x': 2.5, 'y': 4.1})
47
48
# Pre-compile expressions for repeated evaluation
49
compiled_expr = ne.NumExpr("a * b + c")
50
result1 = compiled_expr.run(a, b, c=np.ones(len(a)))
51
result2 = compiled_expr.run(a*2, b*3, c=np.zeros(len(a)))
52
```
53
54
## Architecture
55
56
NumExpr uses a multi-stage compilation and execution pipeline:
57
58
- **Expression Parser**: Converts string expressions into Abstract Syntax Trees (AST)
59
- **Type Inference**: Determines result types and validates operations
60
- **Code Generation**: Translates AST into custom virtual machine opcodes
61
- **Virtual Machine**: Executes opcodes on chunked array data across multiple threads
62
- **Memory Management**: Uses cache-friendly chunking to avoid intermediate allocations
63
64
This architecture enables 0.95x to 15x performance improvements over NumPy by:
65
- Eliminating temporary array allocations
66
- Optimizing cache utilization through data chunking
67
- Parallelizing computation across CPU cores
68
- Leveraging Intel VML when available for transcendental functions
69
70
## Capabilities
71
72
### Expression Evaluation
73
74
Core functionality for evaluating mathematical expressions on NumPy arrays with performance optimization and multi-threading support.
75
76
```python { .api }
77
def evaluate(ex, local_dict=None, global_dict=None, out=None, order='K', casting='safe', **kwargs):
78
"""
79
Evaluate a mathematical expression on arrays.
80
81
Parameters:
82
- ex (str): Mathematical expression string
83
- local_dict (dict, optional): Local variable bindings
84
- global_dict (dict, optional): Global variable bindings
85
- out (ndarray, optional): Output array for result
86
- order (str): Memory layout order ('K', 'A', 'C', 'F')
87
- casting (str): Casting behavior ('no', 'equiv', 'safe', 'same_kind', 'unsafe')
88
89
Returns:
90
ndarray: Result of expression evaluation
91
"""
92
93
def re_evaluate(local_dict=None, **kwargs):
94
"""
95
Re-evaluate the previous expression with new variable values.
96
97
Parameters:
98
- local_dict (dict, optional): New local variable bindings
99
100
Returns:
101
ndarray: Result of re-evaluation
102
"""
103
```
104
105
[Expression Evaluation](./expression-evaluation.md)
106
107
### Compiled Expressions
108
109
Pre-compilation of expressions for repeated evaluation with different data, providing optimal performance for expressions used multiple times.
110
111
```python { .api }
112
class NumExpr:
113
"""
114
Pre-compiled expression object for repeated evaluation.
115
"""
116
def __init__(self, ex, signature=(), sanitize=True, **kwargs):
117
"""
118
Create a compiled expression.
119
120
Parameters:
121
- ex (str): Mathematical expression string
122
- signature (tuple): Variable signature for type checking
123
- sanitize (bool): Enable expression sanitization
124
"""
125
126
def run(self, *args, **kwargs):
127
"""Execute the compiled expression with provided arguments."""
128
```
129
130
[Compiled Expressions](./compiled-expressions.md)
131
132
### Threading and Performance Control
133
134
Configuration of multi-threading behavior and performance optimization settings for CPU-intensive computations.
135
136
```python { .api }
137
def set_num_threads(nthreads):
138
"""
139
Set the number of threads for operations.
140
141
Parameters:
142
- nthreads (int): Number of threads to use
143
144
Returns:
145
int: Previous thread count
146
"""
147
148
def get_num_threads():
149
"""
150
Get the current number of threads in use.
151
152
Returns:
153
int: Current thread count
154
"""
155
156
def detect_number_of_cores():
157
"""
158
Detect the number of CPU cores available.
159
160
Returns:
161
int: Number of detected cores
162
"""
163
```
164
165
[Threading and Performance](./threading-performance.md)
166
167
### VML Integration
168
169
Integration with Intel's Vector Math Library (VML) for hardware-accelerated transcendental functions when available.
170
171
```python { .api }
172
def get_vml_version():
173
"""
174
Get the VML/MKL library version.
175
176
Returns:
177
str or None: VML version string if available
178
"""
179
180
def set_vml_accuracy_mode(mode):
181
"""
182
Set VML accuracy mode.
183
184
Parameters:
185
- mode (str): 'high', 'low', 'fast', or None
186
187
Returns:
188
str: Previous accuracy mode
189
"""
190
191
def set_vml_num_threads(nthreads):
192
"""Set number of threads for VML operations."""
193
```
194
195
[VML Integration](./vml-integration.md)
196
197
### Expression Analysis and Debugging
198
199
Tools for analyzing, validating, and debugging expressions including disassembly of compiled expressions.
200
201
```python { .api }
202
def validate(ex, local_dict=None, global_dict=None, out=None, order='K', casting='safe', **kwargs):
203
"""
204
Validate an expression without evaluating it.
205
206
Parameters:
207
- ex (str): Expression to validate
208
- Additional parameters same as evaluate()
209
210
Returns:
211
tuple: (result_type, result_shape) information
212
"""
213
214
def disassemble(nex):
215
"""
216
Disassemble a NumExpr object showing internal opcodes.
217
218
Parameters:
219
- nex (NumExpr): Compiled expression object
220
221
Returns:
222
str: Human-readable disassembly
223
"""
224
```
225
226
[Expression Analysis](./expression-analysis.md)
227
228
### Testing and Diagnostics
229
230
Development and debugging utilities for testing NumExpr functionality and examining system capabilities.
231
232
```python { .api }
233
def print_versions():
234
"""
235
Print the versions of software that numexpr relies on.
236
237
Displays version information for NumExpr, NumPy, Python, and system
238
details useful for debugging and support.
239
"""
240
241
def test(verbosity=1):
242
"""
243
Run all the tests in the test suite.
244
245
Parameters:
246
- verbosity (int): Test output verbosity level
247
248
Returns:
249
TestResult: Test execution results
250
"""
251
```
252
253
## Module Constants
254
255
```python { .api }
256
# Version information
257
__version__: str # Package version string
258
259
# Threading constants
260
MAX_THREADS: int # Maximum supported threads
261
__BLOCK_SIZE1__: int # Virtual machine block size
262
263
# Feature availability
264
use_vml: bool # Whether VML support is available
265
266
# Runtime state
267
ncores: int # Detected number of CPU cores
268
nthreads: int # Current configured thread count
269
270
# Expression objects and utilities
271
E: object # Expression builder object for creating expressions
272
```
273
274
## Expression Language
275
276
NumExpr supports a comprehensive mathematical expression language including:
277
278
**Arithmetic Operations**: `+`, `-`, `*`, `/`, `%`, `**`
279
**Comparison Operations**: `==`, `!=`, `<`, `<=`, `>`, `>=`
280
**Logical Operations**: `&` (and), `|` (or), `~` (not)
281
**Mathematical Functions**: `sin`, `cos`, `tan`, `arcsin`, `arccos`, `arctan`, `sinh`, `cosh`, `tanh`, `arcsinh`, `arccosh`, `arctanh`, `exp`, `expm1`, `log`, `log1p`, `log10`, `sqrt`, `abs`, `ceil`, `floor`, `arctan2`, `fmod`
282
**Complex Functions**: `real`, `imag`, `complex`, `conjugate`
283
**String Operations**: String comparison and basic operations
284
285
**Data Types**: `bool`, `int`, `long`, `float`, `double`, `complex`, `str`
286
287
All operations support NumPy broadcasting semantics and work with NumPy array types.