0
# Deltalake
1
2
Native Delta Lake Python binding based on delta-rs with Pandas integration. Provides high-performance operations on Delta Lake tables with seamless integration to the Python data ecosystem, enabling ACID transactions, time travel queries, schema evolution, and efficient data storage with multiple backend support.
3
4
## Package Information
5
6
- **Package Name**: deltalake
7
- **Language**: Python
8
- **Installation**: `pip install deltalake`
9
- **Optional Dependencies**: `pip install deltalake[pandas,pyarrow]`
10
11
## Version Information
12
13
```python { .api }
14
__version__: str # Python package version
15
rust_core_version: str # Underlying Rust core version
16
```
17
18
Access package version information to check compatibility and track installed versions.
19
20
## Core Imports
21
22
```python
23
from deltalake import DeltaTable
24
```
25
26
Complete import for all functionality:
27
28
```python
29
from deltalake import (
30
DeltaTable,
31
Metadata,
32
write_deltalake,
33
convert_to_deltalake,
34
QueryBuilder,
35
Schema,
36
Field,
37
DataType,
38
WriterProperties,
39
BloomFilterProperties,
40
ColumnProperties,
41
CommitProperties,
42
PostCommitHookProperties,
43
TableFeatures,
44
Transaction,
45
__version__,
46
rust_core_version
47
)
48
```
49
50
## Basic Usage
51
52
```python
53
from deltalake import DeltaTable, write_deltalake
54
import pandas as pd
55
56
# Reading a Delta table
57
dt = DeltaTable("path/to/delta-table")
58
df = dt.to_pandas()
59
print(f"Table has {dt.version()} versions with {len(dt.files())} files")
60
61
# Writing data to Delta table
62
data = pd.DataFrame({
63
'id': [1, 2, 3],
64
'name': ['Alice', 'Bob', 'Charlie'],
65
'age': [25, 30, 35]
66
})
67
68
write_deltalake("path/to/new-table", data)
69
70
# Updating records
71
dt = DeltaTable("path/to/new-table")
72
dt.update(
73
predicate="age < 30",
74
new_values={"name": "Updated Name"}
75
)
76
77
# Time travel
78
dt.load_as_version(0) # Load first version
79
older_df = dt.to_pandas()
80
```
81
82
## Architecture
83
84
Delta Lake provides ACID transactions on top of object storage through a transaction log that tracks all changes to table metadata and data files. The deltalake package exposes this functionality through several key components:
85
86
- **DeltaTable**: Main interface for reading, writing, and managing Delta tables
87
- **Transaction System**: ACID guarantees with optimistic concurrency control
88
- **Schema Evolution**: Support for adding/modifying columns while maintaining compatibility
89
- **Time Travel**: Access to historical versions of data
90
- **Storage Backends**: Support for local filesystem, S3, Azure Blob Storage, and Google Cloud Storage
91
92
The Rust-based core (delta-rs) provides high-performance operations while the Python binding offers seamless integration with pandas, PyArrow, and the broader Python data ecosystem.
93
94
## Capabilities
95
96
### Table Operations
97
98
Core table management including creation, reading, and metadata access. The DeltaTable class provides the primary interface for interacting with Delta Lake tables.
99
100
```python { .api }
101
class DeltaTable:
102
def __init__(
103
self,
104
table_uri: str | Path,
105
version: int | None = None,
106
storage_options: dict[str, str] | None = None,
107
without_files: bool = False,
108
log_buffer_size: int | None = None
109
): ...
110
111
@classmethod
112
def create(
113
cls,
114
table_uri: str | Path,
115
schema: Schema,
116
mode: Literal["error", "append", "overwrite", "ignore"] = "error",
117
partition_by: list[str] | str | None = None,
118
storage_options: dict[str, str] | None = None
119
) -> DeltaTable: ...
120
121
@staticmethod
122
def is_deltatable(table_uri: str, storage_options: dict[str, str] | None = None) -> bool: ...
123
```
124
125
[Table Operations](./table-operations.md)
126
127
### Data Reading and Conversion
128
129
Converting Delta tables to various formats including pandas DataFrames, PyArrow tables, and streaming readers for efficient data processing.
130
131
```python { .api }
132
def to_pandas(
133
self,
134
columns: list[str] | None = None,
135
filesystem: Any | None = None
136
) -> pd.DataFrame: ...
137
138
def to_pyarrow_table(
139
self,
140
columns: list[str] | None = None,
141
filesystem: Any | None = None
142
) -> pyarrow.Table: ...
143
144
def to_pyarrow_dataset(
145
self,
146
partitions: list[tuple[str, str, Any]] | None = None,
147
filesystem: Any | None = None
148
) -> pyarrow.dataset.Dataset: ...
149
```
150
151
[Data Reading](./data-reading.md)
152
153
### Writing and Data Modification
154
155
Functions for writing data to Delta tables and modifying existing records through update, delete, and merge operations.
156
157
```python { .api }
158
def write_deltalake(
159
table_or_uri: str | Path | DeltaTable,
160
data: Any,
161
*,
162
partition_by: list[str] | str | None = None,
163
mode: Literal["error", "append", "overwrite", "ignore"] = "error",
164
schema_mode: Literal["merge", "overwrite"] | None = None,
165
storage_options: dict[str, str] | None = None,
166
writer_properties: WriterProperties | None = None
167
) -> None: ...
168
169
def update(
170
self,
171
updates: dict[str, str] | None = None,
172
new_values: dict[str, Any] | None = None,
173
predicate: str | None = None,
174
writer_properties: WriterProperties | None = None
175
) -> dict[str, Any]: ...
176
177
def delete(self, predicate: str | None = None) -> dict[str, Any]: ...
178
```
179
180
[Writing and Modification](./writing-modification.md)
181
182
### Schema Management
183
184
Schema definition, evolution, and type system for Delta Lake tables including field definitions and data types.
185
186
```python { .api }
187
class Schema:
188
def __init__(self, fields: list[Field]): ...
189
@property
190
def fields(self) -> list[Field]: ...
191
192
class Field:
193
def __init__(self, name: str, data_type: DataType, nullable: bool = True, metadata: dict | None = None): ...
194
@property
195
def name(self) -> str: ...
196
@property
197
def data_type(self) -> DataType: ...
198
199
# Data types: PrimitiveType, ArrayType, MapType, StructType
200
DataType = Union[PrimitiveType, ArrayType, MapType, StructType]
201
```
202
203
[Schema Management](./schema-management.md)
204
205
### Transaction Management
206
207
Transaction properties, commit configurations, and ACID transaction control for ensuring data consistency.
208
209
```python { .api }
210
class CommitProperties:
211
def __init__(
212
self,
213
max_retry_commit_attempts: int | None = None,
214
app_metadata: dict[str, Any] | None = None
215
): ...
216
217
class PostCommitHookProperties:
218
def __init__(
219
self,
220
create_checkpoint: bool = True,
221
cleanup_expired_logs: bool | None = None
222
): ...
223
224
class Transaction:
225
def commit(
226
self,
227
actions: list[Any],
228
commit_properties: CommitProperties | None = None,
229
post_commit_hook_properties: PostCommitHookProperties | None = None
230
) -> int: ...
231
```
232
233
[Transaction Management](./transaction-management.md)
234
235
### Query Operations
236
237
SQL querying capabilities using Apache DataFusion integration for running analytical queries on Delta tables.
238
239
```python { .api }
240
class QueryBuilder:
241
def __init__(self): ...
242
def register(self, table_name: str, delta_table: DeltaTable) -> QueryBuilder: ...
243
def execute(self, sql: str) -> RecordBatchReader: ...
244
```
245
246
[Query Operations](./query-operations.md)
247
248
### Table Maintenance
249
250
Operations for table optimization, vacuum cleanup, and checkpoint management to maintain table performance and storage efficiency.
251
252
```python { .api }
253
def vacuum(
254
self,
255
retention_hours: int | None = None,
256
dry_run: bool = True,
257
enforce_retention_duration: bool = True
258
) -> list[str]: ...
259
260
def create_checkpoint(self) -> None: ...
261
def cleanup_metadata(self) -> None: ...
262
def optimize(self) -> TableOptimizer: ...
263
```
264
265
[Table Maintenance](./table-maintenance.md)