0
# OpenDAL
1
2
OpenDAL (Open Data Access Layer) is a unified, high-performance data access library that provides a single API for interacting with 50+ different storage services. It enables developers to write storage-agnostic code that works seamlessly across cloud object storage, file systems, databases, and other data backends.
3
4
## Package Information
5
6
- **Package Name**: opendal
7
- **Package Type**: cargo (Rust crate)
8
- **Language**: Rust
9
- **Installation**: `cargo add opendal`
10
11
## Core Imports
12
13
```rust
14
use opendal::{Operator, BlockingOperator, services, layers, Result};
15
```
16
17
Import specific services and layers as needed:
18
19
```rust
20
use opendal::services::S3;
21
use opendal::layers::LoggingLayer;
22
```
23
24
## Basic Usage
25
26
```rust
27
use opendal::{services, Operator, Result};
28
29
#[tokio::main]
30
async fn main() -> Result<()> {
31
// Initialize a service (S3 example)
32
let mut builder = services::S3::default();
33
builder.bucket("my-bucket");
34
35
// Create operator with optional layers
36
let op = Operator::new(builder)?
37
.layer(opendal::layers::LoggingLayer::default())
38
.finish();
39
40
// Perform operations
41
op.write("hello.txt", "Hello, World!").await?;
42
let content = op.read("hello.txt").await?;
43
let metadata = op.stat("hello.txt").await?;
44
45
println!("Content: {}", String::from_utf8_lossy(&content));
46
println!("Size: {}", metadata.content_length());
47
48
Ok(())
49
}
50
```
51
52
## Architecture
53
54
OpenDAL is built around several key architectural components:
55
56
- **Unified API**: Single interface (`Operator`/`BlockingOperator`) across all storage backends
57
- **Service Abstraction**: Pluggable storage backends (50+ services) with consistent behavior
58
- **Layered Architecture**: Composable middleware layers for cross-cutting concerns (logging, retry, metrics, etc.)
59
- **Zero-Cost Abstraction**: Direct mapping to native service APIs without performance overhead
60
- **Type Safety**: Full Rust type system integration with comprehensive error handling
61
- **Async-First Design**: Native async/await support with blocking alternatives
62
63
## Capabilities
64
65
### Core File Operations
66
67
Primary file and directory operations supporting all major storage patterns. Includes basic CRUD operations, metadata access, and directory management.
68
69
```rust { .api }
70
impl Operator {
71
pub async fn read(&self, path: &str) -> Result<opendal::Buffer>;
72
pub async fn write(&self, path: &str, bs: impl Into<opendal::Buffer>) -> Result<()>;
73
pub async fn stat(&self, path: &str) -> Result<opendal::Metadata>;
74
pub async fn delete(&self, path: &str) -> Result<()>;
75
pub async fn list(&self, path: &str) -> Result<Vec<opendal::Entry>>;
76
}
77
```
78
79
[Core Operations](./core-operations.md)
80
81
### Storage Services
82
83
50+ storage backend implementations providing unified access to cloud storage, file systems, and databases. Each service offers the same API interface while handling service-specific protocols internally.
84
85
```rust { .api }
86
// Cloud Object Storage
87
pub struct S3;
88
pub struct Gcs;
89
pub struct Azblob;
90
91
// File Systems
92
pub struct Fs;
93
pub struct Hdfs;
94
95
// Databases
96
pub struct Redis;
97
pub struct Postgresql;
98
```
99
100
[Storage Services](./services.md)
101
102
### Middleware Layers
103
104
Composable middleware system for adding cross-cutting functionality like logging, retry logic, metrics collection, and performance optimization without changing core business logic.
105
106
```rust { .api }
107
impl<A> OperatorBuilder<A> {
108
pub fn layer<L: Layer<A>>(self, layer: L) -> OperatorBuilder<L::LayeredAccessor>;
109
}
110
111
pub struct LoggingLayer;
112
pub struct RetryLayer;
113
pub struct MetricsLayer;
114
```
115
116
[Layers](./layers.md)
117
118
### Type System
119
120
Comprehensive type system providing storage-agnostic data structures, error handling, and metadata management. Supports both owned and borrowed data with zero-copy optimizations.
121
122
```rust { .api }
123
pub struct Buffer;
124
pub struct Entry;
125
pub struct Metadata;
126
pub enum Scheme;
127
pub enum ErrorKind;
128
```
129
130
[Types and Error Handling](./types.md)
131
132
## Error Handling
133
134
OpenDAL uses Rust's `Result<T, Error>` pattern for comprehensive error handling:
135
136
```rust
137
use opendal::{Result, Error, ErrorKind};
138
139
match op.read("file.txt").await {
140
Ok(data) => println!("Success: {} bytes", data.len()),
141
Err(e) if e.kind() == ErrorKind::NotFound => {
142
println!("File not found");
143
},
144
Err(e) => eprintln!("Error: {}", e),
145
}
146
```
147
148
## Configuration Patterns
149
150
All storage services follow a consistent builder pattern for configuration:
151
152
```rust
153
use opendal::services::S3;
154
155
let mut builder = S3::default();
156
builder
157
.bucket("my-bucket")
158
.region("us-west-2")
159
.access_key_id("key")
160
.secret_access_key("secret");
161
162
let op = Operator::new(builder)?.finish();
163
```
164
165
## Advanced Features
166
167
- **Presigned URLs**: Generate time-limited URLs for secure access
168
- **Batch Operations**: Efficient bulk operations for multiple files
169
- **Streaming I/O**: Memory-efficient handling of large files
170
- **Capability Detection**: Runtime discovery of service capabilities
171
- **Layer Composition**: Stack multiple middleware layers for complex functionality