0
# Storage Services
1

2
The storage service provides a unified interface for object storage and blob operations across 20+ storage providers including AWS S3, Azure Blob Storage, Google Cloud Storage, Rackspace Cloud Files, and many others.
3

4
## Providers
5

6
```python { .api }
7
from libcloud.storage.types import Provider
8

9
class Provider:
10
    """Enumeration of supported storage providers"""
11
    S3 = 's3'
12
    S3_US_WEST = 's3_us_west'
13
    S3_EU_WEST = 's3_eu_west'
14
    S3_AP_SOUTHEAST = 's3_ap_southeast'
15
    S3_AP_NORTHEAST = 's3_ap_northeast'
16
    AZURE_BLOBS = 'azure_blobs'
17
    GOOGLE_STORAGE = 'google_storage'
18
    RACKSPACE_CLOUDFILES = 'cloudfiles'
19
    CLOUDFILES_US = 'cloudfiles_us'
20
    CLOUDFILES_UK = 'cloudfiles_uk'
21
    NINEFOLD = 'ninefold'
22
    LOCAL = 'local'
23
    # ... more providers
24
```
25

26
## Driver Factory
27

28
```python { .api }
29
from libcloud.storage.providers import get_driver
30

31
def get_driver(provider: Provider) -> type[StorageDriver]
32
```
33

34
Get the driver class for a specific storage provider.
35

36
**Parameters:**
37
- `provider`: Provider identifier from the Provider enum
38

39
**Returns:**
40
- Driver class for the specified provider
41

42
**Example:**
43
```python
44
from libcloud.storage.types import Provider
45
from libcloud.storage.providers import get_driver
46

47
# Get S3 driver class
48
cls = get_driver(Provider.S3)
49

50
# Initialize driver with credentials
51
driver = cls('access_key', 'secret_key')
52
```
53

54
## Core Classes
55

56
### StorageDriver
57

58
```python { .api }
59
class StorageDriver(BaseDriver):
60
    """Base class for all storage drivers"""
61
    
62
    def list_containers(self) -> List[Container]
63
    def get_container(self, container_name: str) -> Container
64
    def create_container(self, container_name: str) -> Container
65
    def delete_container(self, container: Container) -> bool
66
    def list_container_objects(self, container: Container, ex_prefix: str = None) -> List[Object]
67
    def get_object(self, container_name: str, object_name: str) -> Object
68
    def upload_object(self, file_path: str, container: Container, object_name: str, extra: Dict = None, ex_meta: Dict = None, verify_hash: bool = True) -> Object
69
    def upload_object_via_stream(self, iterator: Iterator[bytes], container: Container, object_name: str, extra: Dict = None, ex_meta: Dict = None) -> Object
70
    def download_object(self, obj: Object, destination_path: str, overwrite_existing: bool = False, delete_on_failure: bool = True) -> bool
71
    def download_object_as_stream(self, obj: Object, chunk_size: int = None) -> Iterator[bytes]
72
    def delete_object(self, obj: Object) -> bool
73
    def ex_iterate_container_objects(self, container: Container, prefix: str = None) -> Iterator[Object]
74
```
75

76
Base class that all storage drivers inherit from. Provides methods for managing containers (buckets) and objects (files/blobs).
77

78
**Key Methods:**
79

80
- `list_containers()`: List all containers/buckets in the account
81
- `create_container()`: Create a new container/bucket
82
- `upload_object()`: Upload a file to storage
83
- `download_object()`: Download a file from storage
84
- `delete_object()`: Delete a stored object
85
- `list_container_objects()`: List objects within a container
86

87
### Container
88

89
```python { .api }
90
class Container:
91
    """Represents a storage container/bucket"""
92
    
93
    name: str
94
    driver: StorageDriver
95
    extra: Dict[str, Any]
96
    
97
    def get_cdn_url(self) -> str
98
    def enable_cdn(self, **kwargs) -> bool
99
    def get_objects(self) -> List[Object]
100
    def upload_object(self, file_path: str, object_name: str) -> Object
101
    def delete_object(self, obj: Object) -> bool
102
```
103

104
Represents a storage container (bucket in S3 terminology) that holds objects.
105

106
**Properties:**
107
- `name`: Container name
108
- `extra`: Provider-specific metadata
109

110
**Methods:**
111
- `get_cdn_url()`: Get CDN URL if CDN is enabled
112
- `enable_cdn()`: Enable CDN for the container
113
- `upload_object()`: Upload an object to this container
114

115
### Object
116

117
```python { .api }
118
class Object:
119
    """Represents a storage object/blob"""
120
    
121
    name: str
122
    size: int
123
    hash: str
124
    container: Container
125
    driver: StorageDriver
126
    meta_data: Dict[str, str]
127
    extra: Dict[str, Any]
128
    
129
    def get_cdn_url(self) -> str
130
    def enable_cdn(self) -> bool
131
    def download(self, destination_path: str, overwrite_existing: bool = False) -> bool
132
    def as_stream(self, chunk_size: int = None) -> Iterator[bytes]
133
    def delete(self) -> bool
134
```
135

136
Represents a stored object/file/blob.
137

138
**Properties:**
139
- `name`: Object name/key
140
- `size`: Size in bytes
141
- `hash`: Content hash (usually MD5 or ETag)
142
- `container`: Parent container
143
- `meta_data`: Custom metadata dictionary
144
- `extra`: Provider-specific metadata
145

146
**Methods:**
147
- `download()`: Download object to local file
148
- `as_stream()`: Get object contents as a stream
149
- `delete()`: Delete the object
150

151
## Constants
152

153
```python { .api }
154
CHUNK_SIZE: int = 8096
155
DEFAULT_CONTENT_TYPE: str = "application/octet-stream"
156
```
157

158
- `CHUNK_SIZE`: Default chunk size for streaming operations
159
- `DEFAULT_CONTENT_TYPE`: Default MIME type for uploaded objects
160

161
## Usage Examples
162

163
### Basic Container and Object Management
164

165
```python
166
from libcloud.storage.types import Provider
167
from libcloud.storage.providers import get_driver
168

169
# Initialize driver
170
cls = get_driver(Provider.S3)
171
driver = cls('access_key', 'secret_key')
172

173
# List existing containers
174
containers = driver.list_containers()
175
for container in containers:
176
    print(f"Container: {container.name}")
177

178
# Create a new container
179
container = driver.create_container('my-data-bucket')
180
print(f"Created container: {container.name}")
181

182
# List objects in container
183
objects = driver.list_container_objects(container)
184
print(f"Container has {len(objects)} objects")
185
```
186

187
### File Upload and Download
188

189
```python
190
# Upload a local file
191
obj = driver.upload_object(
192
    file_path='/path/to/local/file.txt',
193
    container=container,
194
    object_name='documents/file.txt',
195
    extra={'ContentType': 'text/plain'},
196
    ex_meta={'author': 'john_doe', 'version': '1.0'}
197
)
198
print(f"Uploaded object: {obj.name} ({obj.size} bytes)")
199

200
# Download the object
201
success = driver.download_object(
202
    obj, 
203
    destination_path='/path/to/download/file.txt',
204
    overwrite_existing=True
205
)
206
print(f"Download successful: {success}")
207

208
# Alternative: download using object method
209
success = obj.download('/path/to/download/file2.txt')
210
print(f"Downloaded via object method: {success}")
211
```
212

213
### Streaming Operations
214

215
```python
216
# Upload from stream
217
def file_generator():
218
    with open('/path/to/large/file.txt', 'rb') as f:
219
        while True:
220
            chunk = f.read(8192)
221
            if not chunk:
222
                break
223
            yield chunk
224

225
obj = driver.upload_object_via_stream(
226
    iterator=file_generator(),
227
    container=container,
228
    object_name='streaming/large_file.txt'
229
)
230
print(f"Streamed upload complete: {obj.name}")
231

232
# Download as stream
233
stream = driver.download_object_as_stream(obj, chunk_size=4096)
234
with open('/path/to/output.txt', 'wb') as f:
235
    for chunk in stream:
236
        f.write(chunk)
237

238
# Alternative: using object method
239
stream = obj.as_stream(chunk_size=8192)
240
total_size = 0
241
for chunk in stream:
242
    total_size += len(chunk)
243
    # Process chunk
244
print(f"Streamed {total_size} bytes")
245
```
246

247
### Working with Object Metadata
248

249
```python
250
# Upload with custom metadata
251
obj = driver.upload_object(
252
    file_path='/path/to/document.pdf',
253
    container=container,
254
    object_name='docs/report.pdf',
255
    extra={'ContentType': 'application/pdf'},
256
    ex_meta={
257
        'title': 'Monthly Report',
258
        'author': 'Jane Smith',
259
        'department': 'Finance',
260
        'created': '2023-10-15'
261
    }
262
)
263

264
# Access metadata
265
print(f"Object metadata: {obj.meta_data}")
266
print(f"Content type: {obj.extra.get('content_type')}")
267
print(f"Last modified: {obj.extra.get('last_modified')}")
268
```
269

270
### Container Iteration and Filtering
271

272
```python
273
# List objects with prefix filtering
274
objects = driver.list_container_objects(container, ex_prefix='documents/')
275
print(f"Found {len(objects)} objects in documents/ folder")
276

277
# Iterate through large containers efficiently
278
for obj in driver.ex_iterate_container_objects(container, prefix='logs/'):
279
    print(f"Log file: {obj.name} ({obj.size} bytes)")
280
    if obj.size > 1000000:  # Process large files differently
281
        print(f"  Large file detected: {obj.name}")
282
```
283

284
### Multi-Provider Storage Management
285

286
```python
287
from libcloud.storage.types import Provider
288
from libcloud.storage.providers import get_driver
289

290
# Configure multiple storage providers
291
storage_config = {
292
    's3': {
293
        'driver': get_driver(Provider.S3),
294
        'credentials': ('aws_access_key', 'aws_secret_key')
295
    },
296
    'azure': {
297
        'driver': get_driver(Provider.AZURE_BLOBS),
298
        'credentials': ('account_name', 'account_key')
299
    },
300
    'gcs': {
301
        'driver': get_driver(Provider.GOOGLE_STORAGE),
302
        'credentials': ('service_account_email', 'key_file_path')
303
    }
304
}
305

306
# Initialize drivers
307
drivers = {}
308
for name, config in storage_config.items():
309
    cls = config['driver']
310
    drivers[name] = cls(*config['credentials'])
311

312
# Sync data across providers
313
def sync_container(source_driver, dest_driver, container_name):
314
    # Get or create containers
315
    try:
316
        source_container = source_driver.get_container(container_name)
317
    except Exception:
318
        print(f"Source container {container_name} not found")
319
        return
320
    
321
    try:
322
        dest_container = dest_driver.get_container(container_name)
323
    except Exception:
324
        dest_container = dest_driver.create_container(container_name)
325
    
326
    # Sync objects
327
    for obj in source_driver.list_container_objects(source_container):
328
        print(f"Syncing {obj.name}...")
329
        
330
        # Download from source
331
        stream = source_driver.download_object_as_stream(obj)
332
        
333
        # Upload to destination
334
        dest_driver.upload_object_via_stream(
335
            iterator=stream,
336
            container=dest_container,
337
            object_name=obj.name,
338
            ex_meta=obj.meta_data
339
        )
340

341
# Sync from S3 to Azure
342
sync_container(drivers['s3'], drivers['azure'], 'backup-data')
343
```
344

345
### CDN and Public Access
346

347
```python
348
# Enable CDN for container (if supported)
349
try:
350
    success = container.enable_cdn()
351
    if success:
352
        cdn_url = container.get_cdn_url()
353
        print(f"CDN enabled. URL: {cdn_url}")
354
except Exception as e:
355
    print(f"CDN not supported or failed: {e}")
356

357
# Get public URLs for objects
358
try:
359
    public_url = obj.get_cdn_url()
360
    print(f"Public URL: {public_url}")
361
except Exception:
362
    print("Public URLs not supported by this provider")
363
```
364

365
### Error Handling and Best Practices
366

367
```python
368
from libcloud.storage.types import ContainerDoesNotExistError, ObjectDoesNotExistError
369
from libcloud.common.types import InvalidCredsError, LibcloudError
370

371
try:
372
    # Attempt to get a container that might not exist
373
    container = driver.get_container('nonexistent-container')
374
except ContainerDoesNotExistError:
375
    print("Container does not exist, creating it...")
376
    container = driver.create_container('nonexistent-container')
377

378
try:
379
    # Attempt to get an object that might not exist
380
    obj = driver.get_object('my-container', 'nonexistent-file.txt')
381
except ObjectDoesNotExistError:
382
    print("Object does not exist")
383

384
# Verify upload integrity
385
obj = driver.upload_object(
386
    file_path='/path/to/important.dat',
387
    container=container,
388
    object_name='important.dat',
389
    verify_hash=True  # Verify MD5 hash after upload
390
)
391

392
# Safe deletion with confirmation
393
objects_to_delete = driver.list_container_objects(container, ex_prefix='temp/')
394
if objects_to_delete:
395
    print(f"About to delete {len(objects_to_delete)} temporary objects")
396
    for obj in objects_to_delete:
397
        success = driver.delete_object(obj)
398
        print(f"Deleted {obj.name}: {success}")
399
```
400

401
### Advanced Features and Provider-Specific Options
402

403
```python
404
# AWS S3 specific features
405
s3_driver = get_driver(Provider.S3)('access_key', 'secret_key')
406

407
# Upload with S3-specific options
408
obj = s3_driver.upload_object(
409
    file_path='/path/to/file.txt',
410
    container=container,
411
    object_name='data/file.txt',
412
    extra={
413
        'ContentType': 'text/plain',
414
        'ContentEncoding': 'gzip',
415
        'StorageClass': 'GLACIER',  # Use Glacier storage class
416
        'ServerSideEncryption': 'AES256'  # Enable encryption
417
    }
418
)
419

420
# Azure specific features
421
azure_driver = get_driver(Provider.AZURE_BLOBS)('account', 'key')
422

423
# Upload with Azure-specific metadata
424
obj = azure_driver.upload_object(
425
    file_path='/path/to/file.txt',
426
    container=container,
427
    object_name='data/file.txt',
428
    ex_meta={'department': 'IT', 'project': 'backup'},
429
    extra={'blob_type': 'BlockBlob'}
430
)
431

432
# Google Cloud Storage specific features
433
gcs_driver = get_driver(Provider.GOOGLE_STORAGE)('email', 'key_file')
434

435
# Upload with GCS-specific options
436
obj = gcs_driver.upload_object(
437
    file_path='/path/to/file.txt',
438
    container=container,
439
    object_name='data/file.txt',
440
    extra={'storage_class': 'COLDLINE'}  # Use Coldline storage
441
)
442
```
443

444
## Common Use Cases
445

446
### Backup and Archive
447

448
```python
449
import os
450
import datetime
451

452
def backup_directory(driver, container, local_dir, remote_prefix=''):
453
    """Recursively backup a local directory to cloud storage"""
454
    for root, dirs, files in os.walk(local_dir):
455
        for file in files:
456
            local_path = os.path.join(root, file)
457
            relative_path = os.path.relpath(local_path, local_dir)
458
            remote_path = os.path.join(remote_prefix, relative_path).replace('\\', '/')
459
            
460
            print(f"Backing up: {local_path} -> {remote_path}")
461
            
462
            obj = driver.upload_object(
463
                file_path=local_path,
464
                container=container,
465
                object_name=remote_path,
466
                ex_meta={
467
                    'backup_date': datetime.datetime.now().isoformat(),
468
                    'source_path': local_path
469
                }
470
            )
471

472
# Usage
473
backup_container = driver.create_container(f'backup-{datetime.date.today()}')
474
backup_directory(driver, backup_container, '/home/user/documents', 'documents/')
475
```
476

477
### Static Website Hosting
478

479
```python
480
def deploy_static_site(driver, container_name, site_dir):
481
    """Deploy a static website to cloud storage"""
482
    # Create or get container
483
    try:
484
        container = driver.get_container(container_name)
485
    except ContainerDoesNotExistError:
486
        container = driver.create_container(container_name)
487
    
488
    # Enable public access/CDN if supported
489
    try:
490
        container.enable_cdn()
491
    except Exception:
492
        pass
493
    
494
    # Upload all site files
495
    for root, dirs, files in os.walk(site_dir):
496
        for file in files:
497
            local_path = os.path.join(root, file)
498
            relative_path = os.path.relpath(local_path, site_dir)
499
            
500
            # Determine content type
501
            content_type = 'text/html' if file.endswith('.html') else None
502
            if file.endswith('.css'):
503
                content_type = 'text/css'
504
            elif file.endswith('.js'):
505
                content_type = 'application/javascript'
506
            
507
            extra = {}
508
            if content_type:
509
                extra['ContentType'] = content_type
510
            
511
            obj = driver.upload_object(
512
                file_path=local_path,
513
                container=container,
514
                object_name=relative_path,
515
                extra=extra
516
            )
517
            print(f"Deployed: {relative_path}")
518

519
# Usage
520
deploy_static_site(driver, 'my-website', '/path/to/site/')
521
```
522

523
## Exception Types
524

525
```python { .api }
526
from libcloud.storage.types import (
527
    ContainerError,
528
    ObjectError, 
529
    ContainerDoesNotExistError,
530
    ObjectDoesNotExistError,
531
    ContainerAlreadyExistsError,
532
    ObjectHashMismatchError,
533
    InvalidContainerNameError
534
)
535

536
class ContainerError(LibcloudError):
537
    """Base container exception"""
538
    
539
class ObjectError(LibcloudError):
540
    """Base object exception"""
541
    
542
class ContainerDoesNotExistError(ContainerError):
543
    """Container does not exist"""
544
    
545
class ObjectDoesNotExistError(ObjectError):
546
    """Object does not exist"""
547
    
548
class ContainerAlreadyExistsError(ContainerError):
549
    """Container already exists"""
550
    
551
class ObjectHashMismatchError(ObjectError):
552
    """Object hash verification failed"""
553

554
class InvalidContainerNameError(ContainerError):
555
    """Invalid container name provided"""
556
```
557

558
These exceptions provide specific error handling for storage operations, allowing you to handle different failure scenarios appropriately.