0
# Web Integration
1
2
HTTP handlers for serving GridFS files through web frameworks. Motor provides optimized handlers for both Tornado and aiohttp with support for HTTP caching, range requests, content negotiation, and efficient file streaming.
3
4
## Capabilities
5
6
### Tornado GridFS Handler
7
8
RequestHandler for serving GridFS files in Tornado web applications with full HTTP feature support.
9
10
```python { .api }
11
class GridFSHandler(tornado.web.RequestHandler):
12
"""
13
Tornado RequestHandler for serving files from GridFS.
14
15
Supports:
16
- HTTP caching with ETag and Last-Modified headers
17
- Range requests for partial content
18
- Content-Type detection based on file extension
19
- Custom file retrieval logic
20
- Configurable cache behavior
21
"""
22
23
def initialize(
24
self,
25
database: MotorDatabase,
26
root_collection: str = 'fs',
27
get_gridfs_file: Optional[Callable] = None,
28
get_cache_time: Optional[Callable] = None
29
) -> None:
30
"""
31
Initialize the GridFS handler.
32
33
Parameters:
34
- database: MotorDatabase instance for GridFS access
35
- root_collection: GridFS collection name prefix (default: 'fs')
36
- get_gridfs_file: Custom function to retrieve GridFS files
37
- get_cache_time: Custom function to determine cache duration
38
"""
39
40
async def get(self, filename: str, include_body: bool = True) -> None:
41
"""
42
Handle GET requests for GridFS files.
43
44
Parameters:
45
- filename: Name of the file to serve
46
- include_body: Whether to include file content in response
47
"""
48
49
async def head(self, filename: str) -> None:
50
"""Handle HEAD requests (metadata only)."""
51
52
# Customization Methods (override in subclasses)
53
async def get_gridfs_file(
54
self,
55
fs: MotorGridFSBucket,
56
filename: str
57
) -> MotorGridOut:
58
"""
59
Override to customize file retrieval logic.
60
61
Default implementation opens file by name.
62
Override to implement custom logic like file ID lookup,
63
permission checking, or path transformation.
64
65
Parameters:
66
- fs: GridFS bucket instance
67
- filename: Requested filename from URL
68
69
Returns:
70
MotorGridOut instance for the requested file
71
"""
72
73
def get_cache_time(
74
self,
75
filename: str,
76
modified: datetime.datetime,
77
mime_type: str
78
) -> int:
79
"""
80
Override to customize cache duration.
81
82
Parameters:
83
- filename: Name of the file
84
- modified: File modification timestamp
85
- mime_type: MIME type of the file
86
87
Returns:
88
Cache duration in seconds (0 for no cache)
89
"""
90
```
91
92
### aiohttp GridFS Handler
93
94
Handler for serving GridFS files in aiohttp web applications with async/await support.
95
96
```python { .api }
97
class AIOHTTPGridFS:
98
"""
99
aiohttp handler for serving files from GridFS.
100
101
Supports:
102
- HTTP caching with ETag and Last-Modified headers
103
- Streaming file responses
104
- Content-Type detection
105
- Custom file retrieval and cache logic
106
- If-Modified-Since conditional requests
107
"""
108
109
def __init__(
110
self,
111
database: AsyncIOMotorDatabase,
112
root_collection: str = 'fs',
113
get_gridfs_file: Optional[Callable] = None,
114
get_cache_time: Optional[Callable] = None,
115
set_extra_headers: Optional[Callable] = None
116
) -> None:
117
"""
118
Initialize the aiohttp GridFS handler.
119
120
Parameters:
121
- database: AsyncIOMotorDatabase instance
122
- root_collection: GridFS collection name prefix
123
- get_gridfs_file: Custom file retrieval function
124
- get_cache_time: Custom cache duration function
125
- set_extra_headers: Custom header setting function
126
"""
127
128
async def __call__(self, request: aiohttp.web.Request) -> aiohttp.web.Response:
129
"""
130
Handle HTTP requests for GridFS files.
131
132
Parameters:
133
- request: aiohttp Request object
134
135
Returns:
136
aiohttp Response with file content or error
137
"""
138
139
# Customization Functions for aiohttp
140
async def get_gridfs_file(
141
bucket: AsyncIOMotorGridFSBucket,
142
filename: str,
143
request: aiohttp.web.Request
144
) -> AsyncIOMotorGridOut:
145
"""
146
Default file retrieval function for aiohttp handler.
147
148
Override this function to customize file lookup logic:
149
- Retrieve by file ID instead of name
150
- Implement access control
151
- Transform file paths or names
152
- Add logging or metrics
153
154
Parameters:
155
- bucket: GridFS bucket instance
156
- filename: Requested filename from URL path
157
- request: aiohttp request object for context
158
159
Returns:
160
AsyncIOMotorGridOut for the requested file
161
"""
162
163
def get_cache_time(
164
filename: str,
165
modified: datetime.datetime,
166
mime_type: Optional[str]
167
) -> int:
168
"""
169
Default cache duration function for aiohttp handler.
170
171
Override to implement custom cache policies:
172
- Different cache times for different file types
173
- No caching for sensitive files
174
- Long-term caching for static assets
175
176
Parameters:
177
- filename: Name of the file
178
- modified: File upload/modification timestamp
179
- mime_type: MIME type of the file (may be None)
180
181
Returns:
182
Cache duration in seconds (0 for no aggressive caching)
183
"""
184
185
def set_extra_headers(
186
response: aiohttp.web.Response,
187
gridout: AsyncIOMotorGridOut
188
) -> None:
189
"""
190
Default extra headers function for aiohttp handler.
191
192
Override to add custom HTTP headers:
193
- Security headers (CSP, CORS, etc.)
194
- Content-Encoding headers
195
- Custom application headers
196
197
Parameters:
198
- response: aiohttp Response object to modify
199
- gridout: GridFS file being served
200
"""
201
```
202
203
## Usage Examples
204
205
### Basic Tornado GridFS Server
206
207
```python
208
import tornado.web
209
import tornado.ioloop
210
import motor.motor_tornado
211
from motor.web import GridFSHandler
212
213
def make_app():
214
# Connect to MongoDB
215
client = motor.motor_tornado.MotorClient()
216
database = client.my_app_db
217
218
return tornado.web.Application([
219
# Serve GridFS files at /files/<filename>
220
(r"/files/(.*)", GridFSHandler, {
221
"database": database,
222
"root_collection": "fs" # Use default GridFS collection
223
}),
224
# Regular request handlers
225
(r"/", MainHandler),
226
])
227
228
class MainHandler(tornado.web.RequestHandler):
229
def get(self):
230
self.write("""
231
<h1>GridFS File Server</h1>
232
<p>Files available at <a href="/files/">/files/<filename></a></p>
233
""")
234
235
if __name__ == "__main__":
236
app = make_app()
237
app.listen(8888)
238
print("Server running on http://localhost:8888")
239
tornado.ioloop.IOLoop.current().start()
240
```
241
242
### Custom Tornado GridFS Handler
243
244
```python
245
import tornado.web
246
import tornado.ioloop
247
import motor.motor_tornado
248
from motor.web import GridFSHandler
249
import mimetypes
250
from bson import ObjectId
251
252
class CustomGridFSHandler(GridFSHandler):
253
"""Custom GridFS handler with file ID support and custom caching."""
254
255
async def get_gridfs_file(self, fs, filename):
256
"""Support both filename and ObjectId lookup."""
257
258
# Try to parse as ObjectId first
259
try:
260
file_id = ObjectId(filename)
261
return await fs.open_download_stream(file_id)
262
except:
263
# Fall back to filename lookup
264
return await fs.open_download_stream_by_name(filename)
265
266
def get_cache_time(self, filename, modified, mime_type):
267
"""Custom cache policy based on file type."""
268
269
if mime_type:
270
# Long cache for images and static assets
271
if mime_type.startswith('image/'):
272
return 3600 * 24 * 7 # 1 week
273
elif mime_type in ['text/css', 'application/javascript']:
274
return 3600 * 24 # 1 day
275
elif mime_type.startswith('video/'):
276
return 3600 * 2 # 2 hours
277
278
# Default: no aggressive caching
279
return 0
280
281
class UploadHandler(tornado.web.RequestHandler):
282
"""Handler for uploading files to GridFS."""
283
284
async def post(self):
285
database = self.application.settings['database']
286
fs = motor.motor_tornado.MotorGridFSBucket(database)
287
288
# Get uploaded file
289
if 'file' not in self.request.files:
290
self.set_status(400)
291
self.write({"error": "No file uploaded"})
292
return
293
294
file_info = self.request.files['file'][0]
295
filename = file_info['filename']
296
content_type = file_info.get('content_type', 'application/octet-stream')
297
file_data = file_info['body']
298
299
# Upload to GridFS
300
from io import BytesIO
301
file_stream = BytesIO(file_data)
302
303
file_id = await fs.upload_from_stream(
304
filename,
305
file_stream,
306
metadata={'content_type': content_type}
307
)
308
309
self.write({
310
"message": "File uploaded successfully",
311
"file_id": str(file_id),
312
"filename": filename,
313
"url": f"/files/{file_id}"
314
})
315
316
def make_app():
317
client = motor.motor_tornado.MotorClient()
318
database = client.file_server_db
319
320
return tornado.web.Application([
321
(r"/files/(.*)", CustomGridFSHandler, {"database": database}),
322
(r"/upload", UploadHandler),
323
(r"/", MainHandler),
324
], database=database)
325
326
if __name__ == "__main__":
327
app = make_app()
328
app.listen(8888)
329
tornado.ioloop.IOLoop.current().start()
330
```
331
332
### Basic aiohttp GridFS Server
333
334
```python
335
import aiohttp.web
336
import motor.motor_asyncio
337
from motor.aiohttp import AIOHTTPGridFS
338
339
async def create_app():
340
# Connect to MongoDB
341
client = motor.motor_asyncio.AsyncIOMotorClient()
342
database = client.my_app_db
343
344
# Create GridFS handler
345
gridfs_handler = AIOHTTPGridFS(database)
346
347
app = aiohttp.web.Application()
348
349
# Add GridFS route - filename must be in path variable
350
resource = app.router.add_resource("/files/{filename}")
351
resource.add_route("GET", gridfs_handler)
352
resource.add_route("HEAD", gridfs_handler)
353
354
# Add index route
355
async def index(request):
356
return aiohttp.web.Response(
357
text="""
358
<h1>GridFS File Server</h1>
359
<p>Files available at <a href="/files/">/files/<filename></a></p>
360
""",
361
content_type='text/html'
362
)
363
364
app.router.add_get('/', index)
365
366
return app
367
368
if __name__ == "__main__":
369
app = create_app()
370
aiohttp.web.run_app(app, host='localhost', port=8888)
371
```
372
373
### Custom aiohttp GridFS Handler
374
375
```python
376
import aiohttp.web
377
import motor.motor_asyncio
378
from motor.aiohttp import AIOHTTPGridFS, get_gridfs_file, get_cache_time, set_extra_headers
379
from bson import ObjectId
380
import json
381
382
# Custom file retrieval function
383
async def custom_get_gridfs_file(bucket, filename, request):
384
"""Support file ID lookup and access control."""
385
386
# Check authentication (example)
387
auth_header = request.headers.get('Authorization')
388
if not auth_header:
389
raise aiohttp.web.HTTPUnauthorized(text="Authentication required")
390
391
# Try ObjectId lookup first
392
try:
393
file_id = ObjectId(filename)
394
return await bucket.open_download_stream(file_id)
395
except:
396
# Fall back to filename
397
return await bucket.open_download_stream_by_name(filename)
398
399
# Custom cache policy
400
def custom_get_cache_time(filename, modified, mime_type):
401
"""Aggressive caching for static assets."""
402
403
if mime_type:
404
if mime_type.startswith('image/'):
405
return 3600 * 24 * 30 # 30 days for images
406
elif mime_type.startswith('video/'):
407
return 3600 * 24 * 7 # 7 days for videos
408
elif mime_type in ['text/css', 'application/javascript']:
409
return 3600 * 24 # 1 day for CSS/JS
410
411
return 3600 # 1 hour default
412
413
# Custom headers
414
def custom_set_extra_headers(response, gridout):
415
"""Add security and CORS headers."""
416
417
# CORS headers
418
response.headers['Access-Control-Allow-Origin'] = '*'
419
response.headers['Access-Control-Allow-Methods'] = 'GET, HEAD'
420
421
# Security headers
422
response.headers['X-Content-Type-Options'] = 'nosniff'
423
response.headers['X-Frame-Options'] = 'DENY'
424
425
# Custom metadata headers
426
if gridout.metadata:
427
if 'author' in gridout.metadata:
428
response.headers['X-File-Author'] = str(gridout.metadata['author'])
429
430
async def upload_handler(request):
431
"""Handle file uploads to GridFS."""
432
433
database = request.app['database']
434
bucket = motor.motor_asyncio.AsyncIOMotorGridFSBucket(database)
435
436
# Handle multipart upload
437
reader = await request.multipart()
438
439
while True:
440
field = await reader.next()
441
if not field:
442
break
443
444
if field.name == 'file':
445
filename = field.filename
446
content_type = field.headers.get('Content-Type', 'application/octet-stream')
447
448
# Create upload stream
449
upload_stream = bucket.open_upload_stream(
450
filename,
451
metadata={
452
'content_type': content_type,
453
'uploaded_by': request.headers.get('X-User-ID', 'anonymous')
454
}
455
)
456
457
# Stream file data
458
while True:
459
chunk = await field.read_chunk()
460
if not chunk:
461
break
462
await upload_stream.write(chunk)
463
464
await upload_stream.close()
465
466
return aiohttp.web.json_response({
467
'message': 'File uploaded successfully',
468
'file_id': str(upload_stream._id),
469
'filename': filename,
470
'url': f'/files/{upload_stream._id}'
471
})
472
473
return aiohttp.web.json_response(
474
{'error': 'No file uploaded'},
475
status=400
476
)
477
478
async def file_list_handler(request):
479
"""List available files."""
480
481
database = request.app['database']
482
bucket = motor.motor_asyncio.AsyncIOMotorGridFSBucket(database)
483
484
files = []
485
cursor = bucket.find().sort('uploadDate', -1).limit(50)
486
487
async for file_doc in cursor:
488
files.append({
489
'id': str(file_doc._id),
490
'filename': file_doc.filename,
491
'length': file_doc.length,
492
'uploadDate': file_doc.upload_date.isoformat(),
493
'contentType': file_doc.content_type,
494
'url': f'/files/{file_doc._id}'
495
})
496
497
return aiohttp.web.json_response({'files': files})
498
499
async def create_advanced_app():
500
client = motor.motor_asyncio.AsyncIOMotorClient()
501
database = client.advanced_file_server
502
503
# Create custom GridFS handler
504
gridfs_handler = AIOHTTPGridFS(
505
database,
506
get_gridfs_file=custom_get_gridfs_file,
507
get_cache_time=custom_get_cache_time,
508
set_extra_headers=custom_set_extra_headers
509
)
510
511
app = aiohttp.web.Application()
512
app['database'] = database
513
514
# GridFS file serving
515
resource = app.router.add_resource("/files/{filename}")
516
resource.add_route("GET", gridfs_handler)
517
resource.add_route("HEAD", gridfs_handler)
518
519
# File management endpoints
520
app.router.add_post('/upload', upload_handler)
521
app.router.add_get('/api/files', file_list_handler)
522
523
return app
524
525
if __name__ == "__main__":
526
app = create_advanced_app()
527
aiohttp.web.run_app(app, host='localhost', port=8888)
528
```
529
530
### GridFS File Upload Utility
531
532
```python
533
import asyncio
534
import motor.motor_asyncio
535
import aiofiles
536
import mimetypes
537
import os
538
539
async def upload_file_to_gridfs(database, local_path, gridfs_filename=None):
540
"""Utility function to upload local files to GridFS."""
541
542
bucket = motor.motor_asyncio.AsyncIOMotorGridFSBucket(database)
543
544
if not gridfs_filename:
545
gridfs_filename = os.path.basename(local_path)
546
547
# Detect content type
548
content_type, _ = mimetypes.guess_type(local_path)
549
if not content_type:
550
content_type = 'application/octet-stream'
551
552
# Get file stats
553
stat = os.stat(local_path)
554
555
# Upload file
556
async with aiofiles.open(local_path, 'rb') as f:
557
file_id = await bucket.upload_from_stream(
558
gridfs_filename,
559
f,
560
metadata={
561
'content_type': content_type,
562
'original_path': local_path,
563
'file_size': stat.st_size,
564
'upload_tool': 'motor_utility'
565
}
566
)
567
568
print(f"Uploaded {local_path} as {gridfs_filename}")
569
print(f"GridFS ID: {file_id}")
570
return file_id
571
572
async def main():
573
client = motor.motor_asyncio.AsyncIOMotorClient()
574
database = client.file_storage
575
576
# Upload multiple files
577
files_to_upload = [
578
'/path/to/image.jpg',
579
'/path/to/document.pdf',
580
'/path/to/video.mp4'
581
]
582
583
for file_path in files_to_upload:
584
if os.path.exists(file_path):
585
await upload_file_to_gridfs(database, file_path)
586
else:
587
print(f"File not found: {file_path}")
588
589
client.close()
590
591
if __name__ == "__main__":
592
asyncio.run(main())
593
```
594
595
### Performance Optimization Example
596
597
```python
598
import aiohttp.web
599
import motor.motor_asyncio
600
from motor.aiohttp import AIOHTTPGridFS
601
import asyncio
602
import time
603
604
# Optimized file retrieval with caching
605
class CachedGridFSHandler(AIOHTTPGridFS):
606
def __init__(self, *args, **kwargs):
607
super().__init__(*args, **kwargs)
608
self._file_cache = {} # Simple in-memory cache
609
self._cache_ttl = 300 # 5 minutes
610
611
async def _get_cached_file(self, filename):
612
"""Check cache for file metadata."""
613
if filename in self._file_cache:
614
cached_data, timestamp = self._file_cache[filename]
615
if time.time() - timestamp < self._cache_ttl:
616
return cached_data
617
618
return None
619
620
async def _cache_file(self, filename, file_data):
621
"""Cache file metadata."""
622
self._file_cache[filename] = (file_data, time.time())
623
624
async def __call__(self, request):
625
filename = request.match_info['filename']
626
627
# Try cache first for metadata
628
cached = await self._get_cached_file(filename)
629
if cached and request.method == 'HEAD':
630
# Return cached metadata for HEAD requests
631
response = aiohttp.web.Response()
632
response.headers.update(cached['headers'])
633
return response
634
635
# Fall back to parent implementation
636
return await super().__call__(request)
637
638
# Usage with performance monitoring
639
async def create_optimized_app():
640
client = motor.motor_asyncio.AsyncIOMotorClient(
641
maxPoolSize=50, # Increase connection pool
642
minPoolSize=10
643
)
644
database = client.high_performance_files
645
646
# Use cached handler
647
gridfs_handler = CachedGridFSHandler(database)
648
649
app = aiohttp.web.Application()
650
651
# Add middleware for request timing
652
async def timing_middleware(request, handler):
653
start = time.time()
654
response = await handler(request)
655
duration = time.time() - start
656
response.headers['X-Response-Time'] = f"{duration:.3f}s"
657
return response
658
659
app.middlewares.append(timing_middleware)
660
661
# GridFS routes
662
resource = app.router.add_resource("/files/{filename}")
663
resource.add_route("GET", gridfs_handler)
664
resource.add_route("HEAD", gridfs_handler)
665
666
# Health check endpoint
667
async def health_check(request):
668
return aiohttp.web.json_response({"status": "healthy"})
669
670
app.router.add_get('/health', health_check)
671
672
return app
673
674
if __name__ == "__main__":
675
app = create_optimized_app()
676
aiohttp.web.run_app(app, host='localhost', port=8888)
677
```
678
679
## Types
680
681
```python { .api }
682
from typing import Any, Optional, Callable, Union
683
import tornado.web
684
import aiohttp.web
685
from datetime import datetime
686
687
# Handler types
688
GridFSHandlerType = tornado.web.RequestHandler
689
AIOHTTPHandlerType = Callable[[aiohttp.web.Request], aiohttp.web.Response]
690
691
# Customization function types
692
GetGridFSFileFunc = Union[
693
Callable[[Any, str], Any], # Tornado: (fs, filename) -> Future[MotorGridOut]
694
Callable[[Any, str, aiohttp.web.Request], Any] # aiohttp: (bucket, filename, request) -> AsyncIOMotorGridOut
695
]
696
697
GetCacheTimeFunc = Callable[[str, datetime, Optional[str]], int] # (filename, modified, mime_type) -> seconds
698
699
SetExtraHeadersFunc = Union[
700
Callable[[tornado.web.RequestHandler, Any], None], # Tornado
701
Callable[[aiohttp.web.Response, Any], None] # aiohttp
702
]
703
```