0
# Storage and Utilities
1

2
Storage abstractions, utility modules, and debugging tools for Lightning Apps. Provides file system operations, cloud storage integrations, and enhanced debugging capabilities.
3

4
## Capabilities
5

6
### Storage Module
7

8
File system abstractions and cloud storage integrations for Lightning Apps, enabling seamless data management across local and cloud environments.
9

10
```python { .api }
11
class Drive:
12
    """Cloud storage drive abstraction for managing cloud storage resources."""
13
    def __init__(
14
        self,
15
        id: str,
16
        allow_duplicates: bool = False,
17
        component_name: Optional[str] = None,
18
        root_folder: Optional[str] = None,
19
    ):
20
        """
21
        Initialize a Drive for shared file storage.
22
        
23
        Parameters:
24
        - id: Unique identifier for this Drive
25
        - allow_duplicates: Whether to enable files duplication between components
26
        - component_name: Component name which owns this drive (auto-inferred if None)
27
        - root_folder: Mount directory for the drive
28
        """
29
    
30
    def put(self, source: str, destination: str = "") -> str:
31
        """Put a file or directory into the drive."""
32
    
33
    def get(self, source: str, destination: str = "", overwrite: bool = True):
34
        """Get a file or directory from the drive."""
35
    
36
    def list(self, source: str = "") -> List[str]:
37
        """List files and directories in the drive."""
38
    
39
    def delete(self, source: str):
40
        """Delete a file or directory from the drive."""
41

42
class FileSystem:
43
    """File system operations interface for local and remote file systems."""
44
    def __init__(self):
45
        """Initialize filesystem operations."""
46
    
47
    def get(
48
        self,
49
        source: str,
50
        destination: str,
51
        overwrite: bool = True,
52
        on_progress: Optional[Callable] = None
53
    ):
54
        """Copy files from source to destination."""
55
    
56
    def put(
57
        self,
58
        source: str,
59
        destination: str,
60
        overwrite: bool = True,
61
        on_progress: Optional[Callable] = None
62
    ):
63
        """Copy files from source to destination."""
64

65
class Mount:
66
    """Mount point configuration for storage resources in Lightning Apps."""
67
    def __init__(
68
        self,
69
        source: str,
70
        mount_path: str,
71
        read_only: bool = False
72
    ):
73
        """
74
        Initialize a mount configuration.
75
        
76
        Parameters:
77
        - source: Source path to mount
78
        - mount_path: Path where source will be mounted
79
        - read_only: Whether mount is read-only
80
        """
81

82
class StorageOrchestrator:
83
    """Orchestrates storage operations across multiple storage backends."""
84
    def __init__(
85
        self,
86
        app: "LightningApp",
87
        request_queues: Dict[str, BaseQueue],
88
        response_queues: Dict[str, BaseQueue],
89
        copy_request_queue: BaseQueue,
90
        copy_response_queue: BaseQueue,
91
    ):
92
        """Initialize storage orchestrator for managing storage operations."""
93

94
class Path:
95
    """Enhanced path handling utilities optimized for Lightning Apps."""
96
    def __init__(self, *args):
97
        """Initialize a Path object with enhanced Lightning Apps functionality."""
98

99
class Payload:
100
    """Data payload abstraction for efficient storage and transfer operations."""
101
    def __init__(self, value: Any):
102
        """
103
        Initialize a payload for data transfer.
104
        
105
        Parameters:
106
        - value: The data value to wrap as a payload
107
        """
108
```
109

110
The storage module provides utilities for:
111
- Local file system operations via `FileSystem`
112
- Cloud storage integrations (S3, GCS, Azure) via `Drive`
113
- Path management and abstractions via `Path`
114
- Data transfer and synchronization via `StorageOrchestrator`
115
- Mount point management via `Mount`
116
- Payload handling for data transfer via `Payload`
117

118
### Debugging Utilities
119

120
Enhanced debugging tools specifically designed for Lightning Apps development and production debugging.
121

122
```python { .api }
123
# Enhanced debugger for Lightning Apps
124
def set_trace():
125
    """
126
    Set a breakpoint for debugging Lightning Apps.
127
    
128
    Enhanced version of pdb.set_trace() that works correctly
129
    within Lightning Apps execution context, including 
130
    forked processes and distributed environments.
131
    """
132
```
133

134
## Usage Examples
135

136
### Storage Operations
137

138
```python
139
import lightning as L
140

141
class DataManager(L.LightningWork):
142
    def __init__(self):
143
        super().__init__()
144
        
145
    def run(self, data_path):
146
        # Storage operations within Lightning Apps
147
        # The exact API depends on the storage backend
148
        print(f"Managing data at: {data_path}")
149
        
150
        # Example storage operations would be implemented here
151
        # based on the specific storage module functionality
152
```
153

154
### Debugging Lightning Apps
155

156
```python
157
import lightning as L
158
from lightning import pdb
159

160
class DebuggableWork(L.LightningWork):
161
    def __init__(self):
162
        super().__init__()
163
        
164
    def run(self, data):
165
        print("Starting work execution...")
166
        
167
        # Set breakpoint for debugging
168
        pdb.set_trace()  # Enhanced debugger for Lightning Apps
169
        
170
        # Process data
171
        result = self.process_data(data)
172
        return result
173
        
174
    def process_data(self, data):
175
        # Complex processing logic that might need debugging
176
        processed = []
177
        for item in data:
178
            # Another potential breakpoint location
179
            if len(processed) > 10:
180
                pdb.set_trace()
181
            processed.append(item * 2)
182
        return processed
183

184
class DebuggableFlow(L.LightningFlow):
185
    def __init__(self):
186
        super().__init__()
187
        self.worker = DebuggableWork()
188
        
189
    def run(self):
190
        test_data = list(range(20))
191
        result = self.worker.run(test_data)
192
        print(f"Work completed with result: {result}")
193

194
# The enhanced debugger works correctly in Lightning Apps context
195
app = L.LightningApp(DebuggableFlow())
196
app.run()
197
```
198

199
## Integration Notes
200

201
These utilities are automatically available when importing the main Lightning package:
202

203
```python
204
import lightning as L
205

206
# Storage utilities available through L.storage
207
# Enhanced debugging available through L.pdb.set_trace()
208
```
209

210
The storage and utility modules are designed to work seamlessly within the Lightning Apps ecosystem, providing consistent APIs whether running locally or in cloud environments.