0
# Quilt3
1

2
Quilt manages data like code with packages, repositories, browsing and revision history for machine learning, biotech, and other data-driven domains. It provides comprehensive data package management with versioning, metadata management, collaborative workflows, and integrates with cloud storage services like AWS S3.
3

4
## Package Information
5

6
- **Package Name**: quilt3
7
- **Language**: Python
8
- **Installation**: `pip install quilt3`
9
- **Version**: 7.0.0
10

11
## Core Imports
12

13
```python
14
import quilt3
15
```
16

17
For specific functionality:
18

19
```python
20
from quilt3 import Package, Bucket
21
import quilt3.admin
22
```
23

24
## Type Imports
25

26
```python { .api }
27
from typing import Union, Optional, Callable, Any
28

29
# Hook type aliases
30
BuildClientHook = Callable[..., Any]
31
```
32

33
## Basic Usage
34

35
```python
36
import quilt3
37

38
# Configure your Quilt catalog
39
quilt3.config('https://your-catalog-url.com')
40

41
# Create and build a data package
42
pkg = quilt3.Package()
43
pkg.set_dir("data/", "path/to/local/directory/")
44
pkg.set_meta({"description": "My dataset"})
45
pkg.build("my-username/my-package")
46

47
# Browse and install existing packages
48
pkg = quilt3.Package.browse("my-username/my-package")
49
pkg.install("my-username/my-package", dest="./downloaded-data/")
50

51
# Work with S3 buckets
52
bucket = quilt3.Bucket("s3://my-bucket")
53
bucket.put_file("data.csv", "local/path/data.csv")
54
data = bucket.select("data.csv", "SELECT * FROM S3Object LIMIT 10")
55
```
56

57
## Architecture
58

59
Quilt3 is built around several key concepts:
60

61
- **Package**: In-memory representation of a data package with files, metadata, and versioning
62
- **PackageEntry**: Individual file within a package with metadata, hashing, and data access
63
- **Bucket**: Interface for S3 bucket operations and data management
64
- **Registry**: Storage backend for package manifests and versioning (local or remote)
65
- **Session Management**: Authentication and credentials for Quilt catalogs and AWS services
66

67
## Capabilities
68

69
### Data Package Management
70

71
Core functionality for creating, building, installing, and managing data packages. Includes package versioning, metadata handling, and collaborative workflows.
72

73
```python { .api }
74
class Package:
75
    def __init__(self): ...
76
    def build(self, name: str, registry: str = None, message: str = None) -> str: ...
77
    def install(cls, name: str, registry: str = None, top_hash: str = None, dest: str = None): ...
78
    def browse(cls, name: str, registry: str = None, top_hash: str = None): ...
79
    def set_dir(self, lkey: str, path: str = None, meta: dict = None): ...
80
    def set_meta(self, meta: dict): ...
81
```
82

83
[Data Package Management](./package-management.md)
84

85
### Package Data Access
86

87
Methods for accessing, deserializing, and working with data files within packages. Supports various data formats and provides caching and optimization features.
88

89
```python { .api }
90
class PackageEntry:
91
    def get(self) -> str: ...
92
    def get_bytes(self, use_cache_if_available: bool = True) -> bytes: ...
93
    def get_as_json(self, use_cache_if_available: bool = True) -> dict: ...
94
    def deserialize(self, func=None, **format_opts): ...
95
    def fetch(self, dest: str = None): ...
96
```
97

98
[Package Data Access](./data-access.md)
99

100
### S3 Bucket Operations
101

102
Direct S3 bucket interface for file operations, listing, searching, and SQL queries. Provides high-level abstractions over AWS S3 operations.
103

104
```python { .api }
105
class Bucket:
106
    def __init__(self, bucket_uri: str): ...
107
    def put_file(self, key: str, path: str): ...
108
    def put_dir(self, key: str, directory: str): ...
109
    def fetch(self, key: str, path: str): ...
110
    def ls(self, path: str = None, recursive: bool = False): ...
111
    def select(self, key: str, query: str, raw: bool = False): ...
112
```
113

114
[S3 Bucket Operations](./bucket-operations.md)
115

116
### Configuration and Session Management
117

118
Authentication, configuration, and session management for Quilt catalogs and AWS services. Handles login/logout, credentials, and configuration settings.
119

120
```python { .api }
121
def config(*catalog_url, **config_values): ...
122
def login(): ...
123
def logout(): ...
124
def logged_in() -> str: ...
125
def get_boto3_session(fallback: bool = True): ...
126
```
127

128
[Configuration and Session Management](./config-session.md)
129

130
### Administrative Functions
131

132
Administrative capabilities for managing users, roles, SSO configuration, and other Quilt stack administrative tasks.
133

134
```python { .api }
135
import quilt3.admin
136
# Sub-modules: users, roles, sso_config, tabulator
137
# Types: User, ManagedRole, UnmanagedRole, SSOConfig
138
```
139

140
[Administrative Functions](./admin.md)
141

142
### Package Registry Operations
143

144
Functions for working with package registries, including listing packages, searching, copying data, and package deletion.
145

146
```python { .api }
147
def list_packages(registry: str = None) -> list: ...
148
def list_package_versions(name: str, registry: str = None) -> list: ...
149
def search(query: Union[str, dict], limit: int = 10) -> list: ...
150
def copy(src: str, dest: str): ...
151
def delete_package(name: str, registry: str = None, top_hash: str = None): ...
152
```
153

154
[Package Registry Operations](./registry-operations.md)
155

156
### Hooks and Extension System
157

158
Extension system for customizing Quilt3 behavior through configurable hook functions.
159

160
```python { .api }
161
import quilt3.hooks
162

163
def get_build_s3_client_hook() -> Optional[BuildClientHook]: ...
164
def set_build_s3_client_hook(hook: Optional[BuildClientHook]) -> Optional[BuildClientHook]: ...
165
```
166

167
[Hooks and Extension System](./hooks.md)
168

169
## Exception Types
170

171
```python { .api }
172
class PackageException(Exception):
173
    """Exception relating to package validity."""
174
    pass
175

176
class QuiltException(Exception):
177
    """General Quilt operation errors."""
178
    pass
179

180
class QuiltConflictException(QuiltException):
181
    """Conflicts in package operations."""
182
    pass
183
```