0
# Xorbits
1
2
Xorbits is an open-source computing framework that enables seamless scaling of data science and machine learning workloads from single machines to distributed clusters. It provides a familiar Python API that supports popular libraries like pandas, NumPy, PyTorch, and XGBoost, allowing users to scale their existing workflows with minimal code changes.
3
4
## Package Information
5
6
- **Package Name**: xorbits
7
- **Package Type**: pypi
8
- **Language**: Python
9
- **Installation**: `pip install xorbits`
10
- **Python Requires**: >=3.9
11
12
## Core Imports
13
14
```python
15
import xorbits
16
```
17
18
Common imports for specific functionality:
19
20
```python
21
import xorbits.pandas as pd
22
import xorbits.numpy as np
23
import xorbits.sklearn as sk
24
```
25
26
## Basic Usage
27
28
```python
29
import xorbits
30
import xorbits.pandas as pd
31
import xorbits.numpy as np
32
33
# Initialize Xorbits runtime
34
xorbits.init()
35
36
# Create distributed DataFrame (same API as pandas)
37
df = pd.DataFrame({
38
'A': np.random.randn(10000),
39
'B': np.random.randn(10000),
40
'C': np.random.randn(10000)
41
})
42
43
# Perform operations (lazy evaluation)
44
result = df.groupby('A').agg({'B': 'mean', 'C': 'sum'})
45
46
# Execute computation
47
computed_result = xorbits.run(result)
48
print(computed_result)
49
50
# Shutdown when done
51
xorbits.shutdown()
52
```
53
54
## Architecture
55
56
Xorbits leverages a distributed computing architecture built on top of Mars:
57
58
- **DataRef System**: All distributed objects are represented as `DataRef` instances that contain references to underlying Mars entities
59
- **Lazy Evaluation**: Operations are recorded as computation graphs and executed when explicitly triggered
60
- **Drop-in Compatibility**: APIs mirror popular libraries (pandas, numpy, sklearn) with minimal code changes required
61
- **Distributed Execution**: Automatically handles data partitioning, task scheduling, and parallel execution across workers
62
- **Memory Management**: Intelligent memory management with spilling to disk when needed
63
64
## Capabilities
65
66
### Runtime Management
67
68
Core functions for initializing, managing, and shutting down Xorbits runtime environments, including local and distributed cluster configurations.
69
70
```python { .api }
71
from typing import Dict, List, Optional, Union
72
from .._mars.utils import no_default
73
74
def init(
75
address: Optional[str] = None,
76
init_local: bool = no_default,
77
session_id: Optional[str] = None,
78
timeout: Optional[float] = None,
79
n_worker: int = 1,
80
n_cpu: Union[int, str] = "auto",
81
mem_bytes: Union[int, str] = "auto",
82
cuda_devices: Union[List[int], List[List[int]], str] = "auto",
83
web: Union[bool, str] = "auto",
84
new: bool = True,
85
storage_config: Optional[Dict] = None,
86
**kwargs
87
) -> None: ...
88
89
def shutdown(**kw) -> None: ...
90
91
def run(obj, **kwargs): ...
92
```
93
94
[Runtime Management](./runtime-management.md)
95
96
### Configuration
97
98
Configuration management through options system, providing control over execution behavior and runtime settings.
99
100
```python { .api }
101
# Configuration objects and functions
102
options: object
103
def option_context(*args, **kwargs): ...
104
```
105
106
[Configuration](./configuration.md)
107
108
### Pandas Integration
109
110
Drop-in replacement for pandas with distributed computing capabilities, supporting DataFrames, Series, and the full pandas API.
111
112
```python { .api }
113
class DataFrame: ...
114
class Series: ...
115
class Index: ...
116
117
# Data types and constants
118
class Timedelta: ...
119
class DateOffset: ...
120
class Interval: ...
121
class Timestamp: ...
122
NaT: object
123
NA: object
124
```
125
126
[Pandas Integration](./pandas-integration.md)
127
128
### NumPy Integration
129
130
Distributed array computing with NumPy-compatible API, supporting all NumPy operations on large distributed arrays.
131
132
```python { .api }
133
class ndarray: ...
134
135
# NumPy constants and types
136
bool_: type
137
int8: type
138
int16: type
139
int32: type
140
int64: type
141
float16: type
142
float32: type
143
float64: type
144
complex64: type
145
complex128: type
146
dtype: type
147
pi: float
148
e: float
149
inf: float
150
nan: float
151
```
152
153
[NumPy Integration](./numpy-integration.md)
154
155
### Machine Learning
156
157
Distributed machine learning capabilities through sklearn, XGBoost, and LightGBM integrations, enabling scalable model training and prediction.
158
159
```python { .api }
160
# Sklearn submodules
161
from xorbits.sklearn import cluster, datasets, decomposition, ensemble
162
from xorbits.sklearn import linear_model, metrics, model_selection, neighbors
163
from xorbits.sklearn import preprocessing, semi_supervised
164
165
# XGBoost and LightGBM classes dynamically exposed
166
```
167
168
[Machine Learning](./machine-learning.md)
169
170
### Datasets
171
172
Large-scale dataset handling with support for Hugging Face datasets and efficient data loading patterns.
173
174
```python { .api }
175
class Dataset: ...
176
def from_huggingface(dataset_name: str, **kwargs): ...
177
```
178
179
[Datasets](./datasets.md)
180
181
### Remote Computing
182
183
Remote function execution capabilities for distributed computing workloads.
184
185
```python { .api }
186
def spawn(func, **kwargs): ...
187
```
188
189
[Remote Computing](./remote-computing.md)
190
191
## Types
192
193
### Core Data Types
194
195
```python { .api }
196
class Data:
197
"""Base data container class."""
198
199
class DataRef:
200
"""Reference to distributed data object."""
201
202
class DataRefMeta:
203
"""Metaclass for DataRef."""
204
205
from enum import Enum
206
207
class DataType(Enum):
208
"""Enumeration of data types."""
209
object_ = 1
210
scalar = 2
211
tensor = 3
212
dataframe = 4
213
series = 5
214
index = 6
215
categorical = 7
216
dataframe_groupby = 8
217
series_groupby = 9
218
dataset = 10
219
```