CtrlK
BlogDocsLog inGet started
Tessl Logo

tessl/pypi-ibis-framework

The portable Python dataframe library that provides a unified API for data analysis across 20+ different backends

Pending
Overview
Eval results
Files

backends.mddocs/

Backend Management

Backend connection, configuration, and management functions for working with different data processing engines and databases.

Capabilities

Backend Discovery

Get information about available and current backends.

def get_backend(table_or_expr):
    """
    Get the backend for a table or expression.
    
    Parameters:
    - table_or_expr: Table or expression to get backend for
    
    Returns:
    Backend instance
    """

def set_backend(backend):
    """
    Set the default backend for new operations.
    
    Parameters:
    - backend: Backend instance or backend name string
    
    Returns:
    None
    """

def list_backends():
    """
    List all available backend names.
    
    Returns:
    List of backend name strings
    """

Usage Examples:

import ibis

# Get backend for a table
table = ibis.memtable({'x': [1, 2, 3]})
backend = ibis.get_backend(table)
print(backend.name)  # 'duckdb'

# Set default backend
ibis.set_backend('duckdb')

# List available backends
backends = ibis.list_backends()
print(backends)  # ['duckdb', 'postgres', 'sqlite', ...]

Database Backends

Connect to traditional SQL databases.

PostgreSQL

ibis.postgres.connect(
    user=None,
    password=None, 
    host='localhost',
    port=5432,
    database=None,
    **kwargs
):
    """
    Connect to PostgreSQL database.
    
    Parameters:
    - user: str, username
    - password: str, password
    - host: str, hostname
    - port: int, port number
    - database: str, database name
    - **kwargs: additional connection parameters
    
    Returns:
    PostgreSQL backend connection
    """

MySQL

ibis.mysql.connect(
    user=None,
    password=None,
    host='localhost', 
    port=3306,
    database=None,
    **kwargs
):
    """Connect to MySQL database."""

SQLite

ibis.sqlite.connect(path=':memory:', **kwargs):
    """
    Connect to SQLite database.
    
    Parameters:
    - path: str, path to database file or ':memory:' for in-memory
    - **kwargs: additional connection parameters
    
    Returns:
    SQLite backend connection
    """

Cloud Data Warehouses

Connect to cloud-based data processing systems.

BigQuery

ibis.bigquery.connect(
    project_id=None,
    dataset_id=None,
    credentials=None,
    **kwargs
):
    """
    Connect to Google BigQuery.
    
    Parameters:
    - project_id: str, Google Cloud project ID
    - dataset_id: str, default dataset ID
    - credentials: authentication credentials
    - **kwargs: additional connection parameters
    
    Returns:
    BigQuery backend connection
    """

Snowflake

ibis.snowflake.connect(
    user=None,
    password=None,
    account=None,
    database=None,
    warehouse=None,
    **kwargs
):
    """
    Connect to Snowflake data warehouse.
    
    Parameters:
    - user: str, username
    - password: str, password  
    - account: str, Snowflake account identifier
    - database: str, database name
    - warehouse: str, warehouse name
    - **kwargs: additional connection parameters
    
    Returns:
    Snowflake backend connection
    """

Analytical Engines

Connect to specialized analytical and OLAP systems.

DuckDB

ibis.duckdb.connect(path=':memory:', **kwargs):
    """
    Connect to DuckDB database.
    
    Parameters:
    - path: str, path to database file or ':memory:' for in-memory
    - **kwargs: additional connection parameters
    
    Returns:
    DuckDB backend connection
    """

ClickHouse

ibis.clickhouse.connect(
    host='localhost',
    port=9000,
    user='default',
    password='',
    database='default',
    **kwargs
):
    """Connect to ClickHouse database."""

Distributed Systems

Connect to distributed computing frameworks.

Apache Spark

ibis.spark.connect(session=None, **kwargs):
    """
    Connect to Apache Spark.
    
    Parameters:
    - session: existing SparkSession instance
    - **kwargs: SparkSession configuration parameters
    
    Returns:
    Spark backend connection
    """

Trino

ibis.trino.connect(
    host='localhost',
    port=8080,
    user=None,
    catalog=None,
    schema=None,
    **kwargs
):
    """Connect to Trino distributed query engine."""

In-Memory Engines

Connect to in-memory data processing systems.

Polars

ibis.polars.connect(**kwargs):
    """
    Connect to Polars backend.
    
    Returns:
    Polars backend connection
    """

Pandas

ibis.pandas.connect(**kwargs):
    """
    Connect to Pandas backend.
    
    Returns:
    Pandas backend connection
    """

Dask

ibis.dask.connect(client=None, **kwargs):
    """
    Connect to Dask backend.
    
    Parameters:
    - client: Dask client instance
    - **kwargs: additional parameters
    
    Returns:
    Dask backend connection
    """

Backend Operations

Common operations available on all backend connections.

# Backend connection methods
backend.table(name): Table
    """Get existing table by name."""

backend.create_table(name, data): Table
    """Create table from data."""

backend.list_tables(): List[str]  
    """List available table names."""

backend.compile(expr): str
    """Compile expression to backend-specific query."""

backend.execute(expr): Any
    """Execute expression and return results."""

backend.has_operation(operation): bool
    """Check if backend supports operation."""

Usage Examples:

# Connect to different backends
duckdb_con = ibis.duckdb.connect('my_db.duckdb')
pg_con = ibis.postgres.connect(
    user='user', 
    password='pass',
    database='mydb'
)

# Common operations
tables = duckdb_con.list_tables()
my_table = duckdb_con.table('existing_table')

# Create table
import pandas as pd
df = pd.DataFrame({'x': [1, 2, 3], 'y': ['a', 'b', 'c']})
new_table = duckdb_con.create_table('new_data', df)

# Compile and execute
expr = my_table.filter(my_table.x > 1)
sql = duckdb_con.compile(expr)
result = duckdb_con.execute(expr)

Cross-Backend Operations

Move data and expressions between backends.

Usage Examples:

# Create expression on one backend
duckdb_con = ibis.duckdb.connect()
pg_con = ibis.postgres.connect(...)

duckdb_table = duckdb_con.table('source_data')
expr = duckdb_table.filter(duckdb_table.value > 100)

# Execute same expression on different backend
# (assuming compatible table exists)
pg_table = pg_con.table('source_data')
same_expr = pg_table.filter(pg_table.value > 100)
pg_result = pg_con.execute(same_expr)

# Transfer data between backends
data = duckdb_con.execute(expr).to_pandas()
pg_table = pg_con.create_table('transferred_data', data)

Backend-Specific Features

Access backend-specific functionality.

# Backend-specific methods (examples)
bigquery_backend.create_dataset(name): None
spark_backend.spark_session: SparkSession
postgres_backend.raw_sql(query): ResultProxy

Usage Examples:

# BigQuery specific operations
bq_con = ibis.bigquery.connect(project_id='my-project')
bq_con.create_dataset('analytics')

# Spark specific operations  
spark_con = ibis.spark.connect()
spark_session = spark_con.spark_session

# Raw SQL execution
pg_con = ibis.postgres.connect(...)
result = pg_con.raw_sql('SELECT version()')

URL-Based Connections

Connect using database URLs.

backend._from_url(url):
    """
    Connect using database URL.
    
    Parameters:
    - url: str, database connection URL
    
    Returns:
    Backend connection
    """

Usage Examples:

# URL-based connections
pg_con = ibis.postgres._from_url('postgresql://user:pass@localhost/mydb')
sqlite_con = ibis.sqlite._from_url('sqlite:///path/to/database.db')

Install with Tessl CLI

npx tessl i tessl/pypi-ibis-framework

docs

aggregation-windows.md

backends.md

configuration.md

expressions.md

index.md

selectors.md

sql-integration.md

table-construction.md

table-operations.md

temporal.md

udfs.md

tile.json