tessl/pypi-kiln-ai

Kiln AI is a comprehensive platform for building, evaluating, and deploying AI systems with dataset management, model fine-tuning, RAG, and evaluation capabilities.

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Securityby

Pending

The risk profile of this skill

Overview

Eval results

Files

Kiln AI

Name: tessl/pypi-kiln-ai
Author: tessl

Kiln AI is a comprehensive platform for building, evaluating, and deploying AI systems. It provides structured dataset management, model fine-tuning capabilities, retrieval-augmented generation (RAG), embedding support, evaluation systems, and a complete toolkit for working with AI tasks and models. Kiln organizes AI work around projects and tasks, with built-in support for versioning, validation, and collaborative development through a file-based architecture.

Package Information

Package Name: kiln-ai
Package Type: pypi
Language: Python
Installation: pip install kiln-ai

Core Imports

import kiln_ai
from kiln_ai.datamodel import Project, Task, TaskRun, TaskOutput
from kiln_ai.adapters import adapter_for_task

Basic Usage

from kiln_ai.datamodel import Project, Task, TaskRun, TaskOutput
from kiln_ai.adapters import adapter_for_task

# Load an existing project
project = Project.load_from_file("path/to/project.kiln")
print(f"Project: {project.name}")

# Access tasks in the project
tasks = project.tasks()
for task in tasks:
    print(f"Task: {task.name}")
    print(f"Dataset size: {len(task.runs())}")

# Create and run a task
task = Task(
    name="joke_generator",
    instruction="Tell a joke about the given subject."
)

# Create adapter for the task with specific model
adapter = adapter_for_task(task, model_name="gpt_4o", provider="openai")

# Run the task
response = await adapter.invoke("AI")
print(f"Output: {response.output}")

Architecture

Kiln uses a file-based architecture where projects are directories containing JSON files with the .kiln extension. This design enables:

Git compatibility: Easy collaboration with unique IDs preventing conflicts
Direct access: Standard tools (pandas, polars) can read the JSON files
Validation: Python library ensures data integrity

The core hierarchy:

Project: Organizes related tasks in a directory
- Task: Defines instructions, schemas, and requirements
  - TaskRun: Individual execution samples with inputs/outputs
  - Prompt: Saved prompt configurations
  - Finetune: Fine-tuning job configuration and status
  - DatasetSplit: Frozen train/test/validation splits

Capabilities

Core Data Models

Project, Task, TaskRun, and related data structures for organizing AI work, including requirements, ratings, and data sources.

class Project:
    """
    Represents a Kiln project containing related tasks.

    Properties:
    - name (str): Project name
    - description (str): Project description
    - path (str): File system path to project
    """

    def tasks(self) -> list:
        """Get all tasks in the project."""

    @staticmethod
    def load_from_file(path: str) -> 'Project':
        """Load project from .kiln file."""

    def save_to_file(self) -> None:
        """Save project to .kiln file."""

class Task:
    """
    Represents an AI task with instructions and schemas.

    Properties:
    - name (str): Task name
    - description (str): Task description
    - instruction (str): Instructions for completing the task
    - input_json_schema (str | None): JSON schema for inputs
    - output_json_schema (str | None): JSON schema for outputs
    - requirements (list): Task requirements that outputs must satisfy
    """

    def runs(self) -> list:
        """Get all runs for this task."""

    @staticmethod
    def load_from_file(path: str) -> 'Task':
        """Load task from .kiln file."""

    def save_to_file(self) -> None:
        """Save task to .kiln file."""

class TaskRun:
    """
    Single execution/sample of a task.

    Properties:
    - input (str): Input data for the run
    - output (TaskOutput): Output from the run
    - input_source (DataSource | None): Source of input data
    - tags (list[str]): Tags for categorization
    - prompt_id (str | None): Associated prompt identifier
    """

    @staticmethod
    def load_from_file(path: str) -> 'TaskRun':
        """Load task run from .kiln file."""

    def save_to_file(self) -> None:
        """Save task run to .kiln file."""

Data Models

Task Execution

Adapters for running AI tasks with various models and providers, including streaming support and configuration options.

def adapter_for_task(
    task,
    model_name: str,
    provider: str | None = None,
    config: dict | None = None
):
    """
    Create an adapter for executing a task with a specific model.

    Parameters:
    - task: Task instance to run
    - model_name (str): Name of the model to use
    - provider (str | None): Provider name (e.g., "openai", "anthropic")
    - config (dict | None): Additional configuration options

    Returns:
    BaseAdapter instance configured for the task
    """

Task Execution

Model Registry

Comprehensive registry of supported models and providers with capability metadata.

def get_model_by_name(model_name: str):
    """
    Retrieve model definition by name.

    Parameters:
    - model_name (str): Model identifier

    Returns:
    KilnModel instance or None if not found
    """

def built_in_models_from_provider(provider_name: str) -> list:
    """
    List all built-in models for a provider.

    Parameters:
    - provider_name (str): Provider identifier

    Returns:
    list[KilnModel]: List of model definitions
    """

Model Registry

Prompt Building

Multiple prompt builder strategies including few-shot, chain-of-thought, and multi-shot approaches.

def prompt_builder_from_id(prompt_id: str, task):
    """
    Get prompt builder instance from identifier.

    Parameters:
    - prompt_id (str): Prompt builder type identifier
    - task: Task instance for context

    Returns:
    BasePromptBuilder instance
    """

def chain_of_thought_prompt(task) -> str:
    """
    Generate chain-of-thought prompt text for a task.

    Parameters:
    - task: Task instance

    Returns:
    str: Generated CoT prompt
    """

Prompt Builders

Evaluation

Evaluation systems including G-Eval and custom evaluators for assessing task outputs.

class EvalRunner:
    """
    Execute evaluations on task runs.

    Methods:
    - run(): Execute single evaluation
    - run_batch(): Execute batch evaluations
    """

class Eval:
    """
    Evaluation configuration.

    Properties:
    - id (str): Unique identifier
    - name (str): Evaluation name
    - eval_type (str): Type of evaluation
    - config (dict): Evaluation configuration
    """

    @staticmethod
    def load_from_file(path: str) -> 'Eval':
        """Load evaluation from .kiln file."""

    def save_to_file(self) -> None:
        """Save evaluation to .kiln file."""

Evaluation

Fine-tuning

Fine-tuning support for training custom models on task datasets with provider integrations.

class Finetune:
    """
    Fine-tuning job configuration and tracking.

    Properties:
    - id (str): Unique identifier
    - status (str): Job status
    - model_id (str): Base model identifier
    - provider (str): Fine-tuning provider
    """

    def start(self) -> None:
        """Start the fine-tuning job."""

    def check_status(self) -> dict:
        """Check current status of fine-tuning job."""

    @staticmethod
    def load_from_file(path: str) -> 'Finetune':
        """Load fine-tune from .kiln file."""

    def save_to_file(self) -> None:
        """Save fine-tune to .kiln file."""

Fine-tuning

RAG and Embeddings

Retrieval-augmented generation with chunking, embeddings, and vector store integration.

class RagConfig:
    """
    Configuration for RAG (Retrieval-Augmented Generation).

    Properties:
    - vector_store_config: Vector database configuration
    - embedding_config: Embedding model configuration
    - chunker_config: Text chunking configuration
    - top_k (int): Number of results to retrieve
    """

class EmbeddingConfig:
    """
    Configuration for embeddings.

    Properties:
    - model_id (str): Embedding model identifier
    - provider (str): Embedding provider
    - dimensions (int): Embedding vector dimensions
    """

RAG and Embeddings

Tool System

Tool system enabling AI agents to use external functions and APIs.

class KilnTool:
    """
    Base class for tools that AI agents can use.

    Properties:
    - id (str): Tool identifier
    - name (str): Tool name
    - description (str): Tool description
    - input_schema (dict): JSON schema for tool inputs

    Methods:
    - invoke(): Execute the tool synchronously
    - async_invoke(): Execute the tool asynchronously
    """

def tool_from_id(tool_id: str):
    """
    Get tool instance by identifier.

    Parameters:
    - tool_id (str): Tool identifier

    Returns:
    KilnTool instance
    """

Tool System

Configuration and Utilities

Configuration management, formatting utilities, and async lock management.

class Config:
    """
    Configuration management for Kiln.

    Properties:
    - custom_models (list): Custom model identifiers
    - openai_compatible_providers (list): OpenAI-compatible provider configs
    """

    @classmethod
    def shared(cls) -> 'Config':
        """Get singleton configuration instance."""

    def save(self) -> None:
        """Save configuration to disk."""

    def load(self) -> None:
        """Load configuration from disk."""

Configuration

Install with Tessl CLI