0
# Kiln AI
1

2
Kiln AI is a comprehensive platform for building, evaluating, and deploying AI systems. It provides structured dataset management, model fine-tuning capabilities, retrieval-augmented generation (RAG), embedding support, evaluation systems, and a complete toolkit for working with AI tasks and models. Kiln organizes AI work around projects and tasks, with built-in support for versioning, validation, and collaborative development through a file-based architecture.
3

4
## Package Information
5

6
- **Package Name**: kiln-ai
7
- **Package Type**: pypi
8
- **Language**: Python
9
- **Installation**: `pip install kiln-ai`
10

11
## Core Imports
12

13
```python
14
import kiln_ai
15
from kiln_ai.datamodel import Project, Task, TaskRun, TaskOutput
16
from kiln_ai.adapters import adapter_for_task
17
```
18

19
## Basic Usage
20

21
```python
22
from kiln_ai.datamodel import Project, Task, TaskRun, TaskOutput
23
from kiln_ai.adapters import adapter_for_task
24

25
# Load an existing project
26
project = Project.load_from_file("path/to/project.kiln")
27
print(f"Project: {project.name}")
28

29
# Access tasks in the project
30
tasks = project.tasks()
31
for task in tasks:
32
    print(f"Task: {task.name}")
33
    print(f"Dataset size: {len(task.runs())}")
34

35
# Create and run a task
36
task = Task(
37
    name="joke_generator",
38
    instruction="Tell a joke about the given subject."
39
)
40

41
# Create adapter for the task with specific model
42
adapter = adapter_for_task(task, model_name="gpt_4o", provider="openai")
43

44
# Run the task
45
response = await adapter.invoke("AI")
46
print(f"Output: {response.output}")
47
```
48

49
## Architecture
50

51
Kiln uses a file-based architecture where projects are directories containing JSON files with the `.kiln` extension. This design enables:
52

53
- **Git compatibility**: Easy collaboration with unique IDs preventing conflicts
54
- **Direct access**: Standard tools (pandas, polars) can read the JSON files
55
- **Validation**: Python library ensures data integrity
56

57
The core hierarchy:
58

59
- **Project**: Organizes related tasks in a directory
60
  - **Task**: Defines instructions, schemas, and requirements
61
    - **TaskRun**: Individual execution samples with inputs/outputs
62
    - **Prompt**: Saved prompt configurations
63
    - **Finetune**: Fine-tuning job configuration and status
64
    - **DatasetSplit**: Frozen train/test/validation splits
65

66
## Capabilities
67

68
### Core Data Models
69

70
Project, Task, TaskRun, and related data structures for organizing AI work, including requirements, ratings, and data sources.
71

72
```python { .api }
73
class Project:
74
    """
75
    Represents a Kiln project containing related tasks.
76

77
    Properties:
78
    - name (str): Project name
79
    - description (str): Project description
80
    - path (str): File system path to project
81
    """
82

83
    def tasks(self) -> list:
84
        """Get all tasks in the project."""
85

86
    @staticmethod
87
    def load_from_file(path: str) -> 'Project':
88
        """Load project from .kiln file."""
89

90
    def save_to_file(self) -> None:
91
        """Save project to .kiln file."""
92

93
class Task:
94
    """
95
    Represents an AI task with instructions and schemas.
96

97
    Properties:
98
    - name (str): Task name
99
    - description (str): Task description
100
    - instruction (str): Instructions for completing the task
101
    - input_json_schema (str | None): JSON schema for inputs
102
    - output_json_schema (str | None): JSON schema for outputs
103
    - requirements (list): Task requirements that outputs must satisfy
104
    """
105

106
    def runs(self) -> list:
107
        """Get all runs for this task."""
108

109
    @staticmethod
110
    def load_from_file(path: str) -> 'Task':
111
        """Load task from .kiln file."""
112

113
    def save_to_file(self) -> None:
114
        """Save task to .kiln file."""
115

116
class TaskRun:
117
    """
118
    Single execution/sample of a task.
119

120
    Properties:
121
    - input (str): Input data for the run
122
    - output (TaskOutput): Output from the run
123
    - input_source (DataSource | None): Source of input data
124
    - tags (list[str]): Tags for categorization
125
    - prompt_id (str | None): Associated prompt identifier
126
    """
127

128
    @staticmethod
129
    def load_from_file(path: str) -> 'TaskRun':
130
        """Load task run from .kiln file."""
131

132
    def save_to_file(self) -> None:
133
        """Save task run to .kiln file."""
134
```
135

136
[Data Models](./datamodel.md)
137

138
### Task Execution
139

140
Adapters for running AI tasks with various models and providers, including streaming support and configuration options.
141

142
```python { .api }
143
def adapter_for_task(
144
    task,
145
    model_name: str,
146
    provider: str | None = None,
147
    config: dict | None = None
148
):
149
    """
150
    Create an adapter for executing a task with a specific model.
151

152
    Parameters:
153
    - task: Task instance to run
154
    - model_name (str): Name of the model to use
155
    - provider (str | None): Provider name (e.g., "openai", "anthropic")
156
    - config (dict | None): Additional configuration options
157

158
    Returns:
159
    BaseAdapter instance configured for the task
160
    """
161
```
162

163
[Task Execution](./task-execution.md)
164

165
### Model Registry
166

167
Comprehensive registry of supported models and providers with capability metadata.
168

169
```python { .api }
170
def get_model_by_name(model_name: str):
171
    """
172
    Retrieve model definition by name.
173

174
    Parameters:
175
    - model_name (str): Model identifier
176

177
    Returns:
178
    KilnModel instance or None if not found
179
    """
180

181
def built_in_models_from_provider(provider_name: str) -> list:
182
    """
183
    List all built-in models for a provider.
184

185
    Parameters:
186
    - provider_name (str): Provider identifier
187

188
    Returns:
189
    list[KilnModel]: List of model definitions
190
    """
191
```
192

193
[Model Registry](./models.md)
194

195
### Prompt Building
196

197
Multiple prompt builder strategies including few-shot, chain-of-thought, and multi-shot approaches.
198

199
```python { .api }
200
def prompt_builder_from_id(prompt_id: str, task):
201
    """
202
    Get prompt builder instance from identifier.
203

204
    Parameters:
205
    - prompt_id (str): Prompt builder type identifier
206
    - task: Task instance for context
207

208
    Returns:
209
    BasePromptBuilder instance
210
    """
211

212
def chain_of_thought_prompt(task) -> str:
213
    """
214
    Generate chain-of-thought prompt text for a task.
215

216
    Parameters:
217
    - task: Task instance
218

219
    Returns:
220
    str: Generated CoT prompt
221
    """
222
```
223

224
[Prompt Builders](./prompts.md)
225

226
### Evaluation
227

228
Evaluation systems including G-Eval and custom evaluators for assessing task outputs.
229

230
```python { .api }
231
class EvalRunner:
232
    """
233
    Execute evaluations on task runs.
234

235
    Methods:
236
    - run(): Execute single evaluation
237
    - run_batch(): Execute batch evaluations
238
    """
239

240
class Eval:
241
    """
242
    Evaluation configuration.
243

244
    Properties:
245
    - id (str): Unique identifier
246
    - name (str): Evaluation name
247
    - eval_type (str): Type of evaluation
248
    - config (dict): Evaluation configuration
249
    """
250

251
    @staticmethod
252
    def load_from_file(path: str) -> 'Eval':
253
        """Load evaluation from .kiln file."""
254

255
    def save_to_file(self) -> None:
256
        """Save evaluation to .kiln file."""
257
```
258

259
[Evaluation](./evaluation.md)
260

261
### Fine-tuning
262

263
Fine-tuning support for training custom models on task datasets with provider integrations.
264

265
```python { .api }
266
class Finetune:
267
    """
268
    Fine-tuning job configuration and tracking.
269

270
    Properties:
271
    - id (str): Unique identifier
272
    - status (str): Job status
273
    - model_id (str): Base model identifier
274
    - provider (str): Fine-tuning provider
275
    """
276

277
    def start(self) -> None:
278
        """Start the fine-tuning job."""
279

280
    def check_status(self) -> dict:
281
        """Check current status of fine-tuning job."""
282

283
    @staticmethod
284
    def load_from_file(path: str) -> 'Finetune':
285
        """Load fine-tune from .kiln file."""
286

287
    def save_to_file(self) -> None:
288
        """Save fine-tune to .kiln file."""
289
```
290

291
[Fine-tuning](./fine-tuning.md)
292

293
### RAG and Embeddings
294

295
Retrieval-augmented generation with chunking, embeddings, and vector store integration.
296

297
```python { .api }
298
class RagConfig:
299
    """
300
    Configuration for RAG (Retrieval-Augmented Generation).
301

302
    Properties:
303
    - vector_store_config: Vector database configuration
304
    - embedding_config: Embedding model configuration
305
    - chunker_config: Text chunking configuration
306
    - top_k (int): Number of results to retrieve
307
    """
308

309
class EmbeddingConfig:
310
    """
311
    Configuration for embeddings.
312

313
    Properties:
314
    - model_id (str): Embedding model identifier
315
    - provider (str): Embedding provider
316
    - dimensions (int): Embedding vector dimensions
317
    """
318
```
319

320
[RAG and Embeddings](./rag-embeddings.md)
321

322
### Tool System
323

324
Tool system enabling AI agents to use external functions and APIs.
325

326
```python { .api }
327
class KilnTool:
328
    """
329
    Base class for tools that AI agents can use.
330

331
    Properties:
332
    - id (str): Tool identifier
333
    - name (str): Tool name
334
    - description (str): Tool description
335
    - input_schema (dict): JSON schema for tool inputs
336

337
    Methods:
338
    - invoke(): Execute the tool synchronously
339
    - async_invoke(): Execute the tool asynchronously
340
    """
341

342
def tool_from_id(tool_id: str):
343
    """
344
    Get tool instance by identifier.
345

346
    Parameters:
347
    - tool_id (str): Tool identifier
348

349
    Returns:
350
    KilnTool instance
351
    """
352
```
353

354
[Tool System](./tools.md)
355

356
### Configuration and Utilities
357

358
Configuration management, formatting utilities, and async lock management.
359

360
```python { .api }
361
class Config:
362
    """
363
    Configuration management for Kiln.
364

365
    Properties:
366
    - custom_models (list): Custom model identifiers
367
    - openai_compatible_providers (list): OpenAI-compatible provider configs
368
    """
369

370
    @classmethod
371
    def shared(cls) -> 'Config':
372
        """Get singleton configuration instance."""
373

374
    def save(self) -> None:
375
        """Save configuration to disk."""
376

377
    def load(self) -> None:
378
        """Load configuration from disk."""
379
```
380

381
[Configuration](./configuration.md)
382