Build a new AI agent with Olakai monitoring from scratch — project setup, SDK integration, KPI configuration, and end-to-end validation
66
61%
Does it follow best practices?
Impact
Pending
No eval scenarios have been run
Passed
No known issues
Optimize this skill with Tessl
npx tessl skill review --optimize ./content/olakai/skills/new-project/SKILL.mdThis skill guides you through creating a new AI agent that is fully integrated with Olakai for analytics, KPI tracking, and governance.
Before starting, ensure:
npm install -g olakai-cliolakai loginOlakai's core value is tracking business-specific KPIs for your AI agents. Without KPIs, you're tracking events without gaining actionable insights.
What you can measure with KPIs:
Without KPIs configured:
Every agent should have 2-4 KPIs that answer: "How do I know this agent is performing well?"
KPIs created here belong to this specific agent only. If you later create additional agents, each one needs its own KPI definitions — KPIs cannot be shared or reused across agents.
Before diving into implementation, understand how data flows through Olakai:
SDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula → kpiDataSuccessRate * 100)| Rule | Consequence |
|---|---|
| Only CustomDataConfig fields become variables | Unregistered customData fields are NOT usable in KPIs |
| Formula evaluation is case-insensitive | stepCount, STEPCOUNT, StepCount all work in formulas |
| NUMBER configs need numeric values | Don't send "5" (string), send 5 (number) |
| KPIs are unique per agent | Each KPI belongs to exactly one agent — create separately for each |
| Variable | Type | Description |
|---|---|---|
Prompt | string | The prompt text sent to the LLM |
Response | string | The LLM response text |
Documents count | number | Number of attached documents |
PII detected | boolean | Whether PII was detected |
PHI detected | boolean | Whether PHI was detected |
CODE detected | boolean | Whether code was detected |
SECRET detected | boolean | Whether secrets were detected |
Agentic AI (Multi-step autonomous workflows):
Assistive AI (Interactive chatbots/copilots):
Design your KPIs BEFORE writing any SDK code. This ensures only meaningful data is sent and tracked.
What do stakeholders need to know about this agent?
| Business Question | Field Name | Type | KPI Formula | Aggregation |
|---|---|---|---|---|
| Throughput | ItemsProcessed | NUMBER | ItemsProcessed | SUM |
| Reliability | SuccessRate | NUMBER | SuccessRate * 100 | AVERAGE |
| Error count | SuccessRate | NUMBER | IF(SuccessRate < 1, 1, 0) | SUM |
| Correlation | ExecutionId | STRING | (for filtering only) | - |
// ONLY include fields you'll register as CustomDataConfigs
customData: {
// Business KPIs
ItemsProcessed: number, // Count of items handled
SuccessRate: number, // 0-1 success ratio
// Performance KPIs
StepCount: number, // Number of workflow steps
// Identification (for filtering, not KPIs)
ExecutionId: string, // Correlation ID
}IMPORTANT: Only include fields you will register as CustomDataConfigs. Unregistered fields are stored but cannot be used in KPIs.
The Olakai platform automatically tracks these — do NOT duplicate them:
| Already Tracked | Where | Don't Send As customData |
|---|---|---|
| Session ID | Main payload | sessionId |
| Agent ID | API key association | agentId |
| User email | userEmail parameter | email, userEmail |
| Timestamp | Event metadata | timestamp, createdAt |
| Request time | requestTime parameter | duration, latency |
| Token count | tokens parameter | tokenCount |
| Model | Auto-detected | model, modelName |
| Provider | Client config | provider |
customData is ONLY for:
Every agent MUST belong to a workflow, even if it's the only agent in that workflow.
olakai workflows create --name "Your Workflow Name" --json
# Output: { "id": "wfl_xxx...", "name": "Your Workflow Name" }olakai agents create \
--name "Your Agent Name" \
--description "What this agent does" \
--workflow WORKFLOW_ID \
--with-api-key \
--json
# Returns agent details including apiKey:
# {
# "id": "cmkbteqn501kyjy4yu6p6xrrx",
# "name": "Your Agent Name",
# "workflowId": "wfl_xxx...",
# "apiKey": "sk_agent_xxxxx..." <-- Use this in your SDK
# }Agent-Workflow Hierarchy:
Workflow: "Customer Support Pipeline"
├── Agent: "Ticket Classifier"
├── Agent: "Response Generator"
└── Agent: "Quality Checker"
Workflow: "Document Processing"
└── Agent: "Document Summarizer" ← single-agent workflows are validThis step MUST be completed before Step 3 (SDK Integration). Only fields registered here can be used in KPI formulas.
ONLY create configs for data you'll use in KPIs or for filtering. Don't create configs for data already tracked automatically.
# For numeric fields (can be used in KPI calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ItemsProcessed" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "SuccessRate" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "StepCount" --type NUMBER
# For string fields (for filtering/grouping, not calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ExecutionId" --type STRING
# Verify all configs are created
olakai custom-data list --agent-id YOUR_AGENT_IDInstead of writing formulas from scratch, use predefined classifier templates:
# List available templates
olakai kpis templates
# Create a classifier KPI from a template
olakai kpis create --name "User Satisfaction" \
--calculator-id classifier --template-id sentiment_scorer \
--scope CHAT --agent-id $AGENT_ID
# Create a time-saved estimator
olakai kpis create --name "Time Saved" \
--calculator-id classifier --template-id time_saved_estimator \
--scope CHAT --agent-id $AGENT_ID# Variable passthrough
olakai kpis create \
--name "Items Processed" \
--agent-id YOUR_AGENT_ID \
--calculator-id formula \
--formula "ItemsProcessed" \
--unit "items" \
--aggregation SUM
# Percentage calculation
olakai kpis create \
--name "Success Rate" \
--agent-id YOUR_AGENT_ID \
--calculator-id formula \
--formula "SuccessRate * 100" \
--unit "%" \
--aggregation AVERAGE
# Conditional counting
olakai kpis create \
--name "Error Count" \
--agent-id YOUR_AGENT_ID \
--calculator-id formula \
--formula "IF(SuccessRate < 1, 1, 0)" \
--unit "errors" \
--aggregation SUM
# Validate formulas before creating
olakai kpis validate --formula "ItemsProcessed" --agent-id YOUR_AGENT_IDInstall dependencies:
npm install @olakai/sdk openaiInitialize and track:
import { olakaiConfig, olakai } from "@olakai/sdk";
import OpenAI from "openai";
// Initialize Olakai
olakaiConfig({
apiKey: process.env.OLAKAI_API_KEY!,
debug: process.env.NODE_ENV === "development",
});
// Create LLM client
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
// Use wrapped client — monitoring happens automatically
const response = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: userPrompt }],
});Agentic workflow with manual event tracking:
taskExecutionId— Cross-Agent Task Correlation. Generate ONEtaskExecutionIdper task and share it across all agents in a multi-agent workflow. This links events from different agents into a single logical task for analytics.
async function runAgent(input: string): Promise<string> {
const startTime = Date.now();
const executionId = crypto.randomUUID();
const taskExecutionId = crypto.randomUUID();
let totalTokens = 0;
let stepCount = 0;
let itemsProcessed = 0;
try {
// Step 1: Planning
stepCount++;
const plan = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: `Plan: ${input}` }],
});
totalTokens += plan.usage?.total_tokens ?? 0;
// Step 2: Process items
const items = parseItems(plan.choices[0].message.content);
for (const item of items) {
stepCount++;
const result = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: `Process: ${item}` }],
});
totalTokens += result.usage?.total_tokens ?? 0;
itemsProcessed++;
}
// Step 3: Summarize
stepCount++;
const summary = await openai.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Summarize results" }],
});
totalTokens += summary.usage?.total_tokens ?? 0;
const finalResponse = summary.choices[0].message.content ?? "";
// Track the complete workflow as a single event
// Only send fields that have CustomDataConfigs (from Step 2.3)
olakai("event", "ai_activity", {
prompt: input,
response: finalResponse,
tokens: totalTokens,
requestTime: Date.now() - startTime,
taskExecutionId,
task: "Data Processing & Analysis",
customData: {
ExecutionId: executionId,
StepCount: stepCount,
ItemsProcessed: itemsProcessed,
SuccessRate: 1.0,
},
});
return finalResponse;
} catch (error) {
// Track failed execution — same fields, different values
olakai("event", "ai_activity", {
prompt: input,
response: `Error: ${error instanceof Error ? error.message : "Unknown"}`,
tokens: totalTokens,
requestTime: Date.now() - startTime,
taskExecutionId,
task: "Data Processing & Analysis",
customData: {
ExecutionId: executionId,
StepCount: stepCount,
ItemsProcessed: itemsProcessed,
SuccessRate: 0,
},
});
throw error;
}
}Install dependencies:
pip install olakai-sdk openaiInitialize and track:
import os
from olakaisdk import olakai_config, olakai, OlakaiEventParams
from openai import OpenAI
# Initialize Olakai
olakai_config(os.getenv("OLAKAI_API_KEY"))
# Create OpenAI client
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))Agentic workflow:
import time
import uuid
def run_agent(input_text: str) -> str:
start_time = time.time()
execution_id = str(uuid.uuid4())
task_execution_id = str(uuid.uuid4())
total_tokens = 0
step_count = 0
items_processed = 0
try:
# Your workflow steps here...
step_count += 1
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": input_text}]
)
total_tokens += response.usage.total_tokens
final_response = response.choices[0].message.content
# Track successful execution
olakai("event", "ai_activity", OlakaiEventParams(
prompt=input_text,
response=final_response,
tokens=total_tokens,
requestTime=int((time.time() - start_time) * 1000),
taskExecutionId=task_execution_id,
task="Data Processing & Analysis",
customData={
"ExecutionId": execution_id,
"StepCount": step_count,
"ItemsProcessed": items_processed,
"SuccessRate": 1.0,
}
))
return final_response
except Exception as e:
olakai("event", "ai_activity", OlakaiEventParams(
prompt=input_text,
response=f"Error: {str(e)}",
tokens=total_tokens,
requestTime=int((time.time() - start_time) * 1000),
taskExecutionId=task_execution_id,
task="Data Processing & Analysis",
customData={
"ExecutionId": execution_id,
"StepCount": step_count,
"ItemsProcessed": items_processed,
"SuccessRate": 0,
}
))
raiseFor other languages or custom integrations:
curl -X POST "https://app.olakai.ai/api/monitoring/prompt" \
-H "Content-Type: application/json" \
-H "x-api-key: YOUR_API_KEY" \
-d '{
"prompt": "User input here",
"response": "Agent response here",
"app": "your-agent-name",
"task": "Data Processing & Analysis",
"tokens": 1500,
"requestTime": 5000,
"customData": {
"ExecutionId": "abc-123",
"StepCount": 5,
"ItemsProcessed": 10,
"SuccessRate": 1.0
}
}'Always validate your implementation by running a test and inspecting the actual event data.
Execute your agent with test data to generate at least one event.
olakai activity list --agent-id YOUR_AGENT_ID --limit 1 --json
olakai activity get EVENT_ID --jsonCheck customData is present and correct:
olakai activity get EVENT_ID --json | jq '.customData'Check KPIs are numeric (not strings):
olakai activity get EVENT_ID --json | jq '.kpiData'CORRECT — numeric values:
{
"Items Processed": 10,
"Success Rate": 100
}WRONG — string values (broken formula):
{
"Items Processed": "itemsProcessed"
}Fix: olakai kpis update KPI_ID --formula "YourVariable"
WRONG — null values: Fix by verifying:
olakai custom-data list --agent-id ID1. Run agent (generate event)
↓
2. Fetch event: olakai activity get ID --json
↓
3. Check customData present? NO → Fix SDK code
↓
4. Check kpiData numeric? NO → Fix formula
↓
5. Check kpiData not null? NO → Create CustomDataConfig or fix field name
↓
✅ All validations pass — implementation completeBefore deploying to production:
| Category | Operators |
|---|---|
| Arithmetic | +, -, *, / |
| Comparison | <, <=, =, <>, >=, > |
| Logical | AND, OR, NOT |
| Conditional | IF(condition, true_val, false_val), MAP(value, match1, out1, default) |
| Math | ABS, MAX, MIN, AVERAGE, TRUNC |
| Null handling | ISNA(value), ISDEFINED(value), NA() |
--formula "ItemsProcessed" # passthrough
--formula "SuccessRate * 100" # percentage (0-1 to 0-100)
--formula "IF(SuccessRate < 1, 1, 0)" # conditional counting
--formula "IF(PII detected, 1, 0)" # built-in variable
--formula "IF(ISDEFINED(MyField), MyField, 0)" # null-safe
--formula "IF(AND(StepCount > 5, SuccessRate < 0.9), 1, 0)" # compound conditions| Aggregation | Use For | Example |
|---|---|---|
SUM | Totals, counts | Total items processed across all runs |
AVERAGE | Rates, percentages | Average success rate |
Use these predefined task categories for the task field:
| Category | Example Use |
|---|---|
| Research & Intelligence | Competitive intelligence, market research |
| Data Processing & Analysis | Data extraction, statistical analysis |
| Content Development | Blog writing, technical documentation |
| Content Refinement | Editing, proofreading |
| Customer Experience | Complaint resolution, ticket triage |
| Software Development | Code generation, code review, debugging |
| Strategic Planning | Roadmap development, scenario planning |
# CLI Commands
olakai login # Authenticate
olakai workflows create --name "Name" --json # Create workflow
olakai agents create --name "Name" --workflow ID --with-api-key # Register agent
olakai custom-data create --agent-id ID --name X --type NUMBER # Create custom field
olakai kpis create --formula "X" --agent-id ID # Create KPI
olakai activity list --agent-id ID # View events// TypeScript SDK
import { olakaiConfig, olakai } from "@olakai/sdk";
olakaiConfig({ apiKey: process.env.OLAKAI_API_KEY });
olakai("event", "ai_activity", {
prompt: "input",
response: "output",
tokens: 1500,
task: "Data Processing & Analysis",
customData: { StepCount: 3, Success: 1 },
});# Python SDK
from olakaisdk import olakai_config, olakai, OlakaiEventParams
olakai_config(os.getenv("OLAKAI_API_KEY"))
olakai("event", "ai_activity", OlakaiEventParams(
prompt="input",
response="output",
tokens=1500,
task="Data Processing & Analysis",
customData={"StepCount": 3, "Success": 1},
))eeb895b
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.