evals-context

Provides context about the Roo Code evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).

Quality

77%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./bundled/skills/evals-context/SKILL.md

Evals Codebase Context

When to Use This Skill

Use this skill when the task involves:

Modifying or debugging the evals execution infrastructure
Adding new eval exercises or languages
Working with the evals web interface (apps/web-evals)
Modifying the public evals display page on roocode.com
Understanding where evals code lives in this monorepo

When NOT to Use This Skill

Do NOT use this skill when:

Working on unrelated parts of the codebase (extension, webview-ui, etc.)
The task is purely about the VS Code extension's core functionality
Working on the main website pages that don't involve evals

Key Disambiguation: Two "Evals" Locations

This monorepo has two distinct evals-related locations that can cause confusion:

Component	Path	Purpose
Evals Execution System	`packages/evals/`	Core eval infrastructure: CLI, DB schema, Docker configs
Evals Management UI	`apps/web-evals/`	Next.js app for creating/monitoring eval runs (localhost:3446)
Website Evals Page	`apps/web-roo-code/src/app/evals/`	Public roocode.com page displaying eval results
External Exercises Repo	Roo-Code-Evals	Actual coding exercises (NOT in this monorepo)

Directory Structure Reference

`packages/evals/` - Core Evals Package

packages/evals/
├── ARCHITECTURE.md          # Detailed architecture documentation
├── ADDING-EVALS.md          # Guide for adding new exercises/languages
├── README.md                # Setup and running instructions
├── docker-compose.yml       # Container orchestration
├── Dockerfile.runner        # Runner container definition
├── Dockerfile.web           # Web app container
├── drizzle.config.ts        # Database ORM config
├── src/
│   ├── index.ts             # Package exports
│   ├── cli/                 # CLI commands for running evals
│   │   ├── runEvals.ts      # Orchestrates complete eval runs
│   │   ├── runTask.ts       # Executes individual tasks in containers
│   │   ├── runUnitTest.ts   # Validates task completion via tests
│   │   └── redis.ts         # Redis pub/sub integration
│   ├── db/
│   │   ├── schema.ts        # Database schema (runs, tasks)
│   │   ├── queries/         # Database query functions
│   │   └── migrations/      # SQL migrations
│   └── exercises/
│       └── index.ts         # Exercise loading utilities
└── scripts/
    └── setup.sh             # Local macOS setup script

`apps/web-evals/` - Evals Management Web App

apps/web-evals/
├── src/
│   ├── app/
│   │   ├── page.tsx         # Home page (runs list)
│   │   ├── runs/
│   │   │   ├── new/         # Create new eval run
│   │   │   └── [id]/        # View specific run status
│   │   └── api/runs/        # SSE streaming endpoint
│   ├── actions/             # Server actions
│   │   ├── runs.ts          # Run CRUD operations
│   │   ├── tasks.ts         # Task queries
│   │   ├── exercises.ts     # Exercise listing
│   │   └── heartbeat.ts     # Controller health checks
│   ├── hooks/               # React hooks (SSE, models, etc.)
│   └── lib/                 # Utilities and schemas

`apps/web-roo-code/src/app/evals/` - Public Website Evals Page

apps/web-roo-code/src/app/evals/
├── page.tsx      # Fetches and displays public eval results
├── evals.tsx     # Main evals display component
├── plot.tsx      # Visualization component
└── types.ts      # EvalRun type (extends packages/evals types)

This page displays eval results on the public roocode.com website. It imports types from @roo-code/evals but does NOT run evals.

Architecture Overview

The evals system is a distributed evaluation platform that runs AI coding tasks in isolated VS Code environments:

┌─────────────────────────────────────────────────────────────┐
│  Web App (apps/web-evals)  ──────────────────────────────── │
│        │                                                    │
│        ▼                                                    │
│  PostgreSQL ◄────► Controller Container                     │
│        │               │                                    │
│        ▼               ▼                                    │
│     Redis ◄───► Runner Containers (1-25 parallel)           │
└─────────────────────────────────────────────────────────────┘

Key components:

Controller: Orchestrates eval runs, spawns runners, manages task queue (p-queue)
Runner: Isolated Docker container with VS Code + Roo Code extension + language runtimes
Redis: Pub/sub for real-time events (NOT task queuing)
PostgreSQL: Stores runs, tasks, metrics

Common Tasks Quick Reference

Adding a New Eval Exercise

Add exercise to Roo-Code-Evals repo (external)
See packages/evals/ADDING-EVALS.md for structure

Modifying Eval CLI Behavior

Edit files in packages/evals/src/cli/:

runEvals.ts - Run orchestration
runTask.ts - Task execution
runUnitTest.ts - Test validation

Modifying the Evals Web Interface

Edit files in apps/web-evals/src/:

app/runs/new/new-run.tsx - New run form
actions/runs.ts - Run server actions

Modifying the Public Evals Display Page

Edit files in apps/web-roo-code/src/app/evals/:

evals.tsx - Display component
plot.tsx - Charts

Database Schema Changes

Edit packages/evals/src/db/schema.ts
Generate migration: cd packages/evals && pnpm drizzle-kit generate
Apply migration: pnpm drizzle-kit migrate

Running Evals Locally

# From repo root
pnpm evals

# Opens web UI at http://localhost:3446

Ports (defaults):

PostgreSQL: 5433
Redis: 6380
Web: 3446

Testing

# packages/evals tests
cd packages/evals && npx vitest run

# apps/web-evals tests
cd apps/web-evals && npx vitest run

Key Types/Exports from `@roo-code/evals`

The package exports are defined in packages/evals/src/index.ts:

Database queries: getRuns, getTasks, getTaskMetrics, etc.
Schema types: Run, Task, TaskMetrics
Used by both apps/web-evals and apps/web-roo-code

Repository: foryourhealth111-pixel/Vibe-Skills
Path: bundled/skills/evals-context/SKILL.md
Commit: f627ab5

Last updated: about 8 hours ago
First committed: 5 months ago

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.