CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

application-template.mdskills/talk-tyler-team-coding-benchmarks/references/

Team benchmark specification

This is Tessl's recommended application of Tyler's talk. It defines a planned benchmark, not completed measurements.

Decision and candidates

State the decision: model selection, harness choice, workflow routing, or another bounded change. Record each complete stack, including model, harness, configuration, context, tools, budget, and environment. State the accepted outcome and the cost or quality tradeoff that matters.

Task corpus

List authorized repositories, history window, meaningful work types, and excluded trivial changes. Record workflow labels and rough complexity signals. Balance task classes and difficulties so the sample answers the actual decision.

For each task, store the pre-change revision, brief, available context, environment, reference evidence, and grading materials separately. Have a knowledgeable reviewer check that the brief is understandable and solvable. Document known exposure or leakage risk; private does not automatically mean unseen.

Grading and calibration

Define the deterministic checks, static analysis, intent rubric, and quality rubric needed for the task. State whether success requires every component and retain each component result.

For LLM judges, choose human-reviewed examples spanning workflows, model families, and answer quality. Compare judgments and rankings, inspect disagreement, and revise the rubric where appropriate. Keep a separate validation sample as a recommended control.

Repeated comparison

Task classCandidate stackOutcome componentsCost per successFailure costUncertainty

State the repetition plan and appropriate uncertainty method. Do not infer meaningful differences from point estimates alone. Keep changed success definitions clearly labeled.

Trajectory analysis and decision

Inspect repeated searches, re-edits, reversions, stopping behavior, and expensive failures. Connect those patterns to task outcomes rather than rewarding a particular step sequence.

End with the scoped recommendation, unsupported generalizations, and next measurement. Do not claim a universal best model, security guarantee, swarm result, or post-training improvement from this benchmark design.

skills

talk-tyler-team-coding-benchmarks

tile.json