CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

92%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

evaluation-template.mdskills/talk-chanezon-gordon-agent-experience-evaluation/references/

Agent experience evaluation plan

Use the sections relevant to the user's decision. This is a Tessl working template derived from the talk, not an official Scope configuration. Experimental controls, reporting fields, and acceptance thresholds below are recommended applications and must be adapted to the project.

Decision and scenario

  • Decision this experiment should inform:
  • Question: product selection (propensity) or correct use (efficacy):
  • Exact task prompt, including workload and scale:
  • Choices intentionally left open; choices explicitly required:
  • Assumptions and missing inputs:

Fixed profile and starting state

Record harness, model/version, OS, repository revision, directory path, SDK/version, documentation snapshot, extensions/diagnostics, network access, ordinary tools, credentials available, and reset procedure. Use placeholders for secret values. Record changes between runs.

Comparison arms

ArmAdded context/integrationWhat remains identicalQuestion answered
BaselineNo added intervention under testScenario, ordinary tools, model, environment, starting stateWhat happens already?
Skill onlyNamed and versioned skillSame fixed profileDoes the skill help?
MCP onlyNamed and versioned serverSame fixed profileDoes the server help?
Combined, if usefulSame skill plus same serverSame fixed profileIs the combination helpful or conflicting?

Checks before scoring

Define commands or observable checks using the actual project. Do not invent commands where the build or test interface is unknown.

GateConcrete evidence requiredFailure treatment
Language/build validityProject builds or passes the appropriate language checksMark gate failure
TestsTask-specific automated tests passMark gate failure
Runtime behaviorApp performs the required action against controlled data/serviceMark gate failure
Deployment, if requiredDeployment succeeds and deployed behavior is checkedMark failure, or explicitly out of scope

Keep criterion scores diagnostic when gates fail; a good judge score does not override a nonworking application.

Quality rubric

For each criterion, specify expected behavior, the evidence inspected, and pass/fail or scoring rules. For a database workload, consider SDK correctness, workload-appropriate data modeling, partition strategy, query behavior, and safe handling of credentials. These must follow the actual requirements and current product guidance; they are not universal rules from the case study.

Repetition and records

Choose runs per arm, order, budget, stopping rules, and treatment of infrastructure failures before execution. Justify sample size for the decision. Keep raw run IDs, code snapshots, traces, gate outputs, criterion evidence, runtime, token usage, and known cost. Record denominators and missing observations. Any statistical intervals or tests are additions to the talk and should be identified as such.

Results, once measured

ArmCompleted / attemptedGate passes / evaluatedCriterion outcomes (n/N)Time distributionToken distributionCost, if known
BaselineUnmeasuredUnmeasuredUnmeasuredUnmeasuredUnmeasuredUnmeasured
InterventionUnmeasuredUnmeasuredUnmeasuredUnmeasuredUnmeasuredUnmeasured

For each claimed improvement, state the comparable baseline, numerator and denominator, uncertainty, and scope. Report regressions and failures alongside gains. Predetermine a project-specific adoption threshold; do not treat the talk's example percentages as targets or guaranteed outcomes.

Failure to improvement

Observed failure and run IDSource/trace evidenceProposed changeRerun and regression checkOwner
To investigateRequired evidenceSmallest useful changeRelevant scenario and adjacent casesTo assign

Close with the next decision the evidence supports and what it does not yet establish.

skills

talk-chanezon-gordon-agent-experience-evaluation

tile.json