CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

ainativedev/aidevcon-2026-nyc

AI Native DevCon 2026 New York — practical agent skills grounded in conference talks

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

outline.mdskills/talk-shah-training-agent-skills/references/

Source outline

Source: Shrey Shah, Stop Writing Skills. Start Training Them: Self-Improving Skills for Coding Agents, AI Native DevCon New York 2026. Recording. All anchors refer to original numbered cues in the transcript. Short quotations below preserve the supplied transcription, including spoken wording.

TopicSource anchorExact short quoteApplication and scope
Evidence over patching00:03:12–00:06:00, cues 41–80“you never check whether that line actually helped”Unmeasured fixes can create hidden regressions.
Core loop00:06:06–00:07:30, cues 81–103“Accept and edit only if the held out score goes up.”An independent gate controls candidate acceptance.
Four components00:07:36–00:09:25, cues 104–134“tasks.”Training requires tasks, scorer, editor, and acceptance rule.
Three splits00:09:39–00:12:11, cues 137–173“the last four are the test set.”The example separates training, evaluation, and final test cases.
Cheap-first gate00:12:14–00:14:29, cues 174–209“There is no LLM in this decision.”Structural checks and focused rechecks precede paired evaluation.
LLM judge00:14:33–00:16:25, cues 210–237“The judge goes into the score.”A fixed structured judge scores but does not choose acceptance.
Bounded edits and rejections00:17:11–00:18:40, cues 249–272“remember the rejection.”Small edits and concise rejection history reduce repeated mistakes.
Reported run00:18:44–00:21:06, cues 273–305“The skill started weak on purpose”The reported gain concerns a deliberately weak example.
Optimizer checks00:21:37–00:22:44, cues 313–328“If your optimizer improves on pure randomness”Planted defects and null tests probe the trainer.
Consolidation00:22:50–00:24:09, cues 329–347“every eight steps.”The speaker describes periodic consolidation and future recursion.
Output and product definition00:24:19–00:26:25, cues 349–381“It is defining what the good looks like.”The golden dataset defines desired behavior.
Memory Q&A00:27:01–00:28:48, cues 385–409“there is no good answer to this.”Memory curation remains imperfect and personal.
Cost Q&A00:29:02–00:30:30, cues 412–432“it depends on the complexity of the skill”Quota and duration anecdotes do not establish fixed training cost.

Source limitations

  • The reported numerical gains are from specific small experiments, including an intentionally weak baseline; they do not predict general performance.
  • The transcript does not specify the exact paired statistical test, acceptance threshold, or confidence procedure. Repetition reduces but does not cancel all randomness.
  • Portability and zero-added-inference-cost language are high-level claims; runtime behavior still depends on harness support, skill content, and task.
  • The Q&A's quota percentages and training durations are contextual and partly approximate. Do not turn them into fixed prices, schedules, or current model rankings.
  • Promotional interlude cues 238–248 are omitted from the bundled transcript; the remaining cue numbers retain their source positions.
  • This is a spoken transcript; unseen slides, linked repositories, and third-party research were not independently inspected. No outcome beyond the described talk is established.

skills

talk-shah-training-agent-skills

tile.json