CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

llmobs-integrations

dd-trace-py LLMObs integration development guide. Use when creating, modifying, or debugging LLMObs integrations for LLM/AI libraries in the Python tracer. Covers BaseLLMIntegration, stream handling, message extraction, usage metrics, tool call parsing, and VCR-based testing patterns. Triggers: "llmobs", "LLMObs", "BaseLLMIntegration", "llmobs_set_tags", "_llmobs_set_tags", "BaseStreamHandler", "submit_to_llmobs", "integration.trace", "LLM span", "VCR", "cassette", "anthropic", "openai", "google_genai", "claude_agent_sdk", "generative-ai", "LLM integration", "llmobs_enabled".

Quality

94%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

90%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A dense, highly actionable skill body: exact file paths, method signatures, field tables, and constraints with almost no wasted tokens, backed by a well-organized bundle of four reference files. The main gaps are the lack of an explicit numbered workflow in the body (delegated to the implementation guide) and the buried nova-sonic.md reference that is not surfaced in the Reference Files section.

Suggestions

Add a short numbered build sequence to the body (read reference integration -> implement _set_base_span_tags/_llmobs_set_tags -> add subscribers + pyproject entry point -> write test_{name}_llmobs.py -> verify with DD_LLMOBS_ENABLED/DD_TRACE_DEBUG), or surface the Step 1-5 sequence from references/implementation-guide.md so the workflow is explicit rather than implied by section order.

List references/nova-sonic.md in the Reference Files section of SKILL.md — it is currently discoverable only through references/implementation-guide.md, making it two levels deep and easy to miss.

Consider trimming the 'Two-Layer Architecture' heading (it says two layers but enumerates three items including LLMObs Subscribers) and moving the provider-by-provider finish_reason source table into the implementation guide, keeping only the merge/omit/join rules inline.

DimensionReasoningScore

Conciseness

The body is lean and assumes Claude's competence: it opens directly with the two-layer architecture and repo-specific file paths, never explains what LLM observability or tracing is, and every section carries non-obvious repo knowledge (exact entry-point group 'ddtrace.integration_hooks', exact metric key constants, the finish_reason merge/omit/join rules). It matches the 5 anchor — every token earns its place; the 4 anchor's 'minor instances of over-explanation' do not materialize.

5 / 5

Actionability

Guidance is fully concrete and executable: exact method signatures ('_set_base_span_tags(span, **kwargs)', '_llmobs_set_tags(span, args, kwargs, response, operation)'), a copy-paste-ready Message/ToolCall/ToolResult/AudioPart code block, a table of reference integrations with exact file paths, exact env flags ('DD_TRACE_DEBUG=true'), and precise constraints ('submit_to_llmobs=True', 'ctx.dispatch_ended_event()'). Specific examples cover the common cases, matching the 5 anchor.

5 / 5

Workflow Clarity

The workflow is legible and mostly checkpointed: an explicit 'Always read 1-2 references before writing or modifying LLMObs code' directive, the methods to implement, a constraints checklist, and symptom-to-check debugging tips ('No LLMObs spans -- check submit_to_llmobs=True, ctx.dispatch_ended_event(), and llmobs_enabled'). However, the build sequence is implied by section order rather than an explicit numbered step flow, and the validate/retry loop lives in the referenced guides — fitting the 4 anchor (clear sequence, most checkpoints, minor gaps) rather than 5's explicit validation steps and feedback loops, but clearly above 3's implicit checkpoints.

4 / 5

Progressive Disclosure

The SKILL.md is a genuine overview (~150 lines) with clearly signaled one-level references ('[Implementation Guide] ... LLM-specific steps', '[Failure Modes] ... All 8 failure modes', '[Testing Guide] ... VCR cassettes, suitespec') and 875 lines of detail properly pushed to the bundle. Two minor gaps keep it below 5: nova-sonic.md exists in the bundle but is only reachable through implementation-guide.md (two levels deep) and is absent from the Reference Files section, and the ~30-line Response Metadata policy block is fairly dense for an overview file. It is well above the 3 anchor's 'content that should be separate is inline'.

4 / 5

Total

18

/

20

Passed

Description

96%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it states a clear what, an explicit 'Use when' clause, concrete covered capabilities with named classes, and a rich trigger list. The only weakness is that several trigger terms (anthropic, openai, generative-ai, VCR) are broad enough to fire in conversations unrelated to dd-trace-py integration development.

DimensionReasoningScore

Specificity

The description lists multiple concrete capabilities and topics: 'creating, modifying, or debugging LLMObs integrations' plus 'Covers BaseLLMIntegration, stream handling, message extraction, usage metrics, tool call parsing, and VCR-based testing patterns' — named classes and specific technical areas with no vague filler. This matches the anchor for multiple specific concrete actions with comprehensive coverage; nothing significant for this skill's scope is missing, so it is not the 4 anchor ('minor gaps in coverage').

5 / 5

Completeness

It explicitly answers both questions: 'what' via 'dd-trace-py LLMObs integration development guide ... Covers BaseLLMIntegration, stream handling, ...' and 'when' via an explicit 'Use when creating, modifying, or debugging LLMObs integrations for LLM/AI libraries in the Python tracer' clause plus concrete trigger phrases. This matches the 5 anchor exactly; the 4 anchor requires a 'when' that 'could be more explicit', which does not apply here.

5 / 5

Trigger Term Quality

The trigger list is comprehensive and natural for the domain: 'llmobs', 'LLMObs', 'BaseLLMIntegration', 'llmobs_set_tags', 'BaseStreamHandler', 'submit_to_llmobs', 'integration.trace', 'LLM span', 'VCR', 'cassette', 'anthropic', 'openai', 'google_genai', 'claude_agent_sdk', 'generative-ai', 'LLM integration', 'llmobs_enabled' — covering case variants, provider/library names, and both jargon and natural phrases a user would say. It matches the 5 anchor (synonyms and variations included); it is well above 4 ('a few natural terms missing').

5 / 5

Distinctiveness Conflict Risk

The niche is clear and most triggers ('llmobs', 'BaseLLMIntegration', 'submit_to_llmobs', 'integration.trace') are distinct to dd-trace-py LLMObs work, but broad provider terms like 'anthropic', 'openai', 'generative-ai', and 'VCR' would also naturally occur in unrelated conversations about those providers/libraries. This fits the 4 anchor (mostly distinct, minor overlap risk with closely related skills) rather than 5, whose 'minimal conflict risk' is undercut by those broad trigger terms.

4 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

DataDog/dd-trace-py

Path

.claude/skills/llmobs-integrations/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.