tessleng/agent-insight-experiment
Scan a repository to surface actionable findings about agent performance. Analyzes source code, git history, GitHub data, agent logs, and agent context, then synthesizes cross-referenced findings with targeted actions informed by Tessl product awareness. Supports incremental multi-developer contributions and produces a self-contained HTML report.
70
88%
Does it follow best practices?
Impact
—
No eval scenarios have been run
Advisory
Suggest reviewing before use
apex-taxonomy.mdreferences/
APEX Taxonomy — Agent Performance EXperiment
A classification system for insights about how repository characteristics affect coding agent performance. Use this taxonomy to categorize findings — it is a shared language for reports, not a checklist.
Each insight should be assigned exactly one category code (e.g., KCG) and one subcategory code (e.g., KCG-1). If an insight genuinely doesn't fit any existing subcategory, create a new one under the most appropriate parent category and note it as novel.
The taxonomy has five categories: Knowledge Gap (KCG), Conflicting Signals (CAS), Structural Complexity (SCX), Recurrent Failure (RAF), and Tooling Gap (TCG).
Category 1: Knowledge Gap (KCG)
The agent lacks information it needs to perform well. These are the most directly fixable issues — the information exists somewhere, it just hasn't been made available to the agent.
| Code | Subcategory | Description |
|---|---|---|
| KCG-1 | Undocumented internal APIs | Internal libraries, packages, or modules with public APIs but no documentation, JSDoc, docstrings, or usage examples. Includes libraries absent from training data where the agent has no prior knowledge to fall back on. |
| KCG-2 | Undocumented patterns | Architectural patterns, conventions, or coding standards followed in practice but never written down. |
| KCG-3 | Tribal knowledge | Know-how that lives in people's heads — deployment tricks, debugging procedures, historical context for design decisions, known gotchas — that isn't captured anywhere. |
| KCG-4 | Missing type information | Absent or incomplete type definitions, interface contracts, or schema definitions that would help the agent understand data shapes and function signatures. |
| KCG-5 | Stale documentation | Documentation that was once accurate but no longer reflects the current state of the code, actively misleading the agent. |
| KCG-6 | Missing onboarding context | Missing or incomplete setup instructions, build procedures, test commands, or deployment guides that a new contributor (human or agent) would need. |
| KCG-7 | Undocumented dangers | Dangerous patterns or operations (destructive commands, irreversible side effects, risky APIs, data-loss footguns) that are not flagged anywhere — the "unguarded risk" problem. |
Category 2: Conflicting Signals (CAS)
The agent receives contradictory or unclear guidance from the codebase, making it uncertain which approach is correct.
| Code | Subcategory | Description |
|---|---|---|
| CAS-1 | Contradictory context files | Agent configuration files (.cursor/rules, AGENTS.md, CLAUDE.md, .claude/settings, etc.) giving conflicting instructions about the same topic. |
| CAS-2 | Inconsistent code patterns | The same logical operation implemented differently across the codebase — multiple error handling strategies, auth patterns, data access approaches. |
| CAS-3 | Conflicting naming conventions | Inconsistent naming in the same codebase — camelCase vs snake_case in the same language, different terms for the same concept. |
| CAS-4 | Stale inline comments | Comments or inline documentation that contradict the actual code behavior, pointing the agent in the wrong direction. |
| CAS-5 | Ambiguous module boundaries | Unclear ownership or responsibility boundaries between modules, making it hard to know where changes belong. |
| CAS-6 | Conflicting configurations | Style, lint, or build configurations that disagree with each other or with documented standards. |
Category 3: Structural Complexity (SCX)
Repository characteristics — code structure, context organisation, and navigability — that make it inherently harder for agents to navigate, reason about, and make correct changes.
| Code | Subcategory | Description |
|---|---|---|
| SCX-1 | Deep circular dependencies | Circular imports or tightly coupled dependency chains that make it hard to understand the impact of changes. |
| SCX-2 | Non-obvious build process | Build, test, or deployment processes that require non-obvious steps not apparent from the project structure. |
| SCX-3 | Unclear navigation | Confusing code or context organisation, non-obvious cross-project or cross-module relationships, unclear monorepo layout. |
| SCX-4 | High coupling | Components tightly coupled together that can't be understood or modified in isolation, requiring the agent to hold too much context. |
| SCX-5 | Implicit configuration | Important behavior controlled by environment variables, feature flags, magic strings, or runtime configuration not visible in the code. |
| SCX-6 | Complex metaprogramming | Code generation, macros, decorators, or metaprogramming that obscures actual runtime behavior. |
| SCX-7 | Reinvented wheels | Custom implementations of functionality available as well-supported third-party packages. The agent has no docs or community knowledge to draw on for these. |
| SCX-8 | Undocumented complexity | Areas that are genuinely complex (non-trivial algorithms, subtle invariants, performance-critical paths) with no documentation surfacing that complexity — the "hidden complexity" problem. Optional: use when the complexity itself is unavoidable but the lack of signposting is the issue. |
| SCX-9 | Redundant conventions | Context content that restates knowledge the agent already has: standard language conventions, linter-enforced rules, or generic best-practice platitudes. Takes up context budget without changing agent output. |
Category 4: Recurrent Failure (RAF)
Patterns where agents demonstrably and repeatedly struggle, visible from behavioral data (agent logs, PR history, CI results).
| Code | Subcategory | Description |
|---|---|---|
| RAF-1 | Repeated area mistakes | The agent consistently makes errors in specific files, modules, or functional areas. |
| RAF-2 | CI failures post agent | Tests or CI checks that frequently fail after agent-made changes, suggesting the agent doesn't understand the test contract. |
| RAF-3 | Persistent pattern errors | Specific patterns the agent consistently gets wrong — using old APIs, forgetting required steps, misusing internal utilities. |
| RAF-4 | Excessive iteration | Tasks requiring many back-and-forth cycles or corrections before reaching a correct solution. |
| RAF-5 | Frequently reverted changes | Agent-authored code that is frequently reverted or substantially rewritten during review. |
| RAF-6 | User frustration signals | Corrections, expressions of frustration, or abandoned conversations visible in agent logs. |
Category 5: Tooling Gap (TCG)
Issues with the agent's environment, configuration, and tool setup that limit its effectiveness.
| Code | Subcategory | Description |
|---|---|---|
| TCG-1 | Missing context files | Agent configuration files that are absent, incomplete, or too vague to be useful. |
| TCG-2 | Missing tool integrations | MCP servers, extensions, or tools that would help the agent but aren't configured. |
| TCG-3 | Missing skills and rules | Opportunities for skills or rules that would prevent recurring mistakes or handle common tasks more reliably. |
| TCG-4 | Build environment complexity | Complex environment setup that makes it hard for agents to run builds, tests, or tools. |
| TCG-5 | Cross tool inconsistency | Different agent tools (Cursor, Claude Code, etc.) receiving different or conflicting context, leading to inconsistent team experiences. |
| TCG-6 | Unowned content | Context files with no clear maintainer — nobody responsible for keeping them accurate — making drift and staleness likely over time. |
Usage Notes
- Not every data source can populate every category. Source code analysis will find KCG, CAS, SCX, TCG but not RAF. Agent logs are strong for RAF but weaker for SCX. This asymmetry is the point of the experiment.
- Prefer specificity over breadth. An insight that says "the auth library is undocumented" with file paths and examples is far more valuable than "there are documentation gaps in the codebase."
- One insight per distinct issue. Don't bundle multiple unrelated problems into a single insight, even if they share a category.
- Novel subcategories are welcome. If you find something that doesn't fit, create a new subcategory code (e.g., KCG-8) and explain it. Note it as
"novel": truein the report.