Quick Path vs Full Protocol

Quick Path

1. Reproduce the error. Read the full traceback.
2. Check .agents/knowledge/constraints.md for known pitfalls.
3. Write a reproducer test if feasible.
4. Minimal fix — root cause only, don't touch surrounding code.
5. Verify: reproducer passes, pytest tests/<module>/ passes, no regressions across modalities.
6. Run make quality, commit. Run /veomni-review before opening the PR or pushing a substantive update, not per commit.

If not resolved in 15 min → switch to Full Protocol.

Full Protocol

Before You Start

Track the phases with whatever todo/plan tool the running agent provides:

Phase 1: Investigate <symptom>       -> in_progress
Phase 2: Pattern analysis            -> pending
Phase 3: Hypothesis & test           -> pending
Phase 4: Implement fix               -> pending
Phase 5: Knowledge capture           -> pending

Phase 1: Root Cause Investigation

1. Read the FULL error message / symptom. Don't skim. Extract 2-3 keywords.
2. Check constraints first: Read .agents/knowledge/constraints.md — many issues are known constraint violations.
3. Reproduce consistently. If you can't reproduce, you don't understand it.
4. git log --oneline -10 — what changed recently?
5. Trace data flow backward through the call stack.
6. Distributed training specifics:
   • Check if error appears on all ranks or just rank 0.
   • FSDP2: verify sharding plan matches model structure (veomni/distributed/parallel_plan.py).
   • Sequence parallel: check that attention inputs are properly split/gathered (veomni/distributed/sequence_parallel/).
   • MoE: verify expert routing and load balancing (veomni/distributed/moe/).

Phase 2: Pattern Analysis

1. Find a working example (previous commit, different config, reference implementation).

2. Compare completely — diff line by line, not skim. Include config YAML, environment vars, and launcher scripts.

3. Identify ALL differences between working and broken code.

4. Check dependencies — different transformers version? Different PyTorch version?

5. If a package version upgrade is suspected, create isolated uv environments to bisect:

   # Env A: the current default pin (the `transformers-stable` group).
   uv venv .venv-a
   VIRTUAL_ENV=.venv-a uv sync --active --extra gpu --dev
   
   # Env B: the same tree with exactly one package moved.
   uv venv .venv-b
   VIRTUAL_ENV=.venv-b uv sync --active --extra gpu --dev
   VIRTUAL_ENV=.venv-b uv pip install "<package>==<other-version>"

   --active is load-bearing. Without it uv sync runs in project mode and targets .venv/, ignoring VIRTUAL_ENV — so both commands would rebuild the main environment instead of the two you just created, which is the opposite of what this is for. (UV_PROJECT_ENVIRONMENT works too.)

   Confirm that installing the alternate version did not change other packages:

   uv pip freeze --python .venv-a/bin/python > /tmp/veomni-bisect-a.freeze
   uv pip freeze --python .venv-b/bin/python > /tmp/veomni-bisect-b.freeze
   diff -u /tmp/veomni-bisect-a.freeze /tmp/veomni-bisect-b.freeze

   Only the target package may differ. Pin or restore every non-target difference in Env B to Env A's version, then compare again before running the reproducer. If the target cannot run with that dependency set, report the compatibility conflict; a multi-package change is not a one-package bisect.

   Then run the same reproducer in both envs, each with its own env activated — the VIRTUAL_ENV= prefixes above apply only to the uv sync lines they are attached to, not to whatever you run next:

   (source .venv-a/bin/activate && <reproducer>)
   (source .venv-b/bin/activate && <reproducer>)

   If the suspect package is transformers, you need two worktrees and two venvs — one venv per worktree. They isolate different things and neither substitutes for the other: a venv isolates the installed packages, a worktree isolates the checkout. generated/ modeling lives in the checkout, so two venvs in one worktree share a single generated/ and regenerating it for Env B silently changes what Env A runs. Two worktrees without separate venvs share one transformers install, which defeats the bisect outright.

   git worktree add ../bisect-a HEAD && (cd ../bisect-a && uv venv .venv && VIRTUAL_ENV=.venv uv sync --active --extra gpu --dev)
   git worktree add ../bisect-b HEAD && (cd ../bisect-b && uv venv .venv && VIRTUAL_ENV=.venv uv sync --active --extra gpu --dev && VIRTUAL_ENV=.venv uv pip install "transformers==<other-version>")

   Regenerate generated/ inside each worktree against its own pin (make patchgen) before running the reproducer — it is produced against the pinned version, and a stale generated/ is itself a source of failures.

   Compare the package sets here too, using uv pip freeze --python with ../bisect-a/.venv/bin/python and ../bisect-b/.venv/bin/python, and reconcile non-transformers dependency version differences as above. The editable VeOmni and patchgen paths must point to their respective worktrees; normalize those corresponding paths only when comparing the freeze output, without changing either environment's editable installs. Run codegen and the reproducer from each worktree with its own environment activated:

   (cd ../bisect-a && source .venv/bin/activate && make patchgen && <reproducer>)
   (cd ../bisect-b && source .venv/bin/activate && make patchgen && <reproducer>)

Phase 3: Hypothesis and Testing

1. Form ONE specific, falsifiable hypothesis.
2. Design a MINIMAL experiment (change one thing only).
3. Run the experiment. Record the result.
4. If wrong, update understanding and form new hypothesis. No random guess-and-check.

Verification gate — before acting on a conclusion, check:

• Does the evidence actually support this cause, or just correlate?
• Could a different root cause produce the same symptoms?
• What observation would disprove this hypothesis? Have you looked for it?
• If confidence < 80% or the evidence is ambiguous, launch a verification subagent (see Appendix).

Phase 4: Implementation

1. Write a failing test that demonstrates the bug (if feasible).
2. Implement a SINGLE targeted fix addressing the root cause.
3. Verify: test passes, training runs correctly, no regressions.
4. Check for collateral — did the fix break other modalities or trainers?
5. Before opening the PR or pushing a substantive update: run /veomni-review over the branch diff.

Phase 5: Knowledge Capture (mandatory)

Do this immediately after the fix is verified. Knowledge decays fast.

• New hard constraint? → add to .agents/knowledge/constraints.md
• Architecture insight? → add to .agents/knowledge/architecture.md
• Regression guard needed? → prefer adding a case to an existing CI-enumerated test; see .agents/knowledge/testing.md. Only paths not already covered by a directory-level CI entry need a new workflow line. Use the workflow that owns the path, including the e2e workflows for end-to-end tests; account for the GPU/NPU differences in that table.
• Docs outdated? → update docs/ if the fix changes API behavior, config semantics, or usage patterns

If none apply, explicitly note "no new knowledge to capture."

Stop Conditions

Restart from Phase 1 if you catch yourself thinking "let me just try changing X and see", "quick fix for now, clean up later", or "it probably works, moving on".

After 3 consecutive failed fix attempts, stop fixing symptoms. Question whether the underlying approach is wrong, re-examine whether you are solving the right problem, and report the analysis to the user before continuing.

Common Pitfalls

• FSDP2 + gradient accumulation: gradients must be accumulated in the unsharded space — accumulating sharded gradients produces wrong results.
• DCP checkpoint format: model state dict keys must match exactly between save and load — renamed parameters break checkpoint loading silently.
• Multi-modality data collators: text-only collators crash on multimodal data and vice versa — always check data_collator type matches the dataset.
• Sequence parallel: attention outputs must be gathered before loss computation — partial outputs produce incorrect loss values.
• Patchgen: model patches in veomni/models/transformers/*/ are auto-generated — editing generated files directly will be overwritten.

Domain-Specific Checklists

Include the relevant checklist when investigating.

Distributed Training Correctness

• Is the loss identical (within tolerance) between 1-GPU and multi-GPU runs?
• Are ALL model parameters sharded correctly? (check parallel_plan)
• Is gradient clipping applied in the correct coordinate space?
• For sequence parallel: are attention masks split consistently across ranks?
• For MoE: are expert assignments deterministic across runs with the same seed?

Numerical Correctness

• Is there a reference implementation showing the SAME numbers?
• Are ALL weights loaded? (check logs for missing/unexpected keys)
• Is the comparison fair? (same inputs, same dtype, same parallelism)
• Could there be a dtype mismatch? (float32 vs bfloat16 in computation)
• Are there NaN/Inf values being silently masked or replaced?

Appendix: Verification Subagent

When confidence is low or evidence is ambiguous, launch a subagent to challenge your conclusion:

You are a critical reviewer. Your job is to find flaws in the following conclusion.

## Conclusion Under Review
<the specific claim or decision>

## Evidence Presented
<the data, logs, experiments supporting the conclusion>

## Your Task
1. Does the evidence actually support the conclusion, or just correlate?
2. Generate 2+ alternative explanations consistent with the same evidence.
3. What specific observation would DISPROVE this conclusion? Has it been checked?
4. Was the experiment controlled (one variable changed at a time)?

## Output
Verdict: CONFIRMED / CHALLENGED / INSUFFICIENT_EVIDENCE
Findings: [issues found, counter-hypotheses, missing evidence]