Quick Path vs Full Protocol

Quick Path

1. Reproduce the error. Read the full traceback.
2. Check .agents/knowledge/constraints.md for known pitfalls.
3. Write a reproducer test if feasible.
4. Minimal fix — root cause only, don't touch surrounding code.
5. Verify: reproducer passes, pytest tests/<module>/ passes, no regressions across modalities.
6. Run /veomni-review, make quality, commit.

If not resolved in 15 min → switch to Full Protocol.

Full Protocol

Before You Start

Use TodoWrite to track all phases:

Phase 1: Investigate <symptom>       -> in_progress
Phase 2: Pattern analysis            -> pending
Phase 3: Hypothesis & test           -> pending
Phase 4: Implement fix               -> pending
Phase 5: Knowledge capture           -> pending

Phase 1: Root Cause Investigation

1. Read the FULL error message / symptom. Don't skim. Extract 2-3 keywords.
2. Check constraints first: Read .agents/knowledge/constraints.md — many issues are known constraint violations.
3. Reproduce consistently. If you can't reproduce, you don't understand it.
4. git log --oneline -10 — what changed recently?
5. Trace data flow backward through the call stack.
6. Distributed training specifics:
   • Check if error appears on all ranks or just rank 0.
   • FSDP2: verify sharding plan matches model structure (veomni/distributed/parallel_plan.py).
   • Sequence parallel: check that attention inputs are properly split/gathered (veomni/distributed/sequence_parallel/).
   • MoE: verify expert routing and load balancing (veomni/distributed/moe/).

Phase 2: Pattern Analysis

1. Find a working example (previous commit, different config, reference implementation).
2. Compare completely — diff line by line, not skim. Include config YAML, environment vars, and launcher scripts.
3. Identify ALL differences between working and broken code.
4. Check dependencies — different transformers version? Different PyTorch version?
5. If a package version upgrade is suspected, create isolated uv environments to bisect:

   # Create the default env on the default pin (`transformers-stable` → 5.9.0).
   uv venv .venv-default
   VIRTUAL_ENV=.venv-default uv sync --extra gpu --dev

   Run the same reproducer in both envs to confirm the version is the root cause. This avoids polluting the main .venv/.

Phase 3: Hypothesis and Testing

1. Form ONE specific, falsifiable hypothesis.
2. Design a MINIMAL experiment (change one thing only).
3. Run the experiment. Record the result.
4. If wrong, update understanding and form new hypothesis. No random guess-and-check.

Red flags — STOP and restart from Phase 1:

• "Let me just try changing X and see what happens"
• "Quick fix for now, clean up later"
• "It probably works, let me move on"

Verification gate — before acting on a conclusion, check:

• Does the evidence actually support this cause, or just correlate?
• Could a different root cause produce the same symptoms?
• What observation would disprove this hypothesis? Have you looked for it?
• If confidence < 80% or the evidence is ambiguous, launch a verification subagent (see Appendix).

Phase 4: Implementation

1. Write a failing test that demonstrates the bug (if feasible).
2. Implement a SINGLE targeted fix addressing the root cause.
3. Verify: test passes, training runs correctly, no regressions.
4. Check for collateral — did the fix break other modalities or trainers?
5. Before committing: run /veomni-review skill.

Phase 5: Knowledge Capture (mandatory)

Do this immediately after the fix is verified. Knowledge decays fast.

• New hard constraint? → add to .agents/knowledge/constraints.md
• Architecture insight? → add to .agents/knowledge/architecture.md
• New test needed? → add to tests/ for regression prevention
• Docs outdated? → update docs/ if the fix changes API behavior, config semantics, or usage patterns

If none apply, explicitly note "no new knowledge to capture."

Three-Strike Rule

If 3 consecutive fix attempts fail:

• STOP fixing symptoms.
• Question whether the underlying approach/architecture is wrong.
• Step back and re-examine: are you solving the right problem?
• Report to user with analysis before continuing.

Common Pitfalls

• FSDP2 + gradient accumulation: gradients must be accumulated in the unsharded space — accumulating sharded gradients produces wrong results.
• DCP checkpoint format: model state dict keys must match exactly between save and load — renamed parameters break checkpoint loading silently.
• Multi-modality data collators: text-only collators crash on multimodal data and vice versa — always check data_collator type matches the dataset.
• Sequence parallel: attention outputs must be gathered before loss computation — partial outputs produce incorrect loss values.
• Patchgen: model patches in veomni/models/transformers/*/ are auto-generated — editing generated files directly will be overwritten.

Domain-Specific Checklists

Include the relevant checklist when investigating.

Distributed Training Correctness

• Is the loss identical (within tolerance) between 1-GPU and multi-GPU runs?
• Are ALL model parameters sharded correctly? (check parallel_plan)
• Is gradient clipping applied in the correct coordinate space?
• For sequence parallel: are attention masks split consistently across ranks?
• For MoE: are expert assignments deterministic across runs with the same seed?

Numerical Correctness

• Is there a reference implementation showing the SAME numbers?
• Are ALL weights loaded? (check logs for missing/unexpected keys)
• Is the comparison fair? (same inputs, same dtype, same parallelism)
• Could there be a dtype mismatch? (float32 vs bfloat16 in computation)
• Are there NaN/Inf values being silently masked or replaced?

Appendix: Verification Subagent

When confidence is low or evidence is ambiguous, launch a subagent to challenge your conclusion:

You are a critical reviewer. Your job is to find flaws in the following conclusion.

## Conclusion Under Review
<the specific claim or decision>

## Evidence Presented
<the data, logs, experiments supporting the conclusion>

## Your Task
1. Does the evidence actually support the conclusion, or just correlate?
2. Generate 2+ alternative explanations consistent with the same evidence.
3. What specific observation would DISPROVE this conclusion? Has it been checked?
4. Was the experiment controlled (one variable changed at a time)?

## Output
Verdict: CONFIRMED / CHALLENGED / INSUFFICIENT_EVIDENCE
Findings: [issues found, counter-hypotheses, missing evidence]