Code Pattern Extractor

Every codebase has patterns — some intentional (house idioms), some accidental (copy-paste). Finding them tells you how code gets written here, and which duplication should be consolidated.

Three pattern types

The same structural pattern can be any of the three — it depends on whether the repetition is good, accidental, or bad.

Finding clones — structural similarity

Exact-text clones are easy (rg + sort + uniq -c). Near-clones — same structure, different variable names — need normalization:

1. Tokenize each function.
2. Normalize: replace identifiers with placeholders ($1, $2, ...), normalize literals (42 → NUM, "foo" → STR).
3. Hash the normalized token stream.
4. Group by hash. Collisions are clone candidates.

Original A:   user = db.get(user_id);  if user is None: raise NotFound("user")
Original B:   order = db.get(order_id); if order is None: raise NotFound("order")

Normalized:   $1 = db.get($2);         if $1 is None: raise NotFound(STR)
              $1 = db.get($2);         if $1 is None: raise NotFound(STR)

→ Same hash. Clone pair found.

Extractable as: def get_or_404(model, id, name): ...

Finding idioms — frequency of small patterns

Idioms are short (2–5 lines) and appear everywhere. Mine them by n-gram frequency on normalized AST nodes:

The count tells you: high-count idioms are conventions to follow; high-count anti-patterns are systemic problems.

Worked example — extracting a house idiom

Observed in 23 places:

def get_foo(self, foo_id):
    resp = self._client.get(f"/foos/{foo_id}")
    resp.raise_for_status()
    data = resp.json()
    return Foo.from_dict(data)

def get_bar(self, bar_id):
    resp = self._client.get(f"/bars/{bar_id}")
    resp.raise_for_status()
    data = resp.json()
    return Bar.from_dict(data)
# ... ×21 more

Pattern template:

def get_$RESOURCE(self, ${RESOURCE}_id):
    resp = self._client.get(f"/${RESOURCE}s/{${RESOURCE}_id}")
    resp.raise_for_status()
    return $MODEL.from_dict(resp.json())

Parameters: $RESOURCE (string, e.g. foo), $MODEL (class, e.g. Foo).

Verdict: This is a clone, not an idiom. 23 copies of the same 4 lines with 2 parameters → extract:

def _get_resource(self, path: str, model: type[T]) -> T:
    resp = self._client.get(path)
    resp.raise_for_status()
    return model.from_dict(resp.json())

def get_foo(self, foo_id): return self._get_resource(f"/foos/{foo_id}", Foo)

23 × 4 lines → 23 × 1 line + 4 lines shared. And now error handling changes in one place.

Distinguishing idiom from clone

with self._lock: (1 line, 0 params, language-required shape) → idiom. The get_$RESOURCE block above (4 lines, 2 params, nothing requires this shape) → clone.

Do not

• Do not extract every 2-line pattern. if x is None: return None appearing 89 times isn't duplication — it's an idiom, and extracting it to propagate_none(x) makes code less readable.
• Do not report clone pairs without the extraction proposal. "These two functions are similar" is not actionable. "Extract this helper" is.
• Do not ignore the parameter count. A pattern with 6 parameters that differ each time isn't extractable — the "common" part is tiny.
• Do not miss semantic clones that differ textually. if not user vs if user is None — different text, same pattern. Normalize aggressively.

Output format

## Idioms (follow these)
| Pattern | Count | Example location |
| ------- | ----- | ---------------- |

## Clones (extract these)
### <pattern name>
Occurrences: <N>
Template:
<normalized pattern with $PARAMS>
Parameters: <list — what varies>
Proposed extraction:
<function signature + body>
Affected files: <list>

## Anti-patterns (fix these)
| Pattern | Count | Why bad | Locations |
| ------- | ----- | ------- | --------- |