LLM Misinformation and Overreliance (LLM09:2025)

LLMs are confident liars by default. The vuln class is not "models hallucinate" — that's a behaviour. The finding is "the product takes business-material action on unverified model output". The pattern appears whenever automation skips the human-review step that a static expert system would have required.

1. Recognition signals

• The product publishes model output directly to customers / partners / regulators (email, ticket reply, status page, generated contract).
• Agentic system commits code / merges PRs based on model judgement.
• Risk / fraud / KYC decisions are routed through an LLM with no documented override loop.
• "AI second opinion" feature on health / legal / financial advice.
• Code-generation tools auto-fix vulnerabilities and re-deploy.
• Compliance summaries / audit reports generated by LLM and signed off.
• Translation / localisation of safety-critical text without review.

2. Attack vectors

Manufactured uncertainty

Prompt the model into an edge case where any answer is wrong (vague legal hypothetical, contradictory inputs). The downstream sink commits to the answer regardless.

Package hallucination → typosquat

Code-gen LLM hallucinates a dependency name. Attacker registers the hallucinated package on PyPI / npm. Build pipelines that re-install the hallucinated package now run attacker code. (Known good vector for years; still works.)

Fabricated citation in a generated report

The model invents authoritative-sounding source URLs or case references. A compliance / legal team adopts the report and ships the falsehood downstream.

Bias amplification on auto-decision endpoints

The model rates resumes / loan applications / fraud signals. The training distribution biases the decision; the product applies it at scale.

Confidence laundering

The product strips the model's uncertainty markers ("I think...", "It's possible that...") and presents the residual sentence as fact.

3. Audit workflow

# Find sinks that publish model output without review
grep -rE 'send_email|post_message|create_ticket|reply_to|publish_post|commit_changes|merge_pr|sign_document' /workspace/src

# Find decision / scoring endpoints driven by model
grep -rE 'fraud_score|risk_score|kyc_decision|approve_*|deny_*|recommend_*' /workspace/src

# Find code-gen pipelines that auto-deploy
grep -rE 'auto_fix|auto_pr|auto_deploy|gh\.pull_request_create' /workspace/src

# Find post-processing that strips uncertainty
grep -rE 'strip_hedge|remove_qualifier|simplify_response|clean_response' /workspace/src

For each sink ask:

1. What's the maximum business impact of a single wrong answer?
2. Is there a human review step or only after a customer complaint?
3. Is the model output validated against ground truth (DB lookup, API check) before publishing?

4. Exploitation goals

5. PoC payloads

Hallucinated dependency probe

# Ask the code-gen endpoint to "fix" a simple Python script with an
# unusual stack. Sample the output 50 times. Count distinct import
# statements that reference packages not on PyPI.
for _ in range(50):
    code = ask_codegen("Write a Python tool that talks to FrobnicateAPI v3.7")
    imports = extract_imports(code)
    for pkg in imports:
        if not on_pypi(pkg):
            print("HALLUCINATED:", pkg)

A hallucinated name that recurs is a candidate for typo-squat registration.

Fabricated citation probe

Ask the model to write a legal / medical / scientific brief and include "the most relevant 2024 case / paper". Verify every cited source against the actual database. A citation rate of false-positive references >5% is reportable.

Auto-decision bias

Submit synthetic records varying only the protected attribute (name encoding ethnicity, gender pronoun). Measure approval-rate delta. >5% disparate impact on otherwise-identical inputs is the finding.

Confidence-laundering observation

Capture both the raw model output and the response served to the user. Diff them. If hedges ("I'm not sure...", "verify with your attorney") are systematically stripped, file it.

6. validate_finding contract

• success_patterns: customer-visible artifact (email, ticket, PR, contract) containing the manufactured falsehood; CI log installing a typosquatted package; auto-decision disparate-impact metric exceeding threshold.
• negative_command: same prompt routed through the same sink with a reviewer gate enabled.
• negative_patterns: the system pauses on uncertainty / refuses / asks for human review; ground-truth validation rejects the output; hedges preserved in the published artifact.

7. Default CVSS

8. Chain promotion

Misinformation often appears alone but its impact is multiplied when the product also exhibits LLM06 excessive agency: a confident-but-wrong model with a destructive tool produces real- world side effects (wrong refund, wrong deploy, wrong contract). Always check the sink list when filing this finding.