golikovichev/phoenix2pytest
Turn labeled LLM failure traces from an Arize Phoenix project into runnable pytest regression tests using the phoenix2pytest pipeline. Use when the user has an LLM application emitting OpenInference spans to Phoenix and wants a regression suite from real production failures, when extracting test cases from observed LLM bugs (hallucination, format break, off-topic drift, stale data, wrong reasoning, refusal bug), when bridging Phoenix-labeled traces into pytest-based suites for CI, when the user mentions Arize Phoenix MCP, OpenInference instrumentation, LLM observability, Gemini test synthesis, Vertex AI agent evaluation, or wants to react to LLM failures rather than predict them upfront.
88
94%
Does it follow best practices?
Impact
98%
1.63xAverage score across 2 eval scenarios
Low
Low-risk findings worth noting
Low
Low-risk findings.
2 low severity findings. Worth noting, but not necessarily harmful.
W011: Third-party content exposure detected (indirect prompt injection risk).
What this means
The skill exposes the agent to untrusted, user-generated content from public third-party sources, creating a risk of indirect prompt injection. This includes browsing arbitrary URLs, reading social media posts or forum comments, and analyzing content from unknown websites.
Why it was flagged
The required runtime workflow for the web UI `/generate` endpoint ingests OUTSIDER-authored free text from the HTTP form/JSON fields `trace_json` and `details_json` (user-controlled `user_prompt`/`evidence` etc.), which are then embedded into the synthesiser prompt and sent to the agent’s LLM context via `synthesise(...)->build_user_message(...)` and `client.generate_text(...)`.
Where we found it
app.phoenix.arize.com
domain · 5 sites
scripts/vertical_slice.py fetches live Phoenix spans from app.phoenix.arize.com at runtime (line 39/42) and directly embeds the outsider-authored trace content (user_prompt, llm_output) into Gemini LLM prompts (lines 100, 164), creating a prompt-injection path from Phoenix trace data into the agent's LLM context.
scripts/vertical_slice.py
39
base_url=os.environ["PHOENIX_BASE_URL"]
42
return client.spans.get_spans(project_identifier=project, limit=limit)
100
f"USER PROMPT:\n{trace_data['user_prompt']}\n\n"
164
f"USER PROMPT:\n{trace_data['user_prompt']}\n\n"
SKILL.md
27
PHOENIX_BASE_URL=https://app.phoenix.arize.com
api.brightdata.com
domain · 5 sites
scripts/bd_harvest.py fetches Reddit post content at runtime from api.brightdata.com (lines 132, 159, 177) and normalises the outsider-authored post titles and bodies into the dataset schema (line 232), which is then ingested into Phoenix and can flow into LLM prompts.
scripts/bd_harvest.py
58
BD_BASE = "https://api.brightdata.com"
scripts/bd_harvest.py
132
url = f"{BD_BASE}/datasets/v3/trigger"
scripts/bd_harvest.py
159
url = f"{BD_BASE}/datasets/v3/progress/{snapshot_id}"
scripts/bd_harvest.py
177
url = f"{BD_BASE}/datasets/v3/snapshot/{snapshot_id}"
scripts/bd_harvest.py
232
"user_prompt": title
W012: Unverifiable external dependency detected (runtime URL that controls agent).
What this means
The skill fetches instructions or code from an external URL at runtime, and the fetched content directly controls the agent’s prompts or executes code. This dynamic dependency allows the external source to modify the agent’s behavior without any changes to the skill itself.
Why it was flagged
The repo spawns and runs an external MCP server via npx at runtime (see src/phoenix2pytest/mcp_client.py: it runs "npx -y @arizeai/phoenix-mcp@latest", which fetches and executes remote npm package code from the npm registry e.g. https://registry.npmjs.org/@arizeai/phoenix-mcp) — this is executed at runtime and is required for fetching Phoenix traces.
Where we found it
npx -y @arizeai/phoenix-mcp@latest
command · 1 site
The scanner claims src/phoenix2pytest/mcp_client.py runs `npx -y @arizeai/phoenix-mcp@latest`, but that file does not exist in this source tree; the only subprocess.run call in the codebase runs local pytest, not npx.
scripts/vertical_slice.py
198
proc = subprocess.run([sys.executable, "-m", "pytest", str(target), "-v", "--tb=short"])
- Audited
- Security analysis