testland/experiment-sdk-testing
Umbrella for experimentation-SDK test harnesses: the shared offline-datafile / hermetic-init pattern (commit a point-in-time flag/experiment config fixture, initialize the SDK with no network, pin arms per test, assert assignment integrity), with per-vendor references for Statsig (localMode + overrideGate), Optimizely (datafile + forced decisions), Split.io / Harness FME (localhost mode + features map or YAML fixture), Amplitude Experiment (local evaluation + bootstrap), and VWO (settings file + deterministic bucketing). Use when writing tests for application code instrumented with any of these five experimentation SDKs; for experiment DESIGN gates use ab-test-validity-checklist, and to read results use experiment-results-interpreter.
76
96%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
View guide
Passed
No findings from the security scan
optimizely-recipes.mdreferences/
Optimizely extended test recipes
Deeper test recipes for optimizely.md. Its spine
covers install, datafile init, user context + decide, and forced
decisions; this file holds the assignment-integrity and event-tracking
tests plus the run/CI wiring.
Assignment integrity
Same user id must resolve to the same variation, and decide_all must
return a stable key set across calls.
def test_assignment_deterministic():
user_a1 = client.create_user_context("user-1")
user_a2 = client.create_user_context("user-1")
d1 = user_a1.decide("flag-x")
d2 = user_a2.decide("flag-x")
assert d1.variation_key == d2.variation_key
def test_decide_all_returns_consistent_set():
user = client.create_user_context("user-1")
decisions_1 = user.decide_all()
decisions_2 = user.decide_all()
assert decisions_1.keys() == decisions_2.keys()Event tracking
Assert conversion events via the notification listener rather than inspecting network calls.
def test_conversion_event_emitted():
captured_events = []
# Optimizely supports a notification listener
client.notification_center.add_notification_listener(
notification_type="TRACK", notification_callback=lambda *args: captured_events.append(args)
)
user = client.create_user_context("user-1")
user.track_event("checkout_completed")
assert len(captured_events) == 1Running
pytest tests/optimizely/CI integration
jobs:
optimizely-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-python@v5
- run: pip install -e ".[test]"
- run: pytest tests/optimizely/Datafile lives in the repo; no SDK key needed for tests.