unreal-automation-system

Overview

This skill wraps the C++ Automation Test Framework (UE 4.x / 5.x) plus the two most commonly composed sub-systems. Per Epic's Automation Test Framework documentation, the framework spans five test categories (Unit, Feature, Smoke, Content Stress, Screenshot Comparison) and supports multiple authoring styles (traditional, BDD Spec, UI Driver, Functional, Python / Blueprint).

• Automation Spec - BDD-style Describe / It / BeforeEach per Automation Spec documentation.
• Automation Driver - UI input simulation per Automation Driver documentation.

The full authoring code (macros, Spec, Driver) lives in references/authoring-macros-and-apis.md; running, report parsing, and CI wiring live in references/running-and-reporting.md. This file is the decision surface.

Composes with:

• game-test-categories-reference for the canonical six categories Unreal's five categories map to.
• platform-cert-overview-reference for cert-gated requirements automation tests should cover.
• multiplayer-state-machine-coverage for replication / dedicated-server state coverage authored as automation tests.

When to use

• Unit under test is C++ code in an Unreal Engine project (UE 4.x or UE 5.x) that needs the engine runtime, editor APIs, or UMG UI surface.
• You want CI-runnable tests via Unreal's command-line Automation entry point.
• You need BDD-style readable specs (Automation Spec) or scripted UI input simulation (Automation Driver) on top of plain assertion tests.

For Python / Blueprint editor tests outside C++, see Editor Automation in Unreal Engine. For end-to-end pipeline / build-farm orchestration, use Gauntlet on top of the framework this skill covers.

How to use

1. Pick the category from the table below (Unit / Feature / Smoke / Content Stress / Screenshot Comparison) and the matching EAutomationTestFlags filter + application-context mask.
2. Pick the authoring style - plain RunTest macro, data-driven complex macro, BDD Spec, or Automation Driver for UI - and lift the pattern from references/authoring-macros-and-apis.md.
3. Name the test path as dot-separated segments (MyGame.Health.Damage_…); the tree keys the Session Frontend.
4. Run in-editor via Session Frontend to iterate, then run headless via UnrealEditor-Cmd.exe … -ExecCmds="Automation RunTests …" for CI.
5. Parse the -ReportOutputPath JSON - treat any "state": "Fail" as a failed build; full schema in references/running-and-reporting.md.
6. Split the CI matrix - SmokeFilter on PRs (sub-second), ProductFilter | StressFilter plus screenshot comparison nightly.

Test categories and flags

Per the Automation Test Framework page, Epic's five categories are:

Each test declares flags from EAutomationTestFlags that mix:

• Filter - SmokeFilter, EngineFilter, ProductFilter, PerfFilter, StressFilter, NegativeFilter.
• Application context - EditorContext, ClientContext, ServerContext, CommandletContext, plus the convenience mask ApplicationContextMask.

Typical combo for a product-level test runnable in editor / client / server contexts: EAutomationTestFlags::ProductFilter | EAutomationTestFlags::ApplicationContextMask.

Authoring styles

Pick the style that matches the unit under test, then lift the full pattern from references/authoring-macros-and-apis.md:

Running

Iterate in-editor via Window → Test Automation (Session Frontend), then run headless for CI:

UnrealEditor-Cmd.exe MyGame.uproject \
    -ExecCmds="Automation RunTests MyGame.Inventory; Quit" \
    -unattended -nopause -testexit="Automation Test Queue Empty" \
    -ReportOutputPath="artifacts/automation" \
    -log

Command variants, the -ReportOutputPath JSON schema, Gauntlet, and a full GitHub Actions job are in references/running-and-reporting.md.

Worked example

Goal: cover an inventory system's stacking rule as a CI-gating BDD spec, plus a menu-close UI check.

1. Category + flags. Stacking logic is pure C++ product logic, so ProductFilter | ApplicationContextMask; it runs in well under a second, so a second SmokeFilter copy joins the PR job.
2. Authoring. Write FInventorySpec with DEFINE_SPEC and a Describe("AddItem") block holding two It("should …") cases - one asserting count increases by the stack amount, one asserting an over-cap add is rejected (pattern in references/authoring-macros-and-apis.md). Save it as InventorySpec.spec.cpp.
3. UI check. Add FMenuDriverSpec with the Driver enabled in BeforeEach on TaskGraphMainThread, the It running on EAsyncExecution::ThreadPool, By::Id("SubmitButton") clicked, and Disable() in AfterEach.
4. Run headless. UnrealEditor-Cmd.exe MyGame.uproject -ExecCmds="Automation RunTests MyGame.Inventory; Quit" -unattended -nopause -ReportOutputPath="artifacts/automation" -log.
5. Gate. CI reads artifacts/automation/index.json; the over-cap It shows "state": "Fail" when the rule regresses, so the job fails the build. Nightly re-runs the same specs under ProductFilter | StressFilter.

Anti-patterns

Limitations

• C++ build required. Tests live in C++ modules; pure-content Blueprint projects need a code module added to use this framework. (Pure Blueprint projects can use Blueprint Functional Tests in-level - see Epic's docs on Functional Testing.)
• No common exit code definition comparable to Unity's caveat - parse the -ReportOutputPath JSON or scrape the log for LogAutomationController Fail lines.
• Spec parameterised tests are loop-generated per Spec docs; there is no [TestCase] analogue from NUnit. The framework is NUnit-inspired but not NUnit-derived (unlike Unity's UTF).
• Screenshot comparison baseline storage + tolerance configuration is engine-version-specific - consult per-version Epic docs.
• Editor-context tests cannot run in dedicated-server only builds - declare EditorContext on those tests; client / server tests need their own flag set.
• Documentation source. The dev.epicgames.com docs are the public mirror; deeper detail (full macro implementations, precise JSON report schema) lives in the engine source under Engine/Source/Runtime/AutomationController/ and Engine/Source/Developer/AutomationMessages/ - partners with engine source access should consult those for authoritative details.

References

• Automation Test Framework - dev.epicgames.com.
• Automation Spec (BDD) - dev.epicgames.com.
• Automation Driver (UI input) - dev.epicgames.com.
• Full authoring code (macros, Spec, Driver): references/authoring-macros-and-apis.md.
• Running, report parsing, and CI wiring: references/running-and-reporting.md.
• Category mapping: game-test-categories-reference.
• Cert-gated requirements: platform-cert-overview-reference.
• Replication / dedicated-server coverage: multiplayer-state-machine-coverage.