AEM Workflow Triaging — 6.5 LTS / AMS

Classify workflow issues, determine what logs and data to gather, and map to the correct runbook or log search. Optimized for production support on AEM 6.5 LTS and Adobe Managed Services (AMS).

Audience

AEM 6.5 LTS / AMS operators and developers (and the IDE LLM acting on their behalf) classifying workflow incidents across one or more hosts — using host + time-range + logs (direct filesystem, AMS log access, or Splunk) and read-only JMX metrics, before drilling into a single instance. Use this skill for cross-host log mining and symptom classification; switch to workflow-debugging once the instance and root cause are identified.

Variant Scope

• This skill is 6.5-lts-only (includes AMS).
• Log access via direct filesystem (crx-quickstart/logs/error.log), AMS log access, or Splunk (if indexed).
• JMX available via Felix Console (/system/console/jmx) or a JMX client for workflow counts, queue metrics, and remediation.
• Not for AEM as a Cloud Service. If the target is AEMaaCS, stop and use the cloud-service variant of this skill — JMX is not available on cloud production, logs are accessed via Cloud Manager (not the filesystem), and remediation lands through Git + pipeline rather than Felix Console. Several signatures and diagnostic surfaces here do not apply as written on AEMaaCS.

Dependencies

• workflow-debugging — once a symptom is classified and a host/instance is identified, route here for the step-by-step runbook and remediation.
• workflow-debugging/reference.md — diagnostic tool pointers, JMX/config locations, log patterns, and external doc links for 6.5 LTS / AMS.

---

When to use this skill

• User asks: "Workflow errors on <host> for the past X hours", "Workflow activity on <host>", "Why did workflow X fail?", "What should I collect to debug this workflow ticket?"
• User needs: Symptom classification, log patterns to search, Splunk queries, or required inputs for a runbook.
• Context: AEM 6.5 LTS / AMS (author/publish hostname format).

---

Step 1: Classify symptom (symptom_id)

Map the user's description to a symptom_id and runbook.

Each runbook-*.md above is a symptom section in the workflow-debugging skill, not a separate file to open. That skill's Step 1 maps every symptom_id to a first action, and its numbered steps are the runbook body. Classify here, then hand off there.

WorkItem vs. TaskManager task — do not confuse these. Most workflow Inbox items are workflow work items (WorkItem), created by Participant steps and managed by the workflow engine; they are stored under /var/workflow/instances, not in TaskManager. TaskManager (/var/taskmanagement/tasks) only holds tasks created explicitly via the Task API — used by Projects, Assets tasks, and custom integrations. Both paths are browsable in CRXDE Lite. For task_not_in_inbox and user_cannot_see_or_complete_item symptoms on a workflow: investigate the Participant step assignee configuration, Inbox filters, and workflow permissions — not TaskManager storage. Diagnosing the wrong backend wastes significant time.

---

Step 2: Required inputs for triage

Before suggesting a runbook or Splunk search, try to obtain:

If the user only provides host + time, respond with the generic workflow error searches and note that narrowing by model/instance ID will improve accuracy.

---

Step 3: Log patterns and Splunk (what to search)

Logs on 6.5 / AMS are accessible via direct filesystem (crx-quickstart/logs/error.log), AMS log access, or Splunk (if indexed).

Example Splunk searches — <your_aem_index> is a placeholder; substitute your organization's AEM log index:

• All workflow step errors (last 24h): index=<your_aem_index> "Error executing workflow step" | table _time host message | sort - _time
• Process not registered: index=<your_aem_index> "getProcess for" "failed" | table _time host message
• By workflow model or instance: index=<your_aem_index> ("Error executing workflow step" OR WorkflowException) (message=*<modelName>* OR message=*<instanceId>*) | sort - _time
• Lock contention: index=<your_aem_index> "refreshing the session since we had to wait for a lock" | table _time host message
• Thread pool exhaustion (auto-advance impact): index=<your_aem_index> "RejectedExecutionException" | table _time host message

Note: Index names, sourcetypes, and field names are environment-specific — they depend on how your AEM logs are onboarded to Splunk. There is no fixed index; confirm the correct one with whoever owns your Splunk onboarding before running these. Narrow by host, time range, model, and instance ID — both for accuracy and to avoid over-broad data exposure.

---

Step 4: JMX-based diagnostics

On 6.5 / AMS, JMX exposes metrics not available from logs alone, plus remediation operations. All workflow maintenance and diagnostic operations live on one MBean, com.adobe.granite.workflow:type=Maintenance (via /system/console/jmx); a second MBean, com.adobe.granite.workflow:type=Statistics, exposes time-series execution metrics for trend analysis.

Targeting the right MBean: match the object name com.adobe.granite.workflow:type=Maintenance exactly (addressed URL-encoded on the JMX console as .../jmx/com.adobe.granite.workflow:type%3DMaintenance). A "workflow" filter over the JMX registry also returns several com.adobe.granite.workflow.core.* event-listener MBeans — those are not the maintenance surface, so select by the full type=Maintenance object name, not a name substring. The entries in the tables below are operations (methods to invoke, each taking an optional model string), not readable attributes.

Triage is diagnostic-first — classify with read-only operations:

Destructive operations — do NOT run as part of triage. The operations below change or delete production workflow data. They belong in a deliberate remediation step (see workflow-debugging), not in classification:

Always run dryRun=true first, confirm the count and scope, scope by model where possible, and only execute after the root cause is fixed and with change-control approval.

---

Step 5: Example triage prompts and responses

---

Step 6: What logs and JMX can and cannot answer

Can answer (logs + JMX on 6.5 / AMS):

• Step failures: exception type, message, stack (by host, time, model, step).
• Process not registered: which process.label is missing (logs + Felix Console).
• Stuck: step errors, getProcess failures, lock wait, payload/path errors.
• Stale: JMX countStaleWorkflows + "Cannot archive workitem" in logs.
• Queue metrics: JMX returnWorkflowQueueInfo, returnSystemJobInfo, returnWorkflowJobTopicInfo.
• Running / failed instance counts: JMX countRunningWorkflows, returnFailedWorkflowCount.
• Throughput: lock wait, session refresh, JobHandler volume.
• Permission: Terminate/Resume/Suspend failed, AccessControlException.
• Payload/launcher: PathNotFoundException, launcher config errors.
• Purge preview: JMX purgeCompleted(dryRun=true) + "Workflow purge …" in logs.
• Thread pool state: Configuration status ZIP → 039_Sling_Thread_Pools.txt, or the Sling Thread Pools status page.
• Config state: Felix Console or config status ZIP → 003_Configurations.txt.

Cannot answer directly:

• Console state (e.g. "is there a current work item?"). Use the Workflow Console UI (/libs/cq/workflow/admin/console/content/instances.html).
• Runtime process step code behavior. Requires code review + log correlation.

Always pair log-based triage with read-only JMX diagnostics and the appropriate runbook in workflow-debugging.

---

Safety & security guardrails

Triage runs against production. Keep it safe:

• Diagnose read-only; never mutate during triage. Classification and data-gathering use read-only JMX operations, log reads, and console status pages. The mutating operations in Step 4 (retryFailedWorkItems, restartStaleWorkflows, purgeCompleted, purgeActive, terminateFailedInstances) change or delete production workflow data — run them only as a deliberate remediation step (see workflow-debugging): dryRun=true first, scoped by model, after the root cause is fixed, and with change-control approval.
• Least privilege. Felix Console and JMX are admin-level surfaces. Restrict them to operators. Keep /system/console off public networks — on AMS/on-prem it must sit behind the Dispatcher deny rules and admin authentication, never reachable from publish or the internet.
• No secrets in queries or shared logs. Do not embed credentials, tokens, or API keys in Splunk searches or in anything pasted into a ticket; redact them if present.
• Treat logs and thread dumps as sensitive. error.log, configuration status ZIPs, and thread dumps can contain payload paths, user IDs, and application data. Share only with authorized parties and redact customer PII before attaching to a ticket.
• Scope every search. Narrow Splunk/log searches by host, time range, model, and instance ID — for accuracy and to minimize data exposure.

---

References (in repo)

• Step-by-step runbook (per symptom): ../workflow-debugging/SKILL.md
• Diagnostic tool pointers, JMX/config locations, and log patterns: ../workflow-debugging/reference.md
• 6.5 LTS guardrails (paths, service users, OSGi annotations): ../workflow-development/references/workflow-foundation/65-lts-guardrails.md
• Intent router / 6.5 LTS capabilities: ../SKILL.md