AI Pipeline Architecture: Development & Production Pipelines for AI Systems

Generic, brand-neutral engineering capability; deep, sourced playbooks live in references/ and knowledge/. [DOC]

Generic, brand-neutral engineering capability; sourced playbooks in references//knowledge/. [DOC]

TL;DR

AI pipeline architecture defines how data flows through AI systems — from raw ingestion through model training and serving to production monitoring. This skill produces comprehensive pipeline architecture documentation covering development pipelines (experimentation to model artifact), production pipelines (data ingestion to prediction delivery), data store selection, model registry design, CI/CD strategy, and measurable requirements [EXPLICIT]

When to Use

• Designing data and model pipelines for new AI systems
• Evaluating existing pipeline architecture against production requirements
• Selecting data store technologies for AI workloads (relational, object, key-value, graph, vector)
• Designing model registry and versioning strategy
• Implementing CI/CD for ML (Blue and Gold deployment)
• Defining non-functional requirements for AI pipelines (performance, security, compliance)
• Planning pipeline evolution from experimental notebooks to production infrastructure

When NOT to Use

• Internal module boundaries and layer architecture → ai-software-architecture
• CONOPS and operational concept → ai-conops
• Design pattern selection and system tactics → ai-design-patterns
• Testing strategy → ai-testing-strategy
• GenAI/LLM-specific patterns (RAG, agents) → genai-architecture
• Infrastructure provisioning and platform design → infrastructure-architecture

---

Sub-capabilities (resource map)

Deep, evidence-tagged playbooks — open the one the task needs (ICM Layer 3, on-demand). [INFERENCE]

Procedure

1. Resolve the sub-capability; open the matching references/ playbook. [EXPLICIT]
2. Apply its decision tables; pick the strategy explicitly. [EXPLICIT]
3. Validate against the Quality Criteria and tag every claim. [EXPLICIT]

Quality Criteria

• Sub-capability resolved to one playbook. [INFERENCE]
• Claims evidence-tagged. [EXPLICIT]

Contract

• Aceptación: capability resolved to its reference playbook, applied, validated, evidence-tagged. [EXPLICIT]
• Límites: · Focuses on pipeline architecture, not infrastructure provisioning (see infrastructure-architecture) · Does not design internal module structure (see **ai-software-archite. [EXPLICIT]
• Casos borde: Notebook-to-Production Migration: Data scientists work in Jupyter notebooks; production requires orchestrated pipelines. Bridge with notebook-aware orchestrators (Papermill, Pl. [EXPLICIT]
• Supuestos: · Team has or will build experience with ML pipeline orchestration · Infrastructure supports the compute requirements for training and inference · Data sources are identified and a. [SUPUESTO]
• Trade-off: Decision Enables Constrains When to Use --- --- --- --- Batch Pipeline Simple, cost-effective, easy debugging High latency, not real-time Offline analytics,. [EXPLICIT]

Packet

Capas del packet, cargables bajo demanda (disciplina ICM: una capa por vez, nunca todas juntas): references/ guías de profundidad (cargar UNA por etapa) · knowledge/ cuerpo de conocimiento · prompts/ prompts listos · examples/ salida de ejemplo · agents/ subagentes del packet · assets/ recursos estáticos.