Connector Skill

Manages the full lifecycle of Insight Connectors: creation, testing, schema generation, validation, and deployment.

References

Before executing any workflow, read:

• README: src/ingestion/README.md — commands, project structure
• PATTERNS: .cf-studio/config/rules/patterns.md — descriptor shape, mandatory fields, package structure

The test ladder is L0 static → L1 mock → L2 live smoke; workflows/test.md carries the harness.

Non-negotiable wiring invariants

A connector package can be internally perfect and still be half-landed — missing an entry in a repo-wide file that other machinery iterates. That failure mode does not show up on the PR, so it cannot be caught by review alone.

ALWAYS run the guard before opening a PR — it is the only one of these that fails before merge (the Guards job in CI):

python3 scripts/ci/connector_wiring.py

Worked example of getting the first three wrong at once: issue #2048 (active-directory). Details and fixes in create.md §3.8, which also carries the two data-check catalogs (data_quality vs connector_quality) and how to verify a check's selector resolves.

NEVER regenerate connectors-config.yaml wholesale — it overwrites the env: credential references (HubSpot/Salesforce) with fake value: entries. Generate one fragment: cd src/ingestion/scripts/bootstrap-db && ./generate-connectors-config.sh '<category>/<name>'.

NEVER "fix" a legacy non-semver version on a connector that declares no images: block (ai/openai, collaboration/slack — both 2026.05.04). ADR-0015 §"Legacy non-semver values" tolerates them by design; they never reach bump-descriptors.

Command Routing

Parse the user's command and route to the appropriate workflow:

CDK Build

For /connector build <name>, run {INGESTION_DIR}/airbyte-toolkit/build-connector.sh {CONNECTOR_PATH}. This builds the Docker image, pushes to registry (or loads into Kind for local dev), and registers/updates the Airbyte source definition. Only for type: cdk connectors. Use --push and IMAGE_REGISTRY env var for remote clusters.

Connector Reset

For /connector reset <name> <tenant>, run {INGESTION_DIR}/airbyte-toolkit/reset-connector.sh {CONNECTOR_NAME} <tenant>. This deletes the Airbyte connection, source, and definition, drops the Bronze database in ClickHouse, and cleans state files. Use when schema has breaking changes or a full re-sync is needed.

Airbyte Logs

ALWAYS use {INGESTION_DIR}/logs.sh to read Airbyte job logs or Argo workflow logs. NEVER call Airbyte REST API directly for log retrieval.

ALWAYS run logs.sh from {INGESTION_DIR} directory with KUBECONFIG="${KUBECONFIG:-$HOME/.kube/kind-ingestion}".

ALWAYS check logs when a sync fails. Workflow failure → check Argo workflow logs first (./logs.sh <workflow|latest>), then Airbyte job logs (./logs.sh airbyte <job-id>). Common causes: expired credentials in K8s Secret, source API errors, ClickHouse destination unreachable.

NEVER trust the Airbyte job status alone — it lies in both directions:

• A job can report succeeded with 0 records committed while the replication workload actually FAILED (e.g. replication pod unschedulable — FailedScheduling: Insufficient cpu — or the orchestrator OOM-killed mid-stream). "Green-but-empty" syncs look healthy for days.
• A job can stay running forever while the source is silently stalled (e.g. SQLite requests-cache explosion on a heavy substream parent); the Argo poll step times out (Failed) but the orphaned Airbyte job and replication pod keep spinning — cancel the job AND delete the pod.

The real health signal is per-job aggregatedStats.recordsCommitted (jobs API) and bronze freshness (max(_airbyte_extracted_at) per table) — check those, not the status column. A sync that commits only the first stream (e.g. bitbucket repositories) while later substreams show stale _airbyte_extracted_at means the replication died mid-job despite the green status.

E2E Sync

E2E (end-to-end) sync means running the full pipeline through Argo, not just triggering an Airbyte sync via API. The Argo pipeline includes: Airbyte sync → dbt transformations (Bronze → Silver). Without Argo, dbt models are not executed and Silver tables are not populated.

ALWAYS use {INGESTION_DIR}/run-sync.sh <connector> <tenant> for e2e sync. This submits an Argo workflow that runs the complete ingestion pipeline.

ALWAYS use ./logs.sh -f latest or ./logs.sh latest to monitor the Argo workflow (which includes both sync and dbt steps).

NEVER consider a raw Airbyte API sync (/api/v1/connections/sync) as e2e — it only populates Bronze tables.

Airbyte Architecture

Shared Destination

ALWAYS use a single shared ClickHouse destination for all connectors. Do NOT create per-connector destinations.

Each connection controls its own Bronze namespace via the namespaceDefinition and namespaceFormat fields:

The shared destination is configured with a default database (e.g., default or bronze). Each connection overrides the namespace to route data to the correct Bronze database.

Connector Credentials via K8s Secrets

Connector credentials are managed via Kubernetes Secrets, not inline in tenant YAML.

Secret structure:

apiVersion: v1
kind: Secret
metadata:
  name: insight-{connector}-{source-id}       # naming convention
  labels:
    app.kubernetes.io/part-of: insight         # discovery label
  annotations:
    insight.cyberfabric.com/connector: {name}  # matches descriptor.yaml name
    insight.cyberfabric.com/source-id: {id}    # passed as insight_source_id
type: Opaque
stringData:
  {field}: {value}                             # fields from connector.yaml connection_specification

Discovery: connect.sh discovers Secrets by label app.kubernetes.io/part-of=insight and reads connector type from annotation insight.cyberfabric.com/connector.

Tenant YAML: Contains only tenant_id. No connector config, no credentials — everything comes from K8s Secrets. insight_tenant_id is set from tenant YAML tenant_id. insight_source_id is set from Secret annotation insight.cyberfabric.com/source-id.

Multi-instance: Multiple Secrets with the same connector annotation create separate Airbyte sources (e.g., two M365 tenants).

No inline fallback: If no matching Secret is found, the connector is skipped with an error. All parameters (credentials, config fields like start_date) must be in the Secret.

Per-connector docs: Each connector's README.md documents the required Secret fields. See src/ingestion/connectors/*/README.md.

Local development: Create .yaml files in src/ingestion/secrets/connectors/ (gitignored) and run ./secrets/apply.sh to apply them. Secrets must contain ALL connector parameters — there is no inline fallback. See connector READMEs for required Secret fields.

Airbyte Resource Identity

Scripts identify Airbyte resources (definitions, sources, connections) by UUID from the state file — NEVER by name. Name matching is prohibited.

• ID not in state → create resource, save ID to state
• ID in state but Airbyte returns 404 → delete stale ID, recreate, save new ID
• Never search by name — multiple resources can share the same name
• Existing resources: always update config (credentials may have changed since creation)

Password Rotation

When rotating ClickHouse password:

1. Update K8s Secret → ./secrets/apply.sh --infra-only
2. Restart ClickHouse → kubectl rollout restart deployment/clickhouse -n data (strategy: Recreate — avoids PVC conflicts)
3. Sync Airbyte destination → ./scripts/connect.sh <tenant> (updates destination password from Secret)

Service Credentials

ALWAYS obtain credentials from K8s Secrets, not from hardcoded values or ConfigMaps.

ClickHouse

Quick test: kubectl exec -n data deploy/clickhouse -- clickhouse-client --password <password> --query "SELECT currentUser()"

Airbyte

Quick test: curl -s -H "Authorization: Bearer $AIRBYTE_TOKEN" "$AIRBYTE_API/api/v1/health"

Argo

Quick test: kubectl get workflows -n argo --no-headers | tail -5

Argument Parsing

/connector <command> <name> [options]

<name>     Connector name (e.g. m365, bamboohr, jira)
           Or full path: collaboration/m365, hr-directory/bamboohr

If <name> is not a path, search src/ingestion/connectors/ for it.

ALWAYS use the full relative path (e.g. collaboration/m365, not just m365) when calling register.sh directly — it resolves connectors/{path}/connector.yaml.

If <command> is omitted, show available commands and existing connectors.

Context Variables

Set these before routing to workflow: