Dagster Resources -> Orchestra Connections

Overview

Dagster stores credentials in resources — typically ConfigurableResource subclasses (or built-in integration resources like SnowflakeResource, S3Resource, DbtCliResource) wired into Definitions(resources={...}), with secret values supplied via EnvVar. Orchestra connections serve the same purpose but live outside the pipeline code — set up in the Orchestra UI (Settings -> Connections), or through the public API / Orchestra MCP — and are referenced by name in pipeline YAML. This skill maps common Dagster resources to their Orchestra equivalents and covers naming, environment patterns, and secrets.

Connection Name Format

connection: my_snowflake_12345   # format: descriptive-name_XXXXX (5-digit suffix from UI)

The 5-digit suffix is assigned by Orchestra when the connection is created — never invent it. Read it back rather than asking someone to find it: connections are managed through the public API and the Orchestra MCP server as well as the UI, so listing the workspace's connections returns each one's descriptive_name_NNNNN id directly. Fall back to copying it from the UI only where neither is available.

No resource referenced? If the @op/@asset doesn't actually instantiate a credentialed resource — pure computation, no external client, no secrets — don't invent a connection name or a fake env var placeholder just to fill the field:

connection: null   # no distinct resource in the source; Orchestra uses the workspace default for this integration

Only set a specific name_XXXXX or ${{ ENV.VAR }} when the source code actually references a distinct resource/credential.

This extends to task parameters that duplicate connection-level scope, too — e.g. Power BI's workspace_id, or any other parameter whose value is also stored on the Orchestra connection itself. If the source code just reads the same single value everywhere (one env var, one resource-level config field) rather than genuinely varying it per task, leave that parameter null/omitted and let the connection's own configured value apply. Only carry an explicit value through (literal, input, or ${{ ENV.VAR }}) when a specific task truly needs to override it — e.g. targeting a different Power BI workspace than the one configured on the connection.

For environment-specific connections:

connection: ${{ ENV.SNOWFLAKE_CONNECTION_NAME }}

Set SNOWFLAKE_CONNECTION_NAME=my_snowflake_12345 in Orchestra's environment settings.

Resource Type Mapping

Databases

Data Integration

Notifications

Infrastructure & Protocol

Secrets Handling

Never hardcode credentials in YAML. All credentials go in the Orchestra connection — the YAML references only the connection name.

# Correct
task-001:
  integration: SNOWFLAKE
  integration_job: SNOWFLAKE_RUN_QUERY
  connection: snowflake_prod_12345
  parameters:
    statement: 'SELECT 1'

AWS_SECRETS_MANAGER and AZURE_KEY_VAULT are credential stores a connection reads from, not task integrations — neither has an integration_job, so there is no task that fetches a secret mid-run for a later task to consume. A resource that resolves its credential from one of them is handled on the connection, below.

Credentials already held in a secret store

A resource that resolves its secret at runtime from a secret store — SecretsManagerResource / secretsmanager_resource (dagster-aws), or an azure-keyvault-secrets client wrapped in a ConfigurableResource — usually needs no re-keying at all. An Orchestra connection can read its own credentials straight out of AWS Secrets Manager or Azure Key Vault:

• Create the Orchestra connection to the store first. A store only appears in the target connection's Credential Store dropdown once a connection to it exists.
• On the target connection, set Credential Store to that store and identify the secret — the Secret ARN for AWS Secrets Manager, the secret name for Azure Key Vault.
• The secret's value must be a JSON object holding exactly the keys that integration expects — the parameter IDs from its Authentication parameters table, and every required one, not only the sensitive fields. In Azure Key Vault, that whole object is the secret's value — don't split it into one vault secret per field.
• Orchestra reads the secret when the connection is saved and rejects it if it can't be read or its keys don't match, so a wrong key set surfaces there rather than at run time. After that it reads the latest version on each use, so rotating the secret in the store needs no change in Orchestra.

Only those two stores work this way; a resource backed by Hashicorp Vault or GCP Secret Manager still means a manual re-entry. Both store integrations are in beta, so confirm they're enabled on the account before planning the migration around them. See External secret storage and Azure Key Vault.

Linked Connections — Link, Don't Copy

A SnowflakeResource injected into an @op that also runs arbitrary Python becomes a PYTHON_EXECUTE_SCRIPT task (see python-dagster-to-orchestra), which needs those warehouse credentials at runtime.

Decision rule: if the credential already exists as its own Orchestra connection — the Snowflake/Postgres/AWS connection from the table above — link it on the Python/dbt Core connection (Linked connections on that connection's form) instead of copying its user/password into that connection's Secrets. One credential, one place to rotate it.

Each linked connection's fields arrive as environment variables named <INTEGRATION>__<CONNECTION_ID>__<FIELD>: every part upper-cased, each run of non-alphanumeric characters collapsed to a single _, and the parts joined by a double underscore. <CONNECTION_ID> is the id the YAML's connection: field uses (it survives renames), and <FIELD> is the field's ID from that integration's Authentication parameters table — so a Snowflake connection prod_12345 exports SNOWFLAKE__PROD_12345__SNOWFLAKE_PASSWORD.

• Python reads them via the pre-installed SDK: orchestra.get_linked_connection("snowflake", connection_id="prod_12345"), keyed by field ID.
• dbt Core reads them in profiles.yml: {{ env_var('SNOWFLAKE__PROD_12345__SNOWFLAKE_PASSWORD') }}.
• A linked connection that has been deleted, or whose credentials can't be decrypted, is skipped silently — its variables are simply absent. Check a variable exists rather than assuming it.
• Linking is also what opens an approved private connection for a task on Orchestra-hosted compute (MariaDB today); private connections are not opened for connections that aren't linked.

Full reference: Linked connections (same section on dbt Core).

Multi-Environment Pattern

pipeline:
  stage-001:
    tasks:
      task-001:
        integration: SNOWFLAKE
        integration_job: SNOWFLAKE_RUN_QUERY
        connection: ${{ ENV.SNOWFLAKE_CONN }}
        parameters:
          statement: 'SELECT * FROM orders LIMIT 10'

In Orchestra: Settings -> Environments -> set SNOWFLAKE_CONN=snowflake_dev_11111 in dev and snowflake_prod_22222 in prod. This mirrors how Dagster swaps resources per deployment/environment.

Dagster EnvVar / Config -> Orchestra inputs:

Non-credential config (a table name, an environment flag) supplied via EnvVar or a Config field should become inputs::

# Dagster
class Cfg(Config):
    target_table: str = "orders"

# Orchestra
inputs:
  target_table:
    type: string
    default: orders

pipeline:
  stage-001:
    tasks:
      task-001:
        parameters:
          statement: 'SELECT * FROM ${{ inputs.target_table }}'

Connection Setup Checklist

• Identify each Dagster resource and the secret(s) it holds
• Find the equivalent Orchestra connection type above
• Create the connection in Orchestra — UI (Settings -> Connections -> Add Connection), or the public API / Orchestra MCP, which is the faster route when a Definitions block wires up many resources at once
• If the resource resolves its secret from AWS Secrets Manager or Azure Key Vault, point the connection at that store via Credential Store instead of re-keying the value
• Note the full connection name including the 5-digit suffix — read it from the create response or by listing connections, not by hand
• Replace placeholder connection: <name> with the real name
• For environment-specific resources, set env vars and use ${{ ENV.VAR }}
• Never put credentials/tokens in the YAML

Gotchas

• 5-digit suffix is required — connection: snowflake_prod fails; use snowflake_prod_12345.
• Dagster resources are not the connection — the resource holds config in code; the Orchestra connection holds the same credentials in Orchestra. Managing that connection through the API or MCP is not the same as putting it back in code: the secret values are still supplied at creation, never committed.
• dbt Core connections are special — store both Git repo and warehouse credentials. One per repo+warehouse.
• Snowflake resource fields — map directly to Orchestra Snowflake connection fields.
• GCP service account — paste the full JSON into the GCP connection key field.
• Airbyte Cloud vs Server — AirbyteCloudResource -> Cloud connection; AirbyteResource(host=,port=) -> Server connection.
• Don't copy a warehouse credential into a Python/dbt connection's Secrets — if it already exists as its own Orchestra connection, link it and read the injected <INTEGRATION>__<CONNECTION_ID>__<FIELD> variables instead.
• A secret store already in play may remove the re-key entirely — if the resource reads from AWS Secrets Manager or Azure Key Vault, propose Credential Store on the Orchestra connection before telling the user to re-enter the value by hand.
• EnvVar — secrets -> connection; non-secret config -> inputs:.

References

• Orchestra connections: https://docs.getorchestra.io/docs/core-concepts/tasks/connections
• Managing connections programmatically (public API and MCP): https://docs.getorchestra.io/docs/core-concepts/tasks/connections#can-i-manage-connections-programmatically
• Orchestra external secret storage: https://docs.getorchestra.io/docs/core-concepts/tasks/connections#external-secret-storage
• Orchestra linked connections: https://docs.getorchestra.io/docs/integrations/python#linked-connections
• Orchestra environments: https://docs.getorchestra.io/docs/core-concepts/environments
• Dagster resources: https://docs.dagster.io/concepts/resources