or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

admin-api.mdcore-types.mddata-management.mdindex.mdmulti-language.mdplugins.mdtasks-workflows.md
tile.json

tessl/pypi-flyteidl

IDL for Flyte Platform containing protobuf specifications, gRPC API definitions, and generated clients for multiple languages

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/flyteidl@1.16.x

To install, run

npx @tessl/cli install tessl/pypi-flyteidl@1.16.0

index.mddocs/

Flyteidl

The Interface Definition Language (IDL) for the Flyte workflow orchestration platform, providing comprehensive protobuf specifications, gRPC service definitions, and generated client libraries for multiple programming languages. This package serves as the core communication layer for Flyte's distributed architecture, enabling strongly-typed interfaces for workflow definitions, execution metadata, task specifications, and administrative operations.

Package Information

  • Package Name: flyteidl
  • Package Type: pypi (also available as Go module, npm package, Rust crate)
  • Language: Protocol Buffers with generated clients in Python, Go, JavaScript/TypeScript, Rust
  • Installation: pip install flyteidl

Core Imports

Python

# Core protobuf messages
from flyteidl.core import literals_pb2, tasks_pb2, workflow_pb2, types_pb2
from flyteidl.admin import task_pb2, workflow_pb2, execution_pb2

# gRPC service clients
from flyteidl.service import admin_pb2_grpc
from flyteidl.service.admin_pb2_grpc import AdminServiceStub

# Plugin types
from flyteidl.plugins import spark_pb2, pytorch_pb2, array_job_pb2

Go

import (
    "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core"
    "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin"
    "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/service"
    "github.com/flyteorg/flyte/flyteidl/clients/go/admin"
)

JavaScript/TypeScript

import { AdminServiceClient } from '@flyteorg/flyteidl/service/admin_connect';
import { Task, Workflow, Execution } from '@flyteorg/flyteidl/admin/admin_pb';
import { Literal, LiteralType } from '@flyteorg/flyteidl/core/literals_pb';

Rust

use flyteidl::flyteidl::core::{Literal, LiteralType, TaskTemplate};
use flyteidl::flyteidl::admin::{Task, Workflow, Execution};
use flyteidl::flyteidl::service::admin::AdminServiceClient;

Basic Usage

Python Client Example

import grpc
from flyteidl.service.admin_pb2_grpc import AdminServiceStub
from flyteidl.admin.task_pb2 import TaskCreateRequest
from flyteidl.core.identifier_pb2 import Identifier
from flyteidl.core.tasks_pb2 import TaskTemplate

# Create gRPC channel and client
channel = grpc.insecure_channel('localhost:8089')
client = AdminServiceStub(channel)

# Create a task identifier
task_id = Identifier(
    resource_type=0,  # TASK
    project="my-project",
    domain="development", 
    name="my-task",
    version="v1.0.0"
)

# Create task template
task_template = TaskTemplate(
    id=task_id,
    type="python-task",
    metadata=None,
    interface=None,
    custom={}
)

# Create task via Admin API
request = TaskCreateRequest(id=task_id, spec=task_template)
response = client.CreateTask(request)

Go Client Example

package main

import (
    "context"
    
    "github.com/flyteorg/flyte/flyteidl/clients/go/admin"
    "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core"
    adminpb "github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/admin"
)

func main() {
    // Create admin client with configuration
    cfg := admin.GetConfig()
    adminClient, err := admin.NewAdminClient(context.Background(), cfg)
    if err != nil {
        panic(err)
    }

    // Create task identifier
    taskID := &core.Identifier{
        ResourceType: core.ResourceType_TASK,
        Project:      "my-project",
        Domain:       "development",
        Name:         "my-task",
        Version:      "v1.0.0",
    }

    // Get task from Admin API
    task, err := adminClient.GetTask(context.Background(), &adminpb.ObjectGetRequest{
        Id: taskID,
    })
    if err != nil {
        panic(err)
    }
}

Architecture

Flyteidl follows a layered architecture designed for maximum interoperability:

  • Service Layer: gRPC service definitions (AdminService, DataProxyService, etc.) providing the API contract
  • Core Types: Fundamental data structures (Literals, Identifiers, Tasks, Workflows) used across all components
  • Administrative Types: Management entities (Projects, Executions, Resource Attributes) for operational control
  • Plugin System: Extensible task types (Spark, ML frameworks, array jobs) for diverse compute requirements
  • Multi-language Support: Identical APIs across Python, Go, JavaScript/TypeScript, and Rust via protocol buffers

This design enables seamless integration across different technology stacks while maintaining strong type safety and API consistency.

Capabilities

Administrative API

The primary interface for managing Flyte resources including tasks, workflows, launch plans, executions, and projects. Provides comprehensive CRUD operations, state management, and resource attributes configuration.

class AdminServiceStub:
    def CreateTask(self, request: TaskCreateRequest) -> TaskCreateResponse: ...
    def GetTask(self, request: ObjectGetRequest) -> Task: ...
    def ListTasks(self, request: ResourceListRequest) -> TaskList: ...
    def CreateWorkflow(self, request: WorkflowCreateRequest) -> WorkflowCreateResponse: ...
    def CreateExecution(self, request: ExecutionCreateRequest) -> ExecutionCreateResponse: ...
    def GetExecution(self, request: WorkflowExecutionGetRequest) -> Execution: ...
    def ListExecutions(self, request: ResourceListRequest) -> ExecutionList: ...
    def TerminateExecution(self, request: ExecutionTerminateRequest) -> ExecutionTerminateResponse: ...

Administrative API

Core Types and Literals

Foundational type system providing strongly-typed data handling, identifier management, and interface definitions. Includes the Literal system for runtime type safety and data serialization across language boundaries.

class Literal:
    scalar: Scalar
    collection: LiteralCollection
    map: LiteralMap

class LiteralType:
    simple: SimpleType
    blob: BlobType
    collection_type: LiteralType
    map_value_type: LiteralType
    schema: SchemaType
    structured_dataset_type: StructuredDatasetType
    union_type: UnionType

class Identifier:
    resource_type: ResourceType
    project: str
    domain: str
    name: str
    version: str

Core Types and Literals

Task and Workflow Management

Comprehensive definitions for tasks and workflows including templates, specifications, bindings, and execution models. Supports complex workflow composition with conditional branching, parallel execution, and data flow management.

class TaskTemplate:
    id: Identifier
    type: str
    metadata: TaskMetadata
    interface: TypedInterface
    custom: dict
    security_context: SecurityContext
    
class WorkflowTemplate:
    id: Identifier
    metadata: WorkflowMetadata
    interface: TypedInterface
    nodes: List[Node]
    outputs: List[Binding]

class Node:
    id: str
    metadata: NodeMetadata
    inputs: List[Binding]
    upstream_node_ids: List[str]
    output_aliases: List[Alias]
    task_node: TaskNode
    workflow_node: WorkflowNode
    branch_node: BranchNode
    gate_node: GateNode

Task and Workflow Management

Plugin System

Extensible plugin framework supporting diverse compute requirements including Apache Spark, machine learning frameworks (TensorFlow, PyTorch), distributed computing (Dask, Ray), and specialized execution patterns (array jobs, MPI).

class SparkJob:
    spark_conf: dict
    application_file: str
    executor_path: str
    main_class: str
    spark_type: SparkType

class PyTorchJob:
    workers: int
    master_replicas: DistributedPyTorchTrainingReplicaSpec
    worker_replicas: DistributedPyTorchTrainingReplicaSpec

class ArrayJob:
    parallelism: int
    size: int
    min_successes: int
    min_success_ratio: float

Plugin System

Data Management

Data catalog and caching services providing versioned artifact storage, metadata management, tagging systems, and concurrent access coordination. Enables efficient data sharing and lineage tracking across workflow executions.

class Dataset:
    id: DatasetID
    metadata: Metadata
    partition_keys: List[str]

class Artifact:
    id: ArtifactID
    dataset: DatasetID
    data: List[ArtifactData]
    metadata: Metadata
    partitions: List[Partition]
    tags: List[Tag]

# Cache service operations
def Get(request: GetCacheRequest) -> GetCacheResponse: ...
def Put(request: PutCacheRequest) -> PutCacheResponse: ...
def Delete(request: DeleteCacheRequest) -> DeleteCacheResponse: ...

Data Management

Multi-language Support

Identical APIs across Python, Go, JavaScript/TypeScript, and Rust with language-specific client utilities, authentication support, and idiomatic patterns. Includes rich client libraries for advanced features like OAuth2 token management and type conversion helpers.

# Python
from flyteidl.service.admin_pb2_grpc import AdminServiceStub

// Go  
import "github.com/flyteorg/flyte/flyteidl/clients/go/admin"
type AdminClient interface {
    CreateTask(ctx context.Context, request *admin.TaskCreateRequest) (*admin.TaskCreateResponse, error)
    GetTask(ctx context.Context, request *admin.ObjectGetRequest) (*admin.Task, error)
}

// TypeScript
import { AdminServiceClient } from '@flyteorg/flyteidl/service/admin_connect';

// Rust
use flyteidl::flyteidl::service::admin::AdminServiceClient;

Multi-language Support

Types

Core Identifiers

class ResourceType:
    UNSPECIFIED = 0
    TASK = 1
    WORKFLOW = 2 
    LAUNCH_PLAN = 3
    DATASET = 4

class Identifier:
    resource_type: ResourceType
    project: str
    domain: str
    name: str
    version: str
    org: str
    
class WorkflowExecutionIdentifier:
    project: str
    domain: str
    name: str
    org: str
    
class NodeExecutionIdentifier:
    node_id: str
    execution_id: WorkflowExecutionIdentifier
    
class TaskExecutionIdentifier:
    task_id: Identifier
    node_execution_id: NodeExecutionIdentifier
    retry_attempt: int

Data Types

class SimpleType:
    NONE = 0
    INTEGER = 1
    FLOAT = 2
    STRING = 3
    BOOLEAN = 4
    DATETIME = 5
    DURATION = 6
    BINARY = 7
    ERROR = 8
    STRUCT = 9

class Scalar:
    primitive: Primitive
    blob: Blob
    binary: Binary
    schema: Schema
    none_type: Void
    error: Error
    generic: Struct
    structured_dataset: StructuredDataset
    union: Union

class Primitive:
    integer: int
    float_value: float
    string_value: str
    boolean: bool
    datetime: datetime
    duration: timedelta