or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

audio-processing.mdconversational-ai.mdhistory.mdindex.mdprojects-studio.mdstreaming.mdtext-to-speech.mdutilities.mdvoice-management.md
tile.json

conversational-ai.mddocs/

Conversational AI

Overview

The Conversational AI API enables building intelligent voice agents with telephone integration, knowledge base management, and SIP trunk functionality. This comprehensive system allows for creating sophisticated conversational experiences with real-time voice interactions, document-based knowledge retrieval, and telephony integration.

Core Imports

import { 
  ElevenLabsClient,
  type BodySimulatesAConversationV1ConvaiAgentsAgentIdSimulateConversationPost,
  type BodySimulatesAConversationStreamV1ConvaiAgentsAgentIdSimulateConversationStreamPost,
  type BodyAddToKnowledgeBaseV1ConvaiKnowledgeBasePost,
  type BodyCreateTextDocumentV1ConvaiKnowledgeBaseTextPost,
  type BodyCreateFileDocumentV1ConvaiKnowledgeBaseFilePost,
  type BodyCreateUrlDocumentV1ConvaiKnowledgeBaseUrlPost,
  type BodyHandleAnOutboundCallViaSipTrunkV1ConvaiSipTrunkOutboundCallPost,
  type ConversationalAiGetKnowledgeBaseListRequest,
  type ConversationalAiGetDependentAgentsRequest
} from 'elevenlabs';

Agent Management

Creating Conversational Agents

const client = new ElevenLabsClient();

// Create a conversational agent
const agent = await client.conversationalAi.agents.createAgent({
  name: "Customer Support Agent",
  voice_id: "pNInz6obpgDQGcFmaJgB", // Voice to use for agent
  language: "en",
  system_prompt: `You are a helpful customer support agent for TechCorp. 
    You should be friendly, professional, and knowledgeable about our products.
    Always try to resolve customer issues and escalate to human agents when necessary.`,
  initial_message: "Hello! I'm here to help you with any questions about TechCorp products. How can I assist you today?",
  response_engine: "conversational_v1",
  conversation_config: {
    agent: {
      first_message: "Welcome to TechCorp support! How can I help you?",
      system_prompt: "You are a customer support specialist.",
      language: "en",
      tts_config: {
        voice_id: "pNInz6obpgDQGcFmaJgB",
        model_id: "eleven_turbo_v2_5",
        voice_settings: {
          stability: 0.6,
          similarity_boost: 0.8,
          style: 0.2
        }
      }
    },
    conversation: {
      max_duration_seconds: 1800, // 30 minutes
      turn_detection: {
        type: "server_vad",
        threshold: 0.5,
        prefix_padding_ms: 300,
        suffix_padding_ms: 200
      }
    }
  }
});

console.log('Agent created:', agent.agent_id);

Agent Configuration

// Comprehensive agent configuration
const advancedAgent = await client.conversationalAi.agents.createAgent({
  name: "AI Sales Assistant",
  voice_id: "21m00Tcm4TlvDq8ikWAM",
  language: "en",
  system_prompt: `You are an AI sales assistant for an e-commerce company.
    Your goal is to help customers find products, answer questions about pricing,
    availability, and shipping. You should be enthusiastic but not pushy.
    
    Available product categories:
    - Electronics (phones, laptops, accessories)
    - Home & Garden (furniture, appliances, decor)
    - Fashion (clothing, shoes, jewelry)
    
    Always ask clarifying questions to understand customer needs better.`,
  
  conversation_config: {
    agent: {
      first_message: "Hi there! I'm your AI shopping assistant. What are you looking for today?",
      system_prompt: "Be helpful and guide customers to the right products.",
      language: "en",
      
      // Voice configuration
      tts_config: {
        voice_id: "21m00Tcm4TlvDq8ikWAM",
        model_id: "eleven_turbo_v2_5",
        voice_settings: {
          stability: 0.7,
          similarity_boost: 0.8,
          style: 0.3,
          use_speaker_boost: true
        }
      },
      
      // Speech-to-text configuration
      stt_config: {
        model: "whisper-1",
        language: "en"
      }
    },
    
    conversation: {
      max_duration_seconds: 3600, // 1 hour max conversation
      
      // Turn detection settings
      turn_detection: {
        type: "server_vad", // Voice Activity Detection
        threshold: 0.6,
        prefix_padding_ms: 400,
        suffix_padding_ms: 800
      },
      
      // Client events configuration
      client_events: {
        disconnect_on_client_timeout: true,
        client_timeout_seconds: 30
      }
    }
  }
});

List and Manage Agents

// Get all agents
const agents = await client.conversationalAi.agents.getAgents();

console.log('Available agents:');
agents.agents.forEach(agent => {
  console.log(`- ${agent.name} (${agent.agent_id})`);
  console.log(`  Voice: ${agent.voice_id}`);
  console.log(`  Language: ${agent.language}`);
  console.log(`  Created: ${new Date(agent.created_at).toLocaleDateString()}`);
});

// Get specific agent details
const agentDetails = await client.conversationalAi.agents.getAgent("agent_id_here");
console.log('Agent configuration:', agentDetails);

// Update agent
const updatedAgent = await client.conversationalAi.agents.updateAgent(
  "agent_id_here",
  {
    name: "Updated Agent Name",
    system_prompt: "Updated system prompt with new instructions",
    conversation_config: {
      agent: {
        first_message: "Hello! I've been updated with new capabilities."
      }
    }
  }
);

// Delete agent
await client.conversationalAi.agents.deleteAgent("agent_id_here");
console.log('Agent deleted successfully');

Agent Interactions

Simulate Conversations

// Simulate a conversation with an agent
const conversation = await client.conversationalAi.agents.simulateConversation(
  "agent_id_here",
  {
    simulation_specification: {
      simulated_user_config: {
        first_message: "Hi, I'm looking for a new smartphone",
        system_prompt: "You are a customer interested in buying a smartphone. Ask about features, pricing, and availability.",
        response_config: {
          max_tokens: 150,
          temperature: 0.7
        }
      },
      conversation_config: {
        max_turns: 10,
        max_duration_seconds: 300
      }
    }
  }
);

console.log('Conversation simulation:', conversation.conversation_id);
console.log('Messages exchanged:', conversation.messages?.length);

// Process conversation messages
conversation.messages?.forEach((message, index) => {
  console.log(`${index + 1}. [${message.role}]: ${message.content}`);
});

Streaming Conversations

// Stream a real-time conversation
const conversationStream = await client.conversationalAi.agents.simulateConversationStream(
  "agent_id_here",
  {
    simulation_specification: {
      simulated_user_config: {
        first_message: "Hello, I need help with my order",
        system_prompt: "You are a customer with an issue about a recent order"
      },
      conversation_config: {
        max_turns: 5,
        streaming: true
      }
    }
  }
);

// Process streaming responses
for await (const chunk of conversationStream) {
  if (chunk.type === 'message') {
    console.log(`[${chunk.role}]: ${chunk.content}`);
  } else if (chunk.type === 'audio') {
    // Handle audio data for real-time playback
    const audioBuffer = Buffer.from(chunk.audio_base64, 'base64');
    playAudio(audioBuffer);
  }
}

Knowledge Base Management

Create Knowledge Base

// Create a knowledge base for agent context
const knowledgeBase = await client.conversationalAi.knowledgeBase.createKnowledgeBase({
  name: "Product Support KB",
  description: "Comprehensive knowledge base for product support agents",
  is_public: false
});

console.log('Knowledge base created:', knowledgeBase.knowledge_base_id);

Add Documents to Knowledge Base

import * as fs from 'fs';

// Add text document
const textDoc = await client.conversationalAi.knowledgeBase.addTextDocument({
  knowledge_base_id: knowledgeBase.knowledge_base_id,
  name: "Product FAQ",
  text: `
    # Frequently Asked Questions
    
    ## Shipping Information
    Q: How long does shipping take?
    A: Standard shipping takes 3-5 business days, Express shipping takes 1-2 business days.
    
    ## Return Policy
    Q: What is your return policy?
    A: We accept returns within 30 days of purchase for unused items in original packaging.
    
    ## Technical Support
    Q: How do I contact technical support?
    A: You can reach technical support at support@company.com or call 1-800-SUPPORT.
  `,
  metadata: {
    category: "faq",
    last_updated: new Date().toISOString(),
    version: "1.0"
  }
});

// Add file document (PDF, DOCX, etc.)
const fileDoc = await client.conversationalAi.knowledgeBase.addFileDocument({
  knowledge_base_id: knowledgeBase.knowledge_base_id,
  name: "User Manual",
  file: fs.createReadStream('user_manual.pdf'),
  metadata: {
    document_type: "manual",
    product_line: "electronics"
  }
});

// Add URL document (web scraping)
const urlDoc = await client.conversationalAi.knowledgeBase.addUrlDocument({
  knowledge_base_id: knowledgeBase.knowledge_base_id,
  name: "Company Blog Posts",
  url: "https://company.com/blog/product-updates",
  css_selector: "article", // Optional: specific content selector
  metadata: {
    source: "company_blog",
    auto_update: true
  }
});

console.log('Documents added to knowledge base');

Manage Knowledge Base Documents

// List all knowledge bases
const knowledgeBases = await client.conversationalAi.knowledgeBase.getKnowledgeBases({
  page_size: 50
});

console.log('Available knowledge bases:', knowledgeBases.knowledge_bases.length);

// Get specific knowledge base
const kbDetails = await client.conversationalAi.knowledgeBase.getKnowledgeBase(
  knowledgeBase.knowledge_base_id
);

console.log('Knowledge base documents:', kbDetails.documents?.length);

// Update document
await client.conversationalAi.knowledgeBase.updateDocument(
  "document_id_here",
  {
    name: "Updated Document Name",
    text: "Updated document content with new information...",
    metadata: {
      version: "2.0",
      last_updated: new Date().toISOString()
    }
  }
);

// Delete document
await client.conversationalAi.knowledgeBase.deleteDocument("document_id_here");

// Delete entire knowledge base
await client.conversationalAi.knowledgeBase.deleteKnowledgeBase(
  knowledgeBase.knowledge_base_id
);

Connect Knowledge Base to Agent

// Update agent to use knowledge base
const agentWithKB = await client.conversationalAi.agents.updateAgent(
  "agent_id_here",
  {
    knowledge_base_ids: [knowledgeBase.knowledge_base_id],
    system_prompt: `You are a customer support agent with access to our product knowledge base.
      Use the knowledge base to answer customer questions accurately.
      If you cannot find the answer in the knowledge base, clearly state that and offer to escalate.`,
    conversation_config: {
      agent: {
        knowledge_base_config: {
          retrieval_method: "semantic_search",
          max_documents: 5,
          relevance_threshold: 0.7
        }
      }
    }
  }
);

SIP Trunk and Telephony

SIP Trunk Configuration

// Configure SIP trunk for telephony integration
const sipTrunk = await client.conversationalAi.sipTrunk.createSipTrunk({
  name: "Customer Support Line",
  sip_trunk_config: {
    provider: "twilio", // or other SIP providers
    authentication: {
      username: "your_sip_username",
      password: "your_sip_password",
      realm: "sip.provider.com"
    },
    server_config: {
      host: "sip.provider.com",
      port: 5060,
      transport: "UDP"
    }
  }
});

console.log('SIP trunk created:', sipTrunk.sip_trunk_id);

Handle Inbound Calls

// Configure agent to handle inbound phone calls
const phoneAgent = await client.conversationalAi.agents.createAgent({
  name: "Phone Support Agent",
  voice_id: "pNInz6obpgDQGcFmaJgB",
  language: "en",
  system_prompt: `You are a phone support agent. 
    Customers are calling for technical support.
    Be patient and speak clearly since this is a phone conversation.
    Always confirm important information by spelling it out.`,
  
  conversation_config: {
    agent: {
      first_message: "Thank you for calling TechCorp support. My name is AI Assistant. How can I help you today?",
      
      // Phone-optimized settings
      tts_config: {
        voice_id: "pNInz6obpgDQGcFmaJgB",
        model_id: "eleven_turbo_v2_5",
        voice_settings: {
          stability: 0.8, // More stable for phone quality
          similarity_boost: 0.7,
          style: 0.1, // Less expressive for clarity
          speed: 0.9 // Slightly slower for phone
        },
        output_format: "ulaw_8000" // Telephony standard format
      },
      
      // Phone-optimized STT
      stt_config: {
        model: "whisper-1",
        language: "en",
        enhance_audio: true // Improve phone audio quality
      }
    },
    
    conversation: {
      max_duration_seconds: 1800, // 30 min phone call limit
      turn_detection: {
        type: "server_vad",
        threshold: 0.7, // Higher threshold for phone noise
        prefix_padding_ms: 500,
        suffix_padding_ms: 1000 // Longer pause detection for phone
      }
    }
  },
  
  // Connect to SIP trunk
  sip_trunk_id: sipTrunk.sip_trunk_id
});

Make Outbound Calls

// Initiate outbound call via SIP trunk
const outboundCall = await client.conversationalAi.sipTrunk.makeOutboundCall({
  agent_id: phoneAgent.agent_id,
  phone_number: "+1234567890",
  from_number: "+1987654321",
  custom_llm_extra_body: {
    customer_context: {
      name: "John Smith",
      account_id: "ACC123456",
      recent_orders: ["ORDER789", "ORDER456"],
      preferred_contact_method: "phone"
    }
  },
  max_duration_seconds: 600
});

console.log('Outbound call initiated:', outboundCall.call_id);

Call Analytics and Management

// Get call history and analytics
const callHistory = await client.conversationalAi.sipTrunk.getCallHistory({
  agent_id: phoneAgent.agent_id,
  start_date: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000), // Last 7 days
  end_date: new Date(),
  page_size: 100
});

console.log('Recent calls:', callHistory.calls.length);

callHistory.calls.forEach(call => {
  console.log(`Call ${call.call_id}:`);
  console.log(`  Duration: ${call.duration_seconds}s`);
  console.log(`  Status: ${call.status}`);
  console.log(`  Phone: ${call.phone_number}`);
  console.log(`  Started: ${new Date(call.start_time).toLocaleString()}`);
});

// Get specific call details
const callDetails = await client.conversationalAi.sipTrunk.getCall(
  outboundCall.call_id
);

console.log('Call transcript:', callDetails.transcript);
console.log('Call recording URL:', callDetails.recording_url);

Advanced Agent Features

Multi-Language Support

// Create multilingual agent
const multilingualAgent = await client.conversationalAi.agents.createAgent({
  name: "Global Support Agent",
  voice_id: "pNInz6obpgDQGcFmaJgB",
  language: "auto", // Auto-detect language
  system_prompt: `You are a multilingual support agent.
    Detect the customer's language and respond in the same language.
    You can communicate in English, Spanish, French, German, and Italian.
    Always maintain a helpful and professional tone regardless of language.`,
  
  conversation_config: {
    agent: {
      multilingual_config: {
        enabled: true,
        supported_languages: ["en", "es", "fr", "de", "it"],
        auto_detect: true,
        fallback_language: "en"
      },
      
      tts_config: {
        voice_id: "multilingual_voice_id",
        model_id: "eleven_multilingual_v2"
      }
    }
  }
});

Custom Function Integration

// Agent with custom function calling capabilities
const functionAgent = await client.conversationalAi.agents.createAgent({
  name: "E-commerce Assistant",
  voice_id: "21m00Tcm4TlvDq8ikWAM",
  language: "en",
  system_prompt: `You are an e-commerce assistant with access to order management functions.
    You can check order status, process returns, and update shipping addresses.
    Always confirm actions with customers before executing them.`,
  
  conversation_config: {
    agent: {
      function_calling_config: {
        enabled: true,
        functions: [
          {
            name: "check_order_status",
            description: "Check the status of a customer order",
            parameters: {
              type: "object",
              properties: {
                order_id: {
                  type: "string",
                  description: "The order ID to check"
                }
              },
              required: ["order_id"]
            }
          },
          {
            name: "initiate_return",
            description: "Start the return process for an order",
            parameters: {
              type: "object",
              properties: {
                order_id: { type: "string" },
                reason: { type: "string" },
                items: {
                  type: "array",
                  items: { type: "string" }
                }
              },
              required: ["order_id", "reason"]
            }
          }
        ]
      }
    }
  }
});

Real-time Agent Monitoring

// Monitor agent conversations in real-time
class AgentMonitor {
  private activeConversations = new Map<string, any>();
  
  async startMonitoring(agentId: string) {
    // Set up WebSocket connection to monitor agent
    const monitoringWs = new WebSocket(
      `wss://api.elevenlabs.io/v1/convai/agents/${agentId}/monitor`,
      { headers: { 'xi-api-key': process.env.ELEVENLABS_API_KEY } }
    );
    
    monitoringWs.on('message', (data) => {
      const event = JSON.parse(data.toString());
      this.handleAgentEvent(event);
    });
  }
  
  private handleAgentEvent(event: any) {
    switch (event.type) {
      case 'conversation_started':
        console.log(`New conversation: ${event.conversation_id}`);
        this.activeConversations.set(event.conversation_id, {
          startTime: new Date(),
          agentId: event.agent_id,
          participantCount: 1
        });
        break;
        
      case 'message_sent':
        console.log(`Agent message: ${event.message.substring(0, 50)}...`);
        break;
        
      case 'message_received':
        console.log(`User message: ${event.message.substring(0, 50)}...`);
        break;
        
      case 'conversation_ended':
        const conversation = this.activeConversations.get(event.conversation_id);
        if (conversation) {
          const duration = Date.now() - conversation.startTime.getTime();
          console.log(`Conversation ended after ${duration / 1000}s`);
          this.activeConversations.delete(event.conversation_id);
        }
        break;
        
      case 'error':
        console.error('Agent error:', event.error);
        break;
    }
  }
  
  getActiveConversations() {
    return Array.from(this.activeConversations.entries()).map(([id, data]) => ({
      conversationId: id,
      duration: Date.now() - data.startTime.getTime(),
      agentId: data.agentId
    }));
  }
}

// Usage
const monitor = new AgentMonitor();
await monitor.startMonitoring("agent_id_here");

Error Handling and Best Practices

import { ElevenLabsError, ElevenLabsTimeoutError } from 'elevenlabs';

try {
  const agent = await client.conversationalAi.agents.createAgent({
    name: "Test Agent",
    voice_id: "invalid_voice_id", // This will cause an error
    language: "en"
  });
  
} catch (error) {
  if (error instanceof ElevenLabsError) {
    console.error('Agent creation error:', error.statusCode);
    
    if (error.statusCode === 400) {
      console.error('Invalid parameters:', error.body);
    } else if (error.statusCode === 401) {
      console.error('Authentication error - check API key');
    } else if (error.statusCode === 403) {
      console.error('Insufficient permissions for conversational AI');
    } else if (error.statusCode === 422) {
      console.error('Validation error:', error.body);
    }
  }
}

// Robust conversation handling
async function handleConversation(agentId: string, userId: string) {
  try {
    const conversation = await client.conversationalAi.agents.simulateConversation(
      agentId,
      {
        simulation_specification: {
          simulated_user_config: {
            first_message: "Hello, I need assistance"
          }
        }
      }
    );
    
    return conversation;
    
  } catch (error) {
    console.error(`Conversation error for user ${userId}:`, error);
    
    // Implement fallback strategy
    return {
      fallback: true,
      message: "I'm sorry, I'm experiencing technical difficulties. Please try again later or contact human support."
    };
  }
}

Integration Examples

Customer Support Workflow

// Complete customer support agent setup
async function setupCustomerSupportAgent() {
  // 1. Create knowledge base
  const supportKB = await client.conversationalAi.knowledgeBase.createKnowledgeBase({
    name: "Customer Support Knowledge Base"
  });
  
  // 2. Add support documents
  await client.conversationalAi.knowledgeBase.addTextDocument({
    knowledge_base_id: supportKB.knowledge_base_id,
    name: "Common Issues",
    text: "Troubleshooting guide for common customer issues..."
  });
  
  // 3. Create support agent
  const supportAgent = await client.conversationalAi.agents.createAgent({
    name: "AI Customer Support",
    voice_id: "pNInz6obpgDQGcFmaJgB",
    language: "en",
    knowledge_base_ids: [supportKB.knowledge_base_id],
    system_prompt: `You are a customer support agent with access to our knowledge base.
      Help customers resolve issues, answer questions, and escalate complex problems.`
  });
  
  // 4. Set up phone integration
  const sipTrunk = await client.conversationalAi.sipTrunk.createSipTrunk({
    name: "Support Hotline"
  });
  
  return { supportAgent, supportKB, sipTrunk };
}

Best Practices

  1. Agent Design: Create focused agents with clear, specific system prompts
  2. Voice Selection: Choose appropriate voices for your use case (professional, friendly, etc.)
  3. Knowledge Base: Keep knowledge base content up-to-date and well-organized
  4. Phone Integration: Use telephony-optimized settings for SIP trunk calls
  5. Error Handling: Implement comprehensive error handling and fallback strategies
  6. Monitoring: Monitor agent performance and conversation quality
  7. Language Support: Configure appropriate language settings for your audience
  8. Function Calling: Use custom functions to integrate with external systems
  9. Security: Implement proper authentication and access controls
  10. Testing: Thoroughly test agents before deployment in production environments