or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

Files

docs

ai-integrations.mdclient-management.mdcontext-management.mdevent-tracking.mdfeature-flags.mdindex.mduser-group-management.md

ai-integrations.mddocs/

0

# AI Integrations

1

2

LLM provider integrations with automatic usage tracking, cost monitoring, and performance analytics. PostHog's AI integrations provide drop-in replacements for popular LLM clients that automatically capture usage metrics, costs, and performance data while maintaining full API compatibility.

3

4

## Capabilities

5

6

### OpenAI Integration

7

8

Drop-in replacement for OpenAI clients with automatic tracking of usage, costs, and performance metrics.

9

10

```python { .api }

11

class OpenAI:

12

"""

13

PostHog-wrapped OpenAI client with automatic usage tracking.

14

15

Provides identical API to openai.OpenAI with added PostHog telemetry.

16

Automatically tracks token usage, costs, model performance, and errors.

17

"""

18

19

class AsyncOpenAI:

20

"""

21

PostHog-wrapped async OpenAI client with automatic usage tracking.

22

23

Async version of OpenAI wrapper with identical API to openai.AsyncOpenAI.

24

"""

25

26

class AzureOpenAI:

27

"""

28

PostHog-wrapped Azure OpenAI client with automatic usage tracking.

29

30

Provides identical API to openai.AzureOpenAI with added PostHog telemetry.

31

"""

32

33

class AsyncAzureOpenAI:

34

"""

35

PostHog-wrapped async Azure OpenAI client with automatic usage tracking.

36

37

Async version of Azure OpenAI wrapper.

38

"""

39

40

# Utility functions for OpenAI data formatting

41

def format_openai_response(response) -> dict:

42

"""Format OpenAI response for PostHog tracking"""

43

44

def format_openai_input(input) -> dict:

45

"""Format OpenAI input for PostHog tracking"""

46

47

def extract_openai_tools(tools) -> list:

48

"""Extract tools from OpenAI request for tracking"""

49

50

def format_openai_streaming_content(content) -> dict:

51

"""Format OpenAI streaming content for tracking"""

52

```

53

54

### Anthropic Integration

55

56

Drop-in replacement for Anthropic clients with comprehensive usage tracking and cost monitoring.

57

58

```python { .api }

59

class Anthropic:

60

"""

61

PostHog-wrapped Anthropic client with automatic usage tracking.

62

63

Provides identical API to anthropic.Anthropic with added PostHog telemetry.

64

"""

65

66

class AsyncAnthropic:

67

"""

68

PostHog-wrapped async Anthropic client with automatic usage tracking.

69

70

Async version of Anthropic wrapper with identical API to anthropic.AsyncAnthropic.

71

"""

72

73

class AnthropicBedrock:

74

"""

75

PostHog-wrapped Anthropic Bedrock client with automatic usage tracking.

76

77

For use with AWS Bedrock Anthropic models.

78

"""

79

80

class AsyncAnthropicBedrock:

81

"""

82

PostHog-wrapped async Anthropic Bedrock client with automatic usage tracking.

83

"""

84

85

class AnthropicVertex:

86

"""

87

PostHog-wrapped Anthropic Vertex AI client with automatic usage tracking.

88

89

For use with Google Cloud Vertex AI Anthropic models.

90

"""

91

92

class AsyncAnthropicVertex:

93

"""

94

PostHog-wrapped async Anthropic Vertex AI client with automatic usage tracking.

95

"""

96

97

# Utility functions for Anthropic data formatting

98

def format_anthropic_response(response) -> dict:

99

"""Format Anthropic response for PostHog tracking"""

100

101

def format_anthropic_input(input) -> dict:

102

"""Format Anthropic input for PostHog tracking"""

103

104

def extract_anthropic_tools(tools) -> list:

105

"""Extract tools from Anthropic request for tracking"""

106

107

def format_anthropic_streaming_content(content) -> dict:

108

"""Format Anthropic streaming content for tracking"""

109

```

110

111

### Gemini Integration

112

113

Google Gemini client wrapper with automatic usage tracking and performance monitoring.

114

115

```python { .api }

116

class Client:

117

"""

118

PostHog-wrapped Gemini client with automatic usage tracking.

119

120

Provides comprehensive tracking for Google Gemini model interactions.

121

Drop-in replacement for Google's genai.Client.

122

"""

123

124

# genai module compatibility

125

genai = _GenAI() # Contains Client for drop-in replacement

126

127

# Utility functions for Gemini data formatting

128

def format_gemini_response(response) -> dict:

129

"""Format Gemini response for PostHog tracking"""

130

131

def format_gemini_input(input) -> dict:

132

"""Format Gemini input for PostHog tracking"""

133

134

def extract_gemini_tools(tools) -> list:

135

"""Extract tools from Gemini request for tracking"""

136

```

137

138

## Usage Examples

139

140

### Langchain Integration

141

142

Langchain callback handler for comprehensive tracking of complex AI workflows and chains.

143

144

```python { .api }

145

class CallbackHandler:

146

"""

147

PostHog callback handler for Langchain applications.

148

149

Automatically tracks:

150

- Chain executions and performance

151

- LLM calls and token usage

152

- Tool usage and results

153

- Agent actions and decisions

154

- Error handling and debugging info

155

156

Compatible with all Langchain components including chains, agents, and tools.

157

"""

158

```

159

160

## Usage Examples

161

162

### OpenAI Integration

163

164

```python

165

from posthog.ai.openai import OpenAI

166

import posthog

167

168

# Configure PostHog

169

posthog.api_key = 'phc_your_project_api_key'

170

171

# Create OpenAI client with PostHog tracking

172

client = OpenAI(

173

api_key="your-openai-api-key",

174

# All standard OpenAI parameters supported

175

)

176

177

# Use exactly like standard OpenAI client

178

response = client.chat.completions.create(

179

model="gpt-4",

180

messages=[

181

{"role": "user", "content": "Hello, how are you?"}

182

],

183

temperature=0.7,

184

max_tokens=150

185

)

186

187

print(response.choices[0].message.content)

188

189

# PostHog automatically tracks:

190

# - Token usage (prompt + completion tokens)

191

# - Model and parameters used

192

# - Response time and latency

193

# - Costs (when available)

194

# - Errors and failures

195

```

196

197

### Async OpenAI Usage

198

199

```python

200

from posthog.ai.openai import AsyncOpenAI

201

import asyncio

202

import posthog

203

204

posthog.api_key = 'phc_your_project_api_key'

205

206

async def main():

207

client = AsyncOpenAI(api_key="your-openai-api-key")

208

209

response = await client.chat.completions.create(

210

model="gpt-3.5-turbo",

211

messages=[

212

{"role": "user", "content": "Write a haiku about programming"}

213

]

214

)

215

216

print(response.choices[0].message.content)

217

218

asyncio.run(main())

219

```

220

221

### Azure OpenAI Integration

222

223

```python

224

from posthog.ai.openai import AzureOpenAI

225

import posthog

226

227

posthog.api_key = 'phc_your_project_api_key'

228

229

client = AzureOpenAI(

230

azure_endpoint="https://your-resource.openai.azure.com/",

231

api_key="your-azure-openai-key",

232

api_version="2024-02-01"

233

)

234

235

response = client.chat.completions.create(

236

model="gpt-4", # Your deployment name

237

messages=[

238

{"role": "user", "content": "Explain quantum computing"}

239

]

240

)

241

242

print(response.choices[0].message.content)

243

```

244

245

### Anthropic Integration

246

247

```python

248

from posthog.ai.anthropic import Anthropic

249

import posthog

250

251

posthog.api_key = 'phc_your_project_api_key'

252

253

# Create Anthropic client with PostHog tracking

254

client = Anthropic(

255

api_key="your-anthropic-api-key"

256

)

257

258

# Use exactly like standard Anthropic client

259

response = client.messages.create(

260

model="claude-3-opus-20240229",

261

max_tokens=1000,

262

messages=[

263

{"role": "user", "content": "Explain the theory of relativity"}

264

]

265

)

266

267

print(response.content[0].text)

268

269

# Automatic tracking includes:

270

# - Token usage and costs

271

# - Model performance metrics

272

# - Response quality indicators

273

# - Error rates and types

274

```

275

276

### Anthropic Bedrock Integration

277

278

```python

279

from posthog.ai.anthropic import AnthropicBedrock

280

import posthog

281

282

posthog.api_key = 'phc_your_project_api_key'

283

284

client = AnthropicBedrock(

285

aws_access_key="your-aws-access-key",

286

aws_secret_key="your-aws-secret-key",

287

aws_region="us-east-1"

288

)

289

290

response = client.messages.create(

291

model="anthropic.claude-3-sonnet-20240229-v1:0",

292

max_tokens=1000,

293

messages=[

294

{"role": "user", "content": "What is machine learning?"}

295

]

296

)

297

298

print(response.content[0].text)

299

```

300

301

### Gemini Integration

302

303

```python

304

from posthog.ai.gemini import Client, genai

305

import posthog

306

307

posthog.api_key = 'phc_your_project_api_key'

308

309

# Configure Gemini with PostHog tracking

310

genai.configure(api_key="your-gemini-api-key")

311

312

# Create client with PostHog tracking

313

client = genai.GenerativeModel("gemini-pro")

314

315

# Use exactly like standard Gemini client

316

response = client.generate_content("Explain quantum computing")

317

print(response.text)

318

319

# PostHog automatically tracks:

320

# - Token usage and costs

321

# - Model performance metrics

322

# - Response quality indicators

323

# - Error rates and types

324

```

325

326

### Streaming Responses

327

328

```python

329

from posthog.ai.openai import OpenAI

330

import posthog

331

332

posthog.api_key = 'phc_your_project_api_key'

333

client = OpenAI(api_key="your-openai-api-key")

334

335

# Streaming is fully supported with automatic tracking

336

stream = client.chat.completions.create(

337

model="gpt-4",

338

messages=[

339

{"role": "user", "content": "Write a story about AI"}

340

],

341

stream=True

342

)

343

344

full_response = ""

345

for chunk in stream:

346

if chunk.choices[0].delta.content is not None:

347

content = chunk.choices[0].delta.content

348

print(content, end="")

349

full_response += content

350

351

# PostHog tracks complete streaming session including:

352

# - Total tokens used

353

# - Streaming latency and throughput

354

# - Time to first token

355

# - Complete response assembly

356

```

357

358

### Langchain Integration

359

360

```python

361

from posthog.ai.langchain import CallbackHandler

362

from langchain.llms import OpenAI

363

from langchain.chains import LLMChain

364

from langchain.prompts import PromptTemplate

365

import posthog

366

367

posthog.api_key = 'phc_your_project_api_key'

368

369

# Create PostHog callback handler

370

posthog_handler = CallbackHandler()

371

372

# Set up Langchain components

373

llm = OpenAI(temperature=0.7)

374

prompt = PromptTemplate(

375

input_variables=["topic"],

376

template="Write a short article about {topic}"

377

)

378

379

chain = LLMChain(llm=llm, prompt=prompt)

380

381

# Run chain with PostHog tracking

382

result = chain.run(

383

topic="artificial intelligence",

384

callbacks=[posthog_handler]

385

)

386

387

print(result)

388

389

# PostHog automatically tracks:

390

# - Chain execution time and success

391

# - LLM calls and token usage

392

# - Prompt templates and variables

393

# - Output quality and length

394

# - Error handling and retries

395

```

396

397

### Advanced Langchain Usage

398

399

```python

400

from posthog.ai.langchain import CallbackHandler

401

from langchain.agents import initialize_agent, Tool

402

from langchain.agents import AgentType

403

from langchain.llms import OpenAI

404

import posthog

405

406

posthog.api_key = 'phc_your_project_api_key'

407

408

# Custom tools

409

def calculator(expression):

410

"""Calculate mathematical expressions"""

411

try:

412

return str(eval(expression))

413

except:

414

return "Error in calculation"

415

416

def search_tool(query):

417

"""Mock search tool"""

418

return f"Search results for: {query}"

419

420

tools = [

421

Tool(

422

name="Calculator",

423

func=calculator,

424

description="Use for mathematical calculations"

425

),

426

Tool(

427

name="Search",

428

func=search_tool,

429

description="Use for web searches"

430

)

431

]

432

433

# Create agent with PostHog tracking

434

llm = OpenAI(temperature=0)

435

agent = initialize_agent(

436

tools,

437

llm,

438

agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,

439

verbose=True

440

)

441

442

# Create callback handler

443

posthog_handler = CallbackHandler()

444

445

# Run agent with comprehensive tracking

446

result = agent.run(

447

"What is 15 * 27, and then search for information about that number?",

448

callbacks=[posthog_handler]

449

)

450

451

print(result)

452

453

# Tracks agent reasoning, tool usage, and decision making

454

```

455

456

### Context and User Tracking

457

458

```python

459

from posthog.ai.openai import OpenAI

460

import posthog

461

462

posthog.api_key = 'phc_your_project_api_key'

463

464

# Use context for user-specific AI tracking

465

with posthog.new_context():

466

posthog.identify_context('user123')

467

posthog.tag('session_type', 'premium')

468

posthog.tag('use_case', 'content_generation')

469

470

client = OpenAI(api_key="your-openai-api-key")

471

472

# AI usage automatically associated with user and context

473

response = client.chat.completions.create(

474

model="gpt-4",

475

messages=[

476

{"role": "user", "content": "Help me write a blog post"}

477

]

478

)

479

480

# Additional manual tracking

481

posthog.capture('ai_content_generated', {

482

'content_type': 'blog_post',

483

'word_count': len(response.choices[0].message.content.split()),

484

'satisfaction': 'high'

485

})

486

```

487

488

### Error Handling and Monitoring

489

490

```python

491

from posthog.ai.openai import OpenAI

492

import posthog

493

494

posthog.api_key = 'phc_your_project_api_key'

495

client = OpenAI(api_key="your-openai-api-key")

496

497

def safe_ai_call(messages, **kwargs):

498

"""Wrapper for AI calls with comprehensive error tracking"""

499

try:

500

with posthog.new_context():

501

posthog.tag('ai_operation', 'chat_completion')

502

posthog.tag('model', kwargs.get('model', 'gpt-3.5-turbo'))

503

504

response = client.chat.completions.create(

505

messages=messages,

506

**kwargs

507

)

508

509

# Track successful usage

510

posthog.capture('ai_call_success', {

511

'tokens_used': response.usage.total_tokens,

512

'model': response.model,

513

'response_length': len(response.choices[0].message.content)

514

})

515

516

return response

517

518

except Exception as e:

519

# Error is automatically captured by context

520

posthog.capture('ai_call_failed', {

521

'error_type': type(e).__name__,

522

'error_message': str(e),

523

'model': kwargs.get('model', 'unknown')

524

})

525

raise

526

527

# Usage with error tracking

528

try:

529

response = safe_ai_call(

530

messages=[{"role": "user", "content": "Hello"}],

531

model="gpt-4",

532

max_tokens=100

533

)

534

except Exception as e:

535

print(f"AI call failed: {e}")

536

```

537

538

### Privacy Mode

539

540

```python

541

from posthog.ai.openai import OpenAI

542

import posthog

543

544

# Enable privacy mode to exclude prompts and responses from tracking

545

posthog.api_key = 'phc_your_project_api_key'

546

posthog.privacy_mode = True

547

548

client = OpenAI(api_key="your-openai-api-key")

549

550

# With privacy mode enabled, only usage metadata is tracked:

551

# - Token counts

552

# - Model used

553

# - Response times

554

# - Costs

555

# - Error rates

556

#

557

# But NOT:

558

# - Prompt content

559

# - Response content

560

# - User data in messages

561

562

response = client.chat.completions.create(

563

model="gpt-4",

564

messages=[

565

{"role": "user", "content": "Sensitive information here"}

566

]

567

)

568

569

# Only metadata tracked, content is not sent to PostHog

570

```

571

572

### Cost Tracking and Budgets

573

574

```python

575

from posthog.ai.openai import OpenAI

576

import posthog

577

578

posthog.api_key = 'phc_your_project_api_key'

579

580

class CostTracker:

581

def __init__(self, budget_limit=100.0):

582

self.total_cost = 0.0

583

self.budget_limit = budget_limit

584

self.client = OpenAI(api_key="your-openai-api-key")

585

586

def track_usage(self, response):

587

# Calculate approximate cost (varies by model)

588

model_costs = {

589

'gpt-4': {'input': 0.03, 'output': 0.06}, # per 1K tokens

590

'gpt-3.5-turbo': {'input': 0.0015, 'output': 0.002}

591

}

592

593

model = response.model

594

usage = response.usage

595

596

if model in model_costs:

597

input_cost = (usage.prompt_tokens / 1000) * model_costs[model]['input']

598

output_cost = (usage.completion_tokens / 1000) * model_costs[model]['output']

599

total_cost = input_cost + output_cost

600

601

self.total_cost += total_cost

602

603

# Track cost metrics

604

posthog.capture('ai_cost_tracking', {

605

'session_cost': total_cost,

606

'total_cost': self.total_cost,

607

'budget_remaining': self.budget_limit - self.total_cost,

608

'model': model,

609

'tokens_used': usage.total_tokens

610

})

611

612

# Budget alerts

613

if self.total_cost > self.budget_limit * 0.8:

614

posthog.capture('ai_budget_alert', {

615

'alert_type': 'approaching_limit',

616

'usage_percentage': (self.total_cost / self.budget_limit) * 100

617

})

618

619

def generate_text(self, messages, **kwargs):

620

if self.total_cost >= self.budget_limit:

621

raise Exception("Budget limit exceeded")

622

623

response = self.client.chat.completions.create(

624

messages=messages,

625

**kwargs

626

)

627

628

self.track_usage(response)

629

return response

630

631

# Usage

632

tracker = CostTracker(budget_limit=50.0)

633

634

response = tracker.generate_text(

635

messages=[{"role": "user", "content": "Write a summary"}],

636

model="gpt-4"

637

)

638

```

639

640

## Advanced Features

641

642

### Custom Metrics and Analysis

643

644

```python

645

from posthog.ai.openai import OpenAI

646

import posthog

647

import time

648

649

class AIAnalytics:

650

def __init__(self):

651

self.client = OpenAI(api_key="your-openai-api-key")

652

653

def analyze_response_quality(self, prompt, response_text):

654

"""Analyze response quality metrics"""

655

return {

656

'length': len(response_text),

657

'sentences': response_text.count('.'),

658

'complexity_score': len(set(response_text.split())) / len(response_text.split()),

659

'prompt_relevance': self._calculate_relevance(prompt, response_text)

660

}

661

662

def _calculate_relevance(self, prompt, response):

663

# Simple relevance calculation

664

prompt_words = set(prompt.lower().split())

665

response_words = set(response.lower().split())

666

overlap = len(prompt_words.intersection(response_words))

667

return overlap / len(prompt_words) if prompt_words else 0

668

669

def generate_with_analytics(self, messages, **kwargs):

670

start_time = time.time()

671

672

with posthog.new_context():

673

posthog.tag('ai_analytics', True)

674

675

response = self.client.chat.completions.create(

676

messages=messages,

677

**kwargs

678

)

679

680

end_time = time.time()

681

response_time = end_time - start_time

682

683

# Analyze response quality

684

quality_metrics = self.analyze_response_quality(

685

messages[-1]['content'],

686

response.choices[0].message.content

687

)

688

689

# Track comprehensive analytics

690

posthog.capture('ai_detailed_analytics', {

691

'response_time': response_time,

692

'model': response.model,

693

'tokens_per_second': response.usage.total_tokens / response_time,

694

**quality_metrics

695

})

696

697

return response

698

699

# Usage

700

analytics = AIAnalytics()

701

response = analytics.generate_with_analytics(

702

messages=[{"role": "user", "content": "Explain machine learning"}],

703

model="gpt-4"

704

)

705

```

706

707

### A/B Testing AI Models

708

709

```python

710

from posthog.ai.openai import OpenAI

711

import posthog

712

import random

713

714

class ModelABTester:

715

def __init__(self):

716

self.client = OpenAI(api_key="your-openai-api-key")

717

self.models = {

718

'gpt-4': {'weight': 0.3, 'cost_per_token': 0.00003},

719

'gpt-3.5-turbo': {'weight': 0.7, 'cost_per_token': 0.000002}

720

}

721

722

def select_model(self, user_id):

723

"""Select model based on A/B test configuration"""

724

# Use PostHog feature flags for A/B testing

725

variant = posthog.get_feature_flag('ai_model_test', user_id)

726

727

if variant == 'premium':

728

return 'gpt-4'

729

elif variant == 'standard':

730

return 'gpt-3.5-turbo'

731

else:

732

# Fallback to weighted random selection

733

return random.choices(

734

list(self.models.keys()),

735

weights=[m['weight'] for m in self.models.values()]

736

)[0]

737

738

def generate_with_testing(self, user_id, messages, **kwargs):

739

model = self.select_model(user_id)

740

741

with posthog.new_context():

742

posthog.identify_context(user_id)

743

posthog.tag('ab_test', 'ai_model_test')

744

posthog.tag('model_variant', model)

745

746

response = self.client.chat.completions.create(

747

model=model,

748

messages=messages,

749

**kwargs

750

)

751

752

# Track A/B test metrics

753

posthog.capture('ai_ab_test_result', {

754

'model_used': model,

755

'tokens_used': response.usage.total_tokens,

756

'estimated_cost': response.usage.total_tokens * self.models[model]['cost_per_token'],

757

'response_quality': len(response.choices[0].message.content)

758

})

759

760

return response

761

762

# Usage

763

tester = ModelABTester()

764

response = tester.generate_with_testing(

765

'user123',

766

messages=[{"role": "user", "content": "Help me write code"}]

767

)

768

```

769

770

## Best Practices

771

772

### Performance Optimization

773

774

```python

775

# Use appropriate models for different use cases

776

quick_client = OpenAI() # For simple tasks

777

advanced_client = OpenAI() # For complex reasoning

778

779

# Simple tasks

780

simple_response = quick_client.chat.completions.create(

781

model="gpt-3.5-turbo",

782

messages=[{"role": "user", "content": "Summarize this text"}],

783

max_tokens=100

784

)

785

786

# Complex tasks

787

complex_response = advanced_client.chat.completions.create(

788

model="gpt-4",

789

messages=[{"role": "user", "content": "Analyze this complex problem"}],

790

max_tokens=1000

791

)

792

```

793

794

### Error Handling

795

796

```python

797

from posthog.ai.openai import OpenAI

798

import posthog

799

800

client = OpenAI(api_key="your-openai-api-key")

801

802

def robust_ai_call(messages, max_retries=3, **kwargs):

803

for attempt in range(max_retries):

804

try:

805

with posthog.new_context():

806

posthog.tag('attempt', attempt + 1)

807

808

return client.chat.completions.create(

809

messages=messages,

810

**kwargs

811

)

812

except Exception as e:

813

posthog.capture('ai_retry', {

814

'attempt': attempt + 1,

815

'error': str(e),

816

'max_retries': max_retries

817

})

818

819

if attempt == max_retries - 1:

820

raise

821

822

time.sleep(2 ** attempt) # Exponential backoff

823

824

# Usage with automatic retry and tracking

825

response = robust_ai_call(

826

messages=[{"role": "user", "content": "Hello"}],

827

model="gpt-4"

828

)

829

```

830

831

### Data Privacy and Security

832

833

```python

834

import posthog

835

from posthog.ai.openai import OpenAI

836

837

# Enable privacy mode globally

838

posthog.privacy_mode = True

839

840

# Or use environment-specific configuration

841

import os

842

if os.getenv('ENVIRONMENT') == 'production':

843

posthog.privacy_mode = True

844

845

client = OpenAI(api_key="your-openai-api-key")

846

847

# With privacy mode, only metadata is tracked

848

response = client.chat.completions.create(

849

model="gpt-4",

850

messages=[{"role": "user", "content": "Confidential information"}]

851

)

852

853

# Manual tracking of non-sensitive metrics

854

posthog.capture('ai_usage_summary', {

855

'model': 'gpt-4',

856

'tokens_used': response.usage.total_tokens,

857

'use_case': 'content_generation',

858

'user_satisfaction': 'high' # Can be collected separately

859

})

860

```