0
# AI Integrations
1
2
LLM provider integrations with automatic usage tracking, cost monitoring, and performance analytics. PostHog's AI integrations provide drop-in replacements for popular LLM clients that automatically capture usage metrics, costs, and performance data while maintaining full API compatibility.
3
4
## Capabilities
5
6
### OpenAI Integration
7
8
Drop-in replacement for OpenAI clients with automatic tracking of usage, costs, and performance metrics.
9
10
```python { .api }
11
class OpenAI:
12
"""
13
PostHog-wrapped OpenAI client with automatic usage tracking.
14
15
Provides identical API to openai.OpenAI with added PostHog telemetry.
16
Automatically tracks token usage, costs, model performance, and errors.
17
"""
18
19
class AsyncOpenAI:
20
"""
21
PostHog-wrapped async OpenAI client with automatic usage tracking.
22
23
Async version of OpenAI wrapper with identical API to openai.AsyncOpenAI.
24
"""
25
26
class AzureOpenAI:
27
"""
28
PostHog-wrapped Azure OpenAI client with automatic usage tracking.
29
30
Provides identical API to openai.AzureOpenAI with added PostHog telemetry.
31
"""
32
33
class AsyncAzureOpenAI:
34
"""
35
PostHog-wrapped async Azure OpenAI client with automatic usage tracking.
36
37
Async version of Azure OpenAI wrapper.
38
"""
39
40
# Utility functions for OpenAI data formatting
41
def format_openai_response(response) -> dict:
42
"""Format OpenAI response for PostHog tracking"""
43
44
def format_openai_input(input) -> dict:
45
"""Format OpenAI input for PostHog tracking"""
46
47
def extract_openai_tools(tools) -> list:
48
"""Extract tools from OpenAI request for tracking"""
49
50
def format_openai_streaming_content(content) -> dict:
51
"""Format OpenAI streaming content for tracking"""
52
```
53
54
### Anthropic Integration
55
56
Drop-in replacement for Anthropic clients with comprehensive usage tracking and cost monitoring.
57
58
```python { .api }
59
class Anthropic:
60
"""
61
PostHog-wrapped Anthropic client with automatic usage tracking.
62
63
Provides identical API to anthropic.Anthropic with added PostHog telemetry.
64
"""
65
66
class AsyncAnthropic:
67
"""
68
PostHog-wrapped async Anthropic client with automatic usage tracking.
69
70
Async version of Anthropic wrapper with identical API to anthropic.AsyncAnthropic.
71
"""
72
73
class AnthropicBedrock:
74
"""
75
PostHog-wrapped Anthropic Bedrock client with automatic usage tracking.
76
77
For use with AWS Bedrock Anthropic models.
78
"""
79
80
class AsyncAnthropicBedrock:
81
"""
82
PostHog-wrapped async Anthropic Bedrock client with automatic usage tracking.
83
"""
84
85
class AnthropicVertex:
86
"""
87
PostHog-wrapped Anthropic Vertex AI client with automatic usage tracking.
88
89
For use with Google Cloud Vertex AI Anthropic models.
90
"""
91
92
class AsyncAnthropicVertex:
93
"""
94
PostHog-wrapped async Anthropic Vertex AI client with automatic usage tracking.
95
"""
96
97
# Utility functions for Anthropic data formatting
98
def format_anthropic_response(response) -> dict:
99
"""Format Anthropic response for PostHog tracking"""
100
101
def format_anthropic_input(input) -> dict:
102
"""Format Anthropic input for PostHog tracking"""
103
104
def extract_anthropic_tools(tools) -> list:
105
"""Extract tools from Anthropic request for tracking"""
106
107
def format_anthropic_streaming_content(content) -> dict:
108
"""Format Anthropic streaming content for tracking"""
109
```
110
111
### Gemini Integration
112
113
Google Gemini client wrapper with automatic usage tracking and performance monitoring.
114
115
```python { .api }
116
class Client:
117
"""
118
PostHog-wrapped Gemini client with automatic usage tracking.
119
120
Provides comprehensive tracking for Google Gemini model interactions.
121
Drop-in replacement for Google's genai.Client.
122
"""
123
124
# genai module compatibility
125
genai = _GenAI() # Contains Client for drop-in replacement
126
127
# Utility functions for Gemini data formatting
128
def format_gemini_response(response) -> dict:
129
"""Format Gemini response for PostHog tracking"""
130
131
def format_gemini_input(input) -> dict:
132
"""Format Gemini input for PostHog tracking"""
133
134
def extract_gemini_tools(tools) -> list:
135
"""Extract tools from Gemini request for tracking"""
136
```
137
138
## Usage Examples
139
140
### Langchain Integration
141
142
Langchain callback handler for comprehensive tracking of complex AI workflows and chains.
143
144
```python { .api }
145
class CallbackHandler:
146
"""
147
PostHog callback handler for Langchain applications.
148
149
Automatically tracks:
150
- Chain executions and performance
151
- LLM calls and token usage
152
- Tool usage and results
153
- Agent actions and decisions
154
- Error handling and debugging info
155
156
Compatible with all Langchain components including chains, agents, and tools.
157
"""
158
```
159
160
## Usage Examples
161
162
### OpenAI Integration
163
164
```python
165
from posthog.ai.openai import OpenAI
166
import posthog
167
168
# Configure PostHog
169
posthog.api_key = 'phc_your_project_api_key'
170
171
# Create OpenAI client with PostHog tracking
172
client = OpenAI(
173
api_key="your-openai-api-key",
174
# All standard OpenAI parameters supported
175
)
176
177
# Use exactly like standard OpenAI client
178
response = client.chat.completions.create(
179
model="gpt-4",
180
messages=[
181
{"role": "user", "content": "Hello, how are you?"}
182
],
183
temperature=0.7,
184
max_tokens=150
185
)
186
187
print(response.choices[0].message.content)
188
189
# PostHog automatically tracks:
190
# - Token usage (prompt + completion tokens)
191
# - Model and parameters used
192
# - Response time and latency
193
# - Costs (when available)
194
# - Errors and failures
195
```
196
197
### Async OpenAI Usage
198
199
```python
200
from posthog.ai.openai import AsyncOpenAI
201
import asyncio
202
import posthog
203
204
posthog.api_key = 'phc_your_project_api_key'
205
206
async def main():
207
client = AsyncOpenAI(api_key="your-openai-api-key")
208
209
response = await client.chat.completions.create(
210
model="gpt-3.5-turbo",
211
messages=[
212
{"role": "user", "content": "Write a haiku about programming"}
213
]
214
)
215
216
print(response.choices[0].message.content)
217
218
asyncio.run(main())
219
```
220
221
### Azure OpenAI Integration
222
223
```python
224
from posthog.ai.openai import AzureOpenAI
225
import posthog
226
227
posthog.api_key = 'phc_your_project_api_key'
228
229
client = AzureOpenAI(
230
azure_endpoint="https://your-resource.openai.azure.com/",
231
api_key="your-azure-openai-key",
232
api_version="2024-02-01"
233
)
234
235
response = client.chat.completions.create(
236
model="gpt-4", # Your deployment name
237
messages=[
238
{"role": "user", "content": "Explain quantum computing"}
239
]
240
)
241
242
print(response.choices[0].message.content)
243
```
244
245
### Anthropic Integration
246
247
```python
248
from posthog.ai.anthropic import Anthropic
249
import posthog
250
251
posthog.api_key = 'phc_your_project_api_key'
252
253
# Create Anthropic client with PostHog tracking
254
client = Anthropic(
255
api_key="your-anthropic-api-key"
256
)
257
258
# Use exactly like standard Anthropic client
259
response = client.messages.create(
260
model="claude-3-opus-20240229",
261
max_tokens=1000,
262
messages=[
263
{"role": "user", "content": "Explain the theory of relativity"}
264
]
265
)
266
267
print(response.content[0].text)
268
269
# Automatic tracking includes:
270
# - Token usage and costs
271
# - Model performance metrics
272
# - Response quality indicators
273
# - Error rates and types
274
```
275
276
### Anthropic Bedrock Integration
277
278
```python
279
from posthog.ai.anthropic import AnthropicBedrock
280
import posthog
281
282
posthog.api_key = 'phc_your_project_api_key'
283
284
client = AnthropicBedrock(
285
aws_access_key="your-aws-access-key",
286
aws_secret_key="your-aws-secret-key",
287
aws_region="us-east-1"
288
)
289
290
response = client.messages.create(
291
model="anthropic.claude-3-sonnet-20240229-v1:0",
292
max_tokens=1000,
293
messages=[
294
{"role": "user", "content": "What is machine learning?"}
295
]
296
)
297
298
print(response.content[0].text)
299
```
300
301
### Gemini Integration
302
303
```python
304
from posthog.ai.gemini import Client, genai
305
import posthog
306
307
posthog.api_key = 'phc_your_project_api_key'
308
309
# Configure Gemini with PostHog tracking
310
genai.configure(api_key="your-gemini-api-key")
311
312
# Create client with PostHog tracking
313
client = genai.GenerativeModel("gemini-pro")
314
315
# Use exactly like standard Gemini client
316
response = client.generate_content("Explain quantum computing")
317
print(response.text)
318
319
# PostHog automatically tracks:
320
# - Token usage and costs
321
# - Model performance metrics
322
# - Response quality indicators
323
# - Error rates and types
324
```
325
326
### Streaming Responses
327
328
```python
329
from posthog.ai.openai import OpenAI
330
import posthog
331
332
posthog.api_key = 'phc_your_project_api_key'
333
client = OpenAI(api_key="your-openai-api-key")
334
335
# Streaming is fully supported with automatic tracking
336
stream = client.chat.completions.create(
337
model="gpt-4",
338
messages=[
339
{"role": "user", "content": "Write a story about AI"}
340
],
341
stream=True
342
)
343
344
full_response = ""
345
for chunk in stream:
346
if chunk.choices[0].delta.content is not None:
347
content = chunk.choices[0].delta.content
348
print(content, end="")
349
full_response += content
350
351
# PostHog tracks complete streaming session including:
352
# - Total tokens used
353
# - Streaming latency and throughput
354
# - Time to first token
355
# - Complete response assembly
356
```
357
358
### Langchain Integration
359
360
```python
361
from posthog.ai.langchain import CallbackHandler
362
from langchain.llms import OpenAI
363
from langchain.chains import LLMChain
364
from langchain.prompts import PromptTemplate
365
import posthog
366
367
posthog.api_key = 'phc_your_project_api_key'
368
369
# Create PostHog callback handler
370
posthog_handler = CallbackHandler()
371
372
# Set up Langchain components
373
llm = OpenAI(temperature=0.7)
374
prompt = PromptTemplate(
375
input_variables=["topic"],
376
template="Write a short article about {topic}"
377
)
378
379
chain = LLMChain(llm=llm, prompt=prompt)
380
381
# Run chain with PostHog tracking
382
result = chain.run(
383
topic="artificial intelligence",
384
callbacks=[posthog_handler]
385
)
386
387
print(result)
388
389
# PostHog automatically tracks:
390
# - Chain execution time and success
391
# - LLM calls and token usage
392
# - Prompt templates and variables
393
# - Output quality and length
394
# - Error handling and retries
395
```
396
397
### Advanced Langchain Usage
398
399
```python
400
from posthog.ai.langchain import CallbackHandler
401
from langchain.agents import initialize_agent, Tool
402
from langchain.agents import AgentType
403
from langchain.llms import OpenAI
404
import posthog
405
406
posthog.api_key = 'phc_your_project_api_key'
407
408
# Custom tools
409
def calculator(expression):
410
"""Calculate mathematical expressions"""
411
try:
412
return str(eval(expression))
413
except:
414
return "Error in calculation"
415
416
def search_tool(query):
417
"""Mock search tool"""
418
return f"Search results for: {query}"
419
420
tools = [
421
Tool(
422
name="Calculator",
423
func=calculator,
424
description="Use for mathematical calculations"
425
),
426
Tool(
427
name="Search",
428
func=search_tool,
429
description="Use for web searches"
430
)
431
]
432
433
# Create agent with PostHog tracking
434
llm = OpenAI(temperature=0)
435
agent = initialize_agent(
436
tools,
437
llm,
438
agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
439
verbose=True
440
)
441
442
# Create callback handler
443
posthog_handler = CallbackHandler()
444
445
# Run agent with comprehensive tracking
446
result = agent.run(
447
"What is 15 * 27, and then search for information about that number?",
448
callbacks=[posthog_handler]
449
)
450
451
print(result)
452
453
# Tracks agent reasoning, tool usage, and decision making
454
```
455
456
### Context and User Tracking
457
458
```python
459
from posthog.ai.openai import OpenAI
460
import posthog
461
462
posthog.api_key = 'phc_your_project_api_key'
463
464
# Use context for user-specific AI tracking
465
with posthog.new_context():
466
posthog.identify_context('user123')
467
posthog.tag('session_type', 'premium')
468
posthog.tag('use_case', 'content_generation')
469
470
client = OpenAI(api_key="your-openai-api-key")
471
472
# AI usage automatically associated with user and context
473
response = client.chat.completions.create(
474
model="gpt-4",
475
messages=[
476
{"role": "user", "content": "Help me write a blog post"}
477
]
478
)
479
480
# Additional manual tracking
481
posthog.capture('ai_content_generated', {
482
'content_type': 'blog_post',
483
'word_count': len(response.choices[0].message.content.split()),
484
'satisfaction': 'high'
485
})
486
```
487
488
### Error Handling and Monitoring
489
490
```python
491
from posthog.ai.openai import OpenAI
492
import posthog
493
494
posthog.api_key = 'phc_your_project_api_key'
495
client = OpenAI(api_key="your-openai-api-key")
496
497
def safe_ai_call(messages, **kwargs):
498
"""Wrapper for AI calls with comprehensive error tracking"""
499
try:
500
with posthog.new_context():
501
posthog.tag('ai_operation', 'chat_completion')
502
posthog.tag('model', kwargs.get('model', 'gpt-3.5-turbo'))
503
504
response = client.chat.completions.create(
505
messages=messages,
506
**kwargs
507
)
508
509
# Track successful usage
510
posthog.capture('ai_call_success', {
511
'tokens_used': response.usage.total_tokens,
512
'model': response.model,
513
'response_length': len(response.choices[0].message.content)
514
})
515
516
return response
517
518
except Exception as e:
519
# Error is automatically captured by context
520
posthog.capture('ai_call_failed', {
521
'error_type': type(e).__name__,
522
'error_message': str(e),
523
'model': kwargs.get('model', 'unknown')
524
})
525
raise
526
527
# Usage with error tracking
528
try:
529
response = safe_ai_call(
530
messages=[{"role": "user", "content": "Hello"}],
531
model="gpt-4",
532
max_tokens=100
533
)
534
except Exception as e:
535
print(f"AI call failed: {e}")
536
```
537
538
### Privacy Mode
539
540
```python
541
from posthog.ai.openai import OpenAI
542
import posthog
543
544
# Enable privacy mode to exclude prompts and responses from tracking
545
posthog.api_key = 'phc_your_project_api_key'
546
posthog.privacy_mode = True
547
548
client = OpenAI(api_key="your-openai-api-key")
549
550
# With privacy mode enabled, only usage metadata is tracked:
551
# - Token counts
552
# - Model used
553
# - Response times
554
# - Costs
555
# - Error rates
556
#
557
# But NOT:
558
# - Prompt content
559
# - Response content
560
# - User data in messages
561
562
response = client.chat.completions.create(
563
model="gpt-4",
564
messages=[
565
{"role": "user", "content": "Sensitive information here"}
566
]
567
)
568
569
# Only metadata tracked, content is not sent to PostHog
570
```
571
572
### Cost Tracking and Budgets
573
574
```python
575
from posthog.ai.openai import OpenAI
576
import posthog
577
578
posthog.api_key = 'phc_your_project_api_key'
579
580
class CostTracker:
581
def __init__(self, budget_limit=100.0):
582
self.total_cost = 0.0
583
self.budget_limit = budget_limit
584
self.client = OpenAI(api_key="your-openai-api-key")
585
586
def track_usage(self, response):
587
# Calculate approximate cost (varies by model)
588
model_costs = {
589
'gpt-4': {'input': 0.03, 'output': 0.06}, # per 1K tokens
590
'gpt-3.5-turbo': {'input': 0.0015, 'output': 0.002}
591
}
592
593
model = response.model
594
usage = response.usage
595
596
if model in model_costs:
597
input_cost = (usage.prompt_tokens / 1000) * model_costs[model]['input']
598
output_cost = (usage.completion_tokens / 1000) * model_costs[model]['output']
599
total_cost = input_cost + output_cost
600
601
self.total_cost += total_cost
602
603
# Track cost metrics
604
posthog.capture('ai_cost_tracking', {
605
'session_cost': total_cost,
606
'total_cost': self.total_cost,
607
'budget_remaining': self.budget_limit - self.total_cost,
608
'model': model,
609
'tokens_used': usage.total_tokens
610
})
611
612
# Budget alerts
613
if self.total_cost > self.budget_limit * 0.8:
614
posthog.capture('ai_budget_alert', {
615
'alert_type': 'approaching_limit',
616
'usage_percentage': (self.total_cost / self.budget_limit) * 100
617
})
618
619
def generate_text(self, messages, **kwargs):
620
if self.total_cost >= self.budget_limit:
621
raise Exception("Budget limit exceeded")
622
623
response = self.client.chat.completions.create(
624
messages=messages,
625
**kwargs
626
)
627
628
self.track_usage(response)
629
return response
630
631
# Usage
632
tracker = CostTracker(budget_limit=50.0)
633
634
response = tracker.generate_text(
635
messages=[{"role": "user", "content": "Write a summary"}],
636
model="gpt-4"
637
)
638
```
639
640
## Advanced Features
641
642
### Custom Metrics and Analysis
643
644
```python
645
from posthog.ai.openai import OpenAI
646
import posthog
647
import time
648
649
class AIAnalytics:
650
def __init__(self):
651
self.client = OpenAI(api_key="your-openai-api-key")
652
653
def analyze_response_quality(self, prompt, response_text):
654
"""Analyze response quality metrics"""
655
return {
656
'length': len(response_text),
657
'sentences': response_text.count('.'),
658
'complexity_score': len(set(response_text.split())) / len(response_text.split()),
659
'prompt_relevance': self._calculate_relevance(prompt, response_text)
660
}
661
662
def _calculate_relevance(self, prompt, response):
663
# Simple relevance calculation
664
prompt_words = set(prompt.lower().split())
665
response_words = set(response.lower().split())
666
overlap = len(prompt_words.intersection(response_words))
667
return overlap / len(prompt_words) if prompt_words else 0
668
669
def generate_with_analytics(self, messages, **kwargs):
670
start_time = time.time()
671
672
with posthog.new_context():
673
posthog.tag('ai_analytics', True)
674
675
response = self.client.chat.completions.create(
676
messages=messages,
677
**kwargs
678
)
679
680
end_time = time.time()
681
response_time = end_time - start_time
682
683
# Analyze response quality
684
quality_metrics = self.analyze_response_quality(
685
messages[-1]['content'],
686
response.choices[0].message.content
687
)
688
689
# Track comprehensive analytics
690
posthog.capture('ai_detailed_analytics', {
691
'response_time': response_time,
692
'model': response.model,
693
'tokens_per_second': response.usage.total_tokens / response_time,
694
**quality_metrics
695
})
696
697
return response
698
699
# Usage
700
analytics = AIAnalytics()
701
response = analytics.generate_with_analytics(
702
messages=[{"role": "user", "content": "Explain machine learning"}],
703
model="gpt-4"
704
)
705
```
706
707
### A/B Testing AI Models
708
709
```python
710
from posthog.ai.openai import OpenAI
711
import posthog
712
import random
713
714
class ModelABTester:
715
def __init__(self):
716
self.client = OpenAI(api_key="your-openai-api-key")
717
self.models = {
718
'gpt-4': {'weight': 0.3, 'cost_per_token': 0.00003},
719
'gpt-3.5-turbo': {'weight': 0.7, 'cost_per_token': 0.000002}
720
}
721
722
def select_model(self, user_id):
723
"""Select model based on A/B test configuration"""
724
# Use PostHog feature flags for A/B testing
725
variant = posthog.get_feature_flag('ai_model_test', user_id)
726
727
if variant == 'premium':
728
return 'gpt-4'
729
elif variant == 'standard':
730
return 'gpt-3.5-turbo'
731
else:
732
# Fallback to weighted random selection
733
return random.choices(
734
list(self.models.keys()),
735
weights=[m['weight'] for m in self.models.values()]
736
)[0]
737
738
def generate_with_testing(self, user_id, messages, **kwargs):
739
model = self.select_model(user_id)
740
741
with posthog.new_context():
742
posthog.identify_context(user_id)
743
posthog.tag('ab_test', 'ai_model_test')
744
posthog.tag('model_variant', model)
745
746
response = self.client.chat.completions.create(
747
model=model,
748
messages=messages,
749
**kwargs
750
)
751
752
# Track A/B test metrics
753
posthog.capture('ai_ab_test_result', {
754
'model_used': model,
755
'tokens_used': response.usage.total_tokens,
756
'estimated_cost': response.usage.total_tokens * self.models[model]['cost_per_token'],
757
'response_quality': len(response.choices[0].message.content)
758
})
759
760
return response
761
762
# Usage
763
tester = ModelABTester()
764
response = tester.generate_with_testing(
765
'user123',
766
messages=[{"role": "user", "content": "Help me write code"}]
767
)
768
```
769
770
## Best Practices
771
772
### Performance Optimization
773
774
```python
775
# Use appropriate models for different use cases
776
quick_client = OpenAI() # For simple tasks
777
advanced_client = OpenAI() # For complex reasoning
778
779
# Simple tasks
780
simple_response = quick_client.chat.completions.create(
781
model="gpt-3.5-turbo",
782
messages=[{"role": "user", "content": "Summarize this text"}],
783
max_tokens=100
784
)
785
786
# Complex tasks
787
complex_response = advanced_client.chat.completions.create(
788
model="gpt-4",
789
messages=[{"role": "user", "content": "Analyze this complex problem"}],
790
max_tokens=1000
791
)
792
```
793
794
### Error Handling
795
796
```python
797
from posthog.ai.openai import OpenAI
798
import posthog
799
800
client = OpenAI(api_key="your-openai-api-key")
801
802
def robust_ai_call(messages, max_retries=3, **kwargs):
803
for attempt in range(max_retries):
804
try:
805
with posthog.new_context():
806
posthog.tag('attempt', attempt + 1)
807
808
return client.chat.completions.create(
809
messages=messages,
810
**kwargs
811
)
812
except Exception as e:
813
posthog.capture('ai_retry', {
814
'attempt': attempt + 1,
815
'error': str(e),
816
'max_retries': max_retries
817
})
818
819
if attempt == max_retries - 1:
820
raise
821
822
time.sleep(2 ** attempt) # Exponential backoff
823
824
# Usage with automatic retry and tracking
825
response = robust_ai_call(
826
messages=[{"role": "user", "content": "Hello"}],
827
model="gpt-4"
828
)
829
```
830
831
### Data Privacy and Security
832
833
```python
834
import posthog
835
from posthog.ai.openai import OpenAI
836
837
# Enable privacy mode globally
838
posthog.privacy_mode = True
839
840
# Or use environment-specific configuration
841
import os
842
if os.getenv('ENVIRONMENT') == 'production':
843
posthog.privacy_mode = True
844
845
client = OpenAI(api_key="your-openai-api-key")
846
847
# With privacy mode, only metadata is tracked
848
response = client.chat.completions.create(
849
model="gpt-4",
850
messages=[{"role": "user", "content": "Confidential information"}]
851
)
852
853
# Manual tracking of non-sensitive metrics
854
posthog.capture('ai_usage_summary', {
855
'model': 'gpt-4',
856
'tokens_used': response.usage.total_tokens,
857
'use_case': 'content_generation',
858
'user_satisfaction': 'high' # Can be collected separately
859
})
860
```