0
# HTTP Client Class
1
2
Advanced HTTP client providing direct access to connection pooling, custom agents, and request lifecycle management for applications requiring fine-grained control over HTTP behavior.
3
4
## Capabilities
5
6
### HTTPClient Class
7
8
Core HTTP client implementation that wraps Twisted's Agent system with cookie management and request processing.
9
10
```python { .api }
11
class HTTPClient:
12
def __init__(self, agent, cookiejar=None, data_to_body_producer=IBodyProducer):
13
"""
14
Initialize HTTP client with custom agent and configuration.
15
16
Parameters:
17
- agent: IAgent - Twisted agent for making HTTP requests
18
- cookiejar: CookieJar or None - Cookie jar for session management
19
- data_to_body_producer: callable - Function to convert data to IBodyProducer
20
"""
21
22
def request(self, method, url, **kwargs):
23
"""
24
Make an HTTP request with specified method.
25
26
Parameters:
27
- method: str - HTTP method
28
- url: str, bytes, EncodedURL, or DecodedURL - Target URL
29
- **kwargs: Same parameters as module-level request functions
30
31
Returns:
32
Deferred that fires with _Response object
33
"""
34
35
def get(self, url, **kwargs):
36
"""Make a GET request. See module-level get() function."""
37
38
def post(self, url, data=None, **kwargs):
39
"""Make a POST request. See module-level post() function."""
40
41
def put(self, url, data=None, **kwargs):
42
"""Make a PUT request. See module-level put() function."""
43
44
def patch(self, url, data=None, **kwargs):
45
"""Make a PATCH request. See module-level patch() function."""
46
47
def delete(self, url, **kwargs):
48
"""Make a DELETE request. See module-level delete() function."""
49
50
def head(self, url, **kwargs):
51
"""Make a HEAD request. See module-level head() function."""
52
```
53
54
### Connection Pool Management
55
56
Functions for managing HTTP connection pools across requests for improved performance.
57
58
```python { .api }
59
def get_global_pool():
60
"""
61
Get the current global connection pool.
62
63
Returns:
64
HTTPConnectionPool or None
65
"""
66
67
def set_global_pool(pool):
68
"""
69
Set the global connection pool.
70
71
Parameters:
72
- pool: HTTPConnectionPool - Connection pool to set as global
73
"""
74
75
def default_pool(reactor, pool, persistent):
76
"""
77
Get appropriate connection pool based on parameters.
78
79
Parameters:
80
- reactor: Twisted reactor
81
- pool: HTTPConnectionPool or None - Explicit pool
82
- persistent: bool or None - Use persistent connections
83
84
Returns:
85
HTTPConnectionPool - Connection pool to use
86
"""
87
```
88
89
### Reactor Management
90
91
Utility for reactor selection and configuration.
92
93
```python { .api }
94
def default_reactor(reactor):
95
"""
96
Return the specified reactor or import the default reactor.
97
98
Parameters:
99
- reactor: Twisted reactor or None
100
101
Returns:
102
Twisted reactor instance
103
"""
104
```
105
106
## Usage Examples
107
108
### Custom HTTP Client
109
110
```python
111
from twisted.web.client import Agent, HTTPConnectionPool
112
from treq.client import HTTPClient
113
from twisted.internet import reactor
114
from http.cookiejar import CookieJar
115
116
# Create custom agent with specific configuration
117
pool = HTTPConnectionPool(reactor, persistent=True)
118
agent = Agent(reactor, pool=pool, connectTimeout=30)
119
120
# Create client with custom cookie jar
121
cookiejar = CookieJar()
122
client = HTTPClient(agent, cookiejar=cookiejar)
123
124
# Use the client
125
@defer.inlineCallbacks
126
def use_custom_client():
127
response = yield client.get('https://httpbin.org/cookies/set/test/value')
128
cookies_response = yield client.get('https://httpbin.org/cookies')
129
data = yield cookies_response.json()
130
print("Cookies:", data['cookies'])
131
```
132
133
### Connection Pool Configuration
134
135
```python
136
from twisted.web.client import HTTPConnectionPool
137
from treq.client import set_global_pool
138
from twisted.internet import reactor
139
140
# Configure global connection pool
141
pool = HTTPConnectionPool(
142
reactor,
143
persistent=True,
144
maxPersistentPerHost=10,
145
cachedConnectionTimeout=240,
146
retryAutomatically=True
147
)
148
set_global_pool(pool)
149
150
# Now all treq requests will use this pool
151
import treq
152
153
@defer.inlineCallbacks
154
def use_configured_pool():
155
# This request uses the custom pool configuration
156
response = yield treq.get('https://httpbin.org/get')
157
```
158
159
### Advanced Agent Configuration
160
161
```python
162
from twisted.web.client import Agent, BrowserLikePolicyForHTTPS
163
from twisted.internet.ssl import ClientContextFactory
164
from treq.client import HTTPClient
165
import treq
166
167
# Custom SSL context
168
class CustomContextFactory(ClientContextFactory):
169
def getContext(self, hostname, port):
170
context = ClientContextFactory.getContext(self, hostname, port)
171
# Configure SSL context as needed
172
return context
173
174
# Create agent with custom SSL and policy
175
agent = Agent(
176
reactor,
177
contextFactory=CustomContextFactory(),
178
pool=HTTPConnectionPool(reactor, persistent=True),
179
connectTimeout=30
180
)
181
182
client = HTTPClient(agent)
183
184
@defer.inlineCallbacks
185
def secure_request():
186
response = yield client.get('https://secure-api.example.com/data')
187
data = yield response.json()
188
```
189
190
### Custom Body Producer
191
192
```python
193
from twisted.web.iweb import IBodyProducer
194
from zope.interface import implementer
195
from twisted.internet import defer
196
197
@implementer(IBodyProducer)
198
class CustomBodyProducer:
199
def __init__(self, data):
200
self.data = data
201
self.length = len(data)
202
203
def startProducing(self, consumer):
204
consumer.write(self.data)
205
return defer.succeed(None)
206
207
def pauseProducing(self):
208
pass
209
210
def stopProducing(self):
211
pass
212
213
# Custom data-to-body-producer function
214
def custom_data_converter(data):
215
if isinstance(data, dict) and 'custom' in data:
216
return CustomBodyProducer(data['custom'].encode('utf-8'))
217
return IBodyProducer(data) # Use default conversion
218
219
# Client with custom body producer
220
client = HTTPClient(agent, data_to_body_producer=custom_data_converter)
221
222
@defer.inlineCallbacks
223
def use_custom_producer():
224
response = yield client.post(
225
'https://httpbin.org/post',
226
data={'custom': 'special data format'}
227
)
228
```
229
230
### Session Management
231
232
```python
233
from http.cookiejar import CookieJar
234
from treq.client import HTTPClient
235
from twisted.web.client import Agent
236
237
# Create client with persistent cookies
238
cookiejar = CookieJar()
239
client = HTTPClient(Agent(reactor), cookiejar=cookiejar)
240
241
@defer.inlineCallbacks
242
def session_example():
243
# Login request - cookies are automatically stored
244
login_response = yield client.post(
245
'https://httpbin.org/cookies/set/session/abc123',
246
data={'username': 'user', 'password': 'pass'}
247
)
248
249
# Subsequent requests automatically include cookies
250
profile_response = yield client.get('https://httpbin.org/cookies')
251
profile_data = yield profile_response.json()
252
print("Session cookies:", profile_data['cookies'])
253
254
# Logout
255
yield client.post('https://httpbin.org/cookies/delete/session')
256
```
257
258
### Error Handling and Retries
259
260
```python
261
from twisted.internet import defer
262
from twisted.web.client import ResponseFailed
263
264
@defer.inlineCallbacks
265
def robust_client():
266
client = HTTPClient(Agent(reactor))
267
268
max_retries = 3
269
for attempt in range(max_retries):
270
try:
271
response = yield client.get('https://unreliable-api.example.com/data')
272
data = yield response.json()
273
defer.returnValue(data)
274
except ResponseFailed as e:
275
if attempt == max_retries - 1:
276
raise
277
print(f"Attempt {attempt + 1} failed, retrying...")
278
yield defer.sleep(2 ** attempt) # Exponential backoff
279
```
280
281
## Types
282
283
HTTP client related types:
284
285
```python { .api }
286
# Agent interface from Twisted
287
from twisted.web.iweb import IAgent
288
289
# Cookie jar types
290
from http.cookiejar import CookieJar
291
292
# Body producer interface
293
from twisted.web.iweb import IBodyProducer
294
295
# Connection pool
296
from twisted.web.client import HTTPConnectionPool
297
298
# Data to body producer converter
299
DataToBodyProducer = Callable[[Any], IBodyProducer]
300
301
# Response type
302
from treq.response import _Response
303
```
304
305
## Performance Optimization
306
307
### Connection Pooling
308
309
- **Persistent connections**: Enable `persistent=True` for connection reuse
310
- **Pool sizing**: Configure `maxPersistentPerHost` based on expected load
311
- **Timeout management**: Set appropriate `cachedConnectionTimeout` values
312
- **Global pools**: Use `set_global_pool()` to share pools across application
313
314
### Memory Management
315
316
- **Cookie jar cleanup**: Periodically clear unnecessary cookies
317
- **Response buffering**: Use `unbuffered=True` for large responses
318
- **Connection limits**: Set reasonable connection pool limits
319
- **Reactor sharing**: Reuse reactor instances across clients
320
321
### Security Considerations
322
323
- **SSL verification**: Configure appropriate SSL context factories
324
- **Certificate validation**: Use `BrowserLikePolicyForHTTPS` for proper validation
325
- **Cookie security**: Handle secure and HttpOnly cookies appropriately
326
- **Timeout configuration**: Set reasonable timeouts to prevent hanging connections