0
# Stealth Mode and Anti-Detection
1
2
Advanced techniques for avoiding detection including human-like behavior simulation, request timing randomization, header manipulation, and browser fingerprint rotation. Stealth mode helps CloudScraper appear more like a regular browser.
3
4
## Capabilities
5
6
### StealthMode Class
7
8
Main class implementing anti-detection techniques and human-like behavior simulation to reduce the likelihood of being detected as an automated client.
9
10
```python { .api }
11
class StealthMode:
12
def __init__(self, cloudscraper):
13
"""
14
Initialize stealth mode for a CloudScraper instance.
15
16
Parameters:
17
- cloudscraper: CloudScraper instance to apply stealth techniques to
18
"""
19
20
def apply_stealth_techniques(self, method: str, url: str, **kwargs):
21
"""
22
Apply stealth modifications to request parameters.
23
24
Parameters:
25
- method: str, HTTP method (GET, POST, etc.)
26
- url: str, target URL
27
- **kwargs: request parameters to modify
28
29
Returns:
30
dict: Modified request parameters with stealth techniques applied
31
"""
32
33
def set_delay_range(self, min_delay: float, max_delay: float):
34
"""
35
Configure random delay range between requests.
36
37
Parameters:
38
- min_delay: float, minimum delay in seconds
39
- max_delay: float, maximum delay in seconds
40
"""
41
42
def enable_human_like_delays(self, enabled: bool):
43
"""
44
Enable or disable human-like delay patterns.
45
46
Parameters:
47
- enabled: bool, whether to use human-like delays
48
"""
49
50
def enable_randomize_headers(self, enabled: bool):
51
"""
52
Enable or disable header randomization.
53
54
Parameters:
55
- enabled: bool, whether to randomize headers
56
"""
57
58
def enable_browser_quirks(self, enabled: bool):
59
"""
60
Enable or disable browser-specific behavioral quirks.
61
62
Parameters:
63
- enabled: bool, whether to apply browser quirks
64
"""
65
66
def _apply_human_like_delay(self):
67
"""
68
Apply random delay to mimic human behavior between requests.
69
Internal method called automatically during request processing.
70
"""
71
72
def _randomize_headers(self, kwargs):
73
"""
74
Apply header randomization to avoid fingerprinting.
75
76
Parameters:
77
- kwargs: dict, request parameters to modify headers in
78
"""
79
80
def _apply_browser_quirks(self, kwargs):
81
"""
82
Apply browser-specific behavioral quirks to request parameters.
83
84
Parameters:
85
- kwargs: dict, request parameters to modify with browser quirks
86
"""
87
```
88
89
### Basic Stealth Configuration
90
91
Enable stealth mode with default settings for general anti-detection:
92
93
```python
94
# Enable stealth mode with defaults
95
scraper = cloudscraper.create_scraper(enable_stealth=True)
96
97
# Stealth mode disabled
98
scraper = cloudscraper.create_scraper(enable_stealth=False)
99
100
# Basic stealth configuration
101
scraper = cloudscraper.create_scraper(
102
enable_stealth=True,
103
stealth_options={
104
'min_delay': 1.0,
105
'max_delay': 3.0,
106
'human_like_delays': True,
107
'randomize_headers': True,
108
'browser_quirks': True
109
}
110
)
111
```
112
113
### Advanced Stealth Configuration
114
115
Comprehensive stealth setup with fine-tuned parameters for maximum anti-detection effectiveness:
116
117
```python
118
# Maximum stealth configuration
119
scraper = cloudscraper.create_scraper(
120
enable_stealth=True,
121
stealth_options={
122
'min_delay': 2.0, # Minimum 2 seconds between requests
123
'max_delay': 8.0, # Maximum 8 seconds between requests
124
'human_like_delays': True, # Use realistic human timing patterns
125
'randomize_headers': True, # Randomize headers to avoid fingerprinting
126
'browser_quirks': True # Apply browser-specific behaviors
127
},
128
browser={
129
'browser': 'chrome', # Consistent browser fingerprint
130
'platform': 'windows', # Consistent platform
131
'mobile': False # Desktop only
132
}
133
)
134
135
# Conservative stealth for sensitive sites
136
scraper = cloudscraper.create_scraper(
137
enable_stealth=True,
138
stealth_options={
139
'min_delay': 5.0, # Slower, more human-like
140
'max_delay': 15.0, # Long maximum delays
141
'human_like_delays': True,
142
'randomize_headers': True,
143
'browser_quirks': True
144
}
145
)
146
```
147
148
## Stealth Techniques
149
150
### Request Timing and Delays
151
152
Implement human-like timing patterns to avoid detection based on request frequency:
153
154
```python
155
# Configure delay patterns
156
scraper = cloudscraper.create_scraper(
157
enable_stealth=True,
158
stealth_options={
159
'min_delay': 2.0,
160
'max_delay': 6.0,
161
'human_like_delays': True # Realistic timing patterns
162
}
163
)
164
165
# Manual delay configuration
166
scraper.stealth_mode.set_delay_range(1.5, 4.5)
167
scraper.stealth_mode.enable_human_like_delays(True)
168
169
# Different timing for different scenarios
170
fast_scraper = cloudscraper.create_scraper(
171
enable_stealth=True,
172
stealth_options={'min_delay': 0.5, 'max_delay': 2.0}
173
)
174
175
slow_scraper = cloudscraper.create_scraper(
176
enable_stealth=True,
177
stealth_options={'min_delay': 5.0, 'max_delay': 20.0}
178
)
179
```
180
181
#### Usage Examples
182
183
```python
184
# Stealth timing in action
185
scraper = cloudscraper.create_scraper(
186
enable_stealth=True,
187
stealth_options={'min_delay': 2.0, 'max_delay': 5.0},
188
debug=True # See timing information
189
)
190
191
# Each request will have random delays
192
response1 = scraper.get('https://example.com/page1') # Waits 2-5 seconds before
193
response2 = scraper.get('https://example.com/page2') # Waits 2-5 seconds before
194
response3 = scraper.get('https://example.com/page3') # Waits 2-5 seconds before
195
```
196
197
### Header Randomization
198
199
Randomize HTTP headers to avoid consistent fingerprinting patterns:
200
201
```python
202
# Enable header randomization
203
scraper = cloudscraper.create_scraper(
204
enable_stealth=True,
205
stealth_options={
206
'randomize_headers': True # Randomize order and values
207
}
208
)
209
210
# Selective header randomization
211
scraper.stealth_mode.enable_randomize_headers(True)
212
213
# View applied headers with debug mode
214
scraper = cloudscraper.create_scraper(
215
enable_stealth=True,
216
stealth_options={'randomize_headers': True},
217
debug=True
218
)
219
response = scraper.get('https://httpbin.org/headers')
220
print(response.json()) # See randomized headers
221
```
222
223
### Browser Quirks and Behavior
224
225
Apply browser-specific behavioral patterns to mimic real browser usage:
226
227
```python
228
# Enable browser quirks
229
scraper = cloudscraper.create_scraper(
230
enable_stealth=True,
231
stealth_options={
232
'browser_quirks': True # Apply browser-specific behaviors
233
},
234
browser={
235
'browser': 'chrome', # Target specific browser quirks
236
'platform': 'windows'
237
}
238
)
239
240
# Browser-specific configurations
241
chrome_quirks = cloudscraper.create_scraper(
242
enable_stealth=True,
243
stealth_options={'browser_quirks': True},
244
browser={'browser': 'chrome', 'platform': 'windows'}
245
)
246
247
firefox_quirks = cloudscraper.create_scraper(
248
enable_stealth=True,
249
stealth_options={'browser_quirks': True},
250
browser={'browser': 'firefox', 'platform': 'linux'}
251
)
252
```
253
254
## Session Health and Refresh
255
256
### Automatic Session Refresh
257
258
Configure automatic session refresh to maintain session health and avoid detection:
259
260
```python
261
# Session refresh configuration
262
scraper = cloudscraper.create_scraper(
263
enable_stealth=True,
264
session_refresh_interval=3600, # Refresh every hour
265
auto_refresh_on_403=True, # Auto-refresh on 403 errors
266
max_403_retries=3 # Maximum retry attempts
267
)
268
269
# Conservative session management
270
scraper = cloudscraper.create_scraper(
271
enable_stealth=True,
272
session_refresh_interval=1800, # Refresh every 30 minutes
273
auto_refresh_on_403=True,
274
max_403_retries=5
275
)
276
277
# Aggressive session refresh for problematic sites
278
scraper = cloudscraper.create_scraper(
279
enable_stealth=True,
280
session_refresh_interval=900, # Refresh every 15 minutes
281
auto_refresh_on_403=True,
282
max_403_retries=10
283
)
284
```
285
286
### Request Throttling
287
288
Control request frequency to avoid triggering rate limits:
289
290
```python
291
# Request throttling configuration
292
scraper = cloudscraper.create_scraper(
293
enable_stealth=True,
294
min_request_interval=1.0, # Minimum 1 second between requests
295
max_concurrent_requests=1, # Only 1 request at a time
296
rotate_tls_ciphers=True # Rotate TLS fingerprint
297
)
298
299
# More aggressive throttling
300
scraper = cloudscraper.create_scraper(
301
enable_stealth=True,
302
min_request_interval=3.0, # Minimum 3 seconds between requests
303
max_concurrent_requests=1,
304
rotate_tls_ciphers=True
305
)
306
```
307
308
## Advanced Stealth Features
309
310
### TLS Cipher Rotation
311
312
Rotate TLS cipher suites to avoid consistent cryptographic fingerprinting:
313
314
```python
315
# Enable TLS cipher rotation
316
scraper = cloudscraper.create_scraper(
317
enable_stealth=True,
318
rotate_tls_ciphers=True, # Rotate cipher suites
319
debug=True # See cipher rotation
320
)
321
322
# Manual cipher configuration
323
scraper = cloudscraper.create_scraper(
324
enable_stealth=True,
325
cipherSuite='TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256',
326
rotate_tls_ciphers=True
327
)
328
```
329
330
### User Agent Rotation
331
332
Combine stealth mode with user agent rotation for enhanced anti-detection:
333
334
```python
335
# User agent rotation with stealth
336
scraper = cloudscraper.create_scraper(
337
enable_stealth=True,
338
browser={
339
'browser': 'chrome', # Base browser type
340
'platform': 'windows', # Base platform
341
'mobile': False, # Device type
342
'desktop': True
343
},
344
stealth_options={
345
'randomize_headers': True, # This affects user agent selection
346
'browser_quirks': True
347
}
348
)
349
350
# Different user agents for different sessions
351
browsers = ['chrome', 'firefox']
352
for browser in browsers:
353
scraper = cloudscraper.create_scraper(
354
enable_stealth=True,
355
browser={'browser': browser, 'platform': 'windows'},
356
stealth_options={'randomize_headers': True}
357
)
358
response = scraper.get('https://httpbin.org/user-agent')
359
print(f"{browser}: {response.json()['user-agent']}")
360
```
361
362
### Behavioral Patterns
363
364
Implement realistic browsing patterns to avoid detection:
365
366
```python
367
def realistic_browsing_session(scraper, base_url, pages):
368
"""Simulate realistic browsing behavior."""
369
import random
370
import time
371
372
# Start with homepage
373
response = scraper.get(base_url)
374
print(f"Visited homepage: {response.status_code}")
375
376
# Random delay like reading the page
377
time.sleep(random.uniform(5, 15))
378
379
# Visit pages in random order with realistic delays
380
random.shuffle(pages)
381
for page in pages:
382
# Random delay between page visits
383
time.sleep(random.uniform(2, 8))
384
385
url = f"{base_url.rstrip('/')}/{page.lstrip('/')}"
386
response = scraper.get(url)
387
print(f"Visited {page}: {response.status_code}")
388
389
# Occasional longer delays (like reading content)
390
if random.random() < 0.3: # 30% chance
391
time.sleep(random.uniform(10, 30))
392
393
# Use with stealth scraper
394
scraper = cloudscraper.create_scraper(
395
enable_stealth=True,
396
stealth_options={
397
'min_delay': 1.0,
398
'max_delay': 4.0,
399
'human_like_delays': True
400
}
401
)
402
403
pages = ['/about', '/products', '/contact', '/blog']
404
realistic_browsing_session(scraper, 'https://example.com', pages)
405
```
406
407
## Stealth Mode Debugging
408
409
### Debug Output
410
411
Monitor stealth techniques with debug logging:
412
413
```python
414
scraper = cloudscraper.create_scraper(
415
enable_stealth=True,
416
stealth_options={
417
'min_delay': 2.0,
418
'max_delay': 5.0,
419
'human_like_delays': True,
420
'randomize_headers': True,
421
'browser_quirks': True
422
},
423
debug=True
424
)
425
426
response = scraper.get('https://example.com')
427
428
# Debug output shows:
429
# "โฑ๏ธ Request throttling: sleeping 1.2s"
430
# "๐ Rotated TLS cipher suite (rotation #3)"
431
# "๐ญ Applied stealth header randomization"
432
# "๐ถ Human-like delay: 3.4s"
433
```
434
435
### Stealth Effectiveness Testing
436
437
Test stealth effectiveness against detection systems:
438
439
```python
440
def test_stealth_effectiveness(urls, stealth_configs):
441
"""Test different stealth configurations."""
442
results = {}
443
444
for config_name, config in stealth_configs.items():
445
print(f"\nTesting {config_name}...")
446
scraper = cloudscraper.create_scraper(**config)
447
448
config_results = {}
449
for url in urls:
450
try:
451
response = scraper.get(url, timeout=30)
452
config_results[url] = {
453
'status': response.status_code,
454
'success': 200 <= response.status_code < 400,
455
'blocked': response.status_code in [403, 429, 503]
456
}
457
print(f" {url}: {response.status_code}")
458
except Exception as e:
459
config_results[url] = {
460
'status': None,
461
'success': False,
462
'error': str(e)
463
}
464
print(f" {url}: Error - {e}")
465
466
results[config_name] = config_results
467
468
return results
469
470
# Test configurations
471
stealth_configs = {
472
'no_stealth': {
473
'enable_stealth': False
474
},
475
'basic_stealth': {
476
'enable_stealth': True
477
},
478
'maximum_stealth': {
479
'enable_stealth': True,
480
'stealth_options': {
481
'min_delay': 3.0,
482
'max_delay': 8.0,
483
'human_like_delays': True,
484
'randomize_headers': True,
485
'browser_quirks': True
486
},
487
'session_refresh_interval': 1800,
488
'rotate_tls_ciphers': True
489
}
490
}
491
492
test_urls = [
493
'https://example.com',
494
'https://httpbin.org/headers',
495
'https://protected-site.com'
496
]
497
498
results = test_stealth_effectiveness(test_urls, stealth_configs)
499
```
500
501
## Stealth Best Practices
502
503
### Site-Specific Stealth
504
505
Configure stealth settings based on target site characteristics:
506
507
```python
508
# High-security financial sites
509
financial_scraper = cloudscraper.create_scraper(
510
enable_stealth=True,
511
stealth_options={
512
'min_delay': 5.0,
513
'max_delay': 15.0,
514
'human_like_delays': True,
515
'randomize_headers': True,
516
'browser_quirks': True
517
},
518
session_refresh_interval=900, # 15 minutes
519
rotate_tls_ciphers=True
520
)
521
522
# E-commerce sites
523
ecommerce_scraper = cloudscraper.create_scraper(
524
enable_stealth=True,
525
stealth_options={
526
'min_delay': 2.0,
527
'max_delay': 6.0,
528
'human_like_delays': True,
529
'randomize_headers': True
530
},
531
session_refresh_interval=3600 # 1 hour
532
)
533
534
# News/content sites
535
content_scraper = cloudscraper.create_scraper(
536
enable_stealth=True,
537
stealth_options={
538
'min_delay': 1.0,
539
'max_delay': 3.0,
540
'human_like_delays': True
541
}
542
)
543
```
544
545
### Performance vs Stealth Balance
546
547
Balance stealth effectiveness with performance requirements:
548
549
```python
550
# Maximum performance (minimal stealth)
551
fast_scraper = cloudscraper.create_scraper(
552
enable_stealth=True,
553
stealth_options={
554
'min_delay': 0.1,
555
'max_delay': 0.5,
556
'human_like_delays': False,
557
'randomize_headers': False
558
}
559
)
560
561
# Balanced performance and stealth
562
balanced_scraper = cloudscraper.create_scraper(
563
enable_stealth=True,
564
stealth_options={
565
'min_delay': 1.0,
566
'max_delay': 3.0,
567
'human_like_delays': True,
568
'randomize_headers': True
569
}
570
)
571
572
# Maximum stealth (slower performance)
573
stealth_scraper = cloudscraper.create_scraper(
574
enable_stealth=True,
575
stealth_options={
576
'min_delay': 5.0,
577
'max_delay': 20.0,
578
'human_like_delays': True,
579
'randomize_headers': True,
580
'browser_quirks': True
581
},
582
rotate_tls_ciphers=True
583
)
584
```
585
586
### Error Handling with Stealth
587
588
Handle detection-related errors with stealth mode:
589
590
```python
591
try:
592
scraper = cloudscraper.create_scraper(
593
enable_stealth=True,
594
stealth_options={'min_delay': 2.0, 'max_delay': 5.0}
595
)
596
response = scraper.get('https://protected-site.com')
597
598
except cloudscraper.CloudflareLoopProtection:
599
print("Multiple challenges detected - increase stealth settings")
600
# Retry with more aggressive stealth
601
scraper = cloudscraper.create_scraper(
602
enable_stealth=True,
603
stealth_options={
604
'min_delay': 10.0,
605
'max_delay': 30.0,
606
'human_like_delays': True,
607
'randomize_headers': True,
608
'browser_quirks': True
609
}
610
)
611
response = scraper.get('https://protected-site.com')
612
613
except Exception as e:
614
if '403' in str(e) or '429' in str(e):
615
print("Access denied - consider more aggressive stealth settings")
616
raise
617
```