0
# Ingestor SDKs and Integration Libraries
1

2
Metlo provides comprehensive ingestor SDKs for collecting API traffic across multiple programming languages and frameworks. These libraries automatically capture HTTP requests and responses, perform security analysis, and forward data to the Metlo backend for monitoring and vulnerability detection.
3

4
## Capabilities
5

6
### Node.js Ingestor
7

8
#### Core Package (`metlo`)
9

10
The main Node.js ingestor package for automatic traffic collection and analysis.
11

12
```typescript { .api }
13
/**
14
 * Initialize Metlo with backend connection
15
 * @param key - Metlo API key for authentication
16
 * @param host - Metlo backend URL
17
 * @param opts - Optional configuration options
18
 */
19
function init(key: string, host: string, opts?: ConfigOptions): void;
20

21
interface ConfigOptions {
22
  /** Number of requests per second to send to backend (default: 10) */
23
  rps?: number;
24
  /** Disable Metlo collection entirely */
25
  disable?: boolean;
26
  /** Custom configuration for redacting sensitive data */
27
  redactionConfig?: RedactionConfig;
28
  /** Maximum request body size to capture (bytes) */
29
  maxBodySize?: number;
30
  /** Headers to exclude from capture */
31
  excludeHeaders?: string[];
32
  /** Paths to exclude from monitoring */
33
  excludePaths?: string[];
34
}
35

36
interface RedactionConfig {
37
  /** Automatically redact detected PII data */
38
  redactPII?: boolean;
39
  /** Custom field patterns to redact */
40
  customRedactFields?: string[];
41
  /** Replacement text for redacted fields */
42
  redactionText?: string;
43
}
44
```
45

46
**Usage Examples:**
47

48
```typescript
49
import { init } from "metlo";
50

51
// Basic initialization
52
init("your-api-key", "https://your-metlo-backend.com");
53

54
// Advanced configuration
55
init("your-api-key", "https://your-metlo-backend.com", {
56
  rps: 20,
57
  disable: process.env.NODE_ENV === "test",
58
  redactionConfig: {
59
    redactPII: true,
60
    customRedactFields: ["password", "secret", "token"],
61
    redactionText: "[REDACTED]"
62
  },
63
  maxBodySize: 50 * 1024, // 50KB
64
  excludeHeaders: ["authorization", "cookie"],
65
  excludePaths: ["/health", "/metrics"]
66
});
67
```
68

69
#### Express.js Integration
70

71
Automatic middleware injection for Express applications.
72

73
```typescript { .api }
74
// Automatically detects Express usage and injects middleware
75
// No additional configuration required beyond init()
76
```
77

78
**Usage Examples:**
79

80
```typescript
81
import express from "express";
82
import { init } from "metlo";
83

84
// Initialize Metlo before setting up routes
85
init("your-api-key", "https://your-metlo-backend.com");
86

87
const app = express();
88

89
// Your existing routes - Metlo automatically captures traffic
90
app.get("/api/users", (req, res) => {
91
  res.json({ users: [] });
92
});
93

94
app.post("/api/users", (req, res) => {
95
  // Metlo captures both request and response automatically
96
  res.json({ created: true });
97
});
98

99
app.listen(3000);
100
```
101

102
#### Framework Support
103

104
The Node.js ingestor supports popular web frameworks through automatic detection:
105

106
- **Express.js** - Automatic middleware injection
107
- **Koa.js** - Automatic middleware injection  
108
- **Fastify** - Plugin-based integration
109
- **Next.js** - API routes monitoring
110
- **NestJS** - Interceptor-based capture
111

112
### Python Ingestors
113

114
#### Flask Integration
115

116
Flask middleware for automatic API traffic collection.
117

118
```python { .api }
119
from metlo.flask import MetloFlask
120

121
# Initialize Metlo with Flask app
122
metlo = MetloFlask(
123
    app,                          # Flask application instance
124
    api_key="your-api-key",      # Metlo API key
125
    base_url="https://your-metlo-backend.com",  # Metlo backend URL
126
    rps=10,                      # Requests per second limit
127
    disable=False                # Disable flag for testing
128
)
129
```
130

131
**Usage Examples:**
132

133
```python
134
from flask import Flask, request, jsonify
135
from metlo.flask import MetloFlask
136

137
app = Flask(__name__)
138

139
# Initialize Metlo - automatically captures all routes
140
metlo = MetloFlask(
141
    app,
142
    api_key="your-api-key",  
143
    base_url="https://your-metlo-backend.com",
144
    rps=15
145
)
146

147
@app.route("/api/users", methods=["GET"])
148
def get_users():
149
    # Metlo automatically captures this request/response
150
    return jsonify({"users": []})
151

152
@app.route("/api/users", methods=["POST"])  
153
def create_user():
154
    user_data = request.get_json()
155
    # Metlo captures request body and response
156
    return jsonify({"created": True, "id": 123})
157

158
if __name__ == "__main__":
159
    app.run()
160
```
161

162
#### Django Integration
163

164
Django middleware for comprehensive traffic monitoring.
165

166
```python { .api }
167
# settings.py
168
MIDDLEWARE = [
169
    'metlo.django.MetloMiddleware',  # Add as first middleware
170
    # ... other middleware
171
]
172

173
METLO_CONFIG = {
174
    'API_KEY': 'your-api-key',
175
    'BASE_URL': 'https://your-metlo-backend.com',
176
    'RPS': 10,
177
    'DISABLE': False,
178
    'REDACT_PII': True,
179
    'EXCLUDE_PATHS': ['/admin/', '/static/'],
180
}
181
```
182

183
**Usage Examples:**
184

185
```python
186
# settings.py
187
MIDDLEWARE = [
188
    'metlo.django.MetloMiddleware',
189
    'django.middleware.security.SecurityMiddleware',
190
    'django.contrib.sessions.middleware.SessionMiddleware',
191
    # ... other middleware
192
]
193

194
METLO_CONFIG = {
195
    'API_KEY': os.environ.get('METLO_API_KEY'),
196
    'BASE_URL': 'https://your-metlo-backend.com',
197
    'RPS': 20,
198
    'REDACT_PII': True,
199
    'EXCLUDE_PATHS': ['/admin/', '/static/', '/media/'],
200
    'MAX_BODY_SIZE': 51200,  # 50KB
201
}
202

203
# views.py - No changes needed, Metlo captures automatically
204
from django.http import JsonResponse
205
from django.views.decorators.csrf import csrf_exempt
206
import json
207

208
@csrf_exempt
209
def api_users(request):
210
    if request.method == 'GET':
211
        # Metlo captures this response
212
        return JsonResponse({'users': []})
213
    elif request.method == 'POST':
214
        data = json.loads(request.body)
215
        # Metlo captures request body and response
216
        return JsonResponse({'created': True})
217
```
218

219
### Java Ingestors
220

221
#### Spring Boot Integration
222

223
Comprehensive Spring framework integration with automatic configuration.
224

225
```java { .api }
226
// Maven dependency
227
<dependency>
228
    <groupId>com.metlo</groupId>
229
    <artifactId>metlo-spring-boot-starter</artifactId>
230
    <version>1.0.0</version>
231
</dependency>
232

233
// Application properties
234
metlo.api-key=your-api-key
235
metlo.base-url=https://your-metlo-backend.com
236
metlo.rps=10
237
metlo.enabled=true
238
metlo.redact-pii=true
239
metlo.exclude-paths=/actuator/**,/swagger-ui/**
240

241
// Automatic configuration via @EnableMetlo
242
@SpringBootApplication
243
@EnableMetlo
244
public class Application {
245
    public static void main(String[] args) {
246
        SpringApplication.run(Application.class, args);
247
    }
248
}
249
```
250

251
**Usage Examples:**
252

253
```java
254
// Application.java
255
@SpringBootApplication
256
@EnableMetlo  // Enables automatic Metlo integration
257
public class ApiApplication {
258
    public static void main(String[] args) {
259
        SpringApplication.run(ApiApplication.class, args);
260
    }
261
}
262

263
// Controller - No changes needed
264
@RestController
265
@RequestMapping("/api")
266
public class UserController {
267
    
268
    @GetMapping("/users")
269
    public ResponseEntity<List<User>> getUsers() {
270
        // Metlo automatically captures this endpoint
271
        List<User> users = userService.findAll();
272
        return ResponseEntity.ok(users);
273
    }
274
    
275
    @PostMapping("/users")
276
    public ResponseEntity<User> createUser(@RequestBody CreateUserRequest request) {
277
        // Metlo captures request body and response automatically
278
        User user = userService.create(request);
279
        return ResponseEntity.status(HttpStatus.CREATED).body(user);
280
    }
281
}
282

283
// application.yml configuration
284
metlo:
285
  api-key: ${METLO_API_KEY}
286
  base-url: https://your-metlo-backend.com
287
  rps: 15
288
  enabled: true
289
  redact-pii: true
290
  exclude-paths:
291
    - /actuator/**
292
    - /swagger-ui/**
293
    - /health
294
  max-body-size: 51200
295
```
296

297
#### Spring WebFlux (Reactive) Integration
298

299
Support for reactive Spring applications.
300

301
```java { .api }
302
@Configuration
303
@EnableMetloWebFlux
304
public class MetloConfiguration {
305
    
306
    @Bean
307
    public MetloWebFilter metloWebFilter() {
308
        return MetloWebFilter.builder()
309
            .apiKey("your-api-key")
310
            .baseUrl("https://your-metlo-backend.com")
311
            .rps(10)
312
            .redactPII(true)
313
            .build();
314
    }
315
}
316
```
317

318
### Go Ingestors
319

320
#### Core Go Package
321

322
Main Go library for HTTP traffic collection.
323

324
```go { .api }
325
package metlo
326

327
// Initialize Metlo with default settings
328
func InitMetlo(metloHost, metloKey string) *Metlo
329

330
// Initialize Metlo with custom configuration  
331
func InitMetloCustom(metloHost, metloKey string, rps int, disable bool) *Metlo
332

333
type Metlo struct {
334
    // Configuration options
335
    Host     string
336
    APIKey   string
337
    RPS      int
338
    Disabled bool
339
}
340

341
// Middleware for standard http.Handler
342
func (m *Metlo) Middleware(next http.Handler) http.Handler
343

344
// Configuration options
345
type Config struct {
346
    APIKey          string
347
    Host           string
348
    RPS            int
349
    Disable        bool
350
    RedactPII      bool
351
    ExcludePaths   []string
352
    MaxBodySize    int64
353
}
354
```
355

356
**Usage Examples:**
357

358
```go
359
package main
360

361
import (
362
    "net/http"
363
    "github.com/metlo-labs/metlo/golang/metlo"
364
)
365

366
func main() {
367
    // Initialize Metlo
368
    m := metlo.InitMetlo("https://your-metlo-backend.com", "your-api-key")
369
    
370
    // Create your HTTP handlers
371
    mux := http.NewServeMux()
372
    mux.HandleFunc("/api/users", func(w http.ResponseWriter, r *http.Request) {
373
        w.Header().Set("Content-Type", "application/json")
374
        w.Write([]byte(`{"users": []}`))
375
    })
376
    
377
    // Wrap with Metlo middleware
378
    handler := m.Middleware(mux)
379
    
380
    // Start server
381
    http.ListenAndServe(":8080", handler)
382
}
383

384
// Advanced configuration
385
func mainAdvanced() {
386
    config := metlo.Config{
387
        APIKey:       "your-api-key",
388
        Host:         "https://your-metlo-backend.com", 
389
        RPS:          20,
390
        RedactPII:    true,
391
        ExcludePaths: []string{"/health", "/metrics"},
392
        MaxBodySize:  50 * 1024, // 50KB
393
    }
394
    
395
    m := metlo.InitMetloWithConfig(config)
396
    
397
    // Use with your web framework
398
    handler := m.Middleware(yourHandler)
399
    http.ListenAndServe(":8080", handler)
400
}
401
```
402

403
#### Gin Framework Integration
404

405
Dedicated middleware for the Gin web framework.
406

407
```go { .api }
408
import "github.com/metlo-labs/metlo/golang/gin"
409

410
// Gin middleware
411
func MetloGinMiddleware(apiKey, host string) gin.HandlerFunc
412
func MetloGinMiddlewareWithConfig(config metlo.Config) gin.HandlerFunc
413
```
414

415
**Usage Examples:**
416

417
```go
418
package main
419

420
import (
421
    "github.com/gin-gonic/gin"
422
    "github.com/metlo-labs/metlo/golang/gin"
423
)
424

425
func main() {
426
    r := gin.Default()
427
    
428
    // Add Metlo middleware
429
    r.Use(gin_metlo.MetloGinMiddleware("your-api-key", "https://your-metlo-backend.com"))
430
    
431
    // Define routes - Metlo captures automatically
432
    r.GET("/api/users", func(c *gin.Context) {
433
        c.JSON(200, gin.H{"users": []string{}})
434
    })
435
    
436
    r.POST("/api/users", func(c *gin.Context) {
437
        var user map[string]interface{}
438
        c.ShouldBindJSON(&user)
439
        c.JSON(201, gin.H{"created": true, "id": 123})
440
    })
441
    
442
    r.Run(":8080")
443
}
444
```
445

446
#### Gorilla Mux Integration
447

448
Integration with Gorilla Mux router.
449

450
```go { .api }
451
import "github.com/metlo-labs/metlo/golang/mux"
452

453
// Mux middleware
454
func MetloMuxMiddleware(apiKey, host string) mux.MiddlewareFunc
455
```
456

457
### Rust Ingestor
458

459
#### Core Rust Crate
460

461
Common utilities and types for Rust integrations.
462

463
```rust { .api }
464
use metlo_rust::{MetloConfig, MetloClient};
465

466
pub struct MetloConfig {
467
    pub api_key: String,
468
    pub host: String,
469
    pub rps: u32,
470
    pub disable: bool,
471
    pub redact_pii: bool,
472
}
473

474
pub struct MetloClient {
475
    config: MetloConfig,
476
}
477

478
impl MetloClient {
479
    pub fn new(config: MetloConfig) -> Self;
480
    pub async fn capture_request(&self, request: CaptureRequest) -> Result<(), MetloError>;
481
}
482

483
pub struct CaptureRequest {
484
    pub method: String,
485
    pub path: String,
486
    pub headers: Vec<(String, String)>,
487
    pub body: Option<String>,
488
    pub response_status: u16,
489
    pub response_headers: Vec<(String, String)>,
490
    pub response_body: Option<String>,
491
}
492
```
493

494
**Usage Examples:**
495

496
```rust
497
use metlo_rust::{MetloConfig, MetloClient};
498
use warp::Filter;
499

500
#[tokio::main]
501
async fn main() {
502
    // Initialize Metlo
503
    let config = MetloConfig {
504
        api_key: "your-api-key".to_string(),
505
        host: "https://your-metlo-backend.com".to_string(),
506
        rps: 10,
507
        disable: false,
508
        redact_pii: true,
509
    };
510
    
511
    let metlo_client = MetloClient::new(config);
512
    
513
    // Create Warp filter with Metlo middleware
514
    let api = warp::path("api")
515
        .and(warp::path("users"))
516
        .and(warp::get())
517
        .and(with_metlo(metlo_client))
518
        .and_then(get_users);
519
    
520
    warp::serve(api)
521
        .run(([127, 0, 0, 1], 3030))
522
        .await;
523
}
524

525
async fn get_users() -> Result<impl warp::Reply, warp::Rejection> {
526
    Ok(warp::reply::json(&serde_json::json!({"users": []})))
527
}
528
```
529

530
### Kubernetes Integration
531

532
#### Deployment Configuration
533

534
Deploy Metlo ingestors in Kubernetes environments.
535

536
```yaml { .api }
537
# metlo-configmap.yaml
538
apiVersion: v1
539
kind: ConfigMap
540
metadata:
541
  name: metlo-config
542
data:
543
  METLO_API_KEY: "your-api-key"
544
  METLO_HOST: "https://your-metlo-backend.com"
545
  METLO_RPS: "10"
546

547
---
548
# metlo-deployment.yaml  
549
apiVersion: apps/v1
550
kind: Deployment
551
metadata:
552
  name: your-app
553
spec:
554
  template:
555
    spec:
556
      containers:
557
      - name: app
558
        image: your-app:latest
559
        env:
560
        - name: METLO_API_KEY
561
          valueFrom:
562
            configMapKeyRef:
563
              name: metlo-config
564
              key: METLO_API_KEY
565
        - name: METLO_HOST
566
          valueFrom:
567
            configMapKeyRef:
568
              name: metlo-config
569
              key: METLO_HOST
570
```
571

572
#### DaemonSet for Traffic Mirroring
573

574
Deploy traffic mirroring agents across Kubernetes nodes.
575

576
```yaml { .api }
577
apiVersion: apps/v1
578
kind: DaemonSet
579
metadata:
580
  name: metlo-traffic-mirror
581
spec:
582
  selector:
583
    matchLabels:
584
      name: metlo-traffic-mirror
585
  template:
586
    metadata:
587
      labels:
588
        name: metlo-traffic-mirror
589
    spec:
590
      hostNetwork: true
591
      containers:
592
      - name: metlo-mirror
593
        image: metlo/traffic-mirror:latest
594
        env:
595
        - name: METLO_API_KEY
596
          value: "your-api-key"
597
        - name: METLO_HOST
598
          value: "https://your-metlo-backend.com"
599
        securityContext:
600
          privileged: true
601
        volumeMounts:
602
        - name: proc
603
          mountPath: /host/proc
604
          readOnly: true
605
        - name: sys
606
          mountPath: /host/sys
607
          readOnly: true
608
      volumes:
609
      - name: proc
610
        hostPath:
611
          path: /proc
612
      - name: sys  
613
        hostPath:
614
          path: /sys
615
```
616

617
### Burp Suite Integration
618

619
#### Security Testing Extension
620

621
Burp Suite extension for API security testing integration.
622

623
```java { .api }
624
// Extension configuration
625
public class MetloBurpExtension implements IBurpExtender {
626
    
627
    public void registerExtenderCallbacks(IBurpExtenderCallbacks callbacks) {
628
        // Initialize Metlo integration
629
        MetloConfig config = new MetloConfig(
630
            "your-api-key",
631
            "https://your-metlo-backend.com"
632
        );
633
        
634
        // Register HTTP listener
635
        callbacks.registerHttpListener(new MetloHttpListener(config));
636
    }
637
}
638
```
639

640
**Features:**
641
- Automatic request/response capture during security testing
642
- Integration with Metlo test generation
643
- Export scan results to Metlo backend
644
- Custom vulnerability detection rules
645

646
### Docker Compose Integration
647

648
#### Complete Environment Setup
649

650
Docker Compose configuration for development environments.
651

652
```yaml { .api }
653
version: '3.8'
654
services:
655
  app:
656
    build: .
657
    environment:
658
      - METLO_API_KEY=your-api-key
659
      - METLO_HOST=http://metlo-backend:3000
660
      - METLO_RPS=15
661
    depends_on:
662
      - metlo-backend
663
      
664
  metlo-backend:
665
    image: metlo/backend:latest
666
    ports:
667
      - "3000:3000"
668
    environment:
669
      - DATABASE_URL=postgresql://postgres:password@db:5432/metlo
670
      - REDIS_URL=redis://redis:6379
671
    depends_on:
672
      - db
673
      - redis
674
      
675
  db:
676
    image: postgres:13
677
    environment:
678
      - POSTGRES_DB=metlo
679
      - POSTGRES_PASSWORD=password
680
      
681
  redis:
682
    image: redis:6-alpine
683
```
684

685
## Configuration Best Practices
686

687
### Production Configuration
688

689
```typescript
690
// Recommended production settings
691
const productionConfig = {
692
  rps: 50,                    // Higher throughput for production
693
  redactPII: true,            // Always redact sensitive data
694
  maxBodySize: 100 * 1024,    // 100KB limit
695
  excludePaths: [
696
    "/health",
697
    "/metrics", 
698
    "/favicon.ico",
699
    "/robots.txt"
700
  ],
701
  excludeHeaders: [
702
    "authorization",
703
    "cookie",
704
    "x-api-key"
705
  ]
706
};
707
```
708

709
### Development Configuration
710

711
```typescript
712
// Development/testing settings
713
const developmentConfig = {
714
  rps: 10,                    // Lower rate for dev
715
  disable: process.env.NODE_ENV === "test",
716
  redactPII: false,           // Keep data for debugging
717
  maxBodySize: 50 * 1024,     // 50KB limit
718
  verbose: true               // Enable detailed logging
719
};
720
```
721

722
### Performance Optimization
723

724
- **Rate Limiting**: Configure appropriate RPS based on your traffic volume
725
- **Body Size Limits**: Set reasonable limits to avoid memory issues
726
- **Path Exclusion**: Exclude health checks, static assets, and admin paths
727
- **Header Filtering**: Exclude sensitive headers like authentication tokens
728
- **Async Processing**: Use background queues for high-traffic applications
729

730
### Security Considerations
731

732
- **API Key Management**: Use environment variables, never hardcode keys
733
- **Data Redaction**: Enable PII redaction in production environments
734
- **Network Security**: Use HTTPS for all Metlo backend communication
735
- **Access Control**: Restrict Metlo backend access to authorized networks
736
- **Data Retention**: Configure appropriate data retention policies