0
# Python Logstash
1

2
Python logging handler for Logstash that enables applications to send log messages directly to Logstash using multiple transport protocols. The library provides custom logging handlers for UDP, TCP, and AMQP connections with configurable message formatting compatible with different Logstash schema versions.
3

4
## Package Information
5

6
- **Package Name**: python-logstash
7
- **Language**: Python
8
- **Installation**: `pip install python-logstash`
9
- **AMQP Support**: `pip install pika` (required for AMQP handler)
10

11
## Core Imports
12

13
```python
14
import logstash
15
```
16

17
Common usage patterns:
18

19
```python
20
from logstash import UDPLogstashHandler, TCPLogstashHandler, AMQPLogstashHandler
21
from logstash import LogstashHandler  # alias for UDPLogstashHandler
22
from logstash import LogstashFormatterVersion0, LogstashFormatterVersion1
23
```
24

25
For custom formatter usage:
26

27
```python
28
from logstash.formatter import LogstashFormatterBase
29
```
30

31
## Basic Usage
32

33
```python
34
import logging
35
import logstash
36

37
# Configure logger
38
logger = logging.getLogger('my-app')
39
logger.setLevel(logging.INFO)
40

41
# Add UDP handler (default)
42
logger.addHandler(logstash.UDPLogstashHandler('localhost', 5959, version=1))
43

44
# Send log messages
45
logger.info('Application started')
46
logger.error('An error occurred')
47

48
# Add extra fields
49
extra = {
50
    'user_id': 12345,
51
    'request_id': 'abc-123',
52
    'custom_data': {'key': 'value'}
53
}
54
logger.info('User action completed', extra=extra)
55
```
56

57
## Architecture
58

59
The python-logstash library follows a modular handler-formatter architecture:
60

61
- **Handlers**: Transport-specific classes (`UDPLogstashHandler`, `TCPLogstashHandler`, `AMQPLogstashHandler`) that inherit from Python's standard logging handlers and manage network connections
62
- **Formatters**: Schema-specific classes (`LogstashFormatterVersion0`, `LogstashFormatterVersion1`) that convert Python log records into Logstash-compatible JSON format
63
- **Base Classes**: `LogstashFormatterBase` provides common functionality for all formatters
64

65
Each handler automatically selects the appropriate formatter based on the specified schema version, enabling seamless integration with different Logstash configurations while maintaining backward compatibility.
66

67
## Capabilities
68

69
### UDP Logging Handler
70

71
Fast, fire-and-forget logging via UDP. Suitable for high-throughput scenarios where message delivery is not critical.
72

73
```python { .api }
74
class UDPLogstashHandler(TCPLogstashHandler, DatagramHandler):
75
    def __init__(self, host, port=5959, message_type='logstash', tags=None, fqdn=False, version=0):
76
        """
77
        Python logging handler for Logstash. Sends events over UDP.
78
        
79
        Parameters:
80
        - host (str): The host of the logstash server
81
        - port (int): The port of the logstash server (default: 5959)
82
        - message_type (str): The type of the message (default: 'logstash')
83
        - tags (list): List of tags for a logger (default: None)
84
        - fqdn (bool): Show fully qualified domain name (default: False)
85
        - version (int): Version of logstash event schema (default: 0)
86
        """
87
    
88
    def makePickle(self, record):
89
        """
90
        Serialize log record for UDP transmission.
91
        
92
        Parameters:
93
        - record: LogRecord object
94
        
95
        Returns:
96
        - bytes: Formatted message without newline
97
        """
98
```
99

100
### TCP Logging Handler
101

102
Reliable logging via TCP connections. Provides guaranteed message delivery with connection management.
103

104
```python { .api }
105
class TCPLogstashHandler(SocketHandler):
106
    def __init__(self, host, port=5959, message_type='logstash', tags=None, fqdn=False, version=0):
107
        """
108
        Python logging handler for Logstash. Sends events over TCP.
109
        
110
        Parameters:
111
        - host (str): The host of the logstash server
112
        - port (int): The port of the logstash server (default: 5959)
113
        - message_type (str): The type of the message (default: 'logstash')
114
        - tags (list): List of tags for a logger (default: None)
115
        - fqdn (bool): Show fully qualified domain name (default: False)
116
        - version (int): Version of logstash event schema (default: 0)
117
        """
118
    
119
    def makePickle(self, record):
120
        """
121
        Serialize log record for TCP transmission.
122
        
123
        Parameters:
124
        - record: LogRecord object
125
        
126
        Returns:
127
        - bytes: Formatted message with trailing newline
128
        """
129
```
130

131
### AMQP Logging Handler
132

133
Message queue-based logging via RabbitMQ. Provides reliable, asynchronous message delivery with advanced routing capabilities.
134

135
```python { .api }
136
class AMQPLogstashHandler(SocketHandler):
137
    def __init__(self, host='localhost', port=5672, username='guest', password='guest', 
138
                 exchange='logstash', exchange_type='fanout', virtual_host='/', 
139
                 message_type='logstash', tags=None, durable=False, passive=False, 
140
                 version=0, extra_fields=True, fqdn=False, facility=None, 
141
                 exchange_routing_key=''):
142
        """
143
        AMQP Log Format handler for RabbitMQ integration.
144
        
145
        Parameters:
146
        - host (str): AMQP host (default: 'localhost')
147
        - port (int): AMQP port (default: 5672)
148
        - username (str): AMQP username (default: 'guest')
149
        - password (str): AMQP password (default: 'guest')
150
        - exchange (str): AMQP exchange (default: 'logstash')
151
        - exchange_type (str): AMQP exchange type (default: 'fanout')
152
        - virtual_host (str): AMQP virtual host (default: '/')
153
        - message_type (str): The type of the message (default: 'logstash')
154
        - tags (list): List of tags for a logger (default: None)
155
        - durable (bool): AMQP exchange durability (default: False)
156
        - passive (bool): Exchange declaration mode (default: False)
157
        - version (int): Version of logstash event schema (default: 0)
158
        - extra_fields (bool): Send extra fields on log record (default: True)
159
        - fqdn (bool): Use fully qualified domain name (default: False)
160
        - facility (str): Replace facility with specified value (default: None)
161
        - exchange_routing_key (str): AMQP routing key (default: '')
162
        """
163
    
164
    def makeSocket(self):
165
        """
166
        Create AMQP socket connection.
167
        
168
        Returns:
169
        - PikaSocket: AMQP connection wrapper
170
        """
171
    
172
    def makePickle(self, record):
173
        """
174
        Serialize log record for AMQP transmission.
175
        
176
        Parameters:
177
        - record: LogRecord object
178
        
179
        Returns:
180
        - bytes: Formatted message
181
        """
182
```
183

184
### Backward Compatibility Handler
185

186
Alias for UDPLogstashHandler to maintain backward compatibility.
187

188
```python { .api }
189
LogstashHandler = UDPLogstashHandler
190
```
191

192
### Message Formatters
193

194
Control the structure and format of log messages sent to Logstash.
195

196
```python { .api }
197
class LogstashFormatterBase(logging.Formatter):
198
    def __init__(self, message_type='Logstash', tags=None, fqdn=False):
199
        """
200
        Base formatter providing common functionality for all Logstash formatters.
201
        
202
        Parameters:
203
        - message_type (str): Type of message (default: 'Logstash')
204
        - tags (list): List of tags (default: None)
205
        - fqdn (bool): Use fully qualified domain name (default: False)
206
        """
207
    
208
    def get_extra_fields(self, record):
209
        """
210
        Extract extra fields from log record, filtering sensitive data.
211
        
212
        Parameters:
213
        - record: LogRecord object
214
        
215
        Returns:
216
        - dict: Extra fields to include in message
217
        """
218
    
219
    def get_debug_fields(self, record):
220
        """
221
        Extract debug information from log record.
222
        
223
        Parameters:
224
        - record: LogRecord object
225
        
226
        Returns:
227
        - dict: Debug fields including stack trace and process info
228
        """
229
    
230
    @classmethod
231
    def format_source(cls, message_type, host, path):
232
        """
233
        Format source field for Logstash message.
234
        
235
        Parameters:
236
        - message_type (str): Message type
237
        - host (str): Hostname
238
        - path (str): File path
239
        
240
        Returns:
241
        - str: Formatted source string
242
        """
243
    
244
    @classmethod
245
    def format_timestamp(cls, time):
246
        """
247
        Format timestamp in ISO 8601 format.
248
        
249
        Parameters:
250
        - time (float): Unix timestamp
251
        
252
        Returns:
253
        - str: ISO 8601 formatted timestamp
254
        """
255
    
256
    @classmethod
257
    def format_exception(cls, exc_info):
258
        """
259
        Format exception information as string.
260
        
261
        Parameters:
262
        - exc_info: Exception info tuple
263
        
264
        Returns:
265
        - str: Formatted exception string
266
        """
267
    
268
    @classmethod
269
    def serialize(cls, message):
270
        """
271
        Serialize message to JSON bytes.
272
        
273
        Parameters:
274
        - message (dict): Message dictionary
275
        
276
        Returns:
277
        - bytes: JSON-serialized message
278
        """
279

280
class LogstashFormatterVersion0(LogstashFormatterBase):
281
    version = 0
282
    
283
    def format(self, record):
284
        """
285
        Format log record to Logstash v0 schema (@fields structure).
286
        
287
        Parameters:
288
        - record: LogRecord object
289
        
290
        Returns:
291
        - bytes: JSON-formatted message
292
        """
293

294
class LogstashFormatterVersion1(LogstashFormatterBase):
295
    def format(self, record):
296
        """
297
        Format log record to Logstash v1 schema (flat structure).
298
        
299
        Parameters:
300
        - record: LogRecord object
301
        
302
        Returns:
303
        - bytes: JSON-formatted message
304
        """
305
```
306

307
### AMQP Connection Management
308

309
Internal socket wrapper for AMQP connections used by AMQPLogstashHandler.
310

311
```python { .api }
312
class PikaSocket:
313
    def __init__(self, host, port, username, password, virtual_host, exchange, 
314
                 routing_key, durable, passive, exchange_type):
315
        """
316
        AMQP socket wrapper for RabbitMQ connections.
317
        
318
        Parameters:
319
        - host (str): AMQP host
320
        - port (int): AMQP port
321
        - username (str): AMQP username
322
        - password (str): AMQP password
323
        - virtual_host (str): AMQP virtual host
324
        - exchange (str): AMQP exchange name
325
        - routing_key (str): AMQP routing key
326
        - durable (bool): Exchange durability
327
        - passive (bool): Exchange declaration mode
328
        - exchange_type (str): Exchange type
329
        """
330
    
331
    def sendall(self, data):
332
        """
333
        Send data via AMQP channel.
334
        
335
        Parameters:
336
        - data (bytes): Message data to send
337
        """
338
    
339
    def close(self):
340
        """
341
        Close AMQP connection.
342
        """
343
```
344

345
## Usage Examples
346

347
### UDP Handler Example
348

349
```python
350
import logging
351
import logstash
352

353
# Create logger
354
test_logger = logging.getLogger('python-logstash-logger')
355
test_logger.setLevel(logging.INFO)
356

357
# Add UDP handler with version 1 schema
358
test_logger.addHandler(logstash.UDPLogstashHandler('localhost', 5959, version=1))
359

360
# Log messages
361
test_logger.error('python-logstash: test logstash error message.')
362
test_logger.info('python-logstash: test logstash info message.')
363
test_logger.warning('python-logstash: test logstash warning message.')
364

365
# Add extra fields to logstash message
366
extra = {
367
    'test_string': 'python version: ' + repr(sys.version_info),
368
    'test_boolean': True,
369
    'test_dict': {'a': 1, 'b': 'c'},
370
    'test_float': 1.23,
371
    'test_integer': 123,
372
    'test_list': [1, 2, '3'],
373
}
374
test_logger.info('python-logstash: test extra fields', extra=extra)
375
```
376

377
### TCP Handler Example
378

379
```python
380
import logging
381
import logstash
382

383
# Create logger
384
test_logger = logging.getLogger('python-logstash-logger')
385
test_logger.setLevel(logging.INFO)
386

387
# Add TCP handler for reliable delivery
388
test_logger.addHandler(logstash.TCPLogstashHandler('localhost', 5959, version=1))
389

390
# Log messages
391
test_logger.info('Reliable TCP logging message')
392
```
393

394
### AMQP Handler Example
395

396
```python
397
import logging
398
import logstash
399

400
# AMQP configuration
401
host = 'localhost'
402
username = 'guest'
403
password = 'guest'
404
exchange = 'logstash.py'
405

406
# Create logger
407
test_logger = logging.getLogger('python-logstash-logger')
408
test_logger.setLevel(logging.INFO)
409

410
# Add AMQP handler
411
test_logger.addHandler(logstash.AMQPLogstashHandler(
412
    version=1,
413
    host=host,
414
    durable=True,
415
    username=username,
416
    password=password,
417
    exchange=exchange
418
))
419

420
# Log messages
421
test_logger.error('python-logstash: test logstash error message.')
422
test_logger.info('python-logstash: test logstash info message.')
423
test_logger.warning('python-logstash: test logstash warning message.')
424

425
# Log exception with stack trace
426
try:
427
    1/0
428
except:
429
    test_logger.exception('python-logstash: test logstash exception with stack trace')
430
```
431

432
### Django Integration
433

434
Configure in `settings.py`:
435

436
```python
437
LOGGING = {
438
    'version': 1,
439
    'disable_existing_loggers': False,
440
    'handlers': {
441
        'logstash': {
442
            'level': 'DEBUG',
443
            'class': 'logstash.TCPLogstashHandler',
444
            'host': 'localhost',
445
            'port': 5959,
446
            'version': 1,
447
            'message_type': 'logstash',
448
            'fqdn': False,
449
            'tags': ['django', 'production'],
450
        },
451
    },
452
    'loggers': {
453
        'django.request': {
454
            'handlers': ['logstash'],
455
            'level': 'DEBUG',
456
            'propagate': True,
457
        },
458
    },
459
}
460
```
461

462
## Configuration Options
463

464
### Schema Versions
465

466
- **Version 0**: Legacy Logstash format (pre-1.2.x) with `@fields` structure
467
- **Version 1**: Current Logstash format (1.2.x+) with flat field structure
468

469
### Message Fields
470

471
All handlers automatically include:
472
- `@timestamp`: ISO 8601 timestamp
473
- `message`: The log message
474
- `level`/`levelname`: Log level
475
- `logger_name`: Logger name
476
- `host`: Hostname (FQDN if enabled)
477
- `path`: Source file path
478

479
### Extra Fields
480

481
Custom fields can be added via the `extra` parameter in logging calls. Sensitive fields (`auth_token`, `password`, etc.) are automatically filtered out.
482

483
### Error Handling
484

485
- Network connection failures are handled gracefully
486
- Invalid field types are converted to string representations
487
- Exception information is captured when present in log records
488

489
## Dependencies
490

491
- **Core**: Python standard library (`logging`, `socket`, `json`, `datetime`)
492
- **AMQP Handler**: `pika` library for RabbitMQ connectivity
493
- **Compatibility**: Python 2.x and 3.x supported