0
# Test Utilities
1
2
Testing framework for developing and validating custom checkers, including test case base classes, message validation, and functional testing support. Pylint's test utilities enable comprehensive testing of static analysis functionality and custom checker development.
3
4
## Capabilities
5
6
### Base Test Classes
7
8
Foundation classes for testing checkers and pylint functionality.
9
10
```python { .api }
11
class CheckerTestCase:
12
"""
13
Base class for checker unit tests.
14
15
Provides utilities for testing individual checkers
16
with AST nodes and message validation.
17
"""
18
19
CHECKER_CLASS = None # Set to checker class being tested
20
21
def setup_method(self):
22
"""Setup test environment before each test method."""
23
24
def walk(self, node):
25
"""
26
Walk AST node with the checker.
27
28
Args:
29
node: AST node to walk
30
"""
31
32
def assertNoMessages(self):
33
"""Assert that no messages were generated."""
34
35
def assertAddsMessages(self, *messages):
36
"""
37
Assert that specific messages were added.
38
39
Args:
40
*messages: Expected MessageTest instances
41
"""
42
43
class MessageTest:
44
"""
45
Test representation of an expected message.
46
47
Used to verify that checkers generate expected
48
messages for specific code patterns.
49
"""
50
51
def __init__(self, msg_id, node=None, line=None, args=None,
52
confidence=None, col_offset=None, end_line=None,
53
end_col_offset=None):
54
"""
55
Initialize message test.
56
57
Args:
58
msg_id (str): Expected message ID
59
node: Expected AST node (optional)
60
line (int): Expected line number
61
args (tuple): Expected message arguments
62
confidence (str): Expected confidence level
63
col_offset (int): Expected column offset
64
end_line (int): Expected end line
65
end_col_offset (int): Expected end column
66
"""
67
```
68
69
### Functional Testing
70
71
Classes and utilities for functional testing of pylint behavior.
72
73
```python { .api }
74
class FunctionalTestFile:
75
"""
76
Functional test file representation.
77
78
Represents a Python file used for functional testing
79
with expected messages and configuration.
80
"""
81
82
def __init__(self, directory, filename):
83
"""
84
Initialize functional test file.
85
86
Args:
87
directory (str): Directory containing test file
88
filename (str): Test file name
89
"""
90
91
@property
92
def expected_messages(self):
93
"""
94
Get expected messages from test file.
95
96
Returns:
97
list: Expected message objects
98
"""
99
100
@property
101
def pylintrc(self):
102
"""
103
Get pylintrc path for test.
104
105
Returns:
106
str: Path to test-specific pylintrc
107
"""
108
109
class LintModuleTest:
110
"""
111
Module linting test utilities.
112
113
Provides utilities for testing pylint behavior
114
on complete modules and packages.
115
"""
116
117
def __init__(self, test_file):
118
"""
119
Initialize module test.
120
121
Args:
122
test_file: FunctionalTestFile instance
123
"""
124
125
def runTest(self):
126
"""Run the functional test."""
127
128
def _check_result(self, got_messages, expected_messages):
129
"""
130
Check test results against expectations.
131
132
Args:
133
got_messages (list): Actual messages from pylint
134
expected_messages (list): Expected messages
135
"""
136
```
137
138
### Test-Specific Linter
139
140
Specialized linter implementation for testing purposes.
141
142
```python { .api }
143
class UnittestLinter:
144
"""
145
Test-specific linter implementation.
146
147
Simplified linter for unit testing that provides
148
controlled environment and message collection.
149
"""
150
151
def __init__(self):
152
"""Initialize unittest linter."""
153
self.config = None
154
self.reporter = None
155
156
def check(self, files_or_modules):
157
"""
158
Check files for testing.
159
160
Args:
161
files_or_modules: Files or modules to check
162
"""
163
164
def add_message(self, msg_id, line=None, node=None, args=None,
165
confidence=None, col_offset=None):
166
"""
167
Add message during testing.
168
169
Args:
170
msg_id (str): Message identifier
171
line (int): Line number
172
node: AST node
173
args (tuple): Message arguments
174
confidence (str): Confidence level
175
col_offset (int): Column offset
176
"""
177
```
178
179
### Configuration Utilities
180
181
Functions for configuring tests and test environments.
182
183
```python { .api }
184
def set_config(**kwargs):
185
"""
186
Set configuration for tests.
187
188
Provides a convenient way to set pylint configuration
189
options during test execution.
190
191
Args:
192
**kwargs: Configuration options to set
193
194
Example:
195
set_config(max_line_length=100, disable=['missing-docstring'])
196
"""
197
198
def tokenize_str(code):
199
"""
200
Tokenize Python code string.
201
202
Utility function for testing token-based checkers
203
by converting code strings to token streams.
204
205
Args:
206
code (str): Python code to tokenize
207
208
Returns:
209
list: Token objects
210
"""
211
```
212
213
### Test Reporters
214
215
Specialized reporters for testing and validation.
216
217
```python { .api }
218
class GenericTestReporter:
219
"""
220
Generic test reporter.
221
222
Collects messages during testing for validation
223
and provides access to test results.
224
"""
225
226
def __init__(self):
227
"""Initialize generic test reporter."""
228
self.messages = []
229
230
def handle_message(self, msg):
231
"""
232
Handle message during testing.
233
234
Args:
235
msg: Message to collect
236
"""
237
self.messages.append(msg)
238
239
def finalize(self):
240
"""
241
Finalize testing and return results.
242
243
Returns:
244
list: Collected messages
245
"""
246
return self.messages
247
248
class MinimalTestReporter:
249
"""
250
Minimal test reporter.
251
252
Provides minimal message collection for
253
lightweight testing scenarios.
254
"""
255
256
def __init__(self):
257
"""Initialize minimal reporter."""
258
self.messages = []
259
260
class FunctionalTestReporter:
261
"""
262
Functional test reporter.
263
264
Specialized reporter for functional testing
265
with enhanced message formatting and comparison.
266
"""
267
268
def __init__(self):
269
"""Initialize functional test reporter."""
270
self.messages = []
271
272
def display_reports(self, layout):
273
"""Display functional test reports."""
274
pass
275
```
276
277
## Usage Examples
278
279
### Basic Checker Testing
280
281
```python
282
import astroid
283
from pylint.testutils import CheckerTestCase, MessageTest
284
from my_checkers import FunctionNamingChecker
285
286
class TestFunctionNamingChecker(CheckerTestCase):
287
"""Test cases for function naming checker."""
288
289
CHECKER_CLASS = FunctionNamingChecker
290
291
def test_function_with_good_name(self):
292
"""Test that functions with good names pass."""
293
node = astroid.extract_node('''
294
def get_user_name(): #@
295
return "test"
296
''')
297
298
with self.assertNoMessages():
299
self.walk(node)
300
301
def test_function_with_bad_name(self):
302
"""Test that functions with bad names fail."""
303
node = astroid.extract_node('''
304
def userName(): #@
305
return "test"
306
''')
307
308
expected_message = MessageTest(
309
msg_id='invalid-function-name',
310
node=node,
311
args=('userName',),
312
line=2,
313
col_offset=0
314
)
315
316
with self.assertAddsMessages(expected_message):
317
self.walk(node)
318
319
def test_function_with_config(self):
320
"""Test checker with custom configuration."""
321
with self.assertNoMessages():
322
# Set custom configuration
323
self.checker.config.allowed_function_prefixes = ['create', 'build']
324
325
node = astroid.extract_node('''
326
def create_object(): #@
327
pass
328
''')
329
330
self.walk(node)
331
```
332
333
### Functional Testing
334
335
```python
336
from pylint.testutils import FunctionalTestFile, LintModuleTest
337
import os
338
339
class TestMyCheckersFunctional:
340
"""Functional tests for custom checkers."""
341
342
def test_complex_scenarios(self):
343
"""Test complex scenarios with functional tests."""
344
test_dir = 'tests/functional'
345
test_file = 'my_checker_test.py'
346
347
functional_test = FunctionalTestFile(test_dir, test_file)
348
test_case = LintModuleTest(functional_test)
349
test_case.runTest()
350
351
def create_functional_test_file(self):
352
"""Create a functional test file."""
353
test_content = '''
354
"""Test module for custom checker."""
355
356
def bad_function_name(): # [invalid-function-name]
357
"""This should trigger a naming violation."""
358
pass
359
360
def get_value(): # No violation expected
361
"""This should pass naming validation."""
362
return 42
363
364
class BadClassName: # [invalid-class-name]
365
"""This should trigger a class naming violation."""
366
pass
367
'''
368
369
with open('tests/functional/my_checker_test.py', 'w') as f:
370
f.write(test_content)
371
```
372
373
### Custom Test Utilities
374
375
```python
376
from pylint.testutils import UnittestLinter, GenericTestReporter
377
378
class CustomTestFramework:
379
"""Custom testing framework for specific needs."""
380
381
def __init__(self):
382
self.linter = UnittestLinter()
383
self.reporter = GenericTestReporter()
384
self.linter.set_reporter(self.reporter)
385
386
def test_code_snippet(self, code, expected_messages=None):
387
"""
388
Test a code snippet and validate results.
389
390
Args:
391
code (str): Python code to test
392
expected_messages (list): Expected message IDs
393
"""
394
# Create temporary file
395
import tempfile
396
with tempfile.NamedTemporaryFile(mode='w', suffix='.py',
397
delete=False) as f:
398
f.write(code)
399
temp_file = f.name
400
401
try:
402
# Run pylint on the code
403
self.linter.check([temp_file])
404
messages = self.reporter.finalize()
405
406
# Validate results
407
if expected_messages:
408
actual_msg_ids = [msg.msg_id for msg in messages]
409
assert set(actual_msg_ids) == set(expected_messages), \
410
f"Expected {expected_messages}, got {actual_msg_ids}"
411
412
return messages
413
finally:
414
# Clean up
415
import os
416
os.unlink(temp_file)
417
418
def assert_no_violations(self, code):
419
"""Assert that code has no violations."""
420
messages = self.test_code_snippet(code)
421
assert len(messages) == 0, f"Unexpected violations: {messages}"
422
423
def assert_violations(self, code, expected_msg_ids):
424
"""Assert that code has specific violations."""
425
self.test_code_snippet(code, expected_msg_ids)
426
427
# Usage example
428
framework = CustomTestFramework()
429
430
# Test clean code
431
framework.assert_no_violations('''
432
def get_user_data():
433
"""Get user data from database."""
434
return {"name": "test"}
435
''')
436
437
# Test code with violations
438
framework.assert_violations('''
439
def userData(): # Missing docstring, bad naming
440
return {"name": "test"}
441
''', ['missing-docstring', 'invalid-name'])
442
```
443
444
### Mock and Patch Testing
445
446
```python
447
from unittest.mock import patch, MagicMock
448
from pylint.testutils import CheckerTestCase
449
450
class TestCheckerWithMocks(CheckerTestCase):
451
"""Test checker using mocks for external dependencies."""
452
453
CHECKER_CLASS = MyCustomChecker
454
455
@patch('mypackage.external_service.check_api')
456
def test_checker_with_external_service(self, mock_api):
457
"""Test checker that depends on external service."""
458
# Setup mock
459
mock_api.return_value = True
460
461
node = astroid.extract_node('''
462
def process_data(): #@
463
# This would normally call external service
464
pass
465
''')
466
467
with self.assertNoMessages():
468
self.walk(node)
469
470
# Verify mock was called
471
mock_api.assert_called_once()
472
473
def test_checker_with_mock_config(self):
474
"""Test checker with mocked configuration."""
475
# Mock configuration
476
mock_config = MagicMock()
477
mock_config.strict_mode = True
478
mock_config.threshold = 10
479
480
self.checker.config = mock_config
481
482
node = astroid.extract_node('''
483
def test_function(): #@
484
pass
485
''')
486
487
# Test behavior changes based on config
488
with self.assertAddsMessages(
489
MessageTest('strict-mode-violation', node=node)
490
):
491
self.walk(node)
492
```
493
494
### Performance Testing
495
496
```python
497
import time
498
from pylint.testutils import CheckerTestCase
499
500
class TestCheckerPerformance(CheckerTestCase):
501
"""Performance tests for checker efficiency."""
502
503
CHECKER_CLASS = MyPerformanceChecker
504
505
def test_checker_performance(self):
506
"""Test checker performance on large code."""
507
# Generate large test case
508
large_code = '''
509
def large_function():
510
"""Large function for performance testing."""
511
''' + '\n'.join([f' var_{i} = {i}' for i in range(1000)])
512
513
node = astroid.parse(large_code)
514
515
# Measure execution time
516
start_time = time.time()
517
self.walk(node)
518
end_time = time.time()
519
520
execution_time = end_time - start_time
521
522
# Assert reasonable performance
523
assert execution_time < 1.0, \
524
f"Checker too slow: {execution_time:.2f}s"
525
526
# Verify functionality still works
527
self.assertNoMessages() # or expected messages
528
529
def benchmark_checker(self, iterations=100):
530
"""Benchmark checker performance."""
531
node = astroid.extract_node('''
532
def benchmark_function(): #@
533
return "test"
534
''')
535
536
total_time = 0
537
for _ in range(iterations):
538
start = time.time()
539
self.walk(node)
540
total_time += time.time() - start
541
542
avg_time = total_time / iterations
543
print(f"Average execution time: {avg_time*1000:.2f}ms")
544
545
return avg_time
546
```
547
548
## Test Configuration
549
550
### Test Settings
551
552
```python
553
# Configure test environment
554
from pylint.testutils import set_config
555
556
def setup_test_config():
557
"""Setup common test configuration."""
558
set_config(
559
disable=['missing-docstring'], # Disable for test simplicity
560
max_line_length=120, # Longer lines in tests
561
good_names=['i', 'j', 'k', 'x', 'y', 'z', 'test_var'],
562
reports=False, # Disable reports in tests
563
score=False # Disable scoring in tests
564
)
565
566
# Test-specific pylintrc
567
TEST_PYLINTRC = '''
568
[MAIN]
569
load-plugins=my_test_plugin
570
571
[MESSAGES CONTROL]
572
disable=missing-docstring,invalid-name
573
574
[BASIC]
575
good-names=i,j,k,x,y,z,test_var,mock_obj
576
577
[FORMAT]
578
max-line-length=120
579
580
[REPORTS]
581
reports=no
582
score=no
583
'''
584
```
585
586
### Continuous Integration Testing
587
588
```python
589
def run_test_suite():
590
"""Run complete test suite for CI/CD."""
591
import unittest
592
import sys
593
594
# Discover and run all tests
595
loader = unittest.TestLoader()
596
suite = loader.discover('tests/', pattern='test_*.py')
597
598
runner = unittest.TextTestRunner(verbosity=2)
599
result = runner.run(suite)
600
601
# Exit with error code if tests failed
602
if not result.wasSuccessful():
603
sys.exit(1)
604
605
print(f"All tests passed: {result.testsRun} tests")
606
607
# GitHub Actions example
608
'''
609
name: Test Custom Checkers
610
on: [push, pull_request]
611
jobs:
612
test:
613
runs-on: ubuntu-latest
614
strategy:
615
matrix:
616
python-version: [3.8, 3.9, '3.10', 3.11]
617
steps:
618
- uses: actions/checkout@v2
619
- name: Set up Python
620
uses: actions/setup-python@v2
621
with:
622
python-version: ${{ matrix.python-version }}
623
- name: Install dependencies
624
run: |
625
pip install pylint pytest
626
pip install -e .
627
- name: Run tests
628
run: python -m pytest tests/
629
- name: Run functional tests
630
run: python run_test_suite.py
631
'''
632
```