0
# Data Collection
1

2
Azure Monitor Agent data collection rules, endpoints, and associations for custom log and metric collection from virtual machines, containers, and other resources with flexible data routing and transformation capabilities.
3

4
## Capabilities
5

6
### Data Collection Rules
7

8
Manage data collection rules that define what data to collect, how to transform it, and where to send it.
9

10
```python { .api }
11
def create(resource_group_name: str, data_collection_rule_name: str, body: Optional[DataCollectionRuleResource] = None, **kwargs: Any) -> DataCollectionRuleResource:
12
    """
13
    Creates or updates a data collection rule.
14
    
15
    Parameters:
16
    - resource_group_name: str - Name of the resource group
17
    - data_collection_rule_name: str - Name of the data collection rule
18
    - body: Optional[DataCollectionRuleResource] - Data collection rule configuration
19
    
20
    Returns:
21
    DataCollectionRuleResource - The created data collection rule
22
    """
23

24
def get(resource_group_name: str, data_collection_rule_name: str, **kwargs: Any) -> DataCollectionRuleResource:
25
    """
26
    Returns the specified data collection rule.
27
    
28
    Parameters:
29
    - resource_group_name: str - Name of the resource group
30
    - data_collection_rule_name: str - Name of the data collection rule
31
    
32
    Returns:
33
    DataCollectionRuleResource - The data collection rule details
34
    """
35

36
def delete(resource_group_name: str, data_collection_rule_name: str, **kwargs: Any) -> None:
37
    """
38
    Deletes a data collection rule.
39
    
40
    Parameters:
41
    - resource_group_name: str - Name of the resource group
42
    - data_collection_rule_name: str - Name of the data collection rule
43
    """
44

45
def update(resource_group_name: str, data_collection_rule_name: str, body: Optional[ResourceForUpdate] = None, **kwargs: Any) -> DataCollectionRuleResource:
46
    """
47
    Updates part of a data collection rule.
48
    
49
    Parameters:
50
    - resource_group_name: str - Name of the resource group
51
    - data_collection_rule_name: str - Name of the data collection rule
52
    - body: Optional[ResourceForUpdate] - Properties to update
53
    
54
    Returns:
55
    DataCollectionRuleResource - The updated data collection rule
56
    """
57

58
def list_by_resource_group(resource_group_name: str, **kwargs: Any) -> ItemPaged[DataCollectionRuleResource]:
59
    """
60
    Lists all data collection rules in the specified resource group.
61
    
62
    Parameters:
63
    - resource_group_name: str - Name of the resource group
64
    
65
    Returns:
66
    ItemPaged[DataCollectionRuleResource] - List of data collection rules
67
    """
68

69
def list_by_subscription(**kwargs: Any) -> ItemPaged[DataCollectionRuleResource]:
70
    """
71
    Lists all data collection rules in the specified subscription.
72
    
73
    Returns:
74
    ItemPaged[DataCollectionRuleResource] - List of data collection rules
75
    """
76
```
77

78
### Data Collection Endpoints
79

80
Manage data collection endpoints that provide ingestion URLs for custom data collection.
81

82
```python { .api }
83
def create(resource_group_name: str, data_collection_endpoint_name: str, body: Optional[DataCollectionEndpointResource] = None, **kwargs: Any) -> DataCollectionEndpointResource:
84
    """
85
    Creates or updates a data collection endpoint.
86
    
87
    Parameters:
88
    - resource_group_name: str - Name of the resource group
89
    - data_collection_endpoint_name: str - Name of the data collection endpoint
90
    - body: Optional[DataCollectionEndpointResource] - Endpoint configuration
91
    
92
    Returns:
93
    DataCollectionEndpointResource - The created endpoint
94
    """
95

96
def get(resource_group_name: str, data_collection_endpoint_name: str, **kwargs: Any) -> DataCollectionEndpointResource:
97
    """
98
    Returns the specified data collection endpoint.
99
    
100
    Parameters:
101
    - resource_group_name: str - Name of the resource group
102
    - data_collection_endpoint_name: str - Name of the data collection endpoint
103
    
104
    Returns:
105
    DataCollectionEndpointResource - The endpoint details
106
    """
107

108
def delete(resource_group_name: str, data_collection_endpoint_name: str, **kwargs: Any) -> None:
109
    """
110
    Deletes a data collection endpoint.
111
    
112
    Parameters:
113
    - resource_group_name: str - Name of the resource group
114
    - data_collection_endpoint_name: str - Name of the data collection endpoint
115
    """
116

117
def list_by_resource_group(resource_group_name: str, **kwargs: Any) -> ItemPaged[DataCollectionEndpointResource]:
118
    """
119
    Lists all data collection endpoints in the specified resource group.
120
    
121
    Parameters:
122
    - resource_group_name: str - Name of the resource group
123
    
124
    Returns:
125
    ItemPaged[DataCollectionEndpointResource] - List of endpoints
126
    """
127

128
def list_by_subscription(**kwargs: Any) -> ItemPaged[DataCollectionEndpointResource]:
129
    """
130
    Lists all data collection endpoints in the specified subscription.
131
    
132
    Returns:
133
    ItemPaged[DataCollectionEndpointResource] - List of endpoints
134
    """
135
```
136

137
### Data Collection Rule Associations
138

139
Manage associations between data collection rules and target resources.
140

141
```python { .api }
142
def create(resource_uri: str, association_name: str, body: Optional[DataCollectionRuleAssociationProxyOnlyResource] = None, **kwargs: Any) -> DataCollectionRuleAssociationProxyOnlyResource:
143
    """
144
    Creates or updates an association between a resource and a data collection rule.
145
    
146
    Parameters:
147
    - resource_uri: str - Full resource ID of the target resource
148
    - association_name: str - Name of the association
149
    - body: Optional[DataCollectionRuleAssociationProxyOnlyResource] - Association configuration
150
    
151
    Returns:
152
    DataCollectionRuleAssociationProxyOnlyResource - The created association
153
    """
154

155
def get(resource_uri: str, association_name: str, **kwargs: Any) -> DataCollectionRuleAssociationProxyOnlyResource:
156
    """
157
    Returns the specified association between a resource and a data collection rule.
158
    
159
    Parameters:
160
    - resource_uri: str - Full resource ID of the target resource
161
    - association_name: str - Name of the association
162
    
163
    Returns:
164
    DataCollectionRuleAssociationProxyOnlyResource - The association details
165
    """
166

167
def delete(resource_uri: str, association_name: str, **kwargs: Any) -> None:
168
    """
169
    Deletes an association between a resource and a data collection rule.
170
    
171
    Parameters:
172
    - resource_uri: str - Full resource ID of the target resource
173
    - association_name: str - Name of the association
174
    """
175

176
def list_by_resource(resource_uri: str, **kwargs: Any) -> ItemPaged[DataCollectionRuleAssociationProxyOnlyResource]:
177
    """
178
    Lists associations for the specified resource.
179
    
180
    Parameters:
181
    - resource_uri: str - Full resource ID of the target resource
182
    
183
    Returns:
184
    ItemPaged[DataCollectionRuleAssociationProxyOnlyResource] - List of associations
185
    """
186

187
def list_by_rule(resource_group_name: str, data_collection_rule_name: str, **kwargs: Any) -> ItemPaged[DataCollectionRuleAssociationProxyOnlyResource]:
188
    """
189
    Lists associations for the specified data collection rule.
190
    
191
    Parameters:
192
    - resource_group_name: str - Name of the resource group
193
    - data_collection_rule_name: str - Name of the data collection rule
194
    
195
    Returns:
196
    ItemPaged[DataCollectionRuleAssociationProxyOnlyResource] - List of associations
197
    """
198
```
199

200
## Usage Examples
201

202
### Creating a Data Collection Rule for Windows Event Logs
203

204
```python
205
from azure.mgmt.monitor.models import (
206
    DataCollectionRuleResource, DataCollectionRuleDataSources,
207
    DataCollectionRuleDestinations, WindowsEventLogDataSource,
208
    LogAnalyticsDestination, DataFlow
209
)
210

211
# Define Windows Event Log data source
212
windows_events = WindowsEventLogDataSource(
213
    name="WindowsEventLogsDataSource",
214
    streams=["Microsoft-Event"],
215
    x_path_queries=[
216
        "Application!*[System[(Level=1 or Level=2 or Level=3)]]",
217
        "System!*[System[(Level=1 or Level=2 or Level=3)]]"
218
    ]
219
)
220

221
# Define data sources
222
data_sources = DataCollectionRuleDataSources(
223
    windows_event_logs=[windows_events]
224
)
225

226
# Define destination (Log Analytics workspace)
227
log_analytics_destination = LogAnalyticsDestination(
228
    workspace_resource_id=f"/subscriptions/{subscription_id}/resourceGroups/monitoring-rg/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
229
    name="LogAnalyticsDestination"
230
)
231

232
destinations = DataCollectionRuleDestinations(
233
    log_analytics=[log_analytics_destination]
234
)
235

236
# Define data flow
237
data_flow = DataFlow(
238
    streams=["Microsoft-Event"],
239
    destinations=["LogAnalyticsDestination"],
240
    transform_kql="source | where EventLevelName == 'Error'"  # Optional KQL transform
241
)
242

243
# Create data collection rule
244
dcr = DataCollectionRuleResource(
245
    location="East US",
246
    data_sources=data_sources,
247
    destinations=destinations,
248
    data_flows=[data_flow],
249
    description="Collect Windows Event Logs with error filtering"
250
)
251

252
result = client.data_collection_rules.create(
253
    resource_group_name="monitoring-rg",
254
    data_collection_rule_name="windows-events-dcr",
255
    body=dcr
256
)
257
```
258

259
### Creating a Data Collection Endpoint
260

261
```python
262
from azure.mgmt.monitor.models import (
263
    DataCollectionEndpointResource, DataCollectionEndpointLogsIngestion,
264
    DataCollectionEndpointConfigurationAccess, NetworkRuleSet
265
)
266

267
# Define network access rules
268
network_acls = NetworkRuleSet(
269
    public_network_access="Enabled"
270
)
271

272
# Define logs ingestion configuration
273
logs_ingestion = DataCollectionEndpointLogsIngestion(
274
    endpoint="https://my-dce.eastus-1.ingest.monitor.azure.com"
275
)
276

277
# Define configuration access
278
configuration_access = DataCollectionEndpointConfigurationAccess(
279
    endpoint="https://my-dce.eastus-1.handler.control.monitor.azure.com"
280
)
281

282
# Create data collection endpoint
283
dce = DataCollectionEndpointResource(
284
    location="East US",
285
    network_acls=network_acls,
286
    logs_ingestion=logs_ingestion,
287
    configuration_access=configuration_access,
288
    description="Data collection endpoint for custom logs"
289
)
290

291
endpoint_result = client.data_collection_endpoints.create(
292
    resource_group_name="monitoring-rg",
293
    data_collection_endpoint_name="my-data-endpoint",
294
    body=dce
295
)
296
```
297

298
### Associating a Rule with a Virtual Machine
299

300
```python
301
from azure.mgmt.monitor.models import DataCollectionRuleAssociationProxyOnlyResource
302

303
# Create association between VM and data collection rule
304
vm_resource_id = f"/subscriptions/{subscription_id}/resourceGroups/compute-rg/providers/Microsoft.Compute/virtualMachines/web-server"
305

306
association = DataCollectionRuleAssociationProxyOnlyResource(
307
    data_collection_rule_id=result.id,
308
    description="Associate Windows events DCR with web server VM"
309
)
310

311
association_result = client.data_collection_rule_associations.create(
312
    resource_uri=vm_resource_id,
313
    association_name="windows-events-association",
314
    body=association
315
)
316

317
print(f"Association created: {association_result.id}")
318
```
319

320
### Custom Logs Data Collection Rule
321

322
```python
323
from azure.mgmt.monitor.models import (
324
    LogFilesDataSource, LogFileSettings, LogFileSettingsText,
325
    StorageBlob, StorageBlobDestination
326
)
327

328
# Define custom log file data source
329
log_file_source = LogFilesDataSource(
330
    name="CustomAppLogs",
331
    streams=["Custom-AppLogs"],
332
    file_patterns=["/var/log/myapp/*.log"],
333
    format="text",
334
    settings=LogFileSettings(
335
        text=LogFileSettingsText(
336
            record_start_timestamp_format="yyyy-MM-dd HH:mm:ss"
337
        )
338
    )
339
)
340

341
# Define storage destination for raw logs
342
storage_destination = StorageBlobDestination(
343
    storage_account_resource_id=f"/subscriptions/{subscription_id}/resourceGroups/storage-rg/providers/Microsoft.Storage/storageAccounts/logstorage",
344
    container_name="custom-logs",
345
    name="StorageDestination"
346
)
347

348
# Create custom logs DCR
349
custom_dcr = DataCollectionRuleResource(
350
    location="East US",
351
    data_sources=DataCollectionRuleDataSources(
352
        log_files=[log_file_source]
353
    ),
354
    destinations=DataCollectionRuleDestinations(
355
        storage_blobs=[storage_destination]
356
    ),
357
    data_flows=[
358
        DataFlow(
359
            streams=["Custom-AppLogs"],
360
            destinations=["StorageDestination"]
361
        )
362
    ]
363
)
364
```
365

366
## Types
367

368
```python { .api }
369
class DataCollectionRuleResource:
370
    """Data collection rule resource."""
371
    location: str  # Resource location
372
    data_sources: Optional[DataCollectionRuleDataSources]  # Data sources configuration
373
    destinations: Optional[DataCollectionRuleDestinations]  # Destinations configuration
374
    data_flows: Optional[List[DataFlow]]  # Data flow definitions
375
    description: Optional[str]  # Rule description
376
    data_collection_endpoint_id: Optional[str]  # Associated endpoint ID
377
    stream_declarations: Optional[Dict[str, StreamDeclaration]]  # Custom stream definitions
378

379
class DataCollectionRuleDataSources:
380
    """Data sources configuration."""
381
    performance_counters: Optional[List[PerfCounterDataSource]]  # Performance counters
382
    windows_event_logs: Optional[List[WindowsEventLogDataSource]]  # Windows events
383
    syslog: Optional[List[SyslogDataSource]]  # Syslog data
384
    log_files: Optional[List[LogFilesDataSource]]  # Custom log files
385
    extensions: Optional[List[ExtensionDataSource]]  # Extension data sources
386
    iis_logs: Optional[List[IisLogsDataSource]]  # IIS logs
387
    platform_telemetry: Optional[List[PlatformTelemetryDataSource]]  # Platform telemetry
388

389
class DataCollectionRuleDestinations:
390
    """Destinations configuration."""
391
    log_analytics: Optional[List[LogAnalyticsDestination]]  # Log Analytics workspaces
392
    azure_monitor_metrics: Optional[AzureMonitorMetricsDestination]  # Azure Monitor metrics
393
    storage_blobs: Optional[List[StorageBlobDestination]]  # Storage blob destinations
394
    event_hubs: Optional[List[EventHubDestination]]  # Event Hub destinations
395
    event_hubs_direct: Optional[List[EventHubDirectDestination]]  # Direct Event Hub
396

397
class DataFlow:
398
    """Data flow definition."""
399
    streams: List[str]  # Source streams
400
    destinations: List[str]  # Target destinations
401
    transform_kql: Optional[str]  # KQL transformation query
402
    output_stream: Optional[str]  # Output stream name
403

404
class DataCollectionEndpointResource:
405
    """Data collection endpoint resource."""
406
    location: str  # Resource location
407
    description: Optional[str]  # Endpoint description
408
    immutable_id: Optional[str]  # Immutable endpoint identifier
409
    configuration_access: Optional[DataCollectionEndpointConfigurationAccess]  # Configuration access
410
    logs_ingestion: Optional[DataCollectionEndpointLogsIngestion]  # Logs ingestion
411
    metrics_ingestion: Optional[DataCollectionEndpointMetricsIngestion]  # Metrics ingestion
412
    network_acls: Optional[DataCollectionEndpointNetworkAcls]  # Network access rules
413
    private_link_scoped_resources: Optional[List[PrivateLinkScopedResource]]  # Private link resources
414

415
class WindowsEventLogDataSource:
416
    """Windows Event Log data source."""
417
    name: str  # Data source name
418
    streams: List[str]  # Output streams
419
    x_path_queries: List[str]  # XPath queries for event selection
420

421
class LogAnalyticsDestination:
422
    """Log Analytics workspace destination."""
423
    workspace_resource_id: str  # Workspace resource ID
424
    name: str  # Destination name
425

426
class DataCollectionRuleAssociationProxyOnlyResource:
427
    """Data collection rule association."""
428
    data_collection_rule_id: Optional[str]  # Data collection rule ID
429
    data_collection_endpoint_id: Optional[str]  # Data collection endpoint ID
430
    description: Optional[str]  # Association description
431

432
class PerfCounterDataSource:
433
    """Performance counter data source."""
434
    name: str  # Data source name
435
    streams: List[str]  # Output streams
436
    counter_specifiers: List[str]  # Performance counters to collect
437
    sampling_frequency_in_seconds: int  # Sampling frequency
438

439
class SyslogDataSource:
440
    """Syslog data source."""
441
    name: str  # Data source name  
442
    streams: List[str]  # Output streams
443
    facility_names: List[str]  # Syslog facilities
444
    log_levels: List[str]  # Log levels to collect
445

446
class LogFilesDataSource:
447
    """Log files data source."""
448
    name: str  # Data source name
449
    streams: List[str]  # Output streams
450
    file_patterns: List[str]  # File path patterns
451
    format: str  # Log format (text/json)
452
    settings: Optional[LogFileSettings]  # Format-specific settings
453
```