Data collection best practices

  • Article

This section reviews best practices for collecting data using Microsoft Sentinel data connectors. For more information, see Connect data sources, Microsoft Sentinel data connectors reference, and the Microsoft Sentinel solutions catalog.

Prioritize your data connectors

Learn how to prioritize your data connectors as part of the Microsoft Sentinel deployment process.

Filter your logs before ingestion

You may want to filter the logs collected, or even log content, before the data is ingested into Microsoft Sentinel. For example, you may want to filter out logs that are irrelevant or unimportant to security operations, or you may want to remove unwanted details from log messages. Filtering message content may also be helpful when trying to drive down costs when working with Syslog, CEF, or Windows-based logs that have many irrelevant details.

Filter your logs using one of the following methods:

  • The Azure Monitor Agent. Supported on both Windows and Linux to ingest Windows security events. Filter the logs collected by configuring the agent to collect only specified events.

  • Logstash. Supports filtering message content, including making changes to the log messages. For more information, see Connect with Logstash.

Important

Using Logstash to filter your message content will cause your logs to be ingested as custom logs, causing any free-tier logs to become paid-tier logs.

Custom logs also need to be worked into analytics rules, threat hunting, and [workbooks, as they aren't automatically added.

Alternative data ingestion requirements

Standard configuration for data collection may not work well for your organization, due to various challenges. The following tables describe common challenges or requirements, and possible solutions and considerations.

Note

Many solutions listed in the following sections require a custom data connector. For more information, see Resources for creating Microsoft Sentinel custom connectors.

On-premises Windows log collection

Challenge / Requirement Possible solutions Considerations
Requires log filtering Use Logstash

Use Azure Functions

Use LogicApps

Use custom code (.NET, Python)		 While filtering can lead to cost savings, and ingests only the required data, some Microsoft Sentinel features are not supported, such as UEBA, entity pages, and fusion.

When configuring log filtering, make updates in resources such as threat hunting queries and analytics rules
Agent cannot be installed Use Windows Event Forwarding, supported with the Azure Monitor Agent Using Windows Event forwarding lowers load-balancing events per second from the Windows Event Collector, from 10,000 events to 500-1000 events.
Servers do not connect to the internet Use the Log Analytics gateway Configuring a proxy to your agent requires extra firewall rules to allow the Gateway to work.
Requires tagging and enrichment at ingestion Use Logstash to inject a ResourceID

Use an ARM template to inject the ResourceID into on-premises machines

Ingest the resource ID into separate workspaces		 Log Analytics doesn't support RBAC for custom tables

Microsoft Sentinel doesn't support row-level RBAC

Tip: You may want to adopt cross workspace design and functionality for Microsoft Sentinel.
Requires splitting operation and security logs Use the Microsoft Monitor Agent or Azure Monitor Agent multi-home functionality Multi-home functionality requires more deployment overhead for the agent.
Requires custom logs Collect files from specific folder paths

Use API ingestion

Use PowerShell

Use Logstash		 You may have issues filtering your logs.

Custom methods aren't supported.

Custom connectors may require developer skills.

On-premises Linux log collection

Challenge / Requirement Possible solutions Considerations
Requires log filtering Use Syslog-NG

Use Rsyslog

Use FluentD configuration for the agent

Use the Azure Monitor Agent/Microsoft Monitoring Agent

Use Logstash		 Some Linux distributions might not be supported by the agent.

Using Syslog or FluentD requires developer knowledge.

For more information, see Connect to Windows servers to collect security events and Resources for creating Microsoft Sentinel custom connectors.
Agent cannot be installed Use a Syslog forwarder, such as (syslog-ng or rsyslog).
Servers do not connect to the internet Use the Log Analytics gateway Configuring a proxy to your agent requires extra firewall rules to allow the Gateway to work.
Requires tagging and enrichment at ingestion Use Logstash for enrichment, or custom methods, such as API or Event Hubs. You may have extra effort required for filtering.
Requires splitting operation and security logs Use the Azure Monitor Agent with the multi-homing configuration.
Requires custom logs Create a custom collector using the Microsoft Monitoring (Log Analytics) agent.

Endpoint solutions

If you need to collect logs from Endpoint solutions, such as EDR, other security events, Sysmon, and so on, use one of the following methods:

Windows Event Forwarding.

Note

Load balancing cuts down on the events per second that can be processed to the workspace.

Office data

If you need to collect Microsoft Office data, outside of the standard connector data, use one of the following solutions:

Challenge / Requirement Possible solutions Considerations
Collect raw data from Teams, message trace, phishing data, and so on Use the built-in Office 365 connector functionality, and then create a custom connector for other raw data. Mapping events to the corresponding recordID may be challenging.
Requires RBAC for splitting countries/regions, departments, and so on Customize your data collection by adding tags to data and creating dedicated workspaces for each separation needed. Custom data collection has extra ingestion costs.
Requires multiple tenants in a single workspace Customize your data collection using Azure LightHouse and a unified incident view. Custom data collection has extra ingestion costs.

For more information, see Extend Microsoft Sentinel across workspaces and tenants.

Cloud platform data

Challenge / Requirement Possible solutions Considerations
Filter logs from other platforms Use Logstash

Use the Azure Monitor Agent / Microsoft Monitoring (Log Analytics) agent		 Custom collection has extra ingestion costs.

You may have a challenge of collecting all Windows events vs only security events.
Agent cannot be used Use Windows Event Forwarding You may need to load balance efforts across your resources.
Servers are in air-gapped network Use the Log Analytics gateway Configuring a proxy to your agent requires firewall rules to allow the Gateway to work.
RBAC, tagging, and enrichment at ingestion Create custom collection via Logstash or the Log Analytics API. RBAC isn't supported for custom tables

Row-level RBAC isn't supported for any tables.

Next steps

For more information, see: