Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
This article describes how to use the Syslog via AMA and Common Event Format (CEF) via AMA connectors to quickly filter and ingest syslog messages, including messages in Common Event Format (CEF), from Linux machines and from network and security devices and appliances. To learn more about these data connectors, see Syslog and Common Event Format (CEF) via AMA connectors for Microsoft Sentinel.
Before you begin, you must have the resources configured and the appropriate permissions assigned, as described in this section.
Install the appropriate Microsoft Sentinel solution and make sure you have the permissions to complete the steps in this article.
Install the appropriate solution from the Content hub in Microsoft Sentinel. For more information, see Discover and manage Microsoft Sentinel out-of-the-box content.
Identify which data connector the Microsoft Sentinel solution requires — Syslog via AMA or Common Event Format (CEF) via AMA and whether you need to install the Syslog or Common Event Format solution. To fulfill this prerequisite,
In the Content hub, select Manage on the installed solution and review the data connector listed.
If either Syslog via AMA or Common Event Format (CEF) via AMA isn't installed with the solution, identify whether you need to install the Syslog or Common Event Format solution:
Then install either the Syslog or Common Event Format solution from the content hub to get the related AMA data connector.
Have an Azure account with the following Azure role-based access control (Azure RBAC) roles:
Built-in role Scope Reason - Virtual Machine Contributor
- Azure Connected Machine
Resource Administrator- Virtual machines (VM)
- Virtual Machine Scale Sets
- Azure Arc-enabled servers
To deploy the agent Any role that includes the action
Microsoft.Resources/deployments/*- Subscription
- Resource group
- Existing data collection rule
To deploy Azure Resource Manager templates Monitoring Contributor - Subscription
- Resource group
- Existing data collection rule
To create or edit data collection rules
If you're collecting messages from a log forwarder, the following prerequisites apply:
You must have a designated Linux VM as a log forwarder to collect logs.
If your log forwarder isn't an Azure virtual machine, it must have the Azure Arc Connected Machine agent installed on it.
The Linux log forwarder VM must have Python 2.7 or 3 installed. Use the
python --version
orpython3 --version
command to check. If you're using Python 3, make sure it's set as the default command on the machine, or run scripts with the 'python3' command instead of 'python'.The log forwarder must have either the
syslog-ng
orrsyslog
daemon enabled.Your log sources, security devices, and appliances, must be configured to send their log messages to the log forwarder's syslog daemon instead of to their local syslog daemon.
Note
When deploying the AMA to a Virtual Machine Scale Set (VMSS), you're strongly encouraged to use a load balancer that supports the round-robin method to ensure load distribution across all deployed instances.
Configure the machine's security according to your organization's security policy. For example, configure your network to align with your corporate network security policy and change the ports and protocols in the daemon to align with your requirements. To improve your machine security configuration, secure your VM in Azure, or review these best practices for network security.
If your devices are sending syslog and CEF logs over TLS because, for example, your log forwarder is in the cloud, you need to configure the syslog daemon (rsyslog
or syslog-ng
) to communicate in TLS. For more information, see:
The setup process for the Syslog via AMA or Common Event Format (CEF) via AMA data connectors includes the following steps:
- Install the Azure Monitor Agent and create a Data Collection Rule (DCR) by using either of the following methods:
- If you're collecting logs from other machines using a log forwarder, run the "installation" script on the log forwarder to configure the syslog daemon to listen for messages from other machines, and to open the necessary local ports.
Select the appropriate tab for instructions.
Follow the appropriate instructions from the Azure Monitor documentation to install the Azure Monitor Agent on your log forwarder. Remember to use the instructions for Linux, not for Windows.
- Install the AMA using PowerShell
- Install the AMA using the Azure CLI
- Install the AMA using an Azure Resource Manager template
You can create Data Collection Rules (DCRs) using the Azure Monitor Logs Ingestion API. For more information, see Data collection rules in Azure Monitor.
Create a JSON file for the data collection rule, create an API request, and send the request.
Prepare a DCR file in JSON format. The contents of this file is the request body in your API request.
For an example, see Syslog/CEF DCR creation request body. To collect syslog and CEF messages in the same data collection rule, see the example Syslog and CEF streams in the same DCR.
- Verify that the
streams
field is set toMicrosoft-Syslog
for syslog messages, or toMicrosoft-CommonSecurityLog
for CEF messages. - Add the filter and facility log levels in the
facilityNames
andlogLevels
parameters. See Examples of facilities and log levels sections.
- Verify that the
Create an API request in a REST API client of your choosing.
For the request URL and header, copy the following request URL and header.
PUT https://management.chinacloudapi.cn/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}?api-version=2022-06-01
- Substitute the appropriate values for the
{subscriptionId}
and{resourceGroupName}
placeholders. - Enter a name of your choice for the DCR in place of the
{dataCollectionRuleName}
placeholder.
- Substitute the appropriate values for the
For the request body, copy and paste the contents of the DCR JSON file that you created (in step 1 above) into the request body.
Send the request.
For an example of the response that you should receive, see Syslog/CEF DCR creation response.
Now you need to create a DCR Association (DCRA) that ties the DCR to the VM resource that hosts your log forwarder.
Create an API request in a REST API client of your choosing.
For the request URL and header, copy the following request URL and the header.
PUT https://management.chinacloudapi.cn/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachines/{virtualMachineName}/providers/Microsoft.Insights/dataCollectionRuleAssociations/{dataCollectionRuleAssociationName}?api-version=2022-06-01
- Substitute the appropriate values for the
{subscriptionId}
,{resourceGroupName}
, and{virtualMachineName}
placeholders. - Enter a name of your choice for the DCR in place of the
{dataCollectionRuleAssociationName}
placeholder.
- Substitute the appropriate values for the
For the request body, copy the following request body.
{ "properties": { "dataCollectionRuleId": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}" } }
- Substitute the appropriate values for the
{subscriptionId}
and{resourceGroupName}
placeholders. - Enter a name of your choice for the DCR in place of the
{dataCollectionRuleName}
placeholder.
- Substitute the appropriate values for the
Send the request.
Review these examples of the facilities and log levels settings. The name
field includes the filter name.
For CEF message ingestion, the value for "streams"
should be "Microsoft-CommonSecurityLog"
instead of "Microsoft-Syslog"
.
This example collects events from the cron
, daemon
, local0
, local3
and uucp
facilities, with the Warning
, Error
, Critical
, Alert
, and Emergency
log levels:
"dataSources": {
"syslog": [
{
"name": "SyslogStream0",
"streams": [
"Microsoft-Syslog"
],
"facilityNames": [
"cron",
"daemon",
"local0",
"local3",
"uucp"
],
"logLevels": [
"Warning",
"Error",
"Critical",
"Alert",
"Emergency"
]
}
]
}
This example shows how you can collect syslog and CEF messages in the same DCR.
The DCR collects CEF event messages for:
- The
authpriv
andmark
facilities with theInfo
,Notice
,Warning
,Error
,Critical
,Alert
, andEmergency
log levels - The
daemon
facility with theWarning
,Error
,Critical
,Alert
, andEmergency
log levels
It collects syslog event messages for:
- The
kern
,local0
,local5
, andnews
facilities with theCritical
,Alert
, andEmergency
log levels - The
mail
anduucp
facilities with theEmergency
log level
"dataSources": {
"syslog": [
{
"name": "CEFStream1",
"streams": [
"Microsoft-CommonSecurityLog"
],
"facilityNames": [
"authpriv",
"mark"
],
"logLevels": [
"Info",
"Notice",
"Warning",
"Error",
"Critical",
"Alert",
"Emergency"
]
},
{
"name": "CEFStream2",
"streams": [
"Microsoft-CommonSecurityLog"
],
"facilityNames": [
"daemon"
],
"logLevels": [
"Warning",
"Error",
"Critical",
"Alert",
"Emergency"
]
},
{
"name": "SyslogStream3",
"streams": [
"Microsoft-Syslog"
],
"facilityNames": [
"kern",
"local0",
"local5",
"news"
],
"logLevels": [
"Critical",
"Alert",
"Emergency"
]
},
{
"name": "SyslogStream4",
"streams": [
"Microsoft-Syslog"
],
"facilityNames": [
"mail",
"uucp"
],
"logLevels": [
"Emergency"
]
}
]
}
If you're using a log forwarder, configure the syslog daemon to listen for messages from other machines, and open the necessary local ports.
From the connector page, copy the command line that appears under Run the following command to install and apply the CEF collector:
Or copy it from here:
sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python Forwarder_AMA_installer.py
Sign in to the log forwarder machine where you just installed the AMA.
Paste the command you copied in the last step to launch the installation script.
The script configures thersyslog
orsyslog-ng
daemon to use the required protocol and restarts the daemon. The script opens port 514 to listen to incoming messages in both UDP and TCP protocols. To change this setting, refer to the syslog daemon configuration file according to the daemon type running on the machine:- Rsyslog:
/etc/rsyslog.conf
- Syslog-ng:
/etc/syslog-ng/syslog-ng.conf
If you're using Python 3, and it's not set as the default command on the machine, substitute
python3
forpython
in the pasted command. See Log forwarder prerequisites.Note
To avoid Full Disk scenarios where the agent can't function, we recommend that you set the
syslog-ng
orrsyslog
configuration not to store unneeded logs. A Full Disk scenario disrupts the function of the installed AMA. For more information, see RSyslog or Syslog-ng.- Rsyslog:
Verify that logs messages from your linux machine or security devices and appliances are ingested into Microsoft Sentinel.
To validate that the syslog daemon is running on the UDP port and that the AMA is listening, run this command:
netstat -lnptv
You should see the
rsyslog
orsyslog-ng
daemon listening on port 514.To capture messages sent from a logger or a connected device, run this command in the background:
tcpdump -i any port 514 -A -vv &
After you complete the validation, we recommend that you stop the
tcpdump
: Typefg
and then select Ctrl+C.To send demo messages, complete of the following steps:
Use the netcat utility. In this example, the utility reads data posted through the
echo
command with the newline switch turned off. The utility then writes the data to UDP port514
on the localhost with no timeout. To execute the netcat utility, you might need to install another package.echo -n "<164>CEF:0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time" | nc -u -w0 localhost 514
Use the logger. This example writes the message to the
local 4
facility, at severity levelWarning
, to port514
, on the local host, in the CEF RFC format. The-t
and--rfc3164
flags are used to comply with the expected RFC format.logger -p local4.warn -P 514 -n 127.0.0.1 --rfc3164 -t CEF "0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time"
To verify that the connector is installed correctly, run the troubleshooting script with one of these commands:
For CEF logs, run:
sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py&&sudo python Sentinel_AMA_troubleshoot.py --cef
For Cisco Adaptive Security Appliance (ASA) logs, run:
sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py&&sudo python Sentinel_AMA_troubleshoot.py --asa
For Cisco Firepower Threat Defense (FTD) logs, run:
sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py&&sudo python Sentinel_AMA_troubleshoot.py --ftd