Syslog and Common Event Format (CEF) via AMA connectors for Microsoft Sentinel

The Syslog via AMA and Common Event Format (CEF) via AMA data connectors for Microsoft Sentinel filter and ingest Syslog messages, including messages in Common Event Format (CEF), from Linux machines and from network and security devices and appliances. These connectors install the Azure Monitor Agent (AMA) on any Linux machine from which you want to collect Syslog and/or CEF messages. This machine could be the originator of the messages, or it could be a forwarder that collects messages from other machines, such as network or security devices and appliances. The connector sends the agents instructions based on Data Collection Rules (DCRs) that you define. DCRs specify the systems to monitor and the types of logs or messages to collect. They define filters to apply to the messages before they're ingested, for better performance and more efficient querying and analysis.

Syslog and CEF are two common formats for logging data from different devices and applications. They help system administrators and security analysts to monitor and troubleshoot the network and identify potential threats or incidents.

What is Syslog?

Syslog is a standard protocol for sending and receiving messages between different devices or applications over a network. It was originally developed for Unix systems, but it's now widely supported by various platforms and vendors. Syslog messages have a predefined structure that consists of a priority, a timestamp, a hostname, an application name, a process ID, and a message text. Syslog messages can be sent over UDP, TCP, or TLS, depending on the configuration and the security requirements.

The Azure Monitor Agent supports Syslog RFCs 3164 and 5424.

What is Common Event Format (CEF)?

CEF, or Common Event Format, is a vendor-neutral format for logging data from network and security devices and appliances, such as firewalls, routers, detection and response solutions, and intrusion detection systems, as well as from other kinds of systems such as web servers. An extension of Syslog, it was developed especially for security information and event management (SIEM) solutions. CEF messages have a standard header that contains information such as the device vendor, the device product, the device version, the event class, the event severity, and the event ID. CEF messages also have a variable number of extensions that provide more details about the event, such as the source and destination IP addresses, the username, the file name, or the action taken.

Collection of Syslog and CEF messages with AMA

The following diagrams illustrate the architecture of Syslog and CEF message collection in Microsoft Sentinel, using the Syslog via AMA and Common Event Format (CEF) via AMA connectors.

Setup process to collect log messages

From the Content hub in Microsoft Sentinel, install the appropriate solution for Syslog or Common Event Format. This step installs the respective data connectors Syslog via AMA or Common Event Format (CEF) via AMA data connector. For more information, see Discover and manage Microsoft Sentinel out-of-the-box content.

As part of the setup process, create a data collection rule and install the Azure Monitor Agent (AMA) on the log forwarder. Do these tasks either by using the Azure portal or by using the Azure Monitor logs ingestion API.

• When you configure the data connector for the Microsoft Sentinel in the Azure portal, you can create, manage, and delete DCRs per workspace. The AMA is installed automatically on the VMs that you select in the connector configuration.

• Alternatively, send HTTP requests to the Logs Ingestion API. With this setup, you can create, manage, and delete DCRs. This option is more flexible than the portal. For example, with the API, you can filter by specific log levels. In the Azure portal, you can only select a minimum log level. The downside to using this method is that you have to manually install the Azure Monitor Agent on the log forwarder before creating a DCR.

After you create the DCR, and AMA is installed, run the "installation" script on the log forwarder. This script configures the Syslog daemon to listen for messages from other machines, and to open the necessary local ports. Then configure the security devices or appliances as needed.

For more information, see the following articles:

• Ingest Syslog and CEF messages to Microsoft Sentinel with the Azure Monitor Agent
• CEF via AMA data connector - Configure specific appliance or device for Microsoft Sentinel data ingestion

Data ingestion duplication avoidance

Using the same facility for both Syslog and CEF messages might result in data ingestion duplication between the CommonSecurityLog and Syslog tables.

To avoid this scenario, use one of these methods:

• If the source device enables configuration of the target facility: On each source machine that sends logs to the log forwarder in CEF format, edit the Syslog configuration file to remove the facilities used to send CEF messages. This way, the facilities sent in CEF aren't also be sent in Syslog. Make sure that each DCR you configure uses the relevant facility for CEF or Syslog respectively.

  To see an example of how to arrange a DCR to ingest both Syslog and CEF messages from the same agent, go to Syslog and CEF streams in the same DCR.

• If changing the facility for the source appliance isn't applicable: After you create the DCR, add ingestion time transformation to filter out CEF messages from the Syslog stream to avoid duplication. See Tutorial: Edit a data collection rule (DCR). Add KQL transformation similar to the following example:

  "transformKql": "  source\n    |  where ProcessName !contains \"CEF\"\n"
  

Next steps