Integrate Azure Stack Hub with monitoring solutions using syslog forwarding
This article shows you how to use syslog to integrate Azure Stack Hub infrastructure with external security solution(s) already deployed in your datacenter. For example, a security information event management (SIEM) system. The syslog channel exposes audits, alerts, and security logs from all the components of the Azure Stack Hub infrastructure. Use syslog forwarding to integrate with security monitoring solutions and to retrieve all audits, alerts, and security logs to store them for retention.
Starting with the 1809 update, Azure Stack Hub has an integrated syslog client that, once configured, emits syslog messages with the payload in Common Event Format (CEF).
The following diagram describes the integration of Azure Stack Hub with an external SIEM. There are two integration patterns that need to be considered: the first one (the one in blue) is the Azure Stack Hub infrastructure that encompasses the infrastructure virtual machines and the Hyper-V nodes. All the audits, security logs, and alerts from those components are centrally collected and exposed via syslog with CEF payload. This integration pattern is described in this document page. The second integration pattern is the one depicted in orange and covers the baseboard management controllers (BMCs), the hardware lifecycle host (HLH), the virtual machines and virtual appliances that run the hardware partner monitoring and management software, and the top of rack (TOR) switches. Since these components are hardware-partner specific, contact your hardware partner for documentation on how to integrate them with an external SIEM.
Configuring syslog forwarding
The syslog client in Azure Stack Hub supports the following configurations:
Syslog over TCP, with mutual authentication (client and server) and TLS 1.2 encryption: In this configuration, both the syslog server and the syslog client can verify the identity of each other via certificates. The messages are sent over a TLS 1.2 encrypted channel.
Syslog over TCP with server authentication and TLS 1.2 encryption: In this configuration, the syslog client can verify the identity of the syslog server via a certificate. The messages are sent over a TLS 1.2 encrypted channel.
Syslog over TCP, with no encryption: In this configuration, the syslog client and syslog server identities aren't verified. The messages are sent in clear text over TCP.
Syslog over UDP, with no encryption: In this configuration, the syslog client and syslog server identities aren't verified. The messages are sent in clear text over UDP.
Important
We strongly recommend to use TCP using authentication and encryption (configuration #1 or, at the very minimum, #2) for production environments to protect against man-in-the-middle attacks and eavesdropping of messages.
Cmdlets to configure syslog forwarding
Configuring syslog forwarding requires access to the privileged endpoint (PEP). Two PowerShell cmdlets have been added to the PEP to configure the syslog forwarding:
### cmdlet to pass the syslog server information to the client and to configure the transport protocol, the encryption and the authentication between the client and the server
Set-SyslogServer [-ServerName <String>] [-ServerPort <UInt16>] [-NoEncryption] [-SkipCertificateCheck] [-SkipCNCheck] [-UseUDP] [-Remove]
### cmdlet to configure the certificate for the syslog client to authenticate with the server
Set-SyslogClient [-pfxBinary <Byte[]>] [-CertPassword <SecureString>] [-RemoveCertificate] [-OutputSeverity]
Cmdlets parameters
Parameters for Set-SyslogServer cmdlet:
| Parameter | Description | Type | Required |
|---|---|---|---|
| ServerName | FQDN or IP address of the syslog server. | String | yes |
| ServerPort | Port number the syslog server is listening on. | UInt16 | yes |
| NoEncryption | Force the client to send syslog messages in clear text. | flag | no |
| SkipCertificateCheck | Skip validation of the certificate provided by the syslog server during initial TLS handshake. | flag | no |
| SkipCNCheck | Skip validation of the Common Name value of the certificate provided by the syslog server during initial TLS handshake. | flag | no |
| UseUDP | Use syslog with UDP as transport protocol. | flag | no |
| Remove | Remove configuration of the server from the client and stop syslog forwarding. | flag | no |
Parameters for Set-SyslogClient cmdlet:
| Parameter | Description | Type |
|---|---|---|
| pfxBinary | The contents of the pfx file, piped to a Byte[], containing the certificate to be used by the client as identity to authenticate against the syslog server. | Byte[] |
| CertPassword | Password to import the private key that's associated with the pfx file. | SecureString |
| RemoveCertificate | Remove certificate from the client. | flag |
| OutputSeverity | Level of output logging. Values are Default or Verbose. Default includes severity levels: warning, critical, or error. Verbose includes all severity levels: verbose, informational, warning, critical, or error. | String |
Configuring syslog forwarding with TCP, mutual authentication, and TLS 1.2 encryption
In this configuration, the syslog client in Azure Stack Hub forwards the messages to the syslog server over TCP, with TLS 1.2 encryption. During the initial handshake, the client verifies that the server provides a valid, trusted certificate. The client also provides a certificate to the server as proof of its identity. This configuration is the most secure as it provides a full validation of the identity of both the client and the server and it sends messages over an encrypted channel.
Important
We strongly recommend to use this configuration for production environments.
To configure syslog forwarding with TCP, mutual authentication, and TLS 1.2 encryption, run both these cmdlets on a PEP session:
# Configure the server
Set-SyslogServer -ServerName <FQDN or ip address of syslog server> -ServerPort <Port number on which the syslog server is listening on>
# Provide certificate to the client to authenticate against the server
Set-SyslogClient -pfxBinary <Byte[] of pfx file> -CertPassword <SecureString, password for accessing the pfx file>
The client certificate must have the same root as the one provided during the deployment of Azure Stack Hub. It also must contain a private key.
##Example on how to set your syslog client with the certificate for mutual authentication.
##This example script must be run from your hardware lifecycle host or privileged access workstation.
$ErcsNodeName = "<yourPEP>"
$password = ConvertTo-SecureString -String "<your cloudAdmin account password" -AsPlainText -Force
$cloudAdmin = "<your cloudAdmin account name>"
$CloudAdminCred = New-Object System.Management.Automation.PSCredential ($cloudAdmin, $password)
$certPassword = $password
$certContent = Get-Content -Path C:\cert\<yourClientCertificate>.pfx -Encoding Byte
$params = @{
ComputerName = $ErcsNodeName
Credential = $CloudAdminCred
ConfigurationName = "PrivilegedEndpoint"
}
$session = New-PSSession @params
$params = @{
Session = $session
ArgumentList = @($certContent, $certPassword)
}
Write-Verbose "Invoking cmdlet to set syslog client certificate..." -Verbose
Invoke-Command @params -ScriptBlock {
param($CertContent, $CertPassword)
Set-SyslogClient -PfxBinary $CertContent -CertPassword $CertPassword }
Configuring syslog forwarding with TCP, Server authentication, and TLS 1.2 encryption
In this configuration, the syslog client in Azure Stack Hub forwards the messages to the syslog server over TCP, with TLS 1.2 encryption. During the initial handshake, the client also verifies that the server provides a valid, trusted certificate. This configuration prevents the client from sending messages to untrusted destinations. TCP using authentication and encryption is the default configuration and represents the minimum level of security that We recommend for a production environment.
Set-SyslogServer -ServerName <FQDN or ip address of syslog server> -ServerPort <Port number on which the syslog server is listening on>
In case you want to test the integration of your syslog server with the Azure Stack Hub client by using a self-signed or untrusted certificate, you can use these flags to skip the server validation done by the client during the initial handshake.
#Skip validation of the Common Name value in the server certificate. Use this flag if you provide an IP address for your syslog server
Set-SyslogServer -ServerName <FQDN or ip address of syslog server> -ServerPort <Port number on which the syslog server is listening on>
-SkipCNCheck
#Skip entirely the server certificate validation
Set-SyslogServer -ServerName <FQDN or ip address of syslog server> -ServerPort <Port number on which the syslog server is listening on>
-SkipCertificateCheck
Important
We recommend against the use of -SkipCertificateCheck flag for production environments.
Configuring syslog forwarding with TCP and no encryption
In this configuration, the syslog client in Azure Stack Hub forwards the messages to the syslog server over TCP, with no encryption. The client doesn't verify the identity of the server nor does it provide its own identity to the server for verification.
Set-SyslogServer -ServerName <FQDN or ip address of syslog server> -ServerPort <Port number on which the syslog server is listening on> -NoEncryption
Important
We recommend against using this configuration for production environments.
Configuring syslog forwarding with UDP and no encryption
In this configuration, the syslog client in Azure Stack Hub forwards the messages to the syslog server over UDP, with no encryption. The client doesn't verify the identity of the server nor does it provide its own identity to the server for verification.
Set-SyslogServer -ServerName <FQDN or ip address of syslog server> -ServerPort <Port number on which the syslog server is listening on> -UseUDP
While UDP with no encryption is the easiest to configure, it doesn't provide any protection against man-in-the-middle attacks and eavesdropping of messages.
Important
We recommend against using this configuration for production environments.
Removing syslog forwarding configuration
To remove the syslog server configuration altogether and stop syslog forwarding:
Remove the syslog server configuration from the client
Set-SyslogServer -Remove
Remove the client certificate from the client
Set-SyslogClient -RemoveCertificate
Verifying the syslog setup
If you successfully connected the syslog client to your syslog server, you should soon start receiving events. If you don't see any event, verify the configuration of your syslog client by running the following cmdlets:
Verify the server configuration in the syslog client
Get-SyslogServer
Verify the certificate setup in the syslog client
Get-SyslogClient
Syslog message schema
The syslog forwarding of the Azure Stack Hub infrastructure sends messages formatted in Common Event Format (CEF). Each syslog message is structured based on this schema:
<Time> <Host> <CEF payload>
The CEF payload is based on the structure below, but the mapping for each field varies depending on the type of message (Windows Event, Alert created, Alert closed).
# Common Event Format schema
CEF: <Version>|<Device Vendor>|<Device Product>|<Device Version>|<Signature ID>|<Name>|<Severity>|<Extensions>
* Version: 0.0
* Device Vendor: Microsoft
* Device Product: Azure Stack Hub
* Device Version: 1.0
CEF mapping for privileged endpoint events
Prefix fields
* Signature ID: Microsoft-AzureStack-PrivilegedEndpoint: <PEP Event ID>
* Name: <PEP Task Name>
* Severity: mapped from PEP Level (details see the PEP Severity table below)
* Who: account used to connect to the PEP
* WhichIP: IP address of ERCS server hosting the PEP
Table of events for the privileged endpoint:
| Event | PEP event ID | PEP task name | Severity |
|---|---|---|---|
| PrivilegedEndpointAccessed | 1000 | PrivilegedEndpointAccessedEvent | 5 |
| SupportSessionTokenRequested | 1001 | SupportSessionTokenRequestedEvent | 5 |
| SupportSessionDevelopmentTokenRequested | 1002 | SupportSessionDevelopmentTokenRequestedEvent | 5 |
| SupportSessionUnlocked | 1003 | SupportSessionUnlockedEvent | 10 |
| SupportSessionFailedToUnlock | 1004 | SupportSessionFailedToUnlockEvent | 10 |
| PrivilegedEndpointClosed | 1005 | PrivilegedEndpointClosedEvent | 5 |
| NewCloudAdminUser | 1006 | NewCloudAdminUserEvent | 10 |
| RemoveCloudAdminUser | 1007 | RemoveCloudAdminUserEvent | 10 |
| SetCloudAdminUserPassword | 1008 | SetCloudAdminUserPasswordEvent | 5 |
| GetCloudAdminPasswordRecoveryToken | 1009 | GetCloudAdminPasswordRecoveryTokenEvent | 10 |
| ResetCloudAdminPassword | 1010 | ResetCloudAdminPasswordEvent | 10 |
| PrivilegedEndpointSessionTimedOut | 1017 | PrivilegedEndpointSessionTimedOutEvent | 5 |
PEP Severity table:
| Severity | Level | Numerical Value |
|---|---|---|
| 0 | Undefined | Value: 0. Indicates logs at all levels |
| 10 | Critical | Value: 1. Indicates logs for a critical alert |
| 8 | Error | Value: 2. Indicates logs for an error |
| 5 | Warning | Value: 3. Indicates logs for a warning |
| 2 | Information | Value: 4. Indicates logs for an informational message |
| 0 | Verbose | Value: 5. Indicates logs at all levels |
CEF mapping for recovery endpoint events
Prefix fields
* Signature ID: Microsoft-AzureStack-PrivilegedEndpoint: <REP Event ID>
* Name: <REP Task Name>
* Severity: mapped from REP Level (details see the REP Severity table below)
* Who: account used to connect to the REP
* WhichIP: IP address of the device used to connect to the REP
Table of events for the recovery endpoint:
| Event | REP event ID | REP task name | Severity |
|---|---|---|---|
| RecoveryEndpointAccessed | 1011 | RecoveryEndpointAccessedEvent | 5 |
| RecoverySessionTokenRequested | 1012 | RecoverySessionTokenRequestedEvent | 5 |
| RecoverySessionDevelopmentTokenRequested | 1013 | RecoverySessionDevelopmentTokenRequestedEvent | 5 |
| RecoverySessionUnlocked | 1014 | RecoverySessionUnlockedEvent | 10 |
| RecoverySessionFailedToUnlock | 1015 | RecoverySessionFailedToUnlockEvent | 10 |
| RecoveryEndpointClosed | 1016 | RecoveryEndpointClosedEvent | 5 |
REP Severity table:
| Severity | Level | Numerical value |
|---|---|---|
| 0 | Undefined | Value: 0. Indicates logs at all levels |
| 10 | Critical | Value: 1. Indicates logs for a critical alert |
| 8 | Error | Value: 2. Indicates logs for an error |
| 5 | Warning | Value: 3. Indicates logs for a warning |
| 2 | Information | Value: 4. Indicates logs for an informational message |
| 0 | Verbose | Value: 5. Indicates logs at all levels |
CEF mapping for Windows events
* Signature ID: ProviderName:EventID
* Name: TaskName
* Severity: Level (for details, see the severity table below)
* Extension: Custom Extension Name (for details, see the Custom Extension table below)
Severity table for Windows events:
| CEF severity value | Windows event level | Numerical value |
|---|---|---|
| 0 | Undefined | Value: 0. Indicates logs at all levels |
| 10 | Critical | Value: 1. Indicates logs for a critical alert |
| 8 | Error | Value: 2. Indicates logs for an error |
| 5 | Warning | Value: 3. Indicates logs for a warning |
| 2 | Information | Value: 4. Indicates logs for an informational message |
| 0 | Verbose | Value: 5. Indicates logs at all levels |
Custom extension table for Windows events in Azure Stack Hub:
| Custom extension name | Windows event example |
|---|---|
| MasChannel | System |
| MasComputer | test.azurestack.contoso.com |
| MasCorrelationActivityID | C8F40D7C-3764-423B-A4FA-C994442238AF |
| MasCorrelationRelatedActivityID | C8F40D7C-3764-423B-A4FA-C994442238AF |
| MasEventData | svchost!!4132,G,0!!!!EseDiskFlushConsistency!!ESENT!!0x800000 |
| MasEventDescription | The Group Policy settings for the user were processed successfully. There were no changes detected since the last successful processing of Group Policy. |
| MasEventID | 1501 |
| MasEventRecordID | 26637 |
| MasExecutionProcessID | 29380 |
| MasExecutionThreadID | 25480 |
| MasKeywords | 0x8000000000000000 |
| MasKeywordName | Audit Success |
| MasLevel | 4 |
| MasOpcode | 1 |
| MasOpcodeName | info |
| MasProviderEventSourceName | |
| MasProviderGuid | AEA1B4FA-97D1-45F2-A64C-4D69FFFD92C9 |
| MasProviderName | Microsoft-Windows-GroupPolicy |
| MasSecurityUserId | <Windows SID> |
| MasTask | 0 |
| MasTaskCategory | Process Creation |
| MasUserData | KB4093112!!5112!!Installed!!0x0!!WindowsUpdateAgent Xpath: /Event/UserData/* |
| MasVersion | 0 |
CEF mapping for alerts created
* Signature ID: Azure Stack Hub Alert Creation : FaultTypeId
* Name: FaultTypeId : AlertId
* Severity: Alert Severity (for details, see alerts severity table below)
* Extension: Custom Extension Name (for details, see the Custom Extension table below)
Alerts severity table:
| Severity | Level |
|---|---|
| 0 | Undefined |
| 10 | Critical |
| 5 | Warning |
Custom Extension table for Alerts created in Azure Stack Hub:
| Custom extension name | Example |
|---|---|
| MasEventDescription | DESCRIPTION: A user account <TestUser> was created for <TestDomain>. It's a potential security risk. -- REMEDIATION: Contact support. Customer Assistance is required to resolve this issue. Don't try to resolve this issue without their assistance. Before you open a support request, start the log file collection process using the guidance from https://aka.ms/azurestacklogfiles. |
CEF mapping for alerts closed
* Signature ID: Microsoft Azure Stack Hub Alert Creation : FaultTypeId
* Name: FaultTypeId : AlertId
* Severity: Information
The example below shows a syslog message with CEF payload:
2018:05:17:-23:59:28 -07:00 TestHost CEF:0.0|Microsoft|Azure Stack Hub|1.0|3|TITLE: User Account Created -- DESCRIPTION: A user account \<TestUser\> was created for \<TestDomain\>. It's a potential security risk. -- REMEDIATION: Please contact Support. Customer Assistance is required to resolve this issue. Do not try to resolve this issue without their assistance. Before you open a support request, start the log file collection process using the guidance from https://aka.ms/azurestacklogfiles|10
Syslog event types
The table lists all the event types, events, message schema or properties that are send via the syslog channel. Setup verbose switch should only be used if Windows informational events are required for SIEM integration.
| Event Type | Events or message schema | Requires verbose setting | Event Description (optional) |
|---|---|---|---|
| Azure Stack Hub Alerts | For the alert message schema see CEF mapping for alerts closed. A list of all alerts in shared in a separate document. |
No | System health alerts |
| Privileged Endpoint Events | For the privileged endpoint message schema see CEF mapping for privileged endpoint events. PrivilegedEndpointAccessed SupportSessionTokenRequested SupportSessionDevelopmentTokenRequested SupportSessionUnlocked SupportSessionFailedToUnlock PrivilegedEndpointClosed NewCloudAdminUser RemoveCloudAdminUser SetCloudAdminUserPassword GetCloudAdminPasswordRecoveryToken ResetCloudAdminPassword PrivilegedEndpointSessionTimedOut |
No | |
| Recovery Endpoint Events | For the recovery endpoint message schema see CEF mapping for recovery endpoint events. RecoveryEndpointAccessed RecoverySessionTokenRequested RecoverySessionDevelopmentTokenRequested RecoverySessionUnlocked RecoverySessionFailedToUnlock Recovand RecoveryEndpointClosed |
No | |
| Windows Security Events | For the Windows event message schema see CEF mapping for Windows events. |
Yes (To get information events) | Type: - Information - Warning - Error - Critical |
| ARM Events | Message properties: AzsSubscriptionId AzsCorrelationId AzsPrincipalOid AzsPrincipalPuid AzsTenantId AzsOperationName AzsOperationId AzsEventSource AzsDescription AzsResourceProvider AzsResourceUri AzsEventName AzsEventInstanceId AzsChannels AzsEventLevel AzsStatus AzsSubStatus AzsClaims AzsAuthorization AzsHttpRequest AzsProperties AzsEventTimestamp AzsAudience AzsIssuer AzsIssuedAt AzsApplicationId AzsUniqueTokenId AzsArmServiceRequestId AzsEventCategory |
No |
Each registered ARM resource can raise an event. |
| BCDR Events | Message schema: AuditingManualBackup { } AuditingConfig { Interval Retention IsSchedulerEnabled BackupPath } AuditingPruneBackupStore { IsInternalStore } |
No | These events track infra backup admin operations done by customer manually, includes trigger backup, change backup configuration, and prune backup data. |
| Infra Fault Creation and Closing Events | Message schema: InfrastructureFaultOpen { AzsFaultId, AzsFaultTypeName, AzsComponentType, AzsComponentName, AzsFaultHash, AzsCreatedTimeUtc, AzsSource } InfrastructureFaultClose { AzsFaultId, AzsFaultTypeName, AzsComponentType, AzsComponentName, AzsFaultHash, AzsLastUpdatedTimeUtc, AzsSource } |
No | Faults trigger workflows that attempt to remediate errors that can lead to alerts. If a fault has no remediation it does directly lead to an Alert. |
| Service Fault Creation and Closing Events | Message schema: ServiceFaultOpen { AzsFaultId, AzsFaultTypeName, AzsSubscriptionId, AzsResourceGroup, AzsServiceName, AzsResourceId AzsFaultHash, AzsCreatedTimeUtc, AzsSource } ServiceFaultClose { AzsFaultId, AzsFaultTypeName, AzsSubscriptionId, AzsResourceGroup, AzsServiceName, AzsResourceId AzsFaultHash, AzsLastUpdatedTimeUtc, AzsSource } |
No | Faults trigger workflows that attempt to remediate errors that can lead to alerts. If a fault has no remediation it does directly lead to an Alert. |
| PEP WAC events | Message schema: Prefix fields * Signature ID: Microsoft-AzureStack-PrivilegedEndpoint: <PEP Event ID> * Name: <PEP Task Name> * Severity: mapped from PEP Level (details see the PEP Severity table below) * Who: account used to connect to the PEP * WhichIP: IP address of ERCS server hosting the PEP WACServiceStartFailedEvent WACConnectedUserNotRetrievedEvent WACEnableExceptionEvent WACUserAddedEvent WACAddUserToLocalGroupFailedEvent WACIsUserInLocalGroupFailedEvent WACServiceStartTimeoutEvent WACServiceStartInvalidOperationEvent WACGetSidFromUserFailedEvent WACDisableFirewallFailedEvent WACCreateLocalGroupIfNotExistFailedEvent WACEnableFlagIsTrueEvent WACEnableFlagIsFalseEvent WACServiceStartedEvent |
No |