Read NSG flow logs
Learn how to read NSG flow logs entries with PowerShell.
NSG flow logs are stored in a storage account in block blobs. Block blobs are made up of smaller blocks. Each log is a separate block blob that is generated every hour. New logs are generated every hour, the logs are updated with new entries every few minutes with the latest data. In this article you learn how to read portions of the flow logs.
Scenario
In the following scenario, you have an example flow log that is stored in a storage account. You learn how to selectively read the latest events in NSG flow logs. In this article you use PowerShell, however, the concepts discussed in the article aren't limited to the programming language, and are applicable to all languages supported by the Azure Storage APIs.
Setup
Before you begin, you must have Network Security Group Flow Logging enabled on one or many Network Security Groups in your account. For instructions on enabling Network Security flow logs, refer to the following article: Introduction to flow logging for Network Security Groups.
Retrieve the block list
The following PowerShell sets up the variables needed to query the NSG flow log blob and list the blocks within the CloudBlockBlob block blob. Update the script to contain valid values for your environment.
function Get-NSGFlowLogCloudBlockBlob {
[CmdletBinding()]
param (
[string] [Parameter(Mandatory=$true)] $subscriptionId,
[string] [Parameter(Mandatory=$true)] $NSGResourceGroupName,
[string] [Parameter(Mandatory=$true)] $NSGName,
[string] [Parameter(Mandatory=$true)] $storageAccountName,
[string] [Parameter(Mandatory=$true)] $storageAccountResourceGroup,
[string] [Parameter(Mandatory=$true)] $macAddress,
[datetime] [Parameter(Mandatory=$true)] $logTime
)
process {
# Retrieve the primary storage account key to access the NSG logs
$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroup -Name $storageAccountName).Value[0]
# Setup a new storage context to be used to query the logs
$ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
# Container name used by NSG flow logs
$ContainerName = "insights-logs-networksecuritygroupflowevent"
# Name of the blob that contains the NSG flow log
$BlobName = "resourceId=/SUBSCRIPTIONS/${subscriptionId}/RESOURCEGROUPS/${NSGResourceGroupName}/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/${NSGName}/y=$($logTime.Year)/m=$(($logTime).ToString("MM"))/d=$(($logTime).ToString("dd"))/h=$(($logTime).ToString("HH"))/m=00/macAddress=$($macAddress)/PT1H.json"
# Gets the storage blog
$Blob = Get-AzStorageBlob -Context $ctx -Container $ContainerName -Blob $BlobName
# Gets the block blog of type 'Microsoft.Azure.Storage.Blob.CloudBlob' from the storage blob
$CloudBlockBlob = [Microsoft.Azure.Storage.Blob.CloudBlockBlob] $Blob.ICloudBlob
#Return the Cloud Block Blob
$CloudBlockBlob
}
}
function Get-NSGFlowLogBlockList {
[CmdletBinding()]
param (
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
)
process {
# Stores the block list in a variable from the block blob.
$blockList = $CloudBlockBlob.DownloadBlockListAsync()
# Return the Block List
$blockList
}
}
$CloudBlockBlob = Get-NSGFlowLogCloudBlockBlob -subscriptionId "yourSubscriptionId" -NSGResourceGroupName "FLOWLOGSVALIDATIONWESTCENTRALUS" -NSGName "V2VALIDATIONVM-NSG" -storageAccountName "yourStorageAccountName" -storageAccountResourceGroup "ml-rg" -macAddress "000D3AF87856" -logTime "11/11/2018 03:00"
$blockList = Get-NSGFlowLogBlockList -CloudBlockBlob $CloudBlockBlob
The $blockList variable returns a list of the blocks in the blob. Each block blob contains at least two blocks. The first block has a length of 12 bytes, this block contains the opening brackets of the json log. The other block is the closing brackets and has a length of 2 bytes. As you can see the following example log has seven entries in it, each being an individual entry. All new entries in the log are added to the end right before the final block.
Name Length Committed
---- ------ ---------
ZDk5MTk5N2FkNGE0MmY5MTk5ZWViYjA0YmZhODRhYzY= 12 True
NzQxNDA5MTRhNDUzMGI2M2Y1MDMyOWZlN2QwNDZiYzQ= 2685 True
ODdjM2UyMWY3NzFhZTU3MmVlMmU5MDNlOWEwNWE3YWY= 2586 True
ZDU2MjA3OGQ2ZDU3MjczMWQ4MTRmYWNhYjAzOGJkMTg= 2688 True
ZmM3ZWJjMGQ0ZDA1ODJlOWMyODhlOWE3MDI1MGJhMTc= 2775 True
ZGVkYTc4MzQzNjEyMzlmZWE5MmRiNjc1OWE5OTc0OTQ= 2676 True
ZmY2MjUzYTIwYWIyOGU1OTA2ZDY1OWYzNmY2NmU4ZTY= 2777 True
Mzk1YzQwM2U0ZWY1ZDRhOWFlMTNhYjQ3OGVhYmUzNjk= 2675 True
ZjAyZTliYWE3OTI1YWZmYjFmMWI0MjJhNzMxZTI4MDM= 2 True
Read the block blob
Next you need to read the $blocklist variable to retrieve the data. In this example we iterate through the blocklist, read the bytes from each block and story them in an array. Use the DownloadRangeToByteArray method to retrieve the data.
function Get-NSGFlowLogReadBlock {
[CmdletBinding()]
param (
[System.Array] [Parameter(Mandatory=$true)] $blockList,
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
)
# Set the size of the byte array to the largest block
$maxvalue = ($blocklist | measure Length -Maximum).Maximum
# Create an array to store values in
$valuearray = @()
# Define the starting index to track the current block being read
$index = 0
# Loop through each block in the block list
for($i=0; $i -lt $blocklist.count; $i++)
{
# Create a byte array object to story the bytes from the block
$downloadArray = New-Object -TypeName byte[] -ArgumentList $maxvalue
# Download the data into the ByteArray, starting with the current index, for the number of bytes in the current block. Index is increased by 3 when reading to remove preceding comma.
$CloudBlockBlob.DownloadRangeToByteArray($downloadArray,0,$index, $($blockList[$i].Length)) | Out-Null
# Increment the index by adding the current block length to the previous index
$index = $index + $blockList[$i].Length
# Retrieve the string from the byte array
$value = [System.Text.Encoding]::ASCII.GetString($downloadArray)
# Add the log entry to the value array
$valuearray += $value
}
#Return the Array
$valuearray
}
$valuearray = Get-NSGFlowLogReadBlock -blockList $blockList -CloudBlockBlob $CloudBlockBlob
Now the $valuearray array contains the string value of each block. To verify the entry, get the second to the last value from the array by running $valuearray[$valuearray.Length-2]. You don't want the last value, because it's the closing bracket.
The results of this value are shown in the following example:
{
"time": "2017-06-16T20:59:43.7340000Z",
"systemId": "5f4d02d3-a7d0-4ed4-9ce8-c0ae9377951c",
"category": "NetworkSecurityGroupFlowEvent",
"resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/CONTOSORG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/CONTOSONSG",
"operationName": "NetworkSecurityGroupFlowEvents",
"properties": {"Version":1,"flows":[{"rule":"DefaultRule_AllowInternetOutBound","flows":[{"mac":"000D3A18077E","flowTuples":["1497646722,10.0.0.4,168.62.32.14,44904,443,T,O,A","1497646722,10.0.0.4,52.240.48.24,45218,443,T,O,A","1497646725,10.
0.0.4,168.62.32.14,44910,443,T,O,A","1497646725,10.0.0.4,52.240.48.24,45224,443,T,O,A","1497646728,10.0.0.4,168.62.32.14,44916,443,T,O,A","1497646728,10.0.0.4,52.240.48.24,45230,443,T,O,A","1497646732,10.0.0.4,168.62.32.14,44922,443,T,O,A","14976
46732,10.0.0.4,52.240.48.24,45236,443,T,O,A","1497646735,10.0.0.4,168.62.32.14,44928,443,T,O,A","1497646735,10.0.0.4,52.240.48.24,45242,443,T,O,A","1497646738,10.0.0.4,168.62.32.14,44934,443,T,O,A","1497646738,10.0.0.4,52.240.48.24,45248,443,T,O,
A","1497646742,10.0.0.4,168.62.32.14,44942,443,T,O,A","1497646742,10.0.0.4,52.240.48.24,45256,443,T,O,A","1497646745,10.0.0.4,168.62.32.14,44948,443,T,O,A","1497646745,10.0.0.4,52.240.48.24,45262,443,T,O,A","1497646749,10.0.0.4,168.62.32.14,44954
,443,T,O,A","1497646749,10.0.0.4,52.240.48.24,45268,443,T,O,A","1497646753,10.0.0.4,168.62.32.14,44960,443,T,O,A","1497646753,10.0.0.4,52.240.48.24,45274,443,T,O,A","1497646756,10.0.0.4,168.62.32.14,44966,443,T,O,A","1497646756,10.0.0.4,52.240.48
.24,45280,443,T,O,A","1497646759,10.0.0.4,168.62.32.14,44972,443,T,O,A","1497646759,10.0.0.4,52.240.48.24,45286,443,T,O,A","1497646763,10.0.0.4,168.62.32.14,44978,443,T,O,A","1497646763,10.0.0.4,52.240.48.24,45292,443,T,O,A","1497646766,10.0.0.4,
168.62.32.14,44984,443,T,O,A","1497646766,10.0.0.4,52.240.48.24,45298,443,T,O,A","1497646769,10.0.0.4,168.62.32.14,44990,443,T,O,A","1497646769,10.0.0.4,52.240.48.24,45304,443,T,O,A","1497646773,10.0.0.4,168.62.32.14,44996,443,T,O,A","1497646773,
10.0.0.4,52.240.48.24,45310,443,T,O,A","1497646776,10.0.0.4,168.62.32.14,45002,443,T,O,A","1497646776,10.0.0.4,52.240.48.24,45316,443,T,O,A","1497646779,10.0.0.4,168.62.32.14,45008,443,T,O,A","1497646779,10.0.0.4,52.240.48.24,45322,443,T,O,A"]}]}
,{"rule":"DefaultRule_DenyAllInBound","flows":[]},{"rule":"UserRule_ssh-rule","flows":[]},{"rule":"UserRule_web-rule","flows":[{"mac":"000D3A18077E","flowTuples":["1497646738,13.82.225.93,10.0.0.4,1180,80,T,I,A","1497646750,13.82.225.93,10.0.0.4,
1184,80,T,I,A","1497646768,13.82.225.93,10.0.0.4,1181,80,T,I,A","1497646780,13.82.225.93,10.0.0.4,1336,80,T,I,A"]}]}]}
}
This scenario is an example of how to read entries in NSG flow logs without having to parse the entire log. You can read new entries in the log as they're written by using the block ID or by tracking the length of blocks stored in the block blob. This allows you to read only the new entries.
Next steps
Visit Use Elastic Stack, Use Grafana, and Use Graylog to learn more about ways to view NSG flow logs. An Open Source Azure Function approach to consuming the blobs directly and emitting to various log analytics consumers may be found here: Azure Network Watcher NSG Flow Logs Connector.
You can use Azure Traffic Analytics to get insights on your traffic flows. Traffic Analytics uses Log Analytics to make your traffic flow queryable.
To learn more about storage blobs visit: Azure Functions Blob storage bindings