读取 NSG 流日志Read NSG flow logs

了解如何使用 PowerShell 读取 NSG 流日志条目。Learn how to read NSG flow logs entries with PowerShell.

NSG 流日志存储于块 blob 中的存储帐户中。NSG flow logs are stored in a storage account in block blobs. 块 blob 由一些更小的块组成。Block blobs are made up of smaller blocks. 每个日志是每个一小时生成的单独块 blob。Each log is a separate block blob that is generated every hour. 每隔一小时会生成新的日志,每隔几分钟会以包含最新数据的新条目来更新日志。New logs are generated every hour, the logs are updated with new entries every few minutes with the latest data. 本文介绍如何读取部分流日志。In this article you learn how to read portions of the flow logs.

备注

本文进行了更新,以便使用新的 Azure PowerShell Az 模块。This article has been updated to use the new Azure PowerShell Az module. 你仍然可以使用 AzureRM 模块,至少在 2020 年 12 月之前,它将继续接收 bug 修补程序。You can still use the AzureRM module, which will continue to receive bug fixes until at least December 2020. 若要详细了解新的 Az 模块和 AzureRM 兼容性,请参阅新 Azure Powershell Az 模块简介To learn more about the new Az module and AzureRM compatibility, see Introducing the new Azure PowerShell Az module. 有关 Az 模块安装说明,请参阅安装 Azure PowerShellFor Az module installation instructions, see Install Azure PowerShell.

方案Scenario

在如下方案中,你有一个存储在存储帐户的示例流日志。In the following scenario, you have an example flow log that is stored in a storage account. 了解如何选择性地读取 NSG 流日志中的最新事件。You learn how to selectively read the latest events in NSG flow logs. 虽然本文中将使用 PowerShell,但本文中讨论到的概念并不限于编程语言,而是适用于 Azure 存储 API 支持的所有语言。In this article you use PowerShell, however, the concepts discussed in the article are not limited to the programming language, and are applicable to all languages supported by the Azure Storage APIs.

设置Setup

在开始之前,必须在帐户中的一个或多个网络安全组上启用网络安全组流日志记录。Before you begin, you must have Network Security Group Flow Logging enabled on one or many Network Security Groups in your account. 有关如何启用网络安全流日志的说明,请参阅以下文章:Introduction to flow logging for Network Security Groups(网络安全组流日志记录简介)。For instructions on enabling Network Security flow logs, refer to the following article: Introduction to flow logging for Network Security Groups.

检索块列表Retrieve the block list

下方 PowerShell 设置查询 NSG 流日志 blob 和列出 CloudBlockBlob 块 blob 中的块所需的变量。The following PowerShell sets up the variables needed to query the NSG flow log blob and list the blocks within the CloudBlockBlob block blob. 更新脚本以包含适合你环境的有效值。Update the script to contain valid values for your environment.

function Get-NSGFlowLogCloudBlockBlob {
    [CmdletBinding()]
    param (
        [string] [Parameter(Mandatory=$true)] $subscriptionId,
        [string] [Parameter(Mandatory=$true)] $NSGResourceGroupName,
        [string] [Parameter(Mandatory=$true)] $NSGName,
        [string] [Parameter(Mandatory=$true)] $storageAccountName,
        [string] [Parameter(Mandatory=$true)] $storageAccountResourceGroup,
        [string] [Parameter(Mandatory=$true)] $macAddress,
        [datetime] [Parameter(Mandatory=$true)] $logTime
    )

    process {
        # Retrieve the primary storage account key to access the NSG logs
        $StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroup -Name $storageAccountName).Value[0]

        # Setup a new storage context to be used to query the logs
        $ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey

        # Container name used by NSG flow logs
        $ContainerName = "insights-logs-networksecuritygroupflowevent"

        # Name of the blob that contains the NSG flow log
        $BlobName = "resourceId=/SUBSCRIPTIONS/${subscriptionId}/RESOURCEGROUPS/${NSGResourceGroupName}/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/${NSGName}/y=$($logTime.Year)/m=$(($logTime).ToString("MM"))/d=$(($logTime).ToString("dd"))/h=$(($logTime).ToString("HH"))/m=00/macAddress=$($macAddress)/PT1H.json"

        # Gets the storage blog
        $Blob = Get-AzStorageBlob -Context $ctx -Container $ContainerName -Blob $BlobName

        # Gets the block blog of type 'Microsoft.Azure.Storage.Blob.CloudBlob' from the storage blob
        $CloudBlockBlob = [Microsoft.Azure.Storage.Blob.CloudBlockBlob] $Blob.ICloudBlob

        #Return the Cloud Block Blob
        $CloudBlockBlob
    }
}

function Get-NSGFlowLogBlockList  {
    [CmdletBinding()]
    param (
        [Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
    )
    process {
        # Stores the block list in a variable from the block blob.
        $blockList = $CloudBlockBlob.DownloadBlockListAsync()

        # Return the Block List
        $blockList
    }
}

$CloudBlockBlob = Get-NSGFlowLogCloudBlockBlob -subscriptionId "yourSubscriptionId" -NSGResourceGroupName "FLOWLOGSVALIDATIONWESTCENTRALUS" -NSGName "V2VALIDATIONVM-NSG" -storageAccountName "yourStorageAccountName" -storageAccountResourceGroup "ml-rg" -macAddress "000D3AF87856" -logTime "11/11/2018 03:00" 

$blockList = Get-NSGFlowLogBlockList -CloudBlockBlob $CloudBlockBlob

$blockList 变量返回 blob 中块的列表。The $blockList variable returns a list of the blocks in the blob. 每个块 blob 至少包含两个块。Each block blob contains at least two blocks. 第一个块长度为 12 字节,此块包含 json 日志的开括号。The first block has a length of 12 bytes, this block contains the opening brackets of the json log. 另一个块是闭括号,其长度为 2 字节。The other block is the closing brackets and has a length of 2 bytes. 正如你所见,下面的示例日志中具有 7 个条目,其中每个皆为单独条目。As you can see the following example log has seven entries in it, each being an individual entry. 日志中所有新条目会被添加到末尾、最后一个块之前。All new entries in the log are added to the end right before the final block.

Name                                         Length Committed
----                                         ------ ---------
ZDk5MTk5N2FkNGE0MmY5MTk5ZWViYjA0YmZhODRhYzY=     12      True
NzQxNDA5MTRhNDUzMGI2M2Y1MDMyOWZlN2QwNDZiYzQ=   2685      True
ODdjM2UyMWY3NzFhZTU3MmVlMmU5MDNlOWEwNWE3YWY=   2586      True
ZDU2MjA3OGQ2ZDU3MjczMWQ4MTRmYWNhYjAzOGJkMTg=   2688      True
ZmM3ZWJjMGQ0ZDA1ODJlOWMyODhlOWE3MDI1MGJhMTc=   2775      True
ZGVkYTc4MzQzNjEyMzlmZWE5MmRiNjc1OWE5OTc0OTQ=   2676      True
ZmY2MjUzYTIwYWIyOGU1OTA2ZDY1OWYzNmY2NmU4ZTY=   2777      True
Mzk1YzQwM2U0ZWY1ZDRhOWFlMTNhYjQ3OGVhYmUzNjk=   2675      True
ZjAyZTliYWE3OTI1YWZmYjFmMWI0MjJhNzMxZTI4MDM=      2      True

读取块 blobRead the block blob

接下来需要读取 $blocklist 变量以检索数据。Next you need to read the $blocklist variable to retrieve the data. 在此示例中我们循环访问阻止列表,从每个块读取字节并将它们存储在数组中。In this example we iterate through the blocklist, read the bytes from each block and story them in an array. 使用 DownloadRangeToByteArray 方法来检索数据。Use the DownloadRangeToByteArray method to retrieve the data.

function Get-NSGFlowLogReadBlock  {
    [CmdletBinding()]
    param (
        [System.Array] [Parameter(Mandatory=$true)] $blockList,
        [Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob

    )
    # Set the size of the byte array to the largest block
    $maxvalue = ($blocklist | measure Length -Maximum).Maximum

    # Create an array to store values in
    $valuearray = @()

    # Define the starting index to track the current block being read
    $index = 0

    # Loop through each block in the block list
    for($i=0; $i -lt $blocklist.count; $i++)
    {
        # Create a byte array object to story the bytes from the block
        $downloadArray = New-Object -TypeName byte[] -ArgumentList $maxvalue

        # Download the data into the ByteArray, starting with the current index, for the number of bytes in the current block. Index is increased by 3 when reading to remove preceding comma.
        $CloudBlockBlob.DownloadRangeToByteArray($downloadArray,0,$index, $($blockList[$i].Length)) | Out-Null

        # Increment the index by adding the current block length to the previous index
        $index = $index + $blockList[$i].Length

        # Retrieve the string from the byte array

        $value = [System.Text.Encoding]::ASCII.GetString($downloadArray)

        # Add the log entry to the value array
        $valuearray += $value
    }
    #Return the Array
    $valuearray
}
$valuearray = Get-NSGFlowLogReadBlock -blockList $blockList -CloudBlockBlob $CloudBlockBlob

现在 $valuearray 数组包含每个块的字符串值。Now the $valuearray array contains the string value of each block. 若要验证该条目,请通过运行 $valuearray[$valuearray.Length-2] 从数组获取倒数第二个值。To verify the entry, get the second to the last value from the array by running $valuearray[$valuearray.Length-2]. 不需要最后一个值,因为它是闭括号。You do not want the last value, because it is the closing bracket.

此值的结果如下例所示:The results of this value are shown in the following example:

        {
             "time": "2017-06-16T20:59:43.7340000Z",
             "systemId": "5f4d02d3-a7d0-4ed4-9ce8-c0ae9377951c",
             "category": "NetworkSecurityGroupFlowEvent",
             "resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/CONTOSORG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/CONTOSONSG",
             "operationName": "NetworkSecurityGroupFlowEvents",
             "properties": {"Version":1,"flows":[{"rule":"DefaultRule_AllowInternetOutBound","flows":[{"mac":"000D3A18077E","flowTuples":["1497646722,10.0.0.4,168.62.32.14,44904,443,T,O,A","1497646722,10.0.0.4,52.240.48.24,45218,443,T,O,A","1497646725,10.
0.0.4,168.62.32.14,44910,443,T,O,A","1497646725,10.0.0.4,52.240.48.24,45224,443,T,O,A","1497646728,10.0.0.4,168.62.32.14,44916,443,T,O,A","1497646728,10.0.0.4,52.240.48.24,45230,443,T,O,A","1497646732,10.0.0.4,168.62.32.14,44922,443,T,O,A","14976
46732,10.0.0.4,52.240.48.24,45236,443,T,O,A","1497646735,10.0.0.4,168.62.32.14,44928,443,T,O,A","1497646735,10.0.0.4,52.240.48.24,45242,443,T,O,A","1497646738,10.0.0.4,168.62.32.14,44934,443,T,O,A","1497646738,10.0.0.4,52.240.48.24,45248,443,T,O,
A","1497646742,10.0.0.4,168.62.32.14,44942,443,T,O,A","1497646742,10.0.0.4,52.240.48.24,45256,443,T,O,A","1497646745,10.0.0.4,168.62.32.14,44948,443,T,O,A","1497646745,10.0.0.4,52.240.48.24,45262,443,T,O,A","1497646749,10.0.0.4,168.62.32.14,44954
,443,T,O,A","1497646749,10.0.0.4,52.240.48.24,45268,443,T,O,A","1497646753,10.0.0.4,168.62.32.14,44960,443,T,O,A","1497646753,10.0.0.4,52.240.48.24,45274,443,T,O,A","1497646756,10.0.0.4,168.62.32.14,44966,443,T,O,A","1497646756,10.0.0.4,52.240.48
.24,45280,443,T,O,A","1497646759,10.0.0.4,168.62.32.14,44972,443,T,O,A","1497646759,10.0.0.4,52.240.48.24,45286,443,T,O,A","1497646763,10.0.0.4,168.62.32.14,44978,443,T,O,A","1497646763,10.0.0.4,52.240.48.24,45292,443,T,O,A","1497646766,10.0.0.4,
168.62.32.14,44984,443,T,O,A","1497646766,10.0.0.4,52.240.48.24,45298,443,T,O,A","1497646769,10.0.0.4,168.62.32.14,44990,443,T,O,A","1497646769,10.0.0.4,52.240.48.24,45304,443,T,O,A","1497646773,10.0.0.4,168.62.32.14,44996,443,T,O,A","1497646773,
10.0.0.4,52.240.48.24,45310,443,T,O,A","1497646776,10.0.0.4,168.62.32.14,45002,443,T,O,A","1497646776,10.0.0.4,52.240.48.24,45316,443,T,O,A","1497646779,10.0.0.4,168.62.32.14,45008,443,T,O,A","1497646779,10.0.0.4,52.240.48.24,45322,443,T,O,A"]}]}
,{"rule":"DefaultRule_DenyAllInBound","flows":[]},{"rule":"UserRule_ssh-rule","flows":[]},{"rule":"UserRule_web-rule","flows":[{"mac":"000D3A18077E","flowTuples":["1497646738,13.82.225.93,10.0.0.4,1180,80,T,I,A","1497646750,13.82.225.93,10.0.0.4,
1184,80,T,I,A","1497646768,13.82.225.93,10.0.0.4,1181,80,T,I,A","1497646780,13.82.225.93,10.0.0.4,1336,80,T,I,A"]}]}]}
        }

此方案演示了如何无需分析整个日志而读取 NSG 流日志中的条目。This scenario is an example of how to read entries in NSG flow logs without having to parse the entire log. 可以读取日志中新条目,因为它们是通过使用块 ID 或跟踪块 blob 中存储的块的长度而写入的。You can read new entries in the log as they are written by using the block ID or by tracking the length of blocks stored in the block blob. 这可实现仅读取新条目。This allows you to read only the new entries.

后续步骤Next steps

请访问使用弹性堆栈使用 Grafana使用 Graylog 详细了解查看 NSG 流日志的方法。Visit Use Elastic Stack, Use Grafana, and Use Graylog to learn more about ways to view NSG flow logs. 可以在此处找到直接使用 Blob 并发送给各种日志分析使用者的开源 Azure 函数方法:Azure 网络观察程序 NSG 流日志连接器An Open Source Azure Function approach to consuming the blobs directly and emitting to various log analytics consumers may be found here: Azure Network Watcher NSG Flow Logs Connector.

可使用 Azure 流量分析获取有关流量流的见解。You can use Azure Traffic Analytics to get insights on your traffic flows. 流量分析使用 Log Analytics 来使流量流可查询。Traffic Analytics uses Log Analytics to make your traffic flow queryable.

若要详细了解存储 blob,请访问:Azure Functions Blob 存储绑定To learn more about storage blobs visit: Azure Functions Blob storage bindings