重要
网络安全组(NSG)流日志将于 2027 年 9 月 30 日停用。 2025 年 6 月 30 日之后,将无法再创建新的 NSG 流日志。 建议迁移到虚拟网络流日志,这些日志解决了 NSG 流日志的限制。 退休日期后,将不再支持 NSG 流日志中启用的流量分析功能,订阅中的现有 NSG 流日志资源也将被删除。 但是,不会从 Azure 存储中删除现有的 NSG 流日志记录,并且将继续遵循其配置的保留策略。 有关详细信息,请查看官方公告。
本文介绍如何使用 PowerShell 选择性地读取 Azure 网络观察程序流日志的某些部分,而无需分析整个日志。 流日志存储在块 blob 中的存储帐户中。 每个日志都是一个单独的块blob,每小时生成一次,并每隔几分钟用最新数据更新。 使用本文中提供的脚本,可以从流日志中读取最新数据,而无需下载整个日志。
本文中讨论的概念不局限于 PowerShell,适用于 Azure 存储 API 支持的所有语言。
先决条件
具有活动订阅的 Azure 帐户。 创建试用版订阅。
PowerShell 已安装在计算机上。 有关详细信息,请参阅在 Windows、Linux 和 macOS 上安装 PowerShell。 本文需要 Az PowerShell 模块。 有关详细信息,请参阅如何安装 Azure PowerShell。 要查找已安装的版本,请运行
Get-Module -ListAvailable Az
。一个或多个区域中的流日志。 有关详细信息,请参阅创建网络安全组流日志或创建虚拟网络流日志。
流日志和存储帐户的订阅所需的 Azure 角色访问控制(Azure RBAC)权限。 有关详细信息,请参阅网络观察程序 RBAC 权限。
检索阻止列表
在本部分中,你将设置查询流日志 blob 所需的变量,并列出 CloudBlockBlob 块 blob 中的各个块。
将 PowerShell 脚本更新为适合您环境的有效值。
function Get-VNetFlowLogCloudBlockBlob {
[CmdletBinding()]
param (
[string] [Parameter(Mandatory=$true)] $subscriptionId,
[string] [Parameter(Mandatory=$true)] $region,
[string] [Parameter(Mandatory=$true)] $VNetFlowLogName,
[string] [Parameter(Mandatory=$true)] $storageAccountName,
[string] [Parameter(Mandatory=$true)] $storageAccountResourceGroup,
[string] [Parameter(Mandatory=$true)] $macAddress,
[datetime] [Parameter(Mandatory=$true)] $logTime
)
process {
# Retrieve the primary storage account key to access the virtual network flow logs
$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroup -Name $storageAccountName).Value[0]
# Setup a new storage context to be used to query the logs
$ctx = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $StorageAccountKey
# Container name used by virtual network flow logs
$ContainerName = "insights-logs-flowlogflowevent"
# Name of the blob that contains the virtual network flow log
$BlobName = "flowLogResourceID=/$($subscriptionId.ToUpper())_NETWORKWATCHERRG/NETWORKWATCHER_$($region.ToUpper())_$($VNetFlowLogName.ToUpper())/y=$($logTime.Year)/m=$(($logTime).ToString("MM"))/d=$(($logTime).ToString("dd"))/h=$(($logTime).ToString("HH"))/m=00/macAddress=$($macAddress)/PT1H.json"
# Gets the storage blog
$Blob = Get-AzStorageBlob -Context $ctx -Container $ContainerName -Blob $BlobName
# Gets the block blog of type 'Microsoft.Azure.Storage.Blob.CloudBlob' from the storage blob
$CloudBlockBlob = [Microsoft.Azure.Storage.Blob.CloudBlockBlob] $Blob.ICloudBlob
#Return the Cloud Block Blob
$CloudBlockBlob
}
}
function Get-VNetFlowLogBlockList {
[CmdletBinding()]
param (
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
)
process {
# Stores the block list in a variable from the block blob.
$blockList = $CloudBlockBlob.DownloadBlockListAsync()
# Return the Block List
$blockList
}
}
$CloudBlockBlob = Get-VNetFlowLogCloudBlockBlob -subscriptionId "yourSubscriptionId" -region "yourVNetFlowLogRegion" -VNetFlowLogName "yourVNetFlowLogName" -storageAccountName "yourStorageAccountName" -storageAccountResourceGroup "yourStorageAccountRG" -macAddress "0022485D8CF8" -logTime "07/09/2023 03:00"
$blockList = Get-VNetFlowLogBlockList -CloudBlockBlob $CloudBlockBlob
$blockList
变量返回 blob 中块的列表。 每个块 blob 至少包含两个块。 第一个块长度为 12 个字节,并包含 JSON 日志的开括号。 另一个块是闭括号,其长度为 2 个字节。 以下示例日志中有七个单独条目。 日志中所有新条目会被添加到末尾、最后一个块之前。
Name Length Committed
---- ------ ---------
ZDk5MTk5N2FkNGE0MmY5MTk5ZWViYjA0YmZhODRhYzY= 12 True
NzQxNDA5MTRhNDUzMGI2M2Y1MDMyOWZlN2QwNDZiYzQ= 2685 True
ODdjM2UyMWY3NzFhZTU3MmVlMmU5MDNlOWEwNWE3YWY= 2586 True
ZDU2MjA3OGQ2ZDU3MjczMWQ4MTRmYWNhYjAzOGJkMTg= 2688 True
ZmM3ZWJjMGQ0ZDA1ODJlOWMyODhlOWE3MDI1MGJhMTc= 2775 True
ZGVkYTc4MzQzNjEyMzlmZWE5MmRiNjc1OWE5OTc0OTQ= 2676 True
ZmY2MjUzYTIwYWIyOGU1OTA2ZDY1OWYzNmY2NmU4ZTY= 2777 True
Mzk1YzQwM2U0ZWY1ZDRhOWFlMTNhYjQ3OGVhYmUzNjk= 2675 True
ZjAyZTliYWE3OTI1YWZmYjFmMWI0MjJhNzMxZTI4MDM= 2 True
读取块 blob
在本部分中,将读取 $blocklist
变量以检索数据。 在下面的示例中,我们将遍历数据块列表,从每个块读取字节并将其存储在数组中。 使用 DownloadRangeToByteArray 方法来检索数据。
function Get-VNetFlowLogReadBlock {
[CmdletBinding()]
param (
[System.Array] [Parameter(Mandatory=$true)] $blockList,
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
)
$blocklistResult = $blockList.Result
# Set the size of the byte array to the largest block
$maxvalue = ($blocklistResult | Measure-Object Length -Maximum).Maximum
Write-Host "Max value is ${maxvalue}"
# Create an array to store values in
$valuearray = @()
# Define the starting index to track the current block being read
$index = 0
# Loop through each block in the block list
for($i=0; $i -lt $blocklistResult.count; $i++)
{
# Create a byte array object to story the bytes from the block
$downloadArray = New-Object -TypeName byte[] -ArgumentList $maxvalue
# Download the data into the ByteArray, starting with the current index, for the number of bytes in the current block. Index is increased by 3 when reading to remove preceding comma.
$CloudBlockBlob.DownloadRangeToByteArray($downloadArray,0,$index, $($blockListResult[$i].Length)) | Out-Null
# Increment the index by adding the current block length to the previous index
$index = $index + $blockListResult[$i].Length
# Retrieve the string from the byte array
$value = [System.Text.Encoding]::ASCII.GetString($downloadArray)
# Add the log entry to the value array
$valuearray += $value
}
#Return the Array
$valuearray
}
$valuearray = Get-VNetFlowLogReadBlock -blockList $blockList -CloudBlockBlob $CloudBlockBlob
现在 $valuearray
数组包含每个块的字符串值。 若要验证该条目,请通过运行 $valuearray[$valuearray.Length-2]
从数组获取倒数第二个值。 不需要最后一个值,因为它是右中括号。
此值的结果如下例所示:
{
"time": "2023-07-09T03:59:30.2837112Z",
"flowLogVersion": 4,
"flowLogGUID": "abcdef01-2345-6789-0abc-def012345678",
"macAddress": "0022485D8CF8",
"category": "FlowLogFlowEvent",
"flowLogResourceID": "/SUBSCRIPTIONS/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/RESOURCEGROUPS/NETWORKWATCHERRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKWATCHERS/NETWORKWATCHER_China East/FLOWLOGS/MYVNET-MYRESOURCEGROUP-FLOWLOG",
"targetResourceID": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/myResourceGroup/providers/Microsoft.Network/virtualNetworks/myVNet",
"operationName": "FlowLogFlowEvent",
"flowRecords": {
"flows": [
{
"aclID": "00000000-1234-abcd-ef00-c1c2c3c4c5c6",
"flowGroups": [
{
"rule": "BlockHighRiskTCPPortsFromInternet",
"flowTuples": [
"1688875131557,45.119.212.87,192.168.0.4,53018,3389,6,I,D,NX,0,0,0,0"
]
},
{
"rule": "Internet",
"flowTuples": [
"1688875103311,35.203.210.145,192.168.0.4,56688,52113,6,I,D,NX,0,0,0,0",
"1688875119073,162.216.150.87,192.168.0.4,50111,9920,6,I,D,NX,0,0,0,0",
"1688875119910,205.210.31.253,192.168.0.4,54699,1801,6,I,D,NX,0,0,0,0",
"1688875121510,35.203.210.49,192.168.0.4,49250,33013,6,I,D,NX,0,0,0,0",
"1688875121684,162.216.149.206,192.168.0.4,49776,1290,6,I,D,NX,0,0,0,0",
"1688875124012,91.148.190.134,192.168.0.4,57963,40544,6,I,D,NX,0,0,0,0",
"1688875138568,35.203.211.204,192.168.0.4,51309,46956,6,I,D,NX,0,0,0,0",
"1688875142490,205.210.31.18,192.168.0.4,54140,30303,6,I,D,NX,0,0,0,0",
"1688875147864,194.26.135.247,192.168.0.4,53583,20232,6,I,D,NX,0,0,0,0"
]
}
]
}
]
}
}