使用网络观察程序和开源工具执行网络入侵检测Perform network intrusion detection with Network Watcher and open source tools

数据包捕获是一个重要组件,可以实施网络入侵检测系统 (IDS) 并执行网络安全监视 (NSM)。Packet captures are a key component for implementing network intrusion detection systems (IDS) and performing Network Security Monitoring (NSM). 可以借助多种开源 IDS 工具来处理数据包捕获,并检查潜在网络入侵和恶意活动的签名。There are several open source IDS tools that process packet captures and look for signatures of possible network intrusions and malicious activity. 使用网络观察程序提供的数据包捕获,可以分析网络中是否存在任何有害入侵或漏洞。Using the packet captures provided by Network Watcher, you can analyze your network for any harmful intrusions or vulnerabilities.

Suricata 就是这样的一种开源工具,它是一个 IDS 引擎,可使用规则集来监视网络流量,每当出现可疑事件时,它会触发警报。One such open source tool is Suricata, an IDS engine that uses rulesets to monitor network traffic and triggers alerts whenever suspicious events occur. Suricata 提供多线程引擎,意味着它能够以更高的速度和效率执行网络流量分析。Suricata offers a multi-threaded engine, meaning it can perform network traffic analysis with increased speed and efficiency. 有关 Suricata 及其功能的更多详细信息,请访问其网站 https://suricata-ids.org/For more details about Suricata and its capabilities, visit their website at https://suricata-ids.org/.

方案Scenario

本文介绍如何将环境设置为使用网络观察程序、Suricata 和 Elastic Stack 执行网络入侵检测。This article explains how to set up your environment to perform network intrusion detection using Network Watcher, Suricata, and the Elastic Stack. 网络观察程序提供用于执行网络入侵检测的数据包捕获。Network Watcher provides you with the packet captures used to perform network intrusion detection. Suricata 处理数据包捕获,并根据与其给定威胁规则集匹配的数据包触发警报。Suricata processes the packet captures and trigger alerts based on packets that match its given ruleset of threats. 这些警报存储在本地计算机上的某个日志文件中。These alerts are stored in a log file on your local machine. 使用 Elastic Stack 可为 Suricata 生成的日志编制索引,并使用这些日志创建 Kibana 仪表板,提供日志的可视化形式,同时,提供潜在网络漏洞的见解。Using the Elastic Stack, the logs generated by Suricata can be indexed and used to create a Kibana dashboard, providing you with a visual representation of the logs and a means to quickly gain insights to potential network vulnerabilities.

简单的 Web 应用程序方案

可在 Azure VM 上设置这两个开源工具,以便在自己的 Azure 网络环境内部执行此分析。Both open source tools can be set up on an Azure VM, allowing you to perform this analysis within your own Azure network environment.

步骤Steps

安装 SuricataInstall Suricata

有关所有其他安装方法,请访问 https://suricata.readthedocs.io/en/suricata-5.0.2/quickstart.html#installationFor all other methods of installation, visit https://suricata.readthedocs.io/en/suricata-5.0.2/quickstart.html#installation

  1. 在 VM 的命令行终端中运行以下命令:In the command-line terminal of your VM run the following commands:

    sudo add-apt-repository ppa:oisf/suricata-stable
    sudo apt-get update
    sudo sudo apt-get install suricata
    
  2. 若要验证安装,请运行命令 suricata -h 查看命令的完整列表。To verify your installation, run the command suricata -h to see the full list of commands.

下载 Emerging Threats 规则集Download the Emerging Threats ruleset

目前,我们尚未创建运行 Suricata 所需的任何规则。At this stage, we do not have any rules for Suricata to run. 如果想要检测特定的网络威胁,可以创建自己的规则;或者,也可以使用许多提供商开发的规则集,例如 Emerging Threats,或 Snort 开发的 VRT 规则。You can create your own rules if there are specific threats to your network you would like to detect, or you can also use developed rule sets from a number of providers, such as Emerging Threats, or VRT rules from Snort. 本文使用可免费访问的 Emerging Threats 规则集:We use the freely accessible Emerging Threats ruleset here:

下载该规则集,并将其复制到目录:Download the rule set and copy them into the directory:

wget https://rules.emergingthreats.net/open/suricata/emerging.rules.tar.gz
tar zxf emerging.rules.tar.gz
sudo cp -r rules /etc/suricata/

使用 Suricata 处理数据包捕获Process packet captures with Suricata

若要使用 Suricata 处理数据包捕获,请运行以下命令:To process packet captures using Suricata, run the following command:

sudo suricata -c /etc/suricata/suricata.yaml -r <location_of_pcapfile>

若要检查生成的警报,请阅读 fast.log 文件:To check the resulting alerts, read the fast.log file:

tail -f /var/log/suricata/fast.log

设置 Elastic StackSet up the Elastic Stack

尽管 Suricata 生成的日志包含有关网络情况的重要信息,但这些日志文件并不是很容易阅读和理解。While the logs that Suricata produces contain valuable information about what's happening on our network, these log files aren't the easiest to read and understand. 通过将 Suricata 与 Elastic Stack 相连接,可以创建一个 Kibana 仪表板,从而可以搜索、可视化、分析日志并从中获得见解。By connecting Suricata with the Elastic Stack, we can create a Kibana dashboard what allows us to search, graph, analyze, and derive insights from our logs.

安装 ElasticsearchInstall Elasticsearch

  1. Elastic Stack 5.0 及更高版本需要 Java 8。The Elastic Stack from version 5.0 and above requires Java 8. 运行命令 java -version 可以检查版本。Run the command java -version to check your version. 如果尚未安装 java,请参阅 Azure 支持的 JDK 上的文档。If you do not have java installed, refer to documentation on the Azure-suppored JDKs.

  2. 下载适用于系统的正确二进制程序包:Download the correct binary package for your system:

    curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.2.0.deb
    sudo dpkg -i elasticsearch-5.2.0.deb
    sudo /etc/init.d/elasticsearch start
    

    Elasticsearch Installation(Elasticsearch 安装)中介绍了其他安装方法Other installation methods can be found at Elasticsearch Installation

  3. 使用以下命令验证 Elasticsearch 是否正在运行:Verify that Elasticsearch is running with the command:

    curl http://127.0.0.1:9200
    

    应会显示如下所示的响应:You should see a response similar to this:

    {
    "name" : "Angela Del Toro",
    "cluster_name" : "elasticsearch",
    "version" : {
        "number" : "5.2.0",
        "build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe",
        "build_timestamp" : "2016-01-27T13:32:39Z",
        "build_snapshot" : false,
        "lucene_version" : "6.1.0"
    },
    "tagline" : "You Know, for Search"
    }
    

有关安装 Elasticsearch 的其他说明,请参阅 Installation(安装)页For further instructions on installing Elastic search, refer to the page Installation

安装 LogstashInstall Logstash

  1. 若要安装 Logstash,请运行以下命令:To install Logstash run the following commands:

    curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-5.2.0.deb
    sudo dpkg -i logstash-5.2.0.deb
    
  2. 接下来,需要将 Logstash 配置为读取 eve.json 文件的输出。Next we need to configure Logstash to read from the output of eve.json file. 使用以下命令创建 logstash.conf 文件:Create a logstash.conf file using:

    sudo touch /etc/logstash/conf.d/logstash.conf
    
  3. 将以下内容添加到该文件(确保 eve.json 文件的路径正确):Add the following content to the file (make sure that the path to the eve.json file is correct):

    input {
    file {
        path => ["/var/log/suricata/eve.json"]
        codec =>  "json"
        type => "SuricataIDPS"
    }
    
    }
    
    filter {
    if [type] == "SuricataIDPS" {
        date {
        match => [ "timestamp", "ISO8601" ]
        }
        ruby {
        code => "
            if event.get('[event_type]') == 'fileinfo'
            event.set('[fileinfo][type]', event.get('[fileinfo][magic]').to_s.split(',')[0])
            end
        "
        }
    
        ruby{
        code => "
            if event.get('[event_type]') == 'alert'
            sp = event.get('[alert][signature]').to_s.split(' group ')
            if (sp.length == 2) and /\A\d+\z/.match(sp[1])
                event.set('[alert][signature]', sp[0])
            end
            end
            "
        }
    }
    
    if [src_ip]  {
        geoip {
        source => "src_ip"
        target => "geoip"
        #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
        convert => [ "[geoip][coordinates]", "float" ]
        }
        if ![geoip.ip] {
        if [dest_ip]  {
            geoip {
            source => "dest_ip"
            target => "geoip"
            #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
            add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
            add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
            }
            mutate {
            convert => [ "[geoip][coordinates]", "float" ]
            }
        }
        }
    }
    }
    
    output {
    elasticsearch {
        hosts => "localhost"
    }
    }
    
  4. 确保提供对 eve.json 文件的适当权限,使 Logstash 能够引入该文件。Make sure to give the correct permissions to the eve.json file so that Logstash can ingest the file.

    sudo chmod 775 /var/log/suricata/eve.json
    
  5. 若要启动 Logstash,请运行以下命令:To start Logstash run the command:

    sudo /etc/init.d/logstash start
    

有关安装 Logstash 的其他说明,请参阅正式文档For further instructions on installing Logstash, refer to the official documentation

安装 KibanaInstall Kibana

  1. 运行以下命令以安装 Kibana:Run the following commands to install Kibana:

    curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-5.2.0-linux-x86_64.tar.gz
    tar xzvf kibana-5.2.0-linux-x86_64.tar.gz
    
    
  2. 若要运行 Kibana,请使用以下命令:To run Kibana use the commands:

    cd kibana-5.2.0-linux-x86_64/
    ./bin/kibana
    
  3. 若要查看 Kibana Web 界面,请导航到 http://localhost:5601To view your Kibana web interface, navigate to http://localhost:5601

  4. 对于本方案,用于 Suricata 日志的索引模式为“logstash-*”For this scenario, the index pattern used for the Suricata logs is "logstash-*"

  5. 如果想要远程查看 Kibana 仪表板,请创建允许访问端口 5601 的入站 NSG 规则。If you want to view the Kibana dashboard remotely, create an inbound NSG rule allowing access to port 5601.

创建 Kibana 仪表板Create a Kibana dashboard

在本文中,我们提供了一个示例仪表板,用于查看警报中的趋势和详细信息。For this article, we have provided a sample dashboard for you to view trends and details in your alerts.

  1. 此处下载仪表板文件,在此处下载可视化效果文件,在此处下载已保存的搜索文件。Download the dashboard file here, the visualization file here, and the saved search file here.

  2. 在 Kibana 的“Management”(管理)选项卡下,导航到“Saved Objects”(已保存的对象)并导入所有三个文件。 Under the Management tab of Kibana, navigate to Saved Objects and import all three files. 然后,可从“仪表板”选项卡打开并加载示例仪表板。Then from the Dashboard tab you can open and load the sample dashboard.

还可以创建自己的可视化效果,以及根据感兴趣的指标定制的仪表板。You can also create your own visualizations and dashboards tailored towards metrics of your own interest. 阅读 Kibana 的正式文档,详细了解如何创建 Kibana 可视化效果。Read more about creating Kibana visualizations from Kibana's official documentation.

Kibana 仪表板

可视化 IDS 警报日志Visualize IDS alert logs

示例仪表板提供了 Suricata 警报日志的多种可视化效果:The sample dashboard provides several visualizations of the Suricata alert logs:

  1. 按 GeoIP 列出警报 - 基于地理位置,按来源国家/地区(由 IP 确定)显示警报分布的地图Alerts by GeoIP - a map showing the distribution of alerts by their country/region of origin based on geographic location (determined by IP)

    地理 IP

  2. 排名靠前的 10 条警报 - 最常触发的 10 条警报及其说明的摘要。Top 10 Alerts - a summary of the 10 most frequent triggered alerts and their description. 单击单个警报可以进一步筛选仪表板中的内容,以便只显示与该特定警报相关的信息。Clicking an individual alert filters down the dashboard to the information pertaining to that specific alert.

    图 4

  3. 警报数 - 规则集触发的警报总数Number of Alerts - the total count of alerts triggered by the ruleset

    图 5

  4. 排名靠前的 20 个源/目标 IP/端口 – 显示触发警报次数最多的前 20 个 IP 和端口的饼图。Top 20 Source/Destination IPs/Ports - pie charts showing the top 20 IPs and ports that alerts were triggered on. 可以进一步筛选特定的 IP/端口,以查看触发了多少以及哪些类型的警报。You can filter down on specific IPs/ports to see how many and what kind of alerts are being triggered.

    图 6

  5. 警报摘要 - 汇总每个警报的具体详细信息的表格。Alert Summary - a table summarizing specific details of each individual alert. 可以自定义此表,以显示每条警报的其他想要了解的参数。You can customize this table to show other parameters of interest for each alert.

    图 7

有关创建自定义可视化效果和仪表板的更多文档,请参阅 Kibana 的正式文档For more documentation on creating custom visualizations and dashboards, see Kibana's official documentation.

结论Conclusion

通过将网络观察程序提供的数据包捕获与 Suricata 等开源 IDS 工具相结合,可以针对各种威胁执行网络入侵检测。By combining packet captures provided by Network Watcher and open source IDS tools such as Suricata, you can perform network intrusion detection for a wide range of threats. 使用这些仪表板可以快速探查网络中的趋势和异常,以及挖掘数据来发现恶意用户代理或有漏洞的端口触发警报的根本原因。These dashboards allow you to quickly spot trends and anomalies within your network, as well dig into the data to discover root causes of alerts such as malicious user agents or vulnerable ports. 使用这些提取的数据,可以在如何抵御网络中的任何有害入侵企图方面做出明智的决策,并创建规则来防范网络中将来发生入侵。With this extracted data, you can make informed decisions on how to react to and protect your network from any harmful intrusion attempts, and create rules to prevent future intrusions to your network.

后续步骤Next steps

访问 Use packet capture to do proactive network monitoring with Azure Functions(在 Azure Functions 中使用数据包捕获执行主动网络监视),了解如何根据警报触发数据包捕获Learn how to trigger packet captures based on alerts by visiting Use packet capture to do proactive network monitoring with Azure Functions

访问 Visualize NSG flows logs with Power BI(使用 Power BI 可视化 NSG 流日志),了解如何使用 Power BI 可视化 NSG 流日志Learn how to visualize your NSG flow logs with Power BI by visiting Visualize NSG flows logs with Power BI