forked from shashyajoshi/professional-services
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Toptalkers (GoogleCloudPlatform#641)
* VPC FlowLogs Top Talkers initial commit * VPC FlowLogs TopTalkers README + minor cleanup Co-authored-by: Gaspar Chilingarov <[email protected]>
- Loading branch information
1 parent
abb3992
commit 1eb0c45
Showing
14 changed files
with
903 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,74 @@ | ||
# VPC Flow Logs Top Talkers | ||
|
||
This solutions allows to generate reports on the traffic generated by subnets or hosts inside the VPC network. | ||
|
||
Generated reports allow finding top sources of the traffic in a specific VPC. | ||
|
||
### The problem | ||
|
||
For monitoring reasons it may be necessary to determine which hosts or networks in the VPC generate most of the traffic towards other internal hosts or the Internet. | ||
This solution allows to generate a report with desired granularity, label Google API IPs, and custom IP ranges and ports. | ||
|
||
## Deployed resources | ||
|
||
* Logs sink and filter (for collecting only Egress traffic) | ||
* BigQuery dataset (for storing traffic logs) | ||
* BigQuery view (report) | ||
* BigQuery functions (aggregation and labelling of the addresses/ports for the view) | ||
|
||
## Requirements | ||
|
||
The following items should be provisioned before installing these reports: | ||
|
||
* An existing project where the [log sink](https://github.com/terraform-google-modules/terraform-google-log-export) will be created. | ||
* An existing project where [BigQuery dataset](https://github.com/terraform-google-modules/terraform-google-log-export/tree/master/modules/bigquery) will be created. | ||
* [VPC flow logs](https://cloud.google.com/vpc/docs/using-flow-logs) must be already enabled in the target subnets where traffic should be monitored. | ||
|
||
## Setup | ||
|
||
### Terraform configuration | ||
|
||
Please rename `terraform.tfvars.example` to `terraform.tfvars` and change the values to point to the correct log sink project ID and VPC project ID. | ||
|
||
### Google API IP address ranges | ||
|
||
This project comes with a pre-populated list of IP addresses for Google APIs. These IPs change from time to time, so there is a script `update-ip-range-labels.sh` which takes care of updating the list in `ip-range-labels.yaml`. You need to run it manually to update the list, it is not a part of Terraform run. | ||
|
||
### Labelling the traffic | ||
|
||
This solution allows adding custom labels to specific IP ranges and ports. Edit the corresponding sections of the `labels.yaml` to add the mapping between the hosts or subnets and text labels. | ||
|
||
To label the traffic on the specific port add it under the `port_labels` key. | ||
|
||
Re-run the `update-ip-range-labels.sh` script after you changed the `labels.yaml` file to regenerate IP ranges definitions. The `labels.yaml` file will **not** be read by terraform. | ||
Instead, the generated file `ip-range-labels.yaml` will be ingested by Terraform. | ||
|
||
### Report settings | ||
|
||
There are several Terraform input variables which change the report output. They do not affect thevolume of the logs exported to Big Query or the tables scanned to generate the report. | ||
|
||
- `enable_split_by_destination` - set to `false`, if you are interested only in having source IPs in the report | ||
- `enable_split_by_protocol` - set to `false`, if you are not interested in split by the protocol | ||
- `enable_ipv4_traffic` - if set to `false` will exclude all IPv4 traffic from the report. | ||
- `ipv4_ranges_to_include` `ipv4_ranges_to_exclude` - list of IPs or subnets to include or exclude. Specify single IP in `8.8.8.8/32` form. | ||
- `ipv4_aggregate_prefix` - if the subnet is not mentioned in the `labels.yaml`, then at which granularity level aggregate traffic together. I.e. when it is `24`, all IPs in `10.239.1.0/24` network will be labelled `netaddr4-10.239.1.0`. If you want to see per-hosts statistics, please use `32` as a value. | ||
- `enable_ipv6_traffic` - same as above, but for `IPv6` traffic | ||
- `ipv6_ranges_to_include` | ||
- `ipv6_ranges_to_exclude` | ||
- `ipv6_aggregate_prefix` - if the subnet is not mentioned in the `labels.yaml` the resulting label will be `netaddr6-IPV6PREFIX` | ||
|
||
### Time ranges | ||
|
||
Please note that the `current_month_*` and `last_month_*` reports will process only the tables from the corresponding time ranges. If you need historical time ranges - please change the view implementation. | ||
|
||
## Usage | ||
|
||
Once installed with the right configuration values, you'll see several views with the names `top_talkers_report_{current|previous}_month_{daily|weekly|monthly}` under the newly created dataset. This dataset will automatically get populated by Cloud Operations with the VPC flow logs that are enabled in the project where the log sink resides. It may take some minutes for the first entries to appear in the dataset. | ||
|
||
Example of the generated output: | ||
|
||
![example\_bigquery\_report](asset/example_report.png) | ||
|
||
## Costs | ||
|
||
If you enable VPC flow logs, they will be sent by default to the `_Default` log sink. You can either disable the `_Default` log sink (not recommended) or create an exclusion rule that skips VPC flow logs. |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,168 @@ | ||
# Please use update-ip-range-labels.sh to update this file. | ||
# https://www.gstatic.com/ipranges/goog.json published: 2021-04-17T10:01:41.646 | ||
# https://www.gstatic.com/ipranges/cloud.json published: 2021-04-17T10:01:41.646 | ||
ipv4_range_labels: | ||
- ["example-custom-ip-range", "10.0.0.0", "10.0.1.255"] | ||
- ["restricted.googleapis.com", "199.36.153.4", "199.36.153.7"] | ||
- ["private.googleapis.com", "199.36.153.8", "199.36.153.11"] | ||
- ["Google IPv4 CIDR", "8.8.4.0", "8.8.4.255"] | ||
- ["Google IPv4 CIDR", "8.8.8.0", "8.8.8.255"] | ||
- ["Google IPv4 CIDR", "8.35.200.0", "8.35.207.255"] | ||
- ["Google IPv4 CIDR", "34.64.0.0", "34.64.63.255"] | ||
- ["Google IPv4 CIDR", "34.96.0.0", "34.96.63.255"] | ||
- ["Google IPv4 CIDR", "34.98.0.0", "34.98.63.255"] | ||
- ["Google IPv4 CIDR", "34.98.136.0", "34.98.143.255"] | ||
- ["Google IPv4 CIDR", "34.98.144.0", "34.98.159.255"] | ||
- ["Google IPv4 CIDR", "34.98.160.0", "34.98.191.255"] | ||
- ["Google IPv4 CIDR", "34.98.192.0", "34.98.255.255"] | ||
- ["Google IPv4 CIDR", "34.99.0.0", "34.99.255.255"] | ||
- ["Google IPv4 CIDR", "34.100.0.0", "34.100.127.255"] | ||
- ["Google IPv4 CIDR", "34.101.0.0", "34.101.15.255"] | ||
- ["Google IPv4 CIDR", "34.101.16.0", "34.101.17.255"] | ||
- ["Google IPv4 CIDR", "34.101.19.0", "34.101.19.255"] | ||
- ["Google IPv4 CIDR", "34.101.28.0", "34.101.31.255"] | ||
- ["Google IPv4 CIDR", "34.101.32.0", "34.101.63.255"] | ||
- ["Google IPv4 CIDR", "34.103.0.0", "34.103.255.255"] | ||
- ["Google IPv4 CIDR", "34.104.0.0", "34.104.15.255"] | ||
- ["Google IPv4 CIDR", "34.104.16.0", "34.104.23.255"] | ||
- ["Google IPv4 CIDR", "34.104.24.0", "34.104.25.255"] | ||
- ["Google IPv4 CIDR", "34.104.26.0", "34.104.26.255"] | ||
- ["Google IPv4 CIDR", "34.104.28.0", "34.104.31.255"] | ||
- ["Google IPv4 CIDR", "34.104.32.0", "34.104.63.255"] | ||
- ["Google IPv4 CIDR", "34.104.114.0", "34.104.115.255"] | ||
- ["Google IPv4 CIDR", "34.104.120.0", "34.104.123.255"] | ||
- ["Google IPv4 CIDR", "34.108.0.0", "34.111.255.255"] | ||
- ["Google IPv4 CIDR", "34.112.0.0", "34.115.255.255"] | ||
- ["Google IPv4 CIDR", "34.116.8.0", "34.116.15.255"] | ||
- ["Google IPv4 CIDR", "34.116.16.0", "34.116.31.255"] | ||
- ["Google IPv4 CIDR", "34.116.32.0", "34.116.63.255"] | ||
- ["Google IPv4 CIDR", "34.118.128.0", "34.118.255.255"] | ||
- ["Google IPv4 CIDR", "34.119.0.0", "34.119.255.255"] | ||
- ["Google IPv4 CIDR", "34.124.50.0", "34.124.51.255"] | ||
- ["Google IPv4 CIDR", "34.124.56.0", "34.124.59.255"] | ||
- ["Google IPv4 CIDR", "34.124.64.0", "34.124.127.255"] | ||
- ["Google IPv4 CIDR", "34.126.0.0", "34.126.63.255"] | ||
- ["Google IPv4 CIDR", "34.126.192.0", "34.126.255.255"] | ||
- ["Google IPv4 CIDR", "34.127.128.0", "34.127.255.255"] | ||
- ["Google IPv4 CIDR", "34.128.0.0", "34.131.255.255"] | ||
- ["Google IPv4 CIDR", "34.132.0.0", "34.133.255.255"] | ||
- ["Google IPv4 CIDR", "34.135.0.0", "34.135.255.255"] | ||
- ["Google IPv4 CIDR", "34.142.128.0", "34.142.255.255"] | ||
- ["Google IPv4 CIDR", "34.143.0.0", "34.143.255.255"] | ||
- ["Google IPv4 CIDR", "34.144.0.0", "34.144.255.255"] | ||
- ["Google IPv4 CIDR", "34.149.0.0", "34.149.255.255"] | ||
- ["Google IPv4 CIDR", "34.152.64.0", "34.152.127.255"] | ||
- ["Google IPv4 CIDR", "34.152.128.0", "34.152.255.255"] | ||
- ["Google IPv4 CIDR", "34.153.0.0", "34.153.255.255"] | ||
- ["Google IPv4 CIDR", "34.154.0.0", "34.155.255.255"] | ||
- ["Google IPv4 CIDR", "34.156.0.0", "34.159.255.255"] | ||
- ["Google IPv4 CIDR", "34.160.0.0", "34.191.255.255"] | ||
- ["Google IPv4 CIDR", "35.187.128.0", "35.187.143.255"] | ||
- ["Google IPv4 CIDR", "35.190.96.0", "35.190.111.255"] | ||
- ["Google IPv4 CIDR", "35.190.240.0", "35.190.255.255"] | ||
- ["Google IPv4 CIDR", "35.191.0.0", "35.191.255.255"] | ||
- ["Google IPv4 CIDR", "35.199.128.0", "35.199.143.255"] | ||
- ["Google IPv4 CIDR", "35.201.32.0", "35.201.39.255"] | ||
- ["Google IPv4 CIDR", "35.201.40.0", "35.201.40.255"] | ||
- ["Google IPv4 CIDR", "35.201.42.0", "35.201.43.255"] | ||
- ["Google IPv4 CIDR", "35.201.44.0", "35.201.47.255"] | ||
- ["Google IPv4 CIDR", "35.201.48.0", "35.201.63.255"] | ||
- ["Google IPv4 CIDR", "35.203.192.0", "35.203.207.255"] | ||
- ["Google IPv4 CIDR", "35.203.208.0", "35.203.209.255"] | ||
- ["Google IPv4 CIDR", "35.203.220.0", "35.203.223.255"] | ||
- ["Google IPv4 CIDR", "35.203.224.0", "35.203.231.255"] | ||
- ["Google IPv4 CIDR", "35.203.240.0", "35.203.255.255"] | ||
- ["Google IPv4 CIDR", "35.206.0.0", "35.206.31.255"] | ||
- ["Google IPv4 CIDR", "35.218.0.0", "35.218.255.255"] | ||
- ["Google IPv4 CIDR", "35.219.192.0", "35.219.255.255"] | ||
- ["Google IPv4 CIDR", "35.220.28.0", "35.220.29.255"] | ||
- ["Google IPv4 CIDR", "35.220.30.0", "35.220.30.255"] | ||
- ["Google IPv4 CIDR", "35.229.0.0", "35.229.15.255"] | ||
- ["Google IPv4 CIDR", "35.230.192.0", "35.230.223.255"] | ||
- ["Google IPv4 CIDR", "35.230.224.0", "35.230.239.255"] | ||
- ["Google IPv4 CIDR", "35.235.128.0", "35.235.191.255"] | ||
- ["Google IPv4 CIDR", "35.235.192.0", "35.235.207.255"] | ||
- ["Google IPv4 CIDR", "35.235.208.0", "35.235.215.255"] | ||
- ["Google IPv4 CIDR", "35.235.224.0", "35.235.255.255"] | ||
- ["Google IPv4 CIDR", "35.242.28.0", "35.242.29.255"] | ||
- ["Google IPv4 CIDR", "35.242.30.0", "35.242.30.255"] | ||
- ["Google IPv4 CIDR", "35.243.16.0", "35.243.31.255"] | ||
- ["Google IPv4 CIDR", "35.243.48.0", "35.243.55.255"] | ||
- ["Google IPv4 CIDR", "64.15.112.0", "64.15.127.255"] | ||
- ["Google IPv4 CIDR", "64.233.160.0", "64.233.191.255"] | ||
- ["Google IPv4 CIDR", "66.102.0.0", "66.102.15.255"] | ||
- ["Google IPv4 CIDR", "66.249.64.0", "66.249.95.255"] | ||
- ["Google IPv4 CIDR", "70.32.128.0", "70.32.159.255"] | ||
- ["Google IPv4 CIDR", "72.14.192.0", "72.14.255.255"] | ||
- ["Google IPv4 CIDR", "74.114.24.0", "74.114.31.255"] | ||
- ["Google IPv4 CIDR", "74.125.0.0", "74.125.255.255"] | ||
- ["Google IPv4 CIDR", "104.154.0.0", "104.154.15.255"] | ||
- ["Google IPv4 CIDR", "104.154.112.0", "104.154.112.255"] | ||
- ["Google IPv4 CIDR", "104.154.122.0", "104.154.123.255"] | ||
- ["Google IPv4 CIDR", "104.154.124.0", "104.154.127.255"] | ||
- ["Google IPv4 CIDR", "104.155.240.0", "104.155.255.255"] | ||
- ["Google IPv4 CIDR", "104.196.64.0", "104.196.64.255"] | ||
- ["Google IPv4 CIDR", "104.196.72.0", "104.196.79.255"] | ||
- ["Google IPv4 CIDR", "104.196.80.0", "104.196.95.255"] | ||
- ["Google IPv4 CIDR", "104.199.64.0", "104.199.65.255"] | ||
- ["Google IPv4 CIDR", "104.199.240.0", "104.199.241.255"] | ||
- ["Google IPv4 CIDR", "104.237.160.0", "104.237.191.255"] | ||
- ["Google IPv4 CIDR", "107.178.192.0", "107.178.207.255"] | ||
- ["Google IPv4 CIDR", "107.178.224.0", "107.178.239.255"] | ||
- ["Google IPv4 CIDR", "108.170.192.0", "108.170.255.255"] | ||
- ["Google IPv4 CIDR", "108.177.0.0", "108.177.127.255"] | ||
- ["Google IPv4 CIDR", "130.211.0.0", "130.211.3.255"] | ||
- ["Google IPv4 CIDR", "136.112.0.0", "136.127.255.255"] | ||
- ["Google IPv4 CIDR", "142.250.0.0", "142.251.255.255"] | ||
- ["Google IPv4 CIDR", "146.148.0.0", "146.148.1.255"] | ||
- ["Google IPv4 CIDR", "172.110.32.0", "172.110.39.255"] | ||
- ["Google IPv4 CIDR", "172.217.0.0", "172.217.255.255"] | ||
- ["Google IPv4 CIDR", "172.253.0.0", "172.253.255.255"] | ||
- ["Google IPv4 CIDR", "173.194.0.0", "173.194.255.255"] | ||
- ["Google IPv4 CIDR", "192.178.0.0", "192.179.255.255"] | ||
- ["Google IPv4 CIDR", "193.186.4.0", "193.186.4.255"] | ||
- ["Google IPv4 CIDR", "199.36.154.0", "199.36.155.255"] | ||
- ["Google IPv4 CIDR", "199.36.156.0", "199.36.156.255"] | ||
- ["Google IPv4 CIDR", "199.192.112.0", "199.192.113.255"] | ||
- ["Google IPv4 CIDR", "199.192.114.0", "199.192.114.255"] | ||
- ["Google IPv4 CIDR", "199.223.237.0", "199.223.237.255"] | ||
- ["Google IPv4 CIDR", "199.223.238.0", "199.223.239.255"] | ||
- ["Google IPv4 CIDR", "207.223.160.0", "207.223.175.255"] | ||
- ["Google IPv4 CIDR", "208.65.152.0", "208.65.155.255"] | ||
- ["Google IPv4 CIDR", "208.68.108.0", "208.68.111.255"] | ||
- ["Google IPv4 CIDR", "208.81.188.0", "208.81.191.255"] | ||
- ["Google IPv4 CIDR", "208.117.224.0", "208.117.255.255"] | ||
- ["Google IPv4 CIDR", "209.85.128.0", "209.85.255.255"] | ||
- ["Google IPv4 CIDR", "216.58.192.0", "216.58.223.255"] | ||
- ["Google IPv4 CIDR", "216.73.80.0", "216.73.95.255"] | ||
- ["Google IPv4 CIDR", "216.239.32.0", "216.239.63.255"] | ||
ipv6_range_labels: | ||
- ["Google IPv4 CIDR", "2001:4860::", "2001:4860:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2404:6800::", "2404:6800:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2404:f340::", "2404:f340:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1900::", "2600:1900:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:1::", "2600:1901:1:fff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:2::", "2600:1901:3:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:4::", "2600:1901:7:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:8::", "2600:1901:f:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:10::", "2600:1901:1f:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:20::", "2600:1901:3f:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:40::", "2600:1901:7f:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:80::", "2600:1901:ff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:100::", "2600:1901:1ff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:200::", "2600:1901:3ff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:400::", "2600:1901:7ff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:800::", "2600:1901:fff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:1000::", "2600:1901:1fff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:2000::", "2600:1901:3fff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:4000::", "2600:1901:7fff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1901:8000::", "2600:1901:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1902::", "2600:1903:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1904::", "2600:1907:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2600:1908::", "2600:190f:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2607:f8b0::", "2607:f8b0:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2620:11a:a000::", "2620:11a:a0ff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2620:120:e000::", "2620:120:e0ff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2800:3f0::", "2800:3f0:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2a00:1450::", "2a00:1450:ffff:ffff:ffff:ffff:ffff:ffff"] | ||
- ["Google IPv4 CIDR", "2c0f:fb50::", "2c0f:fb50:ffff:ffff:ffff:ffff:ffff:ffff"] |
Oops, something went wrong.