DNS Amplification with All-in-One Monitoring Stack

[bayden10]

When starting the BBB- Monitoring Docker Image and monitoring the traffic on the Gateway it is found that there are TCP connects, one to GoogleContent then the CDN at Fastway.com and then about 15sec later DNS Outbound traffic to Google's DNS 8.8.8.8 is flooding the outbound interface on the Gateway with timeouts starting at 3min.

41.212.43.99 - - [26/Jun/2020:17:44:03 -0400] "GET /bigbluebutton/api/getRecordings?checksum=af67b5d5b199434c9ad664a7b3f1da6875394fec&state=processing HTTP/1.1" 404 152 "-" "python-requests/2.23.0"
41.212.43.99 - - [26/Jun/2020:17:44:03 -0400] "GET /bigbluebutton/api/getRecordings?checksum=92adf9d9eaa7cbf0da88cc317010b236205e03c4&state=published HTTP/1.1" 404 152 "-" "python-requests/2.23.0"
41.212.43.99 - - [26/Jun/2020:17:44:03 -0400] "GET /bigbluebutton/api/getRecordings?checksum=e222b85a2289abe5425b97de032af63db0383aa7&state=deleted HTTP/1.1" 404 152 "-" "python-requests/2.23.0"
41.212.43.99 - - [26/Jun/2020:17:44:17 -0400] "GET /bigbluebutton/api/getMeetings?checksum=7a85bd3ec631a16f45c3056ec805a1c181b75fe0 HTTP/1.1" 404 152 "-" "python-requests/2.23.0"
41.212.43.99 - - [26/Jun/2020:17:44:18 -0400] "GET /bigbluebutton/api/getRecordings?checksum=16589a40ba06eca3c7959008661be61d2475533d&state=unpublished HTTP/1.1" 404 152 "-" "python-requests/2.23.0"
41.212.43.99 - - [26/Jun/2020:17:44:18 -0400] "GET /bigbluebutton/api/getRecordings?checksum=af67b5d5b199434c9ad664a7b3f1da6875394fec&state=processing HTTP/1.1" 404 152 "-" "python-requests/2.23.0"
41.212.43.99 - - [26/Jun/2020:17:44:18 -0400] "GET /bigbluebutton/api/getRecordings?checksum=92adf9d9eaa7cbf0da88cc317010b236205e03c4&state=published HTTP/1.1" 404 152 "-" "python-requests/2.23.0"
41.212.43.99 - - [26/Jun/2020:17:44:18 -0400] "GET /bigbluebutton/api/getRecordings?checksum=e222b85a2289abe5425b97de032af63db0383aa7&state=deleted HTTP/1.1" 404 152 "-" "python-requests/2.23.0"
41.212.43.99 - - [26/Jun/2020:17:44:34 -0400] "GET /bigbluebutton/api/getMeetings?checksum=7a85bd3ec631a16f45c3056ec805a1c181b75fe0 HTTP/1.1" 200 171 "-" "python-requests/2.23.0"

Requested Outbound Connection

Outbound Amplification of DNS (exceeds 100 requests)

Adjusting the prometheus.yaml

Default

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 15s

Changed to

global:
  scrape_interval: 120s
  evaluation_interval: 120s
  scrape_timeout: 10s

Bring the Docker image up and the amount of DNS requests have dropped to about 10.

Somewhere in the scraping API is causing this DNS query issue or a possible configuration requirement to allow for a localhost lookup instead of an external lookup? Just my thoughts.

[greenstatic]

I don't think I understand entirely, you are experiencing a flood of requests to various CDN's and outbound DNS traffic?
It is possible that the CDN traffic could be coming from Grafana, you can check this by opening the network tab in the inspector and clicking around Grafana and looking if it is connecting to any CDN. I doubt Prometheus is making any CDN requests and I'm certain the exporter is not making any external requests (besides the BBB server) - you can check the source.

The DNS traffic appears to be coming from the exporters requests to the BBB server. In my setup (default) the container uses the host's DNS resolver. I suspect your host DNS resolver is Google's DNS (8.8.8.8), the DNS requests for BBB should be cached by your resolver, this seems more like a host misconfiguration to me. If you are hosting the all-in-one monitoring stack on the same host as your BBB server than yes, the DNS traffic can be resolved locally as well. For this to work you will need to either have a local DNS server or create an entry on the host which maps the external DNS of your BBB server to localhost (in /etc/hosts).

[bayden10]

The flooding is outbound from the BBB/BBB-Monitor Stack (same host) to Google's DNS not to the CDNs. No local DNS is required and BBB hosts have been operating without issues without having to redirect DNS to localhost.

Can you provide an example of mapping the DNS in /etc/hosts

[greenstatic]

On mobile right now so I can’t SSH into my BBB host but this should help: http://www.tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap9sec95.html On the left you should have 127.0.0.1 and on the right the FQDN of the BBB server (e.g. bbb.example.com).

…

On Sat, Jun 27, 2020, at 3:33 PM, bayden10 wrote: The flooding is outbound from the BBB/BBB-Monitor Stack (same host) to Google's DNS not to the CDNs. No local DNS is required and BBB hosts have been operating without issues without having to redirect DNS to localhost. Can you provide an example of mapping the DNS in `/etc/hosts` — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADPHVUXPLQFPCGWLGAMIW2DRYXYKZANCNFSM4OJ72RJA>.

[bayden10]

Thank you for your support. I had just the host name and not the FQDN on 127.0.0.1
Resolved...

[greenstatic]

No problem, glad you got it working.

…

On Sat, Jun 27, 2020, at 3:57 PM, bayden10 wrote: Thank you for your support. I had just the host name and not the FQDN on 127.0.0.1 Resolved... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADPHVUQMVPC6YYSO6M3EC6TRYX3EJANCNFSM4OJ72RJA>.