Prometheus monitoring and AlertManager.
Prometheus and AlertManager don't handle authentication, so we run them behind instances of ‘proxy_with_auth’ that enforce being logged in to a restricted set of domains.
In addition, there is webhook_email_proxy, which takes webhook requests from alertmanager and turns them into emails using the gmail api.
skfe | +--------+ | | | V | prom_proxy_with_auth | | | V | prometheus V alert_proxy_with_auth | V alertmanager | V webhook_email_proxy
Metrics from the skolo are brought in via federation. An instance of Prometheus runs on skia-jumphost and collects metrics and then those metrics are gathered by the Prometheus instance on skia-prom by using the federation. The connection between skia-prom and skia-jumphost is over ssh port forwarding initiated via gcloud compute ssh.
skia-prom | +-> prometheus | | | [ssh reverse port forwarding] | V skia-jumphost | +-> prometheus
Push metrics are gathered in a similar manner, with port forwarding from skia-jumphost to the pushgateway running on skia-prom.
skia-prom | +-> pushgateway ^ | | [ssh port forwarding] | | skia-jumphost