Prometheus monitoring and AlertManager.
Prometheus doesn't handle authentication, so we run it behind ‘proxy_with_auth’ that enforce being logged in to a restricted set of domains.
skfe | +--------+ | V prom_proxy_with_auth | V prometheus
Metrics from the skolo are brought in via federation. An instance of Prometheus runs on skia-jumphost and collects metrics and then those metrics are gathered by the Prometheus instance on skia-prom by using the federation. The connection between skia-prom and skia-jumphost is over ssh port forwarding initiated via gcloud compute ssh.
skia-prom | +-> prometheus | | | [ssh reverse port forwarding] | V skia-jumphost | +-> prometheus
Push metrics are gathered in a similar manner, with port forwarding from skia-jumphost to the pushgateway running on skia-prom.
skia-prom | +-> pushgateway ^ | | [ssh port forwarding] | | skia-jumphost