| General Production Manual |
| ========================= |
| |
| This file documents things that don't belong to a specific service. |
| |
| Other Resources (For Googlers only) |
| ----------------------------------- |
| |
| - [https://goto.google.com/skia-infra-gardener] |
| - [https://goto.google.com/skolo-maintenance] |
| - [https://goto.google.com/skolo-playbook] |
| |
| Alerts |
| ====== |
| |
| Items below here should include target links from alerts. |
| |
| DiskSpaceLow |
| ------------ |
| This means a given disk on one of our machines has a low disk. Running out of disk space causes |
| problems, so we try to keep a healthy buffer (which varies depending on the total disk size). |
| For machines running Swarming, this can cause issues when trying to download a task from Isolate, |
| which has been a problem before (). |
| |
| To fix, [connect to the machine](https://skia.org/dev/testing/swarmingbots#connecting-to-swarming-bots), |
| and use `df -h` or a similar command to identify which disk(s) are low. `du -hd 2` can be a useful |
| tool for identifying which folders are taking up a lot of space. |
| - If a /root disk is full, try cleaning out the APT cache `sudo apt-get clean` |
| - If a /var disk is full, try deleting /var/logs/* and restarting the machine. |
| - If a /tmp disk is full, it usually cleans itself up on a reboot. |
| - On a swarming machine, if /b (/mnt/pd0) is full, there are few things to check: |
| - `/b/s/*_cache` folders have gotten very large. If so, stop swarming, delete the folders, and |
| reboot. |
| - /b/docker (the docker cache) can take up 100+ GB. Clean it with `sudo docker system prune -fa`. |
| |
| If many machines are experiencing this, you may want to use the |
| [run_on_swarming_bots](../scripts/run_on_swarming_bots) script to fix them all at once. |
| |
| Key metrics: collectd_df_df_complex |
