| commit | 0fef7d6691c14788697b32c84241d2ad1a90d535 | [log] [tgz] |
|---|---|---|
| author | Leandro Lovisolo <lovisolo@google.com> | Wed Jun 22 23:46:29 2022 +0000 |
| committer | SkCQ <skcq-be@skia-corp.google.com.iam.gserviceaccount.com> | Thu Jun 23 00:20:48 2022 +0000 |
| tree | 1c846e0541bdae5ddd567f96536a6e599280cd87 | |
| parent | 4516a930efcdc6a9d7d59d04ae8f11bb6b1ccf72 [diff] |
[ansible] Windows: sleep for 10 seconds before launching TMM.
skia-e-win-367 and skia-e-368 fail to launch test_machine_monitor.exe with errors such as:
.\test_machine_monitor.exe : F0622 12:21:48.611977 6756 with.go:241] Failed to initialize: oauth2: cannot fetch
token: Post "https://oauth2.googleapis.com/token": dial tcp: lookup oauth2.googleapis.com: no such host
This indicates the network isn't ready before test_machine_monitor.exe is launched. Joe's hypothesis is that because these machines are new and snappy, TMM launches before the network has a chance to become ready. Adding a slight delay before launching TMM seems to fix it.
Note that the above error message isn't accessible from the GCP Logs Viewer. The trick to get log messages in this situation is to pipe both stdout and stderr into a text file from test_machine_monitor.ps1, e.g. "test_machine_monitor.exe --foo --bar ... 2>&1 >> logs.txt".
An alternative I considered is to use the "Event"[1] trigger from the win_scheduled_task[2] Ansible module to launch TMM when the network becomes online, rather than on logon as we do today. However this approach seems way more complicated[3]. Furthermore, it does not take into account the unlikely scenario of the network repeatedly going offline then back online, which would trigger a TMM launch every time the network goes online.
Another alternative would be to add some retry logic to TMM to prevent it from panicking if the network is offline. IMO the "sleep" approach wins due to its simplicity.
[1] https://docs.microsoft.com/en-us/windows/win32/taskschd/eventtrigger
[2] https://docs.ansible.com/ansible/latest/collections/community/windows/win_scheduled_task_module.html
[3] https://www.groovypost.com/howto/automatically-run-script-on-internet-connect-network-connection-drop
Change-Id: Ie8ecebaf00586df2058a04a9819fcdd5d0fff2da
Reviewed-on: https://skia-review.googlesource.com/c/buildbot/+/552396
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
Commit-Queue: Joe Gregorio <jcgregorio@google.com>
Auto-Submit: Leandro Lovisolo <lovisolo@google.com>
diff --git a/skolo/ansible/switchboard/roles/install_test_machine_monitor/templates/test_machine_monitor.ps1 b/skolo/ansible/switchboard/roles/install_test_machine_monitor/templates/test_machine_monitor.ps1 index fc51bce..92964ec 100644 --- a/skolo/ansible/switchboard/roles/install_test_machine_monitor/templates/test_machine_monitor.ps1 +++ b/skolo/ansible/switchboard/roles/install_test_machine_monitor/templates/test_machine_monitor.ps1
@@ -33,6 +33,10 @@ $Env:SWARMING_EXTERNAL_BOT_SETUP = 1 {% endif %} +# Give network a chance to become ready. test_machine_monitor.exe fails to launch if the network is +# still offline. The sleep interval was chosen arbitrarily. +sleep 10 + # Launch test_machine_monitor. .\test_machine_monitor.exe ` --config=prod.json `