[graphite] Catch notifyInUse edge case

Change-Id: If43cca74ed84114839dc7e9e125ae707fb22f31c
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/1015498
Reviewed-by: Michael Ludwig <michaelludwig@google.com>
Commit-Queue: Thomas Smith <thomsmit@google.com>