[infra] Update Linux machines to use Clang 15.0.1

The previous version was Clang 10. This only impacts our GN builds
(Bazel toolchain will need an update as well as a bit of
infrastructure to upload the compiled toolchain).

We need to compile Clang from source for two reasons:
 - https://github.com/llvm/llvm-project/releases/tag/llvmorg-15.0.1
   does not have pre-built amd64 Linux binaries (v14 was
   the last one, it seems).
 - We need to compile libcxx and libcxxabi for MSAN and TSAN
   to work properly [1] [2]

This change compiles Clang using Docker for two reasons:
 - Docker is more reproducible, which is easier to use and
   lets us have better SLSA compliance.
 - Our current Linux compile and test machines are running
   Debian 10, which has libc version 2.28. If a dev (e.g. me)
   compiled Clang on a newer version of Debian, it would
   require a newer version of libc to be on the build machines,
   which is non-trivial to accommodate.

Getting the Docker build to work was a bit tricky, but the
official Clang project has a Dockerfile [3] that I cribbed from.

Building libcxx and libcxxabi in a non-deprecated way required
small, subtle changes [4][5]

We no longer add tools/xsan.supp to the include path because
Clang warns about "unrecognized pragmas", which is what it thinks
the suppression comments are. I think this originally had been
added to force re-compilation if the suppression changed (otherwise
ninja has no way to know something changed). In Bazel, this would
not be an issue (as toolchain contents are accounted for when
deciding to rebuild). For our GN builds, the risk is tolerable
given how infrequently we update that suppression file.

We no longer compile the clang binaries in MinSizeRel mode.
I wasn't able to tell via git history why that was chosen in the
first place, but we have plenty of bandwidth and disk space on
our CI machines; time (i.e. execution time) is the precious
resource.

[1] https://github.com/google/sanitizers/wiki/MemorySanitizer#using-instrumented-libraries
[2] https://github.com/llvm/llvm-project/blob/76fd4bf675b5ceeeca0e4e15cf15d89c7acf4947/llvm/utils/docker/debian10/Dockerfile
[3] https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual#non-instrumented-code
[4] https://libcxx.llvm.org/BuildingLibcxx.html
[5] https://stackoverflow.com/a/73827100/1447621

Change-Id: I5c3c6c9bc61fa3c335647a2ff5b0abe259b0bbbb
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/583776
Reviewed-by: Ravi Mistry <rmistry@google.com>
Reviewed-by: Brian Osman <brianosman@google.com>
diff --git a/gn/skia/BUILD.gn b/gn/skia/BUILD.gn
index 52a7e60..b6f6281 100644
--- a/gn/skia/BUILD.gn
+++ b/gn/skia/BUILD.gn
@@ -68,6 +68,12 @@
     cflags += [ "-Wno-attributes" ]
   }
 
+  if (is_clang && !is_win) {
+    # In Clang 14, this default was changed. We turn this off to (hopefully) make our
+    # GMs more consistent in the transition.
+    cflags += [ "-ffp-contract=off" ]
+  }
+
   if (is_fuchsia && using_fuchsia_sdk) {
     ldflags += [
       "-v",
@@ -312,8 +318,6 @@
 
     if (is_win) {
       cflags += [
-        "/FI$_suppressions",
-
         # On Release builds, we get strange warnings about string literals.
         "/GF-",
       ]
@@ -321,10 +325,7 @@
       assert(clang_win != "")
       libs += [ "$clang_win/lib/clang/$clang_win_version/lib/windows/clang_rt.asan-x86_64.lib" ]
     } else {
-      cflags += [
-        "-include$_suppressions",
-        "-fno-omit-frame-pointer",
-      ]
+      cflags += [ "-fno-omit-frame-pointer" ]
 
       ldflags += [ "-fsanitize=$sanitizers" ]
     }
@@ -618,7 +619,13 @@
     ]
   } else {
     if (skia_enable_optimize_size) {
-      cflags = [ "-Oz" ]
+      cflags = [
+        "-Oz",
+
+        # We currently use bloaty 1.0 on the CI. Newer versions of Bloaty do not
+        # seem to handle newer DWARF data as well.
+        "-gdwarf-3",
+      ]
     } else {
       cflags = [ "-O3" ]
     }
diff --git a/infra/bots/assets/clang_linux/Dockerfile b/infra/bots/assets/clang_linux/Dockerfile
new file mode 100644
index 0000000..8c74f4b
--- /dev/null
+++ b/infra/bots/assets/clang_linux/Dockerfile
@@ -0,0 +1,89 @@
+# This is based off the official LLVM docker container
+# https://github.com/llvm/llvm-project/blob/76fd4bf675b5ceeeca0e4e15cf15d89c7acf4947/llvm/utils/docker/debian10/Dockerfile
+#
+# This was launcher.gcr.io/google/debian10:latest on Sep 22 2022
+# Found by running
+# docker pull launcher.gcr.io/google/debian10:latest && docker images --digests | grep debian10
+FROM launcher.gcr.io/google/debian10@sha256:3242ff21417c7722482c2085f86f28ed4f76cde00bf880f15fc1795975bc2a81
+
+# Install build dependencies of llvm.
+# First, Update the apt's source list and include the sources of the packages.
+RUN grep deb /etc/apt/sources.list | \
+    sed 's/^deb/deb-src /g' >> /etc/apt/sources.list
+# Install compiler, python, etc. We need clang and lld because otherwise we have issues compiling
+# compiler-rt (it fails using the built-in ld).
+#
+# The versions were added after seeing what was available when this image was created on Sep 22 2022
+# Specifying the versions makes this Docker container comply with SLSA level 1.
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends  \
+           ca-certificates=20200601~deb10u2 gnupg=2.2.12-1+deb10u2 \
+           build-essential=12.6 cmake=3.13.4-1 make=4.2.1-1.2 python3=3.7.3-1 \
+           zlib1g=1:1.2.11.dfsg-1+deb10u2 wget=1.20.1-1.1 unzip=6.0-23+deb10u3 \
+           git=1:2.20.1-2+deb10u3 clang=1:7.0-47 lld=1:7.0-47 && \
+    rm -rf /var/lib/apt/lists/*
+# Install a newer ninja release. It seems the older version in the debian repos
+# randomly crashes when compiling llvm.
+RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \
+    echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \
+        | sha256sum -c  && \
+    unzip ninja-linux.zip -d /usr/local/bin && \
+    rm ninja-linux.zip
+
+ENV TARGET_DIR=/tmp/clang_output
+ENV CLANG_RELEASE=llvmorg-15.0.1
+
+RUN mkdir -p /tmp/clang && cd /tmp/clang && \
+    git clone --depth 1 -b ${CLANG_RELEASE} https://llvm.googlesource.com/llvm-project
+
+WORKDIR /tmp/clang/llvm-project
+
+ENV CC=/usr/bin/clang
+ENV CXX=/usr/bin/clang++
+
+# https://libcxx.llvm.org/BuildingLibcxx.html#bootstrapping-build
+# This will build clang first and then use that new clang to build the runtimes
+RUN mkdir ${TARGET_DIR} out && \
+    cmake -G Ninja -S llvm -B out \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DCMAKE_INSTALL_PREFIX=${TARGET_DIR} \
+    -DLLVM_ENABLE_PROJECTS="clang;lld;clang-tools-extra" \
+    -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi" \
+    -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON \
+    -DLLVM_USE_LINKER=lld \
+    -DLLVM_ENABLE_TERMINFO=OFF
+
+# TODO(kjlubick) Previously, we would build clang-tools-extra. Do we still need that?
+
+RUN ninja -C out install
+RUN cp out/bin/llvm-symbolizer out/bin/llvm-profdata out/bin/llvm-cov ${TARGET_DIR}/bin
+RUN cp `c++ -print-file-name=libstdc++.so.6` ${TARGET_DIR}/lib
+
+# Use the newly compiled clang to build TSAN and MSAN libraries.
+ENV CC=${TARGET_DIR}/bin/clang
+ENV CXX=${TARGET_DIR}/bin/clang++
+
+# It is very important to start the build from the runtimes subfolder and not the llvm subfolder
+# like we did above when following the bootstrapping-build instructions.
+# https://stackoverflow.com/a/73827100/1447621
+RUN mkdir tsan_out && \
+    cmake -G Ninja -S runtimes -B tsan_out \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \
+    -DLLVM_USE_SANITIZER=Thread
+
+RUN ninja -C tsan_out cxx cxxabi
+RUN cp -r tsan_out/lib ${TARGET_DIR}/tsan
+
+# We would be following the instructions from
+# https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo
+# but those are currently out of date (https://github.com/google/sanitizers/issues/1574)
+RUN mkdir msan_out && \
+    cmake -GNinja -S runtimes -B msan_out \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \
+    -DLLVM_USE_SANITIZER=MemoryWithOrigins
+
+RUN ninja -C msan_out cxx cxxabi
+
+RUN cp -r msan_out/lib ${TARGET_DIR}/msan
diff --git a/infra/bots/assets/clang_linux/VERSION b/infra/bots/assets/clang_linux/VERSION
index 2edeafb..368f89c 100644
--- a/infra/bots/assets/clang_linux/VERSION
+++ b/infra/bots/assets/clang_linux/VERSION
@@ -1 +1 @@
-20
\ No newline at end of file
+28
\ No newline at end of file
diff --git a/infra/bots/assets/clang_linux/create.py b/infra/bots/assets/clang_linux/create.py
index cdcbd31..a4668a3 100755
--- a/infra/bots/assets/clang_linux/create.py
+++ b/infra/bots/assets/clang_linux/create.py
@@ -12,53 +12,24 @@
 import argparse
 import os
 import subprocess
-import tempfile
-
-
-BRANCH = "release/10.x"
 
 
 def create_asset(target_dir):
-  # CMake will sometimes barf if we pass it a relative path.
-  target_dir = os.path.abspath(target_dir)
+  # Create a local docker image tagged "clang_linux_asset" with a compiled clang for amd64 Linux.
+  # This will take a while. Output is in the image under /tmp/clang_output
+  args = ['docker', 'build', '-t', 'clang_linux_asset', './infra/bots/assets/clang_linux']
+  subprocess.run(args, check=True, encoding='utf8')
 
-  # Build Clang, lld, compiler-rt (sanitizer support) and libc++.
-  os.chdir(tempfile.mkdtemp())
-  subprocess.check_call(["git", "clone", "--depth", "1", "-b", BRANCH,
-                         "https://llvm.googlesource.com/llvm-project"])
-  os.chdir("llvm-project")
-  os.mkdir("out")
-  os.chdir("out")
-  subprocess.check_call(["cmake", "../llvm", "-G", "Ninja",
-                         "-DCMAKE_BUILD_TYPE=MinSizeRel",
-                         "-DCMAKE_INSTALL_PREFIX=" + target_dir,
-                         "-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra;" +
-                             "compiler-rt;libcxx;libcxxabi;lld",
-                         "-DLLVM_INSTALL_TOOLCHAIN_ONLY=ON",
-                         "-DLLVM_ENABLE_TERMINFO=OFF"])
-  subprocess.check_call(["ninja", "install"])
-
-  # Copy a couple extra files we need.
-  subprocess.check_call(["cp", "bin/llvm-symbolizer", target_dir + "/bin"])
-  subprocess.check_call(["cp", "bin/llvm-profdata", target_dir + "/bin"])
-  subprocess.check_call(["cp", "bin/llvm-cov", target_dir + "/bin"])
-  libstdcpp = subprocess.check_output([
-      "c++", "-print-file-name=libstdc++.so.6"]).decode('utf-8')
-  subprocess.check_call(["cp", libstdcpp.strip(), target_dir + "/lib"])
-
-  # Finally, build libc++ for TSAN and MSAN bots using the Clang we just built.
-  for (short,full) in [('tsan','Thread'), ('msan','MemoryWithOrigins')]:
-    os.mkdir("../{}_out".format(short))
-    os.chdir("../{}_out".format(short))
-    subprocess.check_call(
-        ["cmake", "../llvm", "-G", "Ninja",
-         "-DCMAKE_BUILD_TYPE=MinSizeRel",
-         "-DCMAKE_C_COMPILER="   + target_dir + "/bin/clang",
-         "-DCMAKE_CXX_COMPILER=" + target_dir + "/bin/clang++",
-         "-DLLVM_ENABLE_PROJECTS=libcxx;libcxxabi",
-         "-DLLVM_USE_SANITIZER={}".format(full)])
-    subprocess.check_call(["ninja", "cxx"])
-    subprocess.check_call(["cp", "-r", "lib",  target_dir + "/" + short])
+  # Copy the assets out of the container by mounting target_dir
+  print('Copying clang from Docker container into CIPD folder')
+  os.makedirs(target_dir, exist_ok=True)
+  args = ['docker', 'run', '--mount', 'type=bind,source=%s,target=/OUT' % target_dir,
+          'clang_linux_asset', '/bin/sh', '-c',
+          # After copying, we need to make the files write-able by all users.
+          # Docker makes them owned by root by default, and without everyone being
+          # able to write (e.g. delete) them, this can cause issues.
+          'cp -R /tmp/clang_output/* /OUT && chmod -R a+w /OUT']
+  subprocess.run(args, check=True, encoding='utf8')
 
 
 def main():
diff --git a/infra/bots/gen_tasks_logic/gen_tasks_logic.go b/infra/bots/gen_tasks_logic/gen_tasks_logic.go
index b1d6ccf..04e4996 100644
--- a/infra/bots/gen_tasks_logic/gen_tasks_logic.go
+++ b/infra/bots/gen_tasks_logic/gen_tasks_logic.go
@@ -1735,9 +1735,8 @@
 				// Occasional false positives may crop up in the standard library without this.
 				b.envPrefixes("LD_LIBRARY_PATH", "clang_linux/tsan")
 			} else {
-				// This isn't strictly required, but we usually get better sanitizer
-				// diagnostics from libc++ than the default OS-provided libstdc++.
-				b.envPrefixes("LD_LIBRARY_PATH", "clang_linux/lib")
+				// The machines we run on may not have libstdc++ installed.
+				b.envPrefixes("LD_LIBRARY_PATH", "clang_linux/lib/x86_64-unknown-linux-gnu")
 			}
 		}
 	})
diff --git a/infra/bots/recipe_modules/flavor/default.py b/infra/bots/recipe_modules/flavor/default.py
index 6b1c7a8..39601f7 100644
--- a/infra/bots/recipe_modules/flavor/default.py
+++ b/infra/bots/recipe_modules/flavor/default.py
@@ -124,7 +124,7 @@
     ld_library_path = []
 
     workdir = self.m.vars.workdir
-    clang_linux = str(workdir.join('clang_linux'))
+    clang_linux = workdir.join('clang_linux')
     extra_tokens = self.m.vars.extra_tokens
 
     if self.m.vars.is_linux:
@@ -156,18 +156,18 @@
 
     # Find the MSAN/TSAN-built libc++.
     if 'MSAN' in extra_tokens:
-      ld_library_path.append(clang_linux + '/msan')
+      ld_library_path.append(clang_linux.join('msan'))
     elif 'TSAN' in extra_tokens:
-      ld_library_path.append(clang_linux + '/tsan')
+      ld_library_path.append(clang_linux.join('tsan'))
 
     if any('SAN' in t for t in extra_tokens):
       # Sanitized binaries may want to run clang_linux/bin/llvm-symbolizer.
-      path.append(clang_linux + '/bin')
+      path.append(clang_linux.join('bin'))
       # We find that testing sanitizer builds with libc++ uncovers more issues
       # than with the system-provided C++ standard library, which is usually
       # libstdc++. libc++ proactively hooks into sanitizers to help their
       # analyses. We ship a copy of libc++ with our Linux toolchain in /lib.
-      ld_library_path.append(clang_linux + '/lib')
+      ld_library_path.append(clang_linux.join('lib', 'x86_64-unknown-linux-gnu'))
     elif self.m.vars.is_linux:
       cmd = ['catchsegv'] + cmd
 
diff --git a/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Debug-All-MSAN.json b/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Debug-All-MSAN.json
index 71378e1..b72b99b 100644
--- a/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Debug-All-MSAN.json
+++ b/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Debug-All-MSAN.json
@@ -200,7 +200,7 @@
     ],
     "env": {
       "CHROME_HEADLESS": "1",
-      "LD_LIBRARY_PATH": "[START_DIR]/clang_linux/msan:[START_DIR]/clang_linux/lib",
+      "LD_LIBRARY_PATH": "[START_DIR]/clang_linux/msan:[START_DIR]/clang_linux/lib/x86_64-unknown-linux-gnu",
       "PATH": "<PATH>:RECIPE_REPO[depot_tools]:[START_DIR]/clang_linux/bin"
     },
     "name": "nanobench"
diff --git a/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-ASAN.json b/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-ASAN.json
index ecde5be..746608c 100644
--- a/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-ASAN.json
+++ b/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-ASAN.json
@@ -201,7 +201,7 @@
     "env": {
       "ASAN_OPTIONS": "symbolize=1 detect_leaks=1",
       "CHROME_HEADLESS": "1",
-      "LD_LIBRARY_PATH": "[START_DIR]/clang_linux/lib",
+      "LD_LIBRARY_PATH": "[START_DIR]/clang_linux/lib/x86_64-unknown-linux-gnu",
       "LSAN_OPTIONS": "symbolize=1 print_suppressions=1",
       "PATH": "<PATH>:RECIPE_REPO[depot_tools]:[START_DIR]/clang_linux/bin",
       "UBSAN_OPTIONS": "symbolize=1 print_stacktrace=1"
diff --git a/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Win2019-Clang-GCE-CPU-AVX2-x86_64-Debug-All-ASAN.json b/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Win2019-Clang-GCE-CPU-AVX2-x86_64-Debug-All-ASAN.json
index bfcdd33..aad522e 100644
--- a/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Win2019-Clang-GCE-CPU-AVX2-x86_64-Debug-All-ASAN.json
+++ b/infra/bots/recipe_modules/flavor/examples/full.expected/Perf-Win2019-Clang-GCE-CPU-AVX2-x86_64-Debug-All-ASAN.json
@@ -206,9 +206,9 @@
     "env": {
       "ASAN_OPTIONS": "symbolize=1",
       "CHROME_HEADLESS": "1",
-      "LD_LIBRARY_PATH": "[START_DIR]\\clang_linux/lib",
+      "LD_LIBRARY_PATH": "[START_DIR]\\clang_linux\\lib\\x86_64-unknown-linux-gnu",
       "LSAN_OPTIONS": "symbolize=1 print_suppressions=1",
-      "PATH": "<PATH>;RECIPE_REPO[depot_tools];[START_DIR]\\clang_linux/bin",
+      "PATH": "<PATH>;RECIPE_REPO[depot_tools];[START_DIR]\\clang_linux\\bin",
       "UBSAN_OPTIONS": "symbolize=1 print_stacktrace=1"
     },
     "name": "nanobench"
diff --git a/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-TSAN.json b/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-TSAN.json
index 858960c..1eeac5e 100644
--- a/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-TSAN.json
+++ b/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-GCE-CPU-AVX2-x86_64-Release-All-TSAN.json
@@ -200,7 +200,7 @@
     ],
     "env": {
       "CHROME_HEADLESS": "1",
-      "LD_LIBRARY_PATH": "[START_DIR]/clang_linux/tsan:[START_DIR]/clang_linux/lib",
+      "LD_LIBRARY_PATH": "[START_DIR]/clang_linux/tsan:[START_DIR]/clang_linux/lib/x86_64-unknown-linux-gnu",
       "PATH": "<PATH>:RECIPE_REPO[depot_tools]:[START_DIR]/clang_linux/bin",
       "TSAN_OPTIONS": "report_signal_unsafe=0"
     },
diff --git a/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-NUC7i5BNK-GPU-IntelIris640-x86_64-Debug-All-ASAN_Vulkan.json b/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-NUC7i5BNK-GPU-IntelIris640-x86_64-Debug-All-ASAN_Vulkan.json
index 8842e12..1de9b84 100644
--- a/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-NUC7i5BNK-GPU-IntelIris640-x86_64-Debug-All-ASAN_Vulkan.json
+++ b/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Debian10-Clang-NUC7i5BNK-GPU-IntelIris640-x86_64-Debug-All-ASAN_Vulkan.json
@@ -201,7 +201,7 @@
     "env": {
       "ASAN_OPTIONS": "symbolize=1 detect_leaks=1 fast_unwind_on_malloc=0",
       "CHROME_HEADLESS": "1",
-      "LD_LIBRARY_PATH": "[START_DIR]/mesa_intel_driver_linux:[START_DIR]/linux_vulkan_sdk/lib:[START_DIR]/clang_linux/lib",
+      "LD_LIBRARY_PATH": "[START_DIR]/mesa_intel_driver_linux:[START_DIR]/linux_vulkan_sdk/lib:[START_DIR]/clang_linux/lib/x86_64-unknown-linux-gnu",
       "LIBGL_DRIVERS_PATH": "[START_DIR]/mesa_intel_driver_linux",
       "LSAN_OPTIONS": "symbolize=1 print_suppressions=1 fast_unwind_on_malloc=0",
       "PATH": "<PATH>:RECIPE_REPO[depot_tools]:[START_DIR]/linux_vulkan_sdk/bin:[START_DIR]/clang_linux/bin",
diff --git a/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Mac10.13-Clang-MacBookPro11.5-CPU-AVX2-x86_64-Debug-All-ASAN.json b/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Mac10.13-Clang-MacBookPro11.5-CPU-AVX2-x86_64-Debug-All-ASAN.json
index 1bdc891..d145ea4 100644
--- a/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Mac10.13-Clang-MacBookPro11.5-CPU-AVX2-x86_64-Debug-All-ASAN.json
+++ b/infra/bots/recipe_modules/flavor/examples/full.expected/Test-Mac10.13-Clang-MacBookPro11.5-CPU-AVX2-x86_64-Debug-All-ASAN.json
@@ -201,7 +201,7 @@
     "env": {
       "ASAN_OPTIONS": "symbolize=1",
       "CHROME_HEADLESS": "1",
-      "LD_LIBRARY_PATH": "[START_DIR]/clang_linux/lib",
+      "LD_LIBRARY_PATH": "[START_DIR]/clang_linux/lib/x86_64-unknown-linux-gnu",
       "LSAN_OPTIONS": "symbolize=1 print_suppressions=1",
       "PATH": "<PATH>:RECIPE_REPO[depot_tools]:[START_DIR]/clang_linux/bin",
       "UBSAN_OPTIONS": "symbolize=1 print_stacktrace=1"
diff --git a/infra/bots/tasks.json b/infra/bots/tasks.json
index 8ee624f..464c75e 100755
--- a/infra/bots/tasks.json
+++ b/infra/bots/tasks.json
@@ -4078,7 +4078,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -4444,7 +4444,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -5699,7 +5699,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -6242,7 +6242,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -6340,7 +6340,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -6438,7 +6438,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -6536,7 +6536,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -6634,7 +6634,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -6728,7 +6728,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -6849,7 +6849,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -6947,7 +6947,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -7045,7 +7045,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -7143,7 +7143,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -7241,7 +7241,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -7339,7 +7339,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/cmake_linux",
@@ -7442,7 +7442,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/cmake_linux",
@@ -7545,7 +7545,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/cmake_linux",
@@ -7648,7 +7648,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -7746,7 +7746,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -7844,7 +7844,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -7942,7 +7942,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8067,7 +8067,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8164,7 +8164,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8262,7 +8262,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8360,7 +8360,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8458,7 +8458,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8556,7 +8556,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8654,7 +8654,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8783,7 +8783,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8877,7 +8877,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -8975,7 +8975,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -9073,7 +9073,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -9171,7 +9171,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -9292,7 +9292,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -9389,7 +9389,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -9487,7 +9487,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -9585,7 +9585,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -9683,7 +9683,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -9781,7 +9781,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -9879,7 +9879,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/cmake_linux",
@@ -9982,7 +9982,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/cmake_linux",
@@ -10085,7 +10085,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -10183,7 +10183,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -10281,7 +10281,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -10379,7 +10379,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -10477,7 +10477,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -12094,7 +12094,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -12190,7 +12190,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -12286,7 +12286,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -12382,7 +12382,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -12478,7 +12478,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -12574,7 +12574,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -12670,7 +12670,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -12766,7 +12766,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         }
       ],
       "command": [
@@ -21619,7 +21619,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -21668,7 +21668,7 @@
       ],
       "env_prefixes": {
         "LD_LIBRARY_PATH": [
-          "clang_linux/lib"
+          "clang_linux/lib/x86_64-unknown-linux-gnu"
         ],
         "PATH": [
           "clang_linux/bin"
@@ -21683,7 +21683,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -21747,7 +21747,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -22678,7 +22678,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/go",
@@ -28253,7 +28253,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/linux_vulkan_sdk",
@@ -29578,7 +29578,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -43245,7 +43245,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -43558,7 +43558,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -46407,7 +46407,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/linux_vulkan_sdk",
@@ -46742,7 +46742,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/mesa_intel_driver_linux",
@@ -46849,7 +46849,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/linux_vulkan_sdk",
@@ -46961,7 +46961,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/mesa_intel_driver_linux",
@@ -47068,7 +47068,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/mesa_intel_driver_linux",
@@ -47175,7 +47175,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/linux_vulkan_sdk",
@@ -47505,7 +47505,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/mesa_intel_driver_linux",
@@ -47612,7 +47612,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/linux_vulkan_sdk",
@@ -47724,7 +47724,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/mesa_intel_driver_linux",
@@ -47831,7 +47831,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/linux_vulkan_sdk",
@@ -47943,7 +47943,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/mesa_intel_driver_linux",
@@ -48050,7 +48050,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/linux_vulkan_sdk",
@@ -49403,7 +49403,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -49505,7 +49505,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -49607,7 +49607,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -56765,7 +56765,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
@@ -56867,7 +56867,7 @@
         {
           "name": "skia/bots/clang_linux",
           "path": "clang_linux",
-          "version": "version:20"
+          "version": "version:28"
         },
         {
           "name": "skia/bots/skimage",
diff --git a/site/docs/dev/testing/xsan.md b/site/docs/dev/testing/xsan.md
index 93ef252..de15869 100644
--- a/site/docs/dev/testing/xsan.md
+++ b/site/docs/dev/testing/xsan.md
@@ -77,7 +77,7 @@
         cc = "${CLANGDIR}/bin/clang"
         cxx = "${CLANGDIR}/bin/clang++"
         sanitize = "ASAN"
-        extra_ldflags = [ "-fuse-ld=lld", "-Wl,-rpath,${CLANGDIR}/lib" ]
+        extra_ldflags = [ "-fuse-ld=lld", "-Wl,-rpath,${CLANGDIR}/lib/x86_64-unknown-linux-gnu" ]
     EOF
     python3 tools/git-sync-deps
     bin/gn gen out/asan
diff --git a/tools/xsan.supp b/tools/xsan.supp
index 3530e2a..ee5d9f6 100644
--- a/tools/xsan.supp
+++ b/tools/xsan.supp
@@ -1,15 +1,9 @@
-#if 0
-
-#  This file must be a no-op C #include header, and a valid *SAN compile-time suppression file.
-#  Luckily, anything starting with # is a comment to *SAN compile-time suppression files,
-#  and anything inside #if 0 is ignored by C.  Yippee!
+# If you change this file, then you will need to do a clean build (gn clean) in order
+# for it to go into effect. This is not a problem with Bazel builds, only GN ones.
 #
-#  If you want to type '*', type '.*' instead.  Don't make C comments!
-
+# https://clang.llvm.org/docs/SanitizerSpecialCaseList.html#format
 # libpng and zlib both dereference under-aligned pointers.
 # TODO: it'd be nice to tag these as [alignment] only but our Mac toolchain can't yet.
 # [alignment]
 src:.*third_party/externals/libpng/intel/filter_sse2_intrinsics.c
 src:.*third_party/externals/zlib/deflate.c
-
-#endif