Scope the `GroupSse2Impl` `movzx` inline-asm workaround from `MoveMask` to `Match`.
`GroupSse2Impl::MoveMask` previously applied `asm("" : "+r"(mask))` on Clang to prevent an unnecessary `movzx` when iterating over `BitMask` in `Match(h2)`. However, `MoveMask` is also called by `MaskEmpty()`, `MaskFull()`, `MaskNonFull()`, `MaskEmptyOrDeleted()`, and `MaskFullOrSentinel()`.
In non-`Match` callers, the `asm("" : "+r"(mask))` tied-register inline assembly had two negative effects:
1. In `find_or_prepare_insert_large` (such as `IntPolicy::apply<EmplaceDecomposable>`), the tied `+r` constraint on `MoveMask` inside `MaskEmpty()` caused LLVM's `RegisterCoalescer` to coalesce the `vpmovmskb` destination into `%ecx` (which holds `seq.index()` in the probe loop), requiring two extra register-to-register copies (`movq %rcx, %rbx` and `movq %rbx, %rcx`) on every probe loop iteration.
2. In LLVM's `InlineCost` analysis (`visitCallBase`), every inline `asm` statement adds `InlineConstants::CallPenalty` (`+10` cost) per inlined callsite, inflating the perceived inlining cost of functions that call `MaskEmpty()`, `MaskFull()`, or `MaskEmptyOrDeleted()`.
Moving the `asm("" : "+r"(mask))` barrier into `GroupSse2Impl::Match` preserves the `movzx` workaround for `Match(h2)` while eliminating the register coalescing copies and inline-asm call penalty in all other `MoveMask` callers.
PiperOrigin-RevId: 1007541334
diff --git a/absl/container/internal/hashtable_control_bytes.h b/absl/container/internal/hashtable_control_bytes.h
index 34df1d4..9a83c3b 100644
--- a/absl/container/internal/hashtable_control_bytes.h
+++ b/absl/container/internal/hashtable_control_bytes.h
@@ -289,7 +289,15 @@
// Returns a bitmask representing the positions of slots that match hash.
BitMaskType Match(h2_t hash) const {
auto match = _mm_set1_epi8(static_cast<char>(hash));
- return BitMaskType(MoveMask(_mm_cmpeq_epi8(match, ctrl)));
+ auto mask = MoveMask(_mm_cmpeq_epi8(match, ctrl));
+#ifdef __clang__
+ // TODO(b/472522597): Without the inline asm, clang ends up generating an
+ // unnecessary movzx to zero the upper bits of the output when iterating
+ // over matches, even though those bits are already zero. See
+ // https://godbolt.org/z/G6xW1Ecbx.
+ asm("" : "+r"(mask)); // NOLINT
+#endif
+ return BitMaskType(mask);
}
// Returns a bitmask representing the positions of empty slots.
@@ -343,14 +351,7 @@
}
static MaskInt MoveMask(__m128i xmm) {
- auto mask = static_cast<MaskInt>(_mm_movemask_epi8(xmm));
-#ifdef __clang__
- // TODO(b/472522597): Without the inline asm, clang ends up generating an
- // unnecessary movzx to zero the upper bits of the output, but those bits
- // are already zero. See https://godbolt.org/z/G6xW1Ecbx.
- asm("" : "+r"(mask)); // NOLINT
-#endif
- return mask;
+ return static_cast<MaskInt>(_mm_movemask_epi8(xmm));
}
__m128i ctrl;