Reorder operations in GrowToNextCapacityAndPrepareInsert to reduce binary size. In `GrowToNextCapacityAndPrepareInsert`, reading `common` and `policy` fields before `policy.get_char_alloc(common)` and `AllocBackingArray` forces LLVM to spill those values to the stack across both calls. In addition, updating `common.inline_data_` in two separate stages—`set_capacity` and `set_blocked_element_count_to_zero` before `AllocBackingArray`, and `PrepareInsertCommon` and `ResetGrowthLeft(new_capacity, common.size(), common)` at the end of the function—requires loading and storing `common.inline_data_` twice and extracting `common.size()` twice. Reorder the operations in `GrowToNextCapacityAndPrepareInsert` to shrink live ranges across calls and coalesce `common.inline_data_` updates: 1. **Call `policy.get_char_alloc(common)` at function entry**: Because `get_char_alloc` is an indirect call taking a non-const `CommonFields&`, calling it before reading `old_capacity` or `has_infoz` (when only the incoming arguments `common`, `policy`, and `new_hash` are live) eliminates pre-call stack spills and reduces the stack frame from `0x78` to `0x68` bytes. 2. **Defer `old_*` loads until after `AllocBackingArray`**: `AllocBackingArray` receives `new_capacity` and `/*blocked_element_count=*/0` explicitly and does not read `common.capacity()`, `common.control()`, or `common.blocked_element_count()`. Reading `old_ctrl`, `old_slots`, `old_blocked_element_count`, and `old_size` after `AllocBackingArray` avoids keeping them live across both calls. 3. **Read `old_size` unconditionally and group all `common` updates together**: Grouping `set_capacity(new_capacity)`, `set_blocked_element_count_to_zero()`, `set_control(new_ctrl)`, `PrepareInsertCommon(common)`, and `ResetGrowthLeft(new_capacity, old_size + 1, common)` right after `AllocBackingArray` merges all four `common.inline_data_` bitfield updates into a single load-modify-store sequence, shares `old_size` between `policy.transfer_n` and `ResetGrowthLeft`, and allows LLVM to thread the `new_capacity > 127` branch of `InitGrowthLeftNoDeleted` directly into the multi-group `else` branch. PiperOrigin-RevId: 1003492867
diff --git a/absl/container/internal/raw_hash_set.cc b/absl/container/internal/raw_hash_set.cc index b477e60..dcd2cc0 100644 --- a/absl/container/internal/raw_hash_set.cc +++ b/absl/container/internal/raw_hash_set.cc
@@ -1863,29 +1863,39 @@ void* GrowToNextCapacityAndPrepareInsert( CommonFields& common, const PolicyFunctions& __restrict policy, size_t new_hash) { + ABSL_SWISSTABLE_ASSERT( + common.GetGrowthLeftTotalSlow(common.capacity()) == 0); + ABSL_SWISSTABLE_ASSERT(common.capacity() > policy.soo_capacity()); + ABSL_SWISSTABLE_ASSERT(!IsSmallCapacity(common.capacity())); + + // Call `get_char_alloc` first so no local variables need to be spilled + // across the indirect call, which may alias `common`. + void* alloc = policy.get_char_alloc(common); const size_t old_capacity = common.capacity(); - ABSL_SWISSTABLE_ASSERT(common.GetGrowthLeftTotalSlow(old_capacity) == 0); - ABSL_SWISSTABLE_ASSERT(old_capacity > policy.soo_capacity()); - ABSL_SWISSTABLE_ASSERT(!IsSmallCapacity(old_capacity)); ABSL_ASSUME(old_capacity > kMaxSmallCapacity); const size_t new_capacity = NextCapacity(old_capacity); ABSL_ASSUME(new_capacity > kMaxSmallCapacity); - ctrl_t* old_ctrl = common.control(); - void* old_slots = common.slot_array(old_capacity); - size_t old_blocked_element_count = common.blocked_element_count(); - const bool has_infoz = common.has_infoz(); - common.set_capacity(new_capacity); - common.set_blocked_element_count_to_zero(); - const size_t slot_size = policy.slot_size; - const size_t slot_align = policy.slot_align; - void* alloc = policy.get_char_alloc(common); const auto [new_ctrl, new_slots] = AllocBackingArray(common, policy, new_capacity, has_infoz, alloc, /*blocked_element_count=*/0); + // Read old `common` fields after `AllocBackingArray` to avoid spilling them + // across the call. + ctrl_t* old_ctrl = common.control(); + void* old_slots = common.slot_array(old_capacity); + size_t old_blocked_element_count = common.blocked_element_count(); + const size_t old_size = common.size(); + + // Group all `common` updates together so `inline_data_` is modified in a + // single load-store sequence. + common.set_capacity(new_capacity); + common.set_blocked_element_count_to_zero(); common.set_control(new_ctrl); + PrepareInsertCommon(common); + ResetGrowthLeft(new_capacity, old_size + 1, common); + const size_t slot_size = policy.slot_size; SanitizerPoisonMemoryRegion(new_slots, new_capacity * slot_size); h2_t new_h2 = H2(new_hash); @@ -1893,7 +1903,6 @@ FindInfo find_info; if (ABSL_PREDICT_TRUE(is_single_group(new_capacity))) { size_t offset; - const size_t old_size = common.size(); GrowIntoSingleGroupShuffleControlBytes(old_ctrl, old_capacity, old_blocked_element_count, new_ctrl, new_capacity); @@ -1903,11 +1912,11 @@ ABSL_SWISSTABLE_ASSERT(IsEmpty(new_ctrl[offset])); SetCtrlInSingleGroupTable(common, new_capacity, new_ctrl, offset, new_h2, - policy.slot_size); + slot_size); find_info = FindInfo{offset, 0}; // Single group tables have all slots full on resize. So we can transfer // all slots without checking the control bytes. - ABSL_SWISSTABLE_ASSERT(common.size() + old_blocked_element_count == + ABSL_SWISSTABLE_ASSERT(old_size + old_blocked_element_count == old_capacity); void* target = NextSlot(new_slots, slot_size); SanitizerUnpoisonMemoryRegion(target, old_size * slot_size); @@ -1917,22 +1926,20 @@ GrowToNextCapacityDispatch(common, policy, old_ctrl, old_slots); find_info = find_first_non_full(new_capacity, new_ctrl, new_hash); SetCtrl(common, new_capacity, new_ctrl, find_info.offset, new_h2, - policy.slot_size); + slot_size); } ABSL_SWISSTABLE_ASSERT(old_capacity > policy.soo_capacity()); - PrepareInsertCommon(common); - ResetGrowthLeft(new_capacity, common.size(), common); - if (ABSL_PREDICT_FALSE(has_infoz)) { ReportGrowthToInfoz(common, old_ctrl, new_hash, total_probe_length, find_info.probe_length); } // Deallocate after `ReportGrowthToInfoz`, because it reads infoz from // old_ctrl. - DeallocBackingArrayImpl(alloc, old_capacity, old_ctrl, slot_size, slot_align, - has_infoz, old_blocked_element_count, policy.dealloc); - return SlotAddress(new_slots, find_info.offset, policy.slot_size); + DeallocBackingArrayImpl(alloc, old_capacity, old_ctrl, slot_size, + policy.slot_align, has_infoz, + old_blocked_element_count, policy.dealloc); + return SlotAddress(new_slots, find_info.offset, slot_size); } } // namespace