MDEV-35049: Use CRC-32C and avoid allocating heap

For the adaptive hash index, dtuple_fold() and rec_fold() were employing
a slow rolling hash algorithm, computing hash values ("fold") for one
field and one byte at a time, while depending on calls to
rec_get_offsets().

We already have optimized implementations of CRC-32C and have been
successfully using that function in some other InnoDB tables, but not
yet in the adaptive hash index.

Any linear function such as any CRC will fail the avalanche test that
any cryptographically secure hash function is expected to pass:
any single-bit change in the input key should affect on average half
the bits in the output.

But we always were happy with less than cryptographically secure:
in fact, ut_fold_ulint_pair() or ut_fold_binary() are just about as
linear as any CRC, using a combination of multiplication and addition,
partly carry-less. It is worth noting that exclusive-or corresponds to
carry-less subtraction or addition in a binary Galois field, or GF(2).

We only need some way of reducing key prefixes into hash values.
The CRC-32C should be better than a Rabin–Karp rolling hash algorithm.
Compared to the old hash algorithm, it has the drawback that there will
be only 32 bits of entropy before we choose the hash table cell by a
modulus operation. The size of each adaptive hash index array is
(innodb_buffer_pool_size / 512) / innodb_adaptive_hash_index_parts.
With the maximum number of partitions (512), we would not exceed 1<<32
elements per array until the buffer pool size exceeds 1<<50 bytes (1 PiB).
We would hit other limits before that: the virtual address space on many
contemporary 64-bit processor implementations is only 48 bits (256 TiB).
So, we can simply go for the SIMD accelerated CRC-32C.

rec_fold(): Take a combined parameter n_bytes_fields. Determine the
length of each field on the fly, and compute CRC-32C over a single
contiguous range of bytes, from the start of the record payload area
to the end of the last full or partial field. For secondary index records
in ROW_FORMAT=REDUNDANT, also the data area that is reserved for NULL
values (to facilitate in-place updates between NULL and NOT NULL values)
will be included in the count. Luckily, InnoDB always zero-initialized
such unused area; refer to data_write_sql_null() in
rec_convert_dtuple_to_rec_old(). For other than ROW_FORMAT=REDUNDANT,
no space is allocated for NULL values, and therefore the CRC-32C will
only cover the actual payload of the key prefix.

dtuple_fold(): For ROW_FORMAT=REDUNDANT, include the dummy NULL values
in the CRC-32C, so that the values will be comparable with rec_fold().

innodb_ahi-t: A unit test for rec_fold() and dtuple_fold().

btr_search_build_page_hash_index(), btr_search_drop_page_hash_index():
Use a fixed-size stack buffer for computing the fold values, to avoid
dynamic memory allocation.

btr_search_drop_page_hash_index(): Do not release part.latch if we
need to invoke multiple batches of rec_fold().

dtuple_t: Allocate fewer bits for the fields. The maximum number of
data fields is about 1023, so uint16_t will be fine for them. The
info_bits is stored in less than 1 byte.

ut_pair_min(), ut_pair_cmp(): Remove. We can actually combine and compare
int(n_fields << 16 | n_bytes).

PAGE_CUR_LE_OR_EXTENDS, PAGE_CUR_DBG: Remove. These were never defined,
because they would only work with latin1_swedish_ci if at all.

btr_cur_t::check_mismatch(): Replaces !btr_search_check_guess().

cmp_dtuple_rec_bytes(): Replaces cmp_dtuple_rec_with_match_bytes().
Determine the offsets of fields on the fly.

page_cur_try_search_shortcut_bytes(): This caller of
cmp_dtuple_rec_bytes() will not be invoked on the change buffer tree.

cmp_dtuple_rec_leaf(): Replaces cmp_dtuple_rec_with_match()
for comparing leaf-page records.

buf_block_t::ahi_left_bytes_fields: Consolidated Atomic_relaxed<uint32_t>
of curr_left_side << 31 | curr_n_bytes << 16 | curr_n_fields.
The other set of parameters (n_fields, n_bytes, left_side) was removed
as redundant.

btr_search_update_hash_node_on_insert(): Merged to
btr_search_update_hash_on_insert().

btr_search_build_page_hash_index(): Take combined left_bytes_fields
instead of n_fields, n_bytes, left_side.

btr_search_update_block_hash_info(), btr_search_update_hash_ref():
Merged to btr_search_info_update_hash().

btr_cur_t::n_bytes_fields: Replaces n_bytes << 16 | n_fields.

We also remove many redundant checks of btr_search.enabled.
If we are holding any btr_sea::partition::latch, then a nonnull pointer
in buf_block_t::index must imply that the adaptive hash index is enabled.

Reviewed by: Vladislav Lesin

Author: dr-m

storage/innobase/CMakeLists.txt

@@ -362,7 +362,6 @@ SET(INNOBASE_SOURCES
 	include/ut0sort.h
 	include/ut0stage.h
 	include/ut0ut.h
-	include/ut0ut.inl
 	include/ut0vec.h
 	include/ut0vec.inl
 	include/ut0wqueue.h

storage/innobase/btr/btr0btr.cc

@@ -558,8 +558,8 @@ buf_block_t *btr_root_block_sx(dict_index_t *index, mtr_t *mtr, dberr_t *err)
       return root;
   }
 #ifdef BTR_CUR_HASH_ADAPT
-  else
-    ut_ad(!root->index || !root->index->freed());
+  ut_d(else if (dict_index_t *index= root->index))
+    ut_ad(!index->freed());
 #endif
   return root;
 }
@@ -772,7 +772,7 @@ static rec_offs *btr_page_get_parent(rec_offs *offsets, mem_heap_t *heap,
       {
         ut_ad(block->page.lock.have_u_or_x() ||
               (!block->page.lock.have_s() && index->lock.have_x()));
-        ulint up_match= 0, low_match= 0;
+        uint16_t up_match= 0, low_match= 0;
         cursor->page_cur.block= block;
         if (page_cur_search_with_match(tuple, PAGE_CUR_LE, &up_match,
                                        &low_match, &cursor->page_cur,
@@ -1976,7 +1976,7 @@ btr_root_raise_and_insert(
 
 	ut_ad(dtuple_check_typed(tuple));
 	/* Reposition the cursor to the child node */
-	ulint low_match = 0, up_match = 0;
+	uint16_t low_match = 0, up_match = 0;
 
 	if (page_cur_search_with_match(tuple, PAGE_CUR_LE,
 				       &up_match, &low_match,
@@ -2660,7 +2660,7 @@ btr_insert_into_right_sibling(
 		return nullptr;
 	}
 
-	ulint up_match = 0, low_match = 0;
+	uint16_t up_match = 0, low_match = 0;
 
 	if (page_cur_search_with_match(tuple,
 				       PAGE_CUR_LE, &up_match, &low_match,
@@ -3142,7 +3142,7 @@ btr_page_split_and_insert(
 	page_cursor = btr_cur_get_page_cur(cursor);
 	page_cursor->block = insert_block;
 
-	ulint up_match = 0, low_match = 0;
+	uint16_t up_match = 0, low_match = 0;
 
 	if (page_cur_search_with_match(tuple,
 				       PAGE_CUR_LE, &up_match, &low_match,

storage/innobase/btr/btr0cur.cc

@@ -1100,8 +1100,7 @@ dberr_t btr_cur_t::search_leaf(const dtuple_t *tuple, page_cur_mode_t mode,
   MEM_UNDEFINED(&up_bytes, sizeof up_bytes);
   MEM_UNDEFINED(&low_match, sizeof low_match);
   MEM_UNDEFINED(&low_bytes, sizeof low_bytes);
-  ut_d(up_match= ULINT_UNDEFINED);
-  ut_d(low_match= ULINT_UNDEFINED);
+  ut_d(up_match= low_match= uint16_t(~0u));
 
   ut_ad(!(latch_mode & BTR_ALREADY_S_LATCHED) ||
         mtr->memo_contains_flagged(&index()->lock,
@@ -1118,14 +1117,14 @@ dberr_t btr_cur_t::search_leaf(const dtuple_t *tuple, page_cur_mode_t mode,
         || latch_mode == BTR_MODIFY_TREE
         || latch_mode == BTR_MODIFY_ROOT_AND_LEAF);
 
-  flag= BTR_CUR_BINARY;
 #ifndef BTR_CUR_ADAPT
   guess= nullptr;
 #else
   auto info= &index()->search_info;
   guess= info->root_guess;
 
 # ifdef BTR_CUR_HASH_ADAPT
+  flag= BTR_CUR_BINARY;
 #  ifdef UNIV_SEARCH_PERF_STAT
   info->n_searches++;
 #  endif
@@ -1138,9 +1137,9 @@ dberr_t btr_cur_t::search_leaf(const dtuple_t *tuple, page_cur_mode_t mode,
                                     latch_mode, this, mtr))
   {
     /* Search using the hash index succeeded */
-    ut_ad(up_match != ULINT_UNDEFINED || mode != PAGE_CUR_GE);
-    ut_ad(up_match != ULINT_UNDEFINED || mode != PAGE_CUR_LE);
-    ut_ad(low_match != ULINT_UNDEFINED || mode != PAGE_CUR_LE);
+    ut_ad(up_match != uint16_t(~0U) || mode != PAGE_CUR_GE);
+    ut_ad(up_match != uint16_t(~0U) || mode != PAGE_CUR_LE);
+    ut_ad(low_match != uint16_t(~0U) || mode != PAGE_CUR_LE);
     ++btr_cur_n_sea;
 
     return DB_SUCCESS;
@@ -1349,9 +1348,9 @@ dberr_t btr_cur_t::search_leaf(const dtuple_t *tuple, page_cur_mode_t mode,
       if (page_cur_search_with_match(tuple, mode, &up_match, &low_match,
                                      &page_cur, nullptr))
         goto corrupted;
-      ut_ad(up_match != ULINT_UNDEFINED || mode != PAGE_CUR_GE);
-      ut_ad(up_match != ULINT_UNDEFINED || mode != PAGE_CUR_LE);
-      ut_ad(low_match != ULINT_UNDEFINED || mode != PAGE_CUR_LE);
+      ut_ad(up_match != uint16_t(~0U) || mode != PAGE_CUR_GE);
+      ut_ad(up_match != uint16_t(~0U) || mode != PAGE_CUR_LE);
+      ut_ad(low_match != uint16_t(~0U) || mode != PAGE_CUR_LE);
       goto func_exit;
     }
 
@@ -1398,22 +1397,21 @@ dberr_t btr_cur_t::search_leaf(const dtuple_t *tuple, page_cur_mode_t mode,
 
   reached_latched_leaf:
 #ifdef BTR_CUR_HASH_ADAPT
-    if (btr_search.enabled && !(tuple->info_bits & REC_INFO_MIN_REC_FLAG))
+    if (!(tuple->info_bits & REC_INFO_MIN_REC_FLAG) && btr_search.enabled)
     {
-      if (page_cur_search_with_match_bytes(tuple, mode,
-                                           &up_match, &up_bytes,
-                                           &low_match, &low_bytes, &page_cur))
+      if (page_cur_search_with_match_bytes(*tuple, mode, &up_match, &low_match,
+                                           &page_cur, &up_bytes, &low_bytes))
         goto corrupted;
     }
     else
 #endif /* BTR_CUR_HASH_ADAPT */
-    if (page_cur_search_with_match(tuple, mode, &up_match, &low_match,
-                                   &page_cur, nullptr))
-      goto corrupted;
+      if (page_cur_search_with_match(tuple, mode, &up_match, &low_match,
+                                     &page_cur, nullptr))
+        goto corrupted;
 
-    ut_ad(up_match != ULINT_UNDEFINED || mode != PAGE_CUR_GE);
-    ut_ad(up_match != ULINT_UNDEFINED || mode != PAGE_CUR_LE);
-    ut_ad(low_match != ULINT_UNDEFINED || mode != PAGE_CUR_LE);
+    ut_ad(up_match != uint16_t(~0U) || mode != PAGE_CUR_GE);
+    ut_ad(up_match != uint16_t(~0U) || mode != PAGE_CUR_LE);
+    ut_ad(low_match != uint16_t(~0U) || mode != PAGE_CUR_LE);
 
     if (latch_mode == BTR_MODIFY_TREE &&
         btr_cur_need_opposite_intention(block->page, index()->is_clust(),
@@ -1656,9 +1654,9 @@ dberr_t btr_cur_t::pessimistic_search_leaf(const dtuple_t *tuple,
       err= DB_CORRUPTION;
     else
     {
-      ut_ad(up_match != ULINT_UNDEFINED || mode != PAGE_CUR_GE);
-      ut_ad(up_match != ULINT_UNDEFINED || mode != PAGE_CUR_LE);
-      ut_ad(low_match != ULINT_UNDEFINED || mode != PAGE_CUR_LE);
+      ut_ad(up_match != uint16_t(~0U) || mode != PAGE_CUR_GE);
+      ut_ad(up_match != uint16_t(~0U) || mode != PAGE_CUR_LE);
+      ut_ad(low_match != uint16_t(~0U) || mode != PAGE_CUR_LE);
 
 #ifdef BTR_CUR_HASH_ADAPT
       /* We do a dirty read of btr_search.enabled here.  We will recheck in
@@ -1770,8 +1768,9 @@ dberr_t btr_cur_search_to_nth_level(ulint level,
   MEM_UNDEFINED(&cursor->low_bytes, sizeof cursor->low_bytes);
   cursor->up_match= 0;
   cursor->low_match= 0;
+#ifdef BTR_CUR_HASH_ADAPT
   cursor->flag= BTR_CUR_BINARY;
-
+#endif
 #ifndef BTR_CUR_ADAPT
   buf_block_t *block= nullptr;
 #else
@@ -2518,13 +2517,8 @@ btr_cur_optimistic_insert(
 		ut_ad(entry->is_metadata());
 		ut_ad(index->is_instant());
 		ut_ad(flags == BTR_NO_LOCKING_FLAG);
-	} else if (index->table->is_temporary()) {
-	} else {
-		if (!reorg && cursor->flag == BTR_CUR_HASH) {
-			btr_search_update_hash_node_on_insert(cursor);
-		} else {
-			btr_search_update_hash_on_insert(cursor);
-		}
+	} else if (!index->table->is_temporary()) {
+		btr_search_update_hash_on_insert(cursor, reorg);
 	}
 #endif /* BTR_CUR_HASH_ADAPT */
 
@@ -2588,7 +2582,9 @@ btr_cur_pessimistic_insert(
 	      || dict_index_is_clust(index)
 	      || (flags & BTR_CREATE_FLAG));
 
+#ifdef BTR_CUR_HASH_ADAPT
 	cursor->flag = BTR_CUR_BINARY;
+#endif
 
 	/* Check locks and write to undo log, if specified */
 
@@ -2694,9 +2690,8 @@ btr_cur_pessimistic_insert(
 			ut_ad(index->is_instant());
 			ut_ad(flags & BTR_NO_LOCKING_FLAG);
 			ut_ad(!(flags & BTR_CREATE_FLAG));
-		} else if (index->table->is_temporary()) {
-		} else {
-			btr_search_update_hash_on_insert(cursor);
+		} else if (!index->table->is_temporary()) {
+			btr_search_update_hash_on_insert(cursor, false);
 		}
 #endif /* BTR_CUR_HASH_ADAPT */
 		if (inherit && !(flags & BTR_NO_LOCKING_FLAG)) {
@@ -3325,7 +3320,7 @@ static void btr_cur_trim_alter_metadata(dtuple_t* entry,
 	if (n_fields != index->n_uniq) {
 		ut_ad(n_fields
 		      >= index->n_core_fields);
-		entry->n_fields = n_fields;
+		entry->n_fields = uint16_t(n_fields);
 		return;
 	}
 
@@ -3361,7 +3356,7 @@ static void btr_cur_trim_alter_metadata(dtuple_t* entry,
 	ut_ad(n_fields >= index->n_core_fields);
 
 	mtr.commit();
-	entry->n_fields = n_fields + 1;
+	entry->n_fields = uint16_t(n_fields + 1);
 }
 
 /** Trim an update tuple due to instant ADD COLUMN, if needed.
@@ -3417,7 +3412,7 @@ btr_cur_trim(
 			ulint n_fields = upd_get_nth_field(update, 0)
 				->field_no;
 			ut_ad(n_fields + 1 >= entry->n_fields);
-			entry->n_fields = n_fields;
+			entry->n_fields = uint16_t(n_fields);
 		}
 	} else {
 		entry->trim(*index);
@@ -4817,10 +4812,10 @@ class btr_est_cur_t
 
   /** Matched fields and bytes which are used for on-page search, see
   btr_cur_t::(up|low)_(match|bytes) comments for details */
-  ulint m_up_match= 0;
-  ulint m_up_bytes= 0;
-  ulint m_low_match= 0;
-  ulint m_low_bytes= 0;
+  uint16_t m_up_match= 0;
+  uint16_t m_up_bytes= 0;
+  uint16_t m_low_match= 0;
+  uint16_t m_low_bytes= 0;
 
 public:
   btr_est_cur_t(dict_index_t *index, const dtuple_t &tuple,
@@ -4844,12 +4839,7 @@ class btr_est_cur_t
       m_page_mode= PAGE_CUR_LE;
       break;
     default:
-#ifdef PAGE_CUR_LE_OR_EXTENDS
-      ut_ad(mode == PAGE_CUR_L || mode == PAGE_CUR_LE ||
-            mode == PAGE_CUR_LE_OR_EXTENDS);
-#else  /* PAGE_CUR_LE_OR_EXTENDS */
       ut_ad(mode == PAGE_CUR_L || mode == PAGE_CUR_LE);
-#endif /* PAGE_CUR_LE_OR_EXTENDS */
       m_page_mode= mode;
       break;
     }

storage/innobase/btr/btr0pcur.cc

@@ -435,7 +435,7 @@ btr_pcur_t::restore_position(btr_latch_mode restore_latch_mode, mtr_t *mtr)
 	rec_offs_init(offsets);
 	restore_status ret_val= restore_status::NOT_SAME;
 	if (rel_pos == BTR_PCUR_ON && btr_pcur_is_on_user_rec(this)) {
-		ulint n_matched_fields= 0;
+		uint16_t n_matched_fields= 0;
 		if (!cmp_dtuple_rec_with_match(
 		      tuple, btr_pcur_get_rec(this), index,
 		      rec_get_offsets(btr_pcur_get_rec(this), index, offsets,