Optimize Miri for `test_validate_cast_and_convert_metadata`

[joshlf]

Currently, Miri tests are failing (both on main and in PRs) due to test_validate_cast_and_convert_metadata taking too long to run. We need to optimize this under Miri to unblock CI.