Add a cli option for the snapshot cache size

[pawanjay176]

Issue Addressed

N/A

Proposed Changes

Adds a cli option --state-cache-size to change the size of the snapshot cache.
Also makes the default size 3 instead of 4.

[michaelsproul]

Some metrics I collected from Holesky last week and forgot to post.

Resident set size (RSS) has lower peaks:

Blue = cache size 3, yellow = cache size 4

Memory usage is still correlated with snapshot cache size, which is being effectively limited:

Blue = cache size 3, yellow = cache size 4

RSS for cache size 2 is also good:

The main risk of dropping the snapshot cache size is increasing the frequency of snapshot cache misses. Unfortunately on Holesky these are already very frequent, although our changes do not seem to make it worse:

Snapshot cache misses over 7d with cache size=2 from ~Feb 22 12pm (above)

Snapshot cache misses over 7d with cache size=3 from ~Feb 22 12pm (above)

Snapshot cache misses over 7d with cache size=4

[michaelsproul]

@Mergifyio queue

[mergify]

queue

🛑 The pull request has been removed from the queue default

Pull request #5270 has been dequeued by a dequeue command.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

[michaelsproul]

@Mergifyio dequeue

[mergify]

dequeue

✅ The pull request has been removed from the queue default

[michaelsproul]

@Mergifyio requeue

[mergify]

requeue

✅ This pull request will be re-embarked automatically

The followup queue command will be automatically executed to re-embark the pull request

[mergify]

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 3ab9d3a