PRW2 accepts invalid histograms instead of rejecting them

[roidelapluie]

Describe the bug
When a native histogram with a wrong code is received, it is ingested by cortex rather than being rejected (like Prometheus does)

To Reproduce
Steps to reproduce the behavior:

1. Start Cortex 0e9721f
2. Remote write an histogram with bad count (Add remote write 2 receivers compliance tests prometheus/compliance#182)

Expected behavior

Prometheus returns 400:

23 observations found in buckets, but the Count field is 99: histogram's observation count should equal the number of observations found in the buckets (in absence of NaN) for series {name="minimal_hist"}

Cortex returns 200.

[yeya24]

Thanks for reporting the issue. Help wanted to reproduce the issue and provide a fix.

[yeya24]

This seems a valid bug for both PRW v1 and v2 in Cortex. In Prometheus remote write receiver, histogram validation happens in the remote write handler so it returns 4XX right away. https://github.com/prometheus/prometheus/blob/44df626620f93c02485a03134735e6bb3c8b6a10/storage/remote/write_handler.go#L217

Today in Cortex, histogram validation happens in Ingester and we return it as partial error https://github.com/cortexproject/cortex/blob/master/pkg/ingester/ingester.go#L1324, which seems the wrong behavior.

The right way is to validate native histogram in Distributor and return 4XX. https://github.com/cortexproject/cortex/blob/master/pkg/util/validation/validate.go#L370

[yeya24]

@PaurushGarg will help work on it