Binlogs Redacting support + Binlogs forward-compatibility reading support

[JanKrivanek]

Fixes #9147, #9261 and contributes to #8400

Viewer complement PR: KirillOsenkov/MSBuildStructuredLog#732

Important

This contains all changes from

• [Binlogs Redacting] API and support for external redaction functionality #9219
• Binlog forward compatibility and deserialisation-less reading support JanKrivanek/msbuild#1

Context and changes are described in those individual PRs
Feel free to review everything together (this PR), or separately (above 2 PRs) - but not all 3 as you'd be viewing same changes twice

TBD:

• Tests
  • Simulate the fwd compat scenarios by injecting/removing/altering bytes in memory stream underlying under binlog writer and reader.
  • Test the stream utility classes.
• localize all strings Done
• I'm contemplating adding support for recognizing over-reading and mismatched reading during events deserialization Done
• [TBD for next PR] Usage documentation (when, why and how to handle errors during forward compatible reading)

[rokonec]

Request change is only for missing that "backward-compatible-with" field.
Otherwise it looks OK. But I have to admit that I have not time to carefully review full extent of this PR.

[danmoseley]

Is there a design doc or some overview of what the redactor does - it sounds interesting?
Eg., does it attempt to discover security tokens and suchlike, as Github does? It would only be defense in depth as it can never be perfect.

[JanKrivanek]

Is there a design doc or some overview of what the redactor does - it sounds interesting? Eg., does it attempt to discover security tokens and suchlike, as Github does? It would only be defense in depth as it can never be perfect.

Hi @danmoseley,

Design proposal of redactor is located here: https://github.com/JanKrivanek/MSBuildBinlogRedactor/blob/main/docs/DesignProposal.md (ultimate long term wishful plan is here https://github.com/dotnet/msbuild/blob/main/documentation/specs/proposed/security-metadata.md)
Autodetections are in plan - so far we have just some PoC samples - couple token types detector (https://github.com/JanKrivanek/MSBuildBinlogRedactor/blob/main/src/Microsoft.Build.SensitiveDataDetector/PatternsDetector.cs) and username detector (https://github.com/JanKrivanek/MSBuildBinlogRedactor/blob/main/src/Microsoft.Build.SensitiveDataDetector/UsernameDetector.cs). But @michaelcfanning wants to contribute some more comprehensive logic they have been using in 1ES. Plus I'm trying to get CredScan team to OSS their client side library for textual data classification

[ladipro]

I've left comments inline, mostly nits.

[rokonec]

Minor comments

[KirillOsenkov]

No, I increased the version

[JanKrivanek]

No, I increased the version

Thanks @KirillOsenkov - good catch! Not expected indeed! - all the forward compatible flags (allow reading higher version, allow skipping unknown events, allow skiping unknown data in known events) should have same default setting - there was one (allow reading higher version) which was permitted by default - fixed now

[JanKrivanek]

Let's not merge until the linked PRs are merged (to minimize breaking internal teams)