Use 'pixelated' as fallback of 'crisp-edges' in object-position tests

[javifernandez]

The tests fail in Chrome due to small pixel differences between the background-image used in the ref files and the different elements used to test the object-position, like canvas, img or video.

The use of 'image-rendering: crisp-edges' helps to reduce the impact of the different scaling algorithms implemented in the engine's rendering pipelines. However, some web engines don't implement the 'crisp-edges' features; this is the case of the Chrome browser.

It seems using 'pixelated' as fallback solves most of the PNG related test cases.

[dholbert]

The tests fail in Chrome due to small pixel differences between the background-image used in the ref files and the different elements used to test the object-position, like canvas, img or video.

In general, shouldn't that be considered a Chrome bug, and annotated locally as expected-failure or expected-fuzz in Chromium's source tree rather than in the test itself?

The intent in these tests is that there shouldn't be small pixel differences, since we're scaling the same image to the same target-size in all cases, and image-rendering:crisp-edges should prevent antialiasing (and hence should probably prevent uninteresting fuzzy rendering differences) along the edges of the blocks of color in the image.

Additionally, this PR removes the image-rendering: crisp-edges in 2 tests, which is not strictly needed for the purpose of the tests where it's used.

I'm willing to believe it's not useful for some of the cases (maybe for object-fit:none and object-fit: scale-down), but it definitely matters in all the others where upscaling is being done. It makes the blocks-of-color scale crisply, which substantially reduces opportunities for fuzz. I can see the difference visually if I load the reference cases here and toggle image-rendering:crisp-edges off-and-on in devtools.

Also, looking at your PR, it looks like some of the testcases where you're adding fuzz are testcases that Chromium currently passes on wpt.fyi, so I wonder if some of the patch's fuzz is in fact accounting for your image-rendering removal rather than addressing actual current Chrome failures?

[dholbert]

Also, looking at your PR, it looks like some of the testcases where you're adding fuzz are testcases that Chromium currently passes on wpt.fyi, so I wonder if some of the patch's fuzz is in fact accounting for your image-rendering removal rather than addressing actual current Chrome failures?

For example, your PR is adding substantial fuzz (maxDifference=0-107;totalPixels=0-144", i.e. hundreds of pixels differing by quite a lot) to these two tests, which wpt.fyi says Chrome currently passes:
https://wpt.fyi/results/css/css-images/object-fit-cover-png-002e.html
https://wpt.fyi/results/css/css-images/object-fit-cover-png-002i.html

[dholbert]

Looking at the actual current Chromium failures that wpt.fyi shows for these tests (-001 tests, -002 tests), I'm seeing two different categories of issues:
(1) For SVG images: It looks like Chrome isn't honoring object-fit for object and embed at all, e.g. here:
https://wpt.fyi/results/css/css-images/object-fit-contain-svg-001e.html
https://wpt.fyi/results/css/css-images/object-fit-contain-svg-001o.html

(2) For PNG images: It looks like Chrome is incorrectly doing some antialiasing when scaling up an image in these tests, and perhaps doing it an inconsistent way, despite image-rendering:crisp-edges? Or perhaps Chrome is incorrectly positioning the image? At least, you're showing some bits of purple/green color that should not be visible, in the testcase, near the bottom of the 3rd block here:
https://wpt.fyi/results/css/css-images/object-fit-cover-png-001c.html
https://wpt.fyi/results/css/css-images/object-fit-cover-png-002c.html

Screenshot of -002 version, in compare-view, to highlight the differing region and what the pixel-differences look like (purpleish in the left half, greenish in the right half):

There's a similar issue here where the reference case seems to be doing more color-blending than the testcase:
https://wpt.fyi/results/css/css-images/object-fit-fill-png-001c.html
https://wpt.fyi/results/css/css-images/object-fit-fill-png-002c.html
Screenshot of -002 version in compare-view:

[dholbert]

For the two issues I noted above:

• Issue (1) seems like a real bug (in Chrome) that'd be worth annotating as an expected failure.
• Issue (2) also seems like a real bug, too, but subtle enough to annotate with fuzz annotations and perhaps not worry too much about it (though I think those annotations should go in Chromium's internal WPT expected-results rather than in the test itself, since the test's expectations seem legit).
• Neither of those seem to be what your PR is about, which at first glance is just removing image-rendering:crisp-edges and then annotating the fairly-massive fuzziness that results from that removal?

[dholbert]

In general, shouldn't that be considered a Chrome bug, and annotated locally as expected-failure or expected-fuzz in Chromium's source tree rather than in the test itself?

Aha -- it occurred to me that maybe Chromium might lack a way to annotate expected-fuzz (from e.g. Chrome-specific bugs/optimizations/etc) outside of the test itself -- asked in https://matrix.to/#/#wpt:matrix.org and it seems that is indeed not something Chromium's WPT-running/annotating infra supports right now.

Jonathan Lee kindly filed this ticket to add this ability for Chromium: https://crbug.com/434057608

In the meantime, I guess we're stuck using the meta tag inside the test itself to encode any Chromium-specific expected fuzziness, which means the existing fuzzy annotations (which exist for that reason) are ~fine and can be updated as-needed; but my notes in #53930 (comment) still stand as a reason this current PR seems off-track, unless I'm missing something.

[javifernandez]

The intent in these tests is that there shouldn't be small pixel differences, since we're scaling the same image to the same target-size in all cases, and image-rendering:crisp-edges should prevent antialiasing (and hence should probably prevent uninteresting fuzzy rendering differences) along the edges of the blocks of color in the image.

First of all, I agree that there are at least 2 issues in Chrome that make these tests to fail in cases unrelated to the property being tests, which is mainly object-position:

1- the image-rendering:crisp-edge not being implemented
2- rendering differences between 'background-image' and 'canvas'.

With this PR I just wanted to look for alternatives to make these tests useful in Chrome without requiring big implementation changes. In the case of the issue 1, I guess it's going to be implemented eventually, I it may take time.

The issue 2 is trickier and I've been told that it's a long-standing issue in the Chrome rendering pipeline that it's going to be hard to get rid of. So, even with the crisp-edge feature implemented, I'm not sure we would get rid of all the small pixel differences. In fact, the original tests had already a few "fuzzy" tags in some tests.

Additionally, this PR removes the image-rendering: crisp-edges in 2 tests, which is not strictly needed for the purpose of the tests where it's used.

I'm willing to believe it's not useful for some of the cases (maybe for object-fit:none and object-fit: scale-down), but it definitely matters in all the others where upscaling is being done. It makes the blocks-of-color scale crisply, which substantially reduces opportunities for fuzz. I can see the difference visually if I load the reference cases here and toggle image-rendering:crisp-edges off-and-on in devtools.

I agree the crisp-edges has an impact visually, but as far as I understand, it doesn't affect to the test itself, as long as the scaling effects is similar in both, the test and the reference file. This seems to be the case in most of the tests for Firefox at least (I believe it's also the case for Safari). If we can agree on this, we can provide a solution for the failing tests cases; either we change the offset values or assume the differences with the fuzzy tag.

These are the test failing in Firefox just after removing the 'image-rendering: edge-crispy' feature (see the PR for details):

/css/css-images/object-fit-contain-png-001c.html 1 / 1 0 / 1
/css/css-images/object-fit-contain-png-002c.html 1 / 1 0 / 1
/css/css-images/object-fit-scale-down-png-001c.html 1 / 1 0 / 1
/css/css-images/object-fit-scale-down-png-002c.html 1 / 1 0 / 1

Also, looking at your PR, it looks like some of the testcases where you're adding fuzz are testcases that Chromium currently passes on wpt.fyi, so I wonder if some of the patch's fuzz is in fact accounting for your image-rendering removal rather than addressing actual current Chrome failures?

Umm, interesting. This shouldn't be the case, given that the image-rendering property is parsed as invalid in Chrome, so it shouldn't take any effect. Let me study those cases a bit more.

Also, looking at your PR, it looks like some of the testcases where you're adding fuzz are testcases that Chromium currently passes on wpt.fyi, so I wonder if some of the patch's fuzz is in fact accounting for your image-rendering removal rather than addressing actual current Chrome failures?

For example, your PR is adding substantial fuzz (maxDifference=0-107;totalPixels=0-144", i.e. hundreds of pixels differing by quite a lot) to these two tests, which wpt.fyi says Chrome currently passes: https://wpt.fyi/results/css/css-images/object-fit-cover-png-002e.html https://wpt.fyi/results/css/css-images/object-fit-cover-png-002i.html

I think these 2 tests fail in Safari when removing the 'edge-crisy' feature. These would be one of those cases I mentioned we should analyze better, and either assume the pixel difference or change the offset to avoid it.

Regarding the amount of pixels, I believe it's so much because it fails in the same area for all the 7 the cases defined in the test. But yeah, I agree it may cause that we miss some legit test failures, though.

Looking at the actual current Chromium failures that wpt.fyi shows for these tests (-001 tests, -002 tests), I'm seeing two different categories of issues: (1) For SVG images: It looks like Chrome isn't honoring object-fit for object and embed at all, e.g. here: https://wpt.fyi/results/css/css-images/object-fit-contain-svg-001e.html https://wpt.fyi/results/css/css-images/object-fit-contain-svg-001o.html

Yeah, that's true, although I think it the issue about 'object-fit' not being applied it was affecting "embed" only, but I'll check it again. Anyway, I agree this is a legit bug in Chrome and will be reflected as such in wpt-fyi.

(2) For PNG images: It looks like Chrome is incorrectly doing some antialiasing when scaling up an image in these tests, and perhaps doing it an inconsistent way, despite image-rendering:crisp-edges? Or perhaps Chrome is incorrectly positioning the image? At least, you're showing some bits of purple/green color that should not be visible, in the testcase, near the bottom of the 3rd block here: https://wpt.fyi/results/css/css-images/object-fit-cover-png-001c.html https://wpt.fyi/results/css/css-images/object-fit-cover-png-002c.html

These are the cases I would like the test to avoid detecting, given that the main goal of the tests is to determine if the object-position property is applied correctly. I agree that we may miss some failures caused by tiny errors in the position calculations, but perhaps it's something that we can verify with testharness based tests instead.

[dholbert]

1- the image-rendering:crisp-edge not being implemented

MDN says it's supported under a nonstandard name -webkit-optimize-contrast:
https://developer.mozilla.org/en-US/docs/Web/CSS/image-rendering

Testing that briefly locally, I'm not sure if that helps, but I think image-rendering:pixelated does help -- could you try that? Maybe that Just Works here? (We could switch to that instead of crisp-edges, except that pixelated doesn't specify which downscaling algorithm to use, so it still leaves the possibility for fuzziness for the down-scaled cases here. But I'm fine having it as a fallback value for engines that don't support crisp-edges).

In fact, the original tests had already a few "fuzzy" tags in some tests.

Those fuzzy tags were added by another Chromium engineer I think, as part of a Chromium rendering implementation change that introduced some Chromium-specific fuzziness here. (These tests didn't have any fuzzy tags when they were added to WPT, in 2b2a359 ).

I agree the crisp-edges has an impact visually, but as far as I understand, it doesn't affect to the test itself, as long as the scaling effects is similar in both, the test and the reference file.

It definitely affects the validity of the test. The default "auto" image-upscaling-behavior is unspecified and is allowed to be context-dependent (e.g. might be a bit different for video vs. img vs. canvas). Browser engines do take advantage of this flexibility, as can be seen by the fact that some of these tests fail in a fuzzy way in Chromium (without a recognized image-rendering value) and are fuzzy in Firefox when you remove the image-rendering styling -- that's a sign that we're upscaling in different ways in different contexts.

Only when specifying image-rendering do we get a better guarantee about how the upscaled image should actually render (and fill in newly-created pixels in the upscaled image, etc).

(2) For PNG images: It looks like Chrome is incorrectly doing [...] At least, you're showing some bits of purple/green color that should not be visible

These are the cases I would like the test to avoid detecting, given that the main goal of the tests is to determine if the object-position property is applied correctly.

The point is also to confirm that object-position is applied correctly, in the context of an upscaled object-fit-snapped image. These specific subtests in these testcases have an object-position that precisely pushes the pink/green area out of view. So the fact that some pink/green color is actually showing up here is an indication that object-position calculations are ending up incorrect, or (more likely) that some blur is being introduced at the seam in the image (and that blur is being handled differently in the testcase vs. reference case). But good news, it looks like maybe image-rendering:pixelated fixes that for me -- does it fix it for you?

[dholbert]

tl;dr to make these tests perform better on Chromium, I'd be happy swapping all

        image-rendering: crisp-edges;

...for:

        image-rendering: pixelated; /* for UAs that don't support crisp-edges */
        image-rendering: crisp-edges;

Maybe this largely addresses your concerns? If there's still substantial fuzziness/failures at that point, I'd be curious to learn more. (But hopefully any failures at that point are pretty minimal.)

I think crisp-edges is still a bit better to use than pixelated here (hence, I'd like to keep it as the preferred/last CSS keyword to use in these tests), since it prescribes behavior more precisely even when downscaling, which gives these tests the predictability that they need to be as valid as they can be.

(Pixelated allows unspecified UA-defined smoothing when scaling to a non-integer value -- which includes downscaling -- as resolved in w3c/csswg-drafts#5837 and specced in https://drafts.csswg.org/css-images-3/#valdef-image-rendering-pixelated . This is what I was alluding to a bit more hand-wavily in my previous comment when I said " pixelated doesn't specify which downscaling algorithm to use, so it still leaves the possibility for fuzziness for the down-scaled cases here.)

[javifernandez]

Yeah, using image-rendering: pixelated definitively helps. Most of the interop issues caused by small pixel differences in edges are gone with this.

I agree it sounds like a valid fallback for the engines not implementing crisp-edges or even for the tests that while not using it, still cause failures due to the different scaling algorithms implemented. However, I've investigated a bit how "pixelated" is implemented in Chrome and I've found 2 issues:

1- It implies using kInterpolationNone in the HTMLCanvarPainter class, which as far as I could understand would mean applying not interpolation algorithm at all when scaling the image.
2- It deactivates the Display List codepath in the rendering pileline. I don't think this issue has any impact in the expected result of the tests, but I'm going to study it a bit more.

In summary, I agree that using "pixelated" as a fallback would be the best solution to solve the interop issues, at elast until Chrome implements "crisp-edges". I'm not an expert on the Chrome graphics codebase, so I'm going to study a bit more the implications of using "pixelated" and probably asking some Google folks for a review.

In any case, thanks a lot for your feedback Daniel.

[dholbert]

Yeah, using image-rendering: pixelated definitively helps. Most of the interop issues caused by small pixel differences in edges are gone with this.

Hooray! Maybe/hopefully that means the already-existing fuzzy annotations in the tests themselves can be removed, too.

I've investigated a bit how "pixelated" is implemented in Chrome and I've found 2 issues:
1- It implies using kInterpolationNone in the HTMLCanvarPainter class, which as far as I could understand would mean applying not interpolation algorithm at all when scaling the image.

That's intentional & fine, I think - these tests are specifically meant to assert that the sizing & positioning of the image (as influenced by object-{fit,position}) are precise & consistent across all these host-elements/scenarios. Since image-scaling may blend pixels slightly differently across these various scenarios, the test is intentionally trying not to exercise any fancy pixel interpolation (hence the image-rendering decl).

2- It deactivates the Display List codepath in the rendering pileline.

I see. If that means that Chromium's typical user-facing codepath isn't getting exercised by these tests, then that might be a reason to add additional copies of these tests, without image-rendering, to get code-coverage for that pipeline (with whatever fuzzy annotations are needed -- or perhaps with the test simplified to use a single solid color-block, if that's good enough to get some base level of coverage & doesn't require fuzz).

[dholbert]

In any case, thanks a lot for your feedback Daniel.

Sure! Thanks for the discussion & being willing to reevaluate this. :)

[fsoder]

Setting image-rendering: pixelated will certainly eliminate some amount of unpredictability (filter heuristics) here from a Chromium PoV - at least for the <canvas> cases.

The object-position failures are interesting in that they seem to be more "consistent", so I'd suspect that the differences there (in Chromium) may come from the reference side (background-image rendering is riddled heuristics, so it's difficult to say what the source may be without inspecting the emitted draw commands - based on the diffs I'd guess a slight difference in the source rectangle of the draws).

The "Display List codepath" is nothing to worry about here I think, since that should only apply when printing AFAIU.

[fsoder]

Oh, I guess the <canvas>es may be promoted to layers as well, which could be another source of difference for those.

[javifernandez]

The object-position failures are interesting in that they seem to be more "consistent", so I'd suspect that the differences there (in Chromium) may come from the reference side (background-image rendering is riddled heuristics, so it's difficult to say what the source may be without inspecting the emitted draw commands - based on the diffs I'd guess a slight difference in the source rectangle of the draws).

This is certainly what happens in the case of the SVG tests; for instance:

• object-position-svg-001i.html vs object-position-svg-001-ref.html

[javifernandez]

I have changed the title and the commit, so that this PR just adds 'pixelated' as fallback. It seems it helps to get rid of a bunch of interop issues, at least for now.

If @dholbert, @fsoder and @annevk accept the change we could merge this and I'd investigate the other issues (eg, SVG pixel differences and lack of support in elements) in a different PR.

[dholbert]

Looks good to me! One request - could you adjust the template versions of these tests in the tools subdirectory here? If we ever wanted to regenerate these tests, we'd want to have those templates up-to-date.

(I'm also surprised the object-position tests didn't already have image-rendering: crisp-edges - I guess when I added these tests, they didn't need that in Firefox. But they do do some image-scaling and hence may need that.)