Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too small test images leads to falsely successful tests #426

Open
strindhaug opened this issue Aug 29, 2023 · 0 comments
Open

Too small test images leads to falsely successful tests #426

strindhaug opened this issue Aug 29, 2023 · 0 comments
Labels

Comments

@strindhaug
Copy link

I was writing a fix and test case for
#425
when I noticed that despite my fix being stashed all the tests succeeded!
So I assumed maybe my fix wasn't needed after all somehow, but just to be sure I changed the test to be definitely breaking by including the wrong svg; but to my surprise it still succeded even though it printed 404 not found messages in the log.

I then discovered that the compareToRefImage function that most of the tests use compare the two images with a 0.1 threshold and counts it as okay if less than 100 px are different.
Which would be fine if most of the test images were quite large so that 100px would account for small font-rendering and anti aliasing differences (though I would have thought that's what the threshold is for).

But since the test images are quite small, and several of them, such as test/resources/svg-use-tag/image:

testImage

which I was basing my new test on, is mostly transparent pixels even when the generated image is completely empty the test still succeeds.
I manually roughly counted the non-transparent pixels of the test image which seems to be around 127px and presumably at least around 60 of these are below the 0.1 threshold to be considered identical to a fully transparent pixel, meaning when I changed thecompareToRefImage to expect less than 1 px not 100px it now failed with only 68px difference from the reference image when generating a completely missing SVG.

Expected Behavior

Failing tests should fail

Current Behavior

Failing tests sometimes succeed

Possible Solution

Rewrite the tests to use much larger test images where the most of the image is non-transparent pixels so that a completely different or missing image cannot accidentally "pass" the test.

Also consider changing that 100px allowed difference to something much smaller too.

But with the current 100px allowance, all the test images and reference images should have at least 1000 non-transparent pixels so that there is no way that a completely missing image could be less than a 100 px difference.

Steps To Reproduce

  1. ...
  2. ...
  3. ...
Error Message & Stack Trace

<!-- Provide a log message if relevant -->

Additional Context

Your Environment

  • html-to-image: [e.g. 0.1.0]
  • OS: [e.g. macOS Sierra 10.12.3]
  • Browser: [e.g. chrome 78.0.3904.108]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant