Overview
The Reftest harness compares the display of two web pages. If the bitmaps resulting from displaying the two files in an 800x1000 window are identical, the test passes. If they differ, the test fails. Or, alternatively, the conditions can be reversed. The power of the tool comes from the fact that there is more than one way to achieve any given visual effect in a browser. So, if the effect of complex markup is being tested, put that complex markup into a page and create another page that uses simple markup to achieve the same visual effect. Reftest will then compare them and verify whether they produce the same bitmap.
This idea can seem odd when first encountered. Automated testing usually
compares output against an invariant, a "gold standard", that is determined
to be correct. If one has software that multiplies numbers, one wants a
regression test to show that 2 * 2 continues to be calculated to be 4, not
something similar to but not quite exactly 4. But an operating system does
change with time. It is not invariant. And a browser may change the visual
effect produced by a tag while still being compliant with relevant
standards. For example, the HTML 4.01 specification at the W3C specifies
that text inside of a <blockquote> will be indented, but
it does not specify the number of pixels of the indentation. If a browser
changes the depth of the indenting and the visual construct is tested against
an invariant, the test would appear to fail. But the test should not fail,
unless the <blockquote> element did not cause any
indentation at all. If a regression test harness has false failures, it makes
the harness not trustworthy and the harness will not be used.
Reftests cannot rely on behaviour that requires elevated privileges. This is because build process runs them in a profile that does not automatically grant them. Requesting elevated privileges will display the standard security alert, and will likely cause the test to hang or timeout. Any tests that require such privileges to work correctly would most likely be better written using the Mochitest framework.
Source code
- {{ Source("layout/tools/reftest", "Harness") }}
- {{ Source("layout/reftests", "Tests") }}