Simon Niederberger wrote:

> I'm sure you're aware that the term "best practice" is coined as such and
> not to be taken to seriously.

Taking "Best Practice" terms seriously is a Best Practice.

> How a human would determine a correct document? Take a hard copy of the
> the expected result, put the actual result on top and hold the two against
> a light. If there are no mismatches, test is good.
>
> Can I automate that?

That's called a "golden copy" unit test. Just byte-compare the two files.
Then get ready for all this technique's special risks.

Each time the copy improves, you should hit a button that updates the copy
in your test data. Check the copy into your version controller, and reject
any change that makes the copy fail unexpectedly. If you indeed upgrade the
image, you then expect the test to fail. You manually inspect the output to
confirm a bug did not slip in at the same time (and this adds risk), and
then you upgrade the golden copy.

Time stamps and other stochastic things will make the test fail, so you will
need to fudge, or remove the time stamps going into the file, or something.

Golden copy is very fast. Some folks do it for JPG files and such. Videogame
programmers may even do it with output movies. You may even upgrade your
byte comparison into a comparison that renders the output into a raster, and
applies a "close-enough" test to each color.

And it's super-high risk because a test failure will not indicate which
module broke. So run the tests with your unit tests, as your developers edit
their code. Then the module that broke must perforce be the one they edited
most recently. And augment the tests with white-box tests.

Next, you can output a simpler version of the system, just for testing. A
test on PNG files might start by testing a simple black square. Or a test on
GraphViz's DOT system would switch its output from PNG to SVG. You can parse
that and compare the results as text, not as bytes. Tests on PDF might have
the option to output raw Postscript.

Such tests should still be considered legacy retro-fits. A system designed
for testing would not need the golden-copy system. If you must output PDF,
it's not designed for testing, so use golden-copy.

--
Phlip
http://c2.com/cgi/wiki?ZeekLand <-- NOT a blog!!