Sorry for the double post. Sent the first one per mail and when it didn't appear
I posted again on the group page, which then told me that the post was awaiting
moderation.

Anyway, I just got an answer from Holger Will and thought I'd repeat it here for
all the other souls seeking for a solution to this problem:

The solution is that SVG by default uses the input in the LinearRGB color space,
which strangely means that the input isn't interpreted linearly. Simply switch
to the sRGB color mode and everything works like any sane person would expect it
to work in the first place:

<svg:feDisplacementMap color-interpolation-filters="sRGB" in="image"
in2="displacement" scale="255" xChannelSelector="R" yChannelSelector="G" />