Try a different material from the MXM Gallery instead, you will probably have better luck with it than this one.

i think there's atleast one similar to this one on there you can test.
It's probably a combination of both the 8bit problem with the displacement and a possible polygonal problem with your mesh, as you said. Both..
/ Max