Thanks for poking with the test ! It is good to have skepticism, because it helps reveal potential issues. The point raised earlier about patterning has some merit and will be investigated.markps wrote:As I understand the histogram will give you an average of brightness of the entire image and will not indicate the noise contrast on a patch of the image right?
However, this last point you are trying to make, although I can see the validity of your individual components, they do not tie in holistically into this test (at least not in a glaring way).
Yes the histogram will give an average and when the two scenes are setup to be the same then there is little expectation of them having wildly different histogram values on non corresponding portions. If they have rough areas then they tend to be on the same locations (as indicated by noise isolates in the test).
Now, if you can manage to render a scene (the same identical scene) once in Beta and once in V1.0 and show opposing noise contrast areas and reverse opposing contrast areas (on the other render) in non-corresponding locations so as to average in the same exact histogram mean/median values (+/- 1), then it would be nice to see. (Even if the noise areas somehow manage to be swapped the level of entropy is still the same and this is the metric that matters ... the amount of overal "heat" of the images at a specific time. We are cooling the same pie in two refrigerators of the same company).
Also .... in your example:


The left image has a mean/median of 152/127
The right has a mean/median of 190/127
The test accounts for both values simultaneously. In your example both values do not correspond within +/- 1