shithub: dav1d

Info  •  Files  •  Log  •  Branches

Re-add imax(v, 0) in SGR calculation

Apparently this can happen for bitdepth > 8. I haven't seen it
happen for bitdepth==8. Fixes #161.

--- a/src/looprestoration_tmpl.c

+++ b/src/looprestoration_tmpl.c

@@ -441,7 +441,7 @@

             const int b =

                 (BB[i] + (1 << (BITDEPTH - 8) >> 1)) >> (BITDEPTH - 8);

-            const unsigned p = a * n - b * b;

+            const unsigned p = imax(a * n - b * b, 0);

             const unsigned z = (p * s + (1 << 19)) >> 20;

             const int x = dav1d_sgr_x_by_xplus1[imin(z, 255)];