# Pyramid Vector Quantization for Video Coding

Mozilla

           331 E. Evelyn Avenue
           Mountain View
           CA
           94041
           USA

         jmvalin@jmvalin.ca

This proposes applying pyramid vector quantization (PVQ) to video coding.

# Introduction

This draft describes a proposal for adapting the Opus RFC 6716 energy conservation principle to video coding based on a pyramid vector quantizer (PVQ) . One potential advantage of conserving energy of the AC coefficients in video coding is preserving textures rather than low-passing them. Also, by introducing a fixed-resolution PVQ-type quantizer, we automatically gain a simple activity masking model.
The main challenge of adapting this scheme to video is that we have a good prediction (the reference frame), so we are essentially starting from a point that is already on the PVQ hyper-sphere, rather than at the origin like in CELT. Other challenges are the introduction of a quantization matrix and the fact that we want the reference (motion predicted) data to perfectly correspond to one of the entries in our codebook.

# Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

# Gain-Shape Coding and Activity Masking

The main idea behind the proposed video coding scheme is to code groups of DCT coefficient as a scalar gain and a unit-norm "shape" vector. A block's AC coefficients may all be part of the same group, or may be divided by frequency (e.g. by octave) and/or by directionality (horizontal vs vertical).
It is desirable for a single quality parameter to control the resolution of both the gain and the shape. Ideally, that quality parameter should also take into account activity masking, that is, the fact that the eye is less sensitive to regions of an image that have more details. According to Jason Garrett-Glaser, the perceptual analysis in the x264 encoder uses a resolution proportional to the variance of the AC coefficients raised to the power a, with a=0.173. For gain-shape quantization, this is equivalent to using a resolution of g^(2a), where g is the gain. We can derive a scalar quantizer that follows this resolution: b g=Q_g gamma , where gamma is the gain quantization index, b=1/(1-2*a) and Q_g is the gain resolution and main quality parameter.
An important aspect of the current proposal is the use of prediction. In the case of the gain, there is usually a significant correlation with the gain of neighboring blocks. One way to predict the gain of a block is to compute the gain of the coefficients obtained through intra or inter prediction. Another way is to use the encoded gain of the neighboring blocks to explicitly predict the gain of the current block.

# Householder Reflection

Let vector x_d denote the (pre-normalization) DCT band to be coded in the current block and vector r_d denote the corresponding reference (based on intra prediction or motion compensation), the encoder computes and encodes the "band gain" g = sqrt(x_d^T x_d). The normalized band is computed as x_d x = --------- , || x_d || with the normalized reference r similarly computed based on r_d. The encoder then finds the position and sign of the maximum value in r: m = argmax_i | r_i | s = sign(r_m) and computes the Householder reflection that reflects r to -s e_m. The reflection vector is given by v = r + s e_m . The encoder reflects the normalized band to find the unit-norm vector v^T x z = x - 2 ----- v . v^T v
The closer the current band is from the reference band, the closer z is from -s e_m. This can be represented either as an angle, or as a coordinate on a projected pyramid.

# Angle-Based Encoding

Assuming no quantization, the similarity can be represented by the angle theta = arccos(-s z_m) . If theta is quantized and transmitted to the decoder, then z can be reconstructed as z = -s cos(theta) e_m + sin(theta) z_r , where z_r is a unit vector based on z that excludes dimension m.
The vector z_r can be quantized using PVQ. Let y be a vector of integers that satisfies sum_i(|y[i]|) = K , with K determined in advance, then the PVQ search finds the vector y that maximizes y^T z_r / (y^T y) . The quantized version of z_r is y z_rq = ------- . || y ||
If we assume that MSE is a good criterion for optimizing the resolution, then the angle quantization resolution should be (roughly) dg 1 b Q_theta = ---------*----- = ------ . d(gamma) g gamma To derive the optimal K we need to consider the normalized distortion for a Laplace-distributed variable found experimentally to be approximately (N-1)^2 + C*(N-1) D_p = ----------------- , 24*K^2 with C ~= 4.2. The distortion due to the gain is b^2*Q_g^2*gamma^(2*b-2) D_g = ----------------------- . 12 Since PVQ codes N-2 degrees of freedom, its distortion should also be (N-2) times the gain distortion, which eventually leads us to the optimal number of pulses gamma*sin(theta) / N + C - 2 \ K = ---------------- sqrt | --------- | . b \ 2 /
The value of K does not need to be coded because all the variables it depends on are known to the decoder. However, because Q_theta depends on the gain, this can lead to unacceptable loss propagation behavior in the case where inter prediction is used for the gain. This problem can be worked around by making the approximation sin(theta)~=theta. With this approximation, then tau is equal to the inverse of the theta quantization index, with no dependency on the gain. Alternatively, instead of quantizing theta, we can quantize sin(theta) which also removes the dependency on the gain. In the general case, we quantize f(theta) and then assume that sin(theta)~=f(theta). A possible choice of f(theta) is a quadratic function of the form: 2 f(theta) = a1 theta - a2 theta. where a1 and a2 are two constants satisfying the constraint that f(pi/2)=pi/2. The value of f(theta) can also be predicted, but in case where we care about error propagation, it should only be predicted from information coded in the current frame.

# Bi-prediction

We can use this scheme for bi-prediction by introducing a second theta parameter. For the case of two (normalized) reference frames r1 and r2, we introduce s1=(r1+r2)/2 and s2=(r1-r2)/2. We start by using s1 as a reference, apply the Householder reflection to both x and s2, and evaluate theta1. From there, we derive a second Householder reflection from the reflected version of s2 and apply it to z. The result is that the theta2 parameter controls how the current image compares to the two reference images. It should even be possible to use this in the case of fades, using two references that are before the frame being encoded.

# Coefficient coding

Encoding coefficients quantized with PVQ differs from encoding scalar-quantized coefficients from the fact that the sum of the coefficients magnitude is known (equal to K). It is possible to take advantage of the known K value either through modeling the distribution of coefficient magnitude or by modeling the zero runs. In the case of magnitude modeling, the expection of the magnitude of coefficient n is modeled as K_n E(|y_n|) = alpha * ----- , N - n where K_n is the number of number of pulses left after encoding coeffients from 0 to n-1 and alpha depends on the distribution of the coefficients. For run-length modeling, the expection of the position of the next non-zero coefficient is given by N - n E(|run|) = beta * ----- , K_n where beta also models the coefficient distribution.

# Development Repository

The algorithms in this proposal are being developed as part of Xiph.Org's Daala project. The code is available in the Daala git repository at . See for more information.

# IANA Considerations

This document makes no request of IANA.

# Security Considerations

This draft has no security considerations.

# Acknowledgements

Thanks to Jason Garrett-Glaser, Timothy Terriberry, Greg Maxwell, and Nathan Egge for their contribution to this document.

Normative References

Informative References

# A Pyramid Vector Quantizer