Motion vector refinement for high-performance transcoding

TLDR: This paper shows that the incoming motion vectors become nonoptimal due to the reconstruction errors, and proposes a fast-search adaptive motion vector refinement scheme capable of providing video quality comparable to that can be achieved by performing a new full-scale motion estimation but with much less computation.

Abstract: In transcoding, simply reusing the motion vectors extracted from an incoming video bit stream may not result in the best quality. In this paper, we show that the incoming motion vectors become nonoptimal due to the reconstruction errors. To achieve the best video quality possible, a new motion estimation should be performed in the transcoder. We propose a fast-search adaptive motion vector refinement scheme that is capable of providing video quality comparable to that can be achieved by performing a new full-scale motion estimation but with much less computation. We discuss the case when some incoming frames are dropped for frame-rate conversions, and propose motion vector composition method to compose a motion vector from the incoming motion vectors. The composed motion vector can also be refined using the proposed motion vector refinement scheme to achieve better results.

Figures

TABLE I PERFORMANCE OBTAINED USING DIFFERENT MOTION VECTOR COMPOSITION METHODS. INCOMING BIT-STREAM OF 128 Kb/sAND 30 frames/s WAS TRANSCODED INTO 50 Kb/s AND 10 frames/s

Fig. 9. Performance of motion vector refinement with frame-rate conversion (“Foreman”). Incoming bit stream at 128 Kb/s with 30 Kb/s (300 frames) was transcoded to 32 Kb/s with 10 frames/s (search range of 2 pixels applied).

Fig. 1. Transcoding by the cascade of a decoder and an encoder.

Fig. 2. Motion estimation.

Fig. 4. Performance of motion vector refinement (“Carphone” of QCIF format, 300 frames). Incoming bit stream at 128 Kb/s was transcoded to 32 Kb/s with 30 frames/s. The search range for full-scale motion estimation is 15 pixels, and the search range for motion vector refinement is 2 pixels throughout the paper.

Fig. 3. Structure of a cascaded transcoder.

Frequently asked questions

Q1. What have the authors contributed in "Motion vector refinement for high-performance transcoding" ?

In this paper, the authors show that the incoming motion vectors become nonoptimal due to the reconstruction errors. The authors propose a fast-search adaptive motion vector refinement scheme that is capable of providing video quality comparable to that can be achieved by performing a new full-scale motion estimation but with much less computation. The authors discuss the case when some incoming frames are dropped for frame-rate conversions, and propose motion vector composition method to compose a motion vector from the incoming motion vectors. 

Q2. What is the effect of a coarser quantization step size on the composed motion vector?

the composed motion vector may have degraded performance due to the effect of reconstruction errors when a coarser quantization step size is applied during the transcoding. 

Q3. What is the need for motion vector refinement?

In general, the need for motion vector refinement depends on the effect of the reconstruction errors relative to the strength of the motioncompensated prediction residual signal as shown previously in (5), and thus is signal dependent. 

Q4. What is the advantage of FDVS over the bilinear interpolation scheme?

Another advantage of FDVS over the bilinear interpolation scheme is that when multiple frames are dropped, it can be processed in the forward order, eliminating the multiple memories needed to store the incoming motion vectors of all the dropped frames. 

Q5. How is the performance of the proposed FDVS method?

The performance of the proposed FDVS method is about 1.7 dB (foreman) and 0.8dB (carphone) better than the bilinear interpolation. 

Q6. What is the way to find a motion vector pointing to a block in frame?

Since frame ( ) is dropped, for MB , the authors need to find a motion vector pointing to a block in frame ( ) which matches well with MB . 

Q7. What is the performance of motion vector refinement?

the refinement of the incoming motion vectors using a small search window (e.g., search range of 2 pixels) increases the performance close to that of the full-scale fullsearch motion estimation. 

Q8. What is the way to compute the delta motion vector?

A delta motion vector ( )can be estimated within a new search window , around the point indicated by the base motion vector:SAD (6)SAD(7)The new search window can be set much smaller than the full-scale window (e.g., a search range of 2 pixels instead of 15 pixels or larger) and still produce almost the same video quality as the full-scale motion estimation. 

Q9. What is the effect of the quantization step size difference?

Fig. 11 implies that when the quantization step size difference is small, the distortion caused by the reuse of incoming motion vector is small. 

Q10. What is the way to obtain the optimal motion vector?

the optimal motion vector can be easily obtained by refining the incoming motion vector within a small range as opposed to applying a full-scale motion estimation [16], [17]. 

Q11. What is the effect of the quantization error on the incoming motion vector?

since in general there is no guarantee that the effect is negligible all the time, there are nonzero probabilities that the quantization errors may cause the incoming motion vector to be nonoptimal [i.e., the authors can find a better motion vector which minimizes (4)]. 

Q12. What is the speed-up ratio for the motion vector refinement scheme?

Starting with the base motion vector, the motion vector refinement scheme searches for a delta motion vector within a search area much smaller than that of the full-scale motion estimation.