Pyramidal implementation of the Lucas Kanade feature tracker description of the algorithm

TLDR: It is essential to define the notion of similarity in a 2D neighborhood sense and the image velocity d is defined as being the vector that minimizes the residual function defined as follows.

Abstract: 1 Problem Statement Let I and J be two 2D grayscaled images. The two quantities I(x) = I(x, y) and J(x) = J(x, y) are then the grayscale value of the two images are the location x = [x y] , where x and y are the two pixel coordinates of a generic image point x. The image I will sometimes be referenced as the first image, and the image J as the second image. For practical issues, the images I and J are discret function (or arrays), and the upper left corner pixel coordinate vector is [0 0] . Let nx and ny be the width and height of the two images. Then the lower right pixel coordinate vector is [nx − 1 ny − 1] . Consider an image point u = [ux uy] on the first image I. The goal of feature tracking is to find the location v = u + d = [ux+dx uy +dy] on the second image J such as I(u) and J(v) are “similar”. The vector d = [dx dy] is the image velocity at x, also known as the optical flow at x. Because of the aperture problem, it is essential to define the notion of similarity in a 2D neighborhood sense. Let ωx and ωy two integers. We define the image velocity d as being the vector that minimizes the residual function defined as follows: