Optical Flow 的简单介绍。

光流法是利用图像序列中像素在时间域上的变化以及相邻帧之间的相关性来找到上一帧跟当前帧之间存在的对应关系，从而计算出相邻帧之间物体的运动信息的一种方法。

一言以概之：所谓光流就是瞬时速率，在时间间隔很小（比如视频的连续前后两帧之间）时，也等同于目标点的位移

Recovering motion

• Feature-tracking – Extract visual features (corners, textured areas) and “track” them over multiple frames

• Optical flow – Recover image motion at each pixel from spatio-temporal image brightness variations (optical flow)

Estimating the motion of every pixel in a sequence of images is a problem with many applications in computer vision, such as image segmentation, object classification,visual odometry, and driver assistance.

In general, optical flow describes a sparse or dense vector field, where a displacement vector is assigned to certain pixel position, that points to where that pixel can be found in another image. In the context of scene flow estimation, which is performed on images with additional depth values, every pixel is assigned a depth displacement as well.

光流（optic flow）是什么呢？名字很专业，感觉很陌生，但本质上，我们是最熟悉不过的了。因为这种视觉现象我们每天都在经历。从本质上说，光流就是你在这个运动着的世界里感觉到的明显的视觉运动（呵呵，相对论，没有绝对的静止，也没有绝对的运动）。例如，当你坐在火车上，然后往窗外看。你可以看到树、地面、建筑等等，他们都在往后退。这个运动就是光流。而且，我们都会发现，他们的运动速度居然不一样？这就给我们提供了一个挺有意思的信息：通过不同目标的运动速度判断它们与我们的距离。一些比较远的目标，例如云、山，它们移动很慢，感觉就像静止一样。但一些离得比较近的物体，例如建筑和树，就比较快的往后退，然后离我们的距离越近，它们往后退的速度越快。一些非常近的物体，例如路面的标记啊，草地啊等等，快到好像在我们耳旁发出嗖嗖的声音。

光流除了提供远近外，还可以提供角度信息。与咱们的眼睛正对着的方向成90度方向运动的物体速度要比其他角度的快，当小到0度的时候，也就是物体朝着我们的方向直接撞过来，我们就是感受不到它的运动（光流）了，看起来好像是静止的。当它离我们越近，就越来越大（当然了，我们平时看到感觉还是有速度的，因为物体较大，它的边缘还是和我们人眼具有大于0的角度的）。

光流的计算较为耗时，以往我们能够实时应用的光流基本都是稀疏光流（如Lucas Kanade光流法，可用于物体跟踪等。其基本思想是提取一些好的特征点 GoodFeaturestoTrack，对这些特征点利用邻域光流一致性的假设进行计算）。而稠密光流（Dense flow）则只是一种理论上的东西。这样的尴尬局面被GPU完全打破了！

使用范围

Uses of motion

• Improving video quality – Motion stabilization – Super resolution • Segmenting objects based on motion cues • Tracking objects • Recognizing events and activities

Segmenting objects based on motion cues

Background subtraction – A static camera is observing a scene – Goal: separate the static background from the moving foreground

<src=“https://ftp.bmp.ovh/imgs/2020/05/3896b9b068c28ca4.png" heigth=“80%” width =“80%">

Tracking objects

• Facing tracking on openCV

<src =“https://ftp.bmp.ovh/imgs/2020/05/7476ec4549282426.png" height=“80%” width=“80%">

displacement: 迁移，displacement vector

Optical flow works on several assumptions:

  1. The pixel intensities of an object do not change between consecutive frames. （这种假设在… 中是不一定存在的）
  2. Neighbouring pixels have similar motion.

Recent breakthroughs in computer vision research have allowed machines to perceive its surrounding world through techniques such as object detection for detecting instances of objects belonging to a certain class and semantic segmentation for pixel-wise classification.

Let us begin with a high-level understanding of optical flow.

Sparse optical flow gives the flow vectors of some “interesting features” (say few pixels depicting the edges or corners of an object) within the frame while Dense optical flow, which gives the flow vectors of the entire frame (all pixels) - up to one flow vector per pixel. As you would’ve guessed, Dense optical flow has higher accuracy at the cost of being slow/computationally expensive.

Sparse optical flow selects a sparse feature set of pixels (e.g. interesting features such as edges and corners) to track its velocity vectors (motion). The extracted features are passed in the optical flow function from frame to frame to ensure that the same points are being tracked. There are various implementations of sparse optical flow, including the Lucas–Kanade method, the Horn–Schunck method, the Buxton–Buxton method, and more. We will be using the Lucas-Kanade method with OpenCV, an open source library of computer vision algorithms, for implementation.

We’ve previously computed the optical flow for a sparse feature set of pixels. Dense optical flow attempts to compute the optical flow vector for every pixel of each frame. While such computation may be slower, it gives a more accurate result and a denser result suitable for applications such as learning structure from motion and video segmentation. There are various implementations of dense optical flow. We will be using the Farneback method, one of the most popular implementations, with using OpenCV, an open source library of computer vision algorithms, for implementation.

Introduction to Motion Estimation with Optical Flow