goal
In this chapter, it will be studied:
- Understanding the concept of optical flow and its estimation using Lucas-Kanade methods
- utilization
()and other functions to track feature points in the video - utilization
()method creates a dense optical flow field
optical current
Optical flow is a pattern of apparent movement of an image object between two consecutive frames caused by the motion of an object or camera. It is a two-dimensional vector field where each vector is a displacement vector showing the movement of points from the first frame to the second. It is shown in the figure below.
It shows a ball moving in 5 consecutive frames. The arrows indicate its displacement vector. Optical flow has many applications in the following areas:
- Movement structure
- video compression
- Video Stabilization
- ...
Optical flow is based on several assumptions:
- The pixel intensity of an object does not change between consecutive frames
- Neighboring pixels have similar motion
(Check the similarity of the inverse matrices with the Harris corner point detector. It indicates that corner points are better tracking points.)
So from the user's point of view, the idea is simple, giving some points to be tracked and receiving the optical flow vectors for those points. However, there are some problems. So far, what is being processed is small movements, so it fails when there are very large movements. To deal with this, a pyramid needs to be used. When sampling on the pyramid, small movements are eliminated and large movements become smaller. So, by applying Lucas-Kanade there, the optical flow is combined with scaling.
Lucas-Kanade optical flow in OpenCV
OpenCV() All these features are provided. Below create a simple application to track certain points in a video. To determine the points, use the()The first frame is taken and some of the Shi-Tomasi corner points are detected. The first frame is taken, some of the Shi-Tomasi corner points are detected in it, and then these points are tracked iteratively using the Lucas-Kanade optical flow. For the function()that passes the previous frame, the point before it, and the next frame. It returns the point after it along with some state number that has a value of 1 if the point after it is found and zero otherwise. Iteratively pass these points as preceding points in the next step, and the iteration proceeds.
import cv2
import numpy as np
video_file = 'slow_traffic_small.mp4'
cap = (video_file)
# params for ShiTomasi corner detection
feature_params = dict( maxCorners = 100,
qualityLevel = 0.3,
minDistance = 7,
blockSize = 7 )
# Parameters for lucas kanade optical flow
lk_params = dict(
winSize = (15, 15),
maxLevel = 2,
criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))
# create some random colors
color = (0, 255, (100, 3))
# Take first frame and find corners in it
ret, old_frame = ()
old_gray = (old_frame, cv2.COLOR_BGR2GRAY)
p0 = (old_gray, mask=None, **feature_params)
# create a mask image for drawing purpose
mask = np.zeros_like(old_frame)
while True:
ret, frame = ()
if ret:
frame_gray = (frame, cv2.COLOR_BGR2GRAY)
# calculate optical flow
p1, st, err = (old_gray, frame_gray, p0, None, **lk_params)
# Select good points
if p1 is not None:
good_new = p1[st==1]
good_old = p0[st==1]
# draw the tracks
for i,(new, old) in enumerate(zip(good_new, good_old)):
a, b = ()
c, d = ()
mask = (mask, (int(a), int(b)), (int(c), int(d)), color[i].tolist(), 2)
frame = (frame, (int(a), int(b)), 5, color[i].tolist(), -1)
img = (frame,mask)
('frame',img)
k = (30) & 0xff
if k == 27:
()
break
# Now update the previous frame and previous points
old_gray = frame_gray.copy()
p0 = good_new.reshape(-1,1,2)
else:
()
break
The above code does not check the correctness of the next key point. Therefore, even if any feature point in the image disappears, it is possible for the optical flow to find the next point that looks like it might be close to it. (In fact, for robust tracking, corner points should detect points at specific time intervals. openCV sample presents an example where it finds a feature point every 5 frames, and also runs a backward check on the optical flow points, selecting only the good ones.) The code is available at (/opencv/open…)
The results are as follows:
Dense optical flow in OpenCV
Lucas-Kanade method for computing optical flow with sparse feature sets(In the example, corner points detected using the Shi-Tomasi algorithm).OpenCV provides another algorithm to find dense optical flow.It calculates the optical flow at all points in the frame. ItAlgorithm based on Gunner FarnebackThe algorithm is explained in Gunner Farneback's "Two-Frame Motion Estimation Based on Polynomial Expansion" in 2003.
The following example shows how to find a dense optical flow using the above algorithm. First a 2-channel vector with optical flow vectors (u,v)(u,v)(u,v) is obtained and their magnitudes and directions are found. The results are color coded for better visualization. The direction corresponds to the hue value (Hue) of the image and the magnitude corresponds to the value screen. The code is as follows:
import cv2
import numpy as np
cap = ("")
ret, frame1 = ()
frame1_gray = (frame1, cv2.COLOR_BGR2GRAY)
hsv = np.zeros_like(frame1)
hsv[...,1] = 255
while(1):
ret, frame2 = ()
if ret:
frame2_gray = (frame2, cv2.COLOR_BGR2GRAY)
flow = (frame1_gray, frame2_gray, None, 0.5, 3, 15, 3, 5, 1.2, 0)
mag, ang = (flow[...,0], flow[...,1])
hsv[...,0] = ang*180//2
hsv[...,2] = (mag, None, 0, 255, cv2.NORM_MINMAX)
bgr = (hsv, cv2.COLOR_HSV2BGR)
('frame2', bgr)
k = (30) & 0xff
if k == 27:
()
break
elif k == ord('s'):
('',frame2)
('',bgr)
frame1_gray = frame2_gray
else:
()
break
Additional resources
- /4.1.2/d4/de…
- ()
- ()
- /wiki/Optica…
This is the OpenCV Optical Flow example of the details, more information about the OpenCV Optical Flow, please pay attention to my other related articles!