An ASABE Meeting Presentation

DOI: 10.13031/aim.201700817
Paper Number: 1700817

An improved multiple target tracking method in the pigpen

based on kernel correlation filter by thermal infrared video

Li Maa,b, Gang Liua,*, Yixin Caia, Ze Zonga

a
Key Laboratory of Modern Precision Agriculture System Integration Research, Ministry of Education,
China Agricultural University, Beijing 100083, China
b
College of Information Science and Technology, Agricultural University of Hebei, Baoding 071001,
China
Written for presentation at the
2017 ASABE Annual International Meeting
Sponsored by ASABE
Spokane, Washington
July 16-19, 2017

ABSTRACT. The rapid turning movement of the pig often results in the tracking frame drift or loss of the target, resulting in
reduced tracking accuracy. In this paper, an improved multi-objective KCF tracking algorithm is proposed using thermal
infrared video as a video source. The tracking window is optimized to be rounded to avoid the effects of motion steering;
the traditional KCF algorithm is improved so that it can achieve multiple pig target tracking. The experimental data are
three videos containing two, three, and five pigs, respectively. A total of 90 videos, including pitching, walking, steering,
adhesion, non-blocking and occlusion, were selected for testing, and the DCF and MOSSE algorithms were also improved
to multi-target tracking. The results show that the improved tracking accuracy of MKCF algorithm is better than that of
MDCF and MMOSSE, and the tracking time satisfies the requirement of real - time.
Keywords. Kernel correlation filter, Multi - target tracking, Target detection, Pigsty scene.

Instructions
Pig behavior detection, especially pig target detection and tracking is a hot topic in fine animal husbandry research
recently (Duan et al., 2015; Gronskyte et al., 2015) . At present and most of the sows, piglets, fattening pigs are kept in small
pigpen. Scatter breeding is an effective way to protect animal welfare (Guo et al., 2015) . There are a large number of sows,
piglets and fattening pigs are raised in a small circle. Researchers on the study of animal behavior, there is no effective
means of detection, can only rely on artificial observation method to observe the animal behavior. The use of machine vision
technology to detect animal behavior can avoid stress responses and reduce the observer's physical labor and the probabilities
of contact between human and animal (Kaixuan et al., 2017). The technology can also be used to develop intelligent
monitoring equipment for modern intensive farms to improve the level of animal husbandry industry intelligence.
Thermal infrared vision is one of the most significant advantages of machine vision technology. Compared to

The authors are solely responsible for the content of this meeting presentation. The presentation does not necessarily reflect the official position of the
American Society of Agricultural and Biological Engineers (ASABE), and its printing and distribution does not constitute an endorsement of views
which may be expressed. Meeting presentations are not subject to the formal peer review process by ASABE editorial committees; therefore, they are
not to be presented as refereed publications. Publish your paper in our journal after successfully completing the peer review process. See
www.asabe.org/JournalSubmission for details. Citation of this work should state that it is from an ASABE meeting paper. EXAMPLE: Author’s Last
Name, Initials. 2017. Title of presentation. ASABE Paper No. ---. St. Joseph, MI.: ASABE. For information about securing permission to reprint or
reproduce a meeting presentation, please contact ASABE at www.asabe.org/permissions (2950 Niles Road, St. Joseph, MI 49085-9659 USA).
conventional visible light video surveillance, it can easily obtain animal targets at night without being affected by the lack
of light at night and the complex background environment (Ma et al., 2015) . Moreover, the infrared temperature
measurement technology is based on infrared radiation energy of the detected objects to determine their temperature, which
obtain the target surface radiation temperature by non-contact temperature measurement methods. Therefore, the use of
thermal infrared vision can not only achieve full-day detection of the target can also get the target surface temperature.
Thermal infrared target detection and tracking technology has important research significance and application value (Xiu-
Wei et al., 2011) .
Although the target tracking study has been carried out for decades and has made considerable progress in recent years,
it is still a challenging subject in the machine learning field. The performance of the tracking algorithm is mainly measured
from the aspects of computing speed and tracking accuracy.
Early target tracking techniques focused on motion models such as kalman filter, particle filter, and Camshift tracking
(Zhang et al., 2016) . This method is susceptible to a number of factors, including changes in light, occlusion and background
mixed, often tracking the loss of the problem.
The discriminant target tracking methods in recent years (Li et al., 2016) , such as the structured output tracking with
kernel (Struck) (Hare et al., 2016) , the tracking-learning-detection (TLD) (Kalal et al., 2012) , the multiple instance learning
(MIL) (Boris et al., 2011) and the compressive tracking (CT) (Zhang et al., 2012) . Tracking is seen as a binary classification
problem in which the use of training data to learn a classifier, the target separated from the background, can significantly
improve the tracking performance. Tracking is seen as a binary classification problem in such methods. It uses training data
to learn a classifier, which separates the target from the background and can significantly improve tracking performance.
However, because the process of training the sample is very time-consuming, these algorithms usually select a limited
number of training samples and candidate samples, so as to achieve real-time target tracking effect, which makes the tracking
accuracy and computational efficiency are obviously inadequate.
The correlation filter is initially used for target detection and recognition, and can achieve precise positioning of the target,
but can not meet the real-time tracking because of the time-consuming. (Bolme et al., 2010) proposed a tracking algorithm
for learning a minimum output sum of squared error (MOSSE) correlation filter on grayscale images. The method only
needs to use the sample image of the target area to train the target appearance model, which convert the similarity calculation
between the target and all candidate regions to the frequency domain through the discrete Fourier transform. MOSSE
achieved the high-speed tracking of the relevant filter for the first time, but it also selected a limited number of candidate
samples to train the classifier model (Li et al., 2017). On this basis, (Henriques et al., 2015) presented the Circulat Structure
of tracking-by-detection with Kernel (CSK), which constructs a large number of candidate regions for the detection process
of the classifier. Then he proposed a kernel correlation filter (KCF) based on Histogram of Oriented Gradients (HOG) instead
of raw pixels as the target feature representation (Henriques et al., 2012). KCF uses a large number of positive and negative
target samples for training, which has a high computational efficiency, and excellent tracking results, when dealing with
tracking changes in light, background interference, target deformation, occlusion. A linear multi-channel filter with very
low computational complexity which almost matches the performance of non-linear kernels was also proposed with the
name of Dual Correlation Filter (DCF). However, KCF is a tracking algorithm for single-target, and the tracking performance
decreases when the target is rotated and deformed.
On the basis of KCF, this paper proposes an improved multi-objective KCF tracking method, which increases the number
of tracking targets and optimizes the tracking window scale in order to reduce the influence on the accuracy and stability
caused by rotation and deformation.

Proposed method
Review: Kernel Correlation Filter
The core idea of the KCF tracking algorithm is to cyclically shift the tracking target area to construct a large number of
samples to train the classifier (Li et al., 2017). The similarity degree of the candidate region is calculated by the kernel
function, and the candidate region with the largest similarity is selected as the new tracking target. At the same time, the
discrete Fourier transform is used to reduce the amount of computation in the classifier training and detection process.
A．The expression of the sample
The outstanding advantage of the KCF algorithm is the use of cyclic matrices (Bibi and Ghanem, 2015). Assume an n×
1 vector representing a patch with the object of interest, denoted x. We consider it as the base sample. In order to train the
classifier, we use a circular matrix P for cyclic shift, which is the permutation matrix

ASABE 2017 Annual International Meeting Page 1

 0 0 0  1
1 0 0  0 

P = 0 1 0  0 (1)
 
    
0 0  1 0 

The cyclic shift of x can be represented as Px = [x n , x1 , x 2 , , x n −1 ]T . For all the training samples x i which is
periodically every i shift, can constitute a cyclic matrix X , which can be expressed as C(x) :
 x1 x2 x3  x n 
x x1 x 2  x n −1 
 n
= =  x n −1
X C(x) x n x1  x 3  (2)
 
      
 x 2 x 3 x 4  x1 
= [X 0 X1  X n −1 ]T

All of the circular matrix X can be given the diagonalization using Discrete Fourier transform ( DFT ), as follows，
X = Fdiag(x)F
ˆ H (3)
where F is a coefficient matrix of Discrete Fourier Transform that does not depend on x , The symbol ^ represents the
DFT of the corresponding vector. FFH = I .
B．Training and detection
(Henriques et al., 2015) points out that the purpose of the KCF algorithm is to find a function f (z) that minimize the
square of the error of the sample X i with the regression function yi for all samples X , thus calculating w

min ∑ (f (X i ) − yi ) + λ w
2 2
(4)
w
i
where λ is a regularization parameter that controls overfitting. Rifkin[14] pointed out that W can be expressed as
w= ∑ α ϕ(X )
i
i i (5)

where αi is optimize variables, which is in the dual space, instead of w . ϕ(X i ) is the function of mapping the training
X and X∗ in the high latitude
samples into the high-dimensional feature space. The correlation between any two samples
∗
T T
k(x, x ∗ ) , where K is the Gaussian kernel function. Construct the kernel matrix K, a is
feature space is ϕ (x)ϕ (x ) =
given as follows，
a (K + λI) −1 y
= (6)
where a = [a1 , a 2 ,  , a n ] . The Fourier transform of a is shown by the following equation
F (y)
F (a)= aˆ= (7)
F (k xx ) + λ
xx
where â is the Discrete Fourier transform of a ; k is the first row vector of the kernel matrix K .

For the updated image block z , the probability which is determined to be objective is y′
y′ = ∑ a i k(X i , z) (8)
i
i −1
The candidate region sample set P z is the result of cyclic shifts for z . Define the matrix K z = k(Pi −1z, P j−1x)
which satisfies the conditions of the cyclic matrix,
K z = C(k xz ) (9)

ASABE 2017 Annual International Meeting Page 2

 xz
Where k is the first row vector of the matrix K z . f (z) is the probability distribution of the input image block as a
tracking target at all candidate region positions,
f (z) = F −1 ( F (k xz )  a)
ˆ (10)
The area corresponding to the element that gets the maximum value in f (z) is the tracking target.

Improved Method
According to the pig's body shape and behavior characteristics, this article has improved the detection box instead of the
regular square check box. In order to adapt to the pig's body shape, (Duan et al., 2015) used fixed-size oval box to improve
the tracking accuracy. However, due to the limited space in the small pigsty, the moving direction of the pig is often drastic
changes and the movement speed is relatively fast, resulting in reduced tracking accuracy. The original rectangular track
box is changed to a circular box so that the search range can be kept at the same scale. When the pig is deflected, the
algorithm can extract as many edge features as possible.
A single target tracking algorithm is given in this paper. In production practice, there are often 2 or 3 sows in a small pig
pigsty, and there will be five or even more cases in the fattening pig pigsty. In order to obtain the motion behavior of each
animal, this paper increases the number of detection boxes on the basis of the original algorithm to achieve multi-target
tracking in the same pig house.

Results and analysis

Experimental setup
Infrared monitoring system platform is built next to the pigsty. The hardware platform includes the infrared camera
( FLIR T620 ), a laptop and a rocker tripod. The size of the pigsty is 3 m ×4 m. Each video continues to record for 5 minutes,
its video sampling rate is 15 frame/s and the pixel resolution is 640 pixel × 480 pixel. Software experimental environment:
The proposed algorithm is implemented on the PC with Windows 7 operating system, Intel Xeon E5 - 1620 0 @ 3.60 GHz,
16.0 G RAM.

Results and discussion

Compare the tracking box
Select 20 videos of five pigs, which include a variety of behaviors such as prone, moving, turning, sticking, blocking,
and occlusion. Respectively, using rectangular box, ellipse box and circular box to track, the tracking results shown in Figure
1.

a b c
Figure 1 Compare the tracking results of different tracking boxes: (a) the rectangular box; (b) the ellipse box; and (c) the circular box.

This paper uses the precision curve to measure and compare. (Liyang et al., 2015) stated that the precision curve is the
ASABE 2017 Annual International Meeting Page 3
ratio of the number of frames whose distance between the coordinate center of the tracking target result and the coordinate
center of the real artificial mark which is less than a certain threshold (1-50) and the length of the whole test video sequence
image.
Calculate the average accuracy of all the tracking boxes in the video, and the tracking accuracy results for the different
tracking boxes are plotted in Table 1. Select the curve at a threshold of 20 to compare the tracking results of different tracking
boxes. The results show that the tracking accuracy of circular box is much higher than that of rectangular box and ellipse
box, and it has the ability of steering adaptability. The circular box is superior to the other two boxes in the anti-steering
performance.
Table 1. The accuracy of the three tracking boxes
tracking boxes Mean precision (20px) Mean FPS
Rectangular box 27.02% 39.50
ellipse box 27.02% 38.61
circular box 80.26% 51.83

Accuracy and runtime analysis

In order to evaluate the performance of the multi-target tracking algorithm proposed in this paper, data from two, three
and five pigs were collected and 90 clear and accurate videos (30 videos per pig) were selected for the tracking. These videos
include a variety of behaviors such as prone, moving, turning, sticking, blocking, and occlusion. The tracking precision
result of the multi-objective KCF ( M-KCF ) algorithm is shown and compared with the multi-objective DCF ( M-DCF )
and multi-objective MOSSE ( M-MOSSE )in Table 2. It can be seen that the MKCF is the best precision performance in the
three algorithms. Because M-KCF is based on the feature HOG, its detection effect is higher than MMOSSER based on
Raw pixels.

Table 2. Summary of experimental results on the 90 videos

Mean precision (20px) Mean FPS
algorithm
Two pigs three pigs Five pigs Two pigs three pigs Five pigs
M-KCF 86.15% 89.18% 81.76% 53.12 80.46 52.08
M-DCF 86.03% 78.28% 81.01% 87.92 85.21 80.31
M-MOSSE 53.88% 50.83% 57.84% 235.63 252.50 229.34

Table 2 shows that M-MOSSE is the fastest at run time, followed by M-DCF, and M-KCF is the slowest. The reason is
that M-MOSSE and M-DCF are linear filters, so the speed is faster than M-KCF, which is the correlation filter. Although
M-KCF is running slowly, but the tracking effect is good. M-KCF is a high-speed detection algorithm that can meet the real-
time tracking needs of farms ( the number of frames processed per second in 15 to 20 frames can be regarded as real-time
tracking usually).

Conclusion
Aiming at the problem of tracking multiple pigs in pigsty, this paper presents a multi - target tracking algorithm based on
KCF. In order to reduce the negative effects of animal steering motion on tracking and improve the accuracy of tracking, a
circular approach is proposed as a tracking box. The experimental results show that the algorithm can effectively track the
multi - pig target in the pen and can meet the real - time requirement.

Acknowledgements (uses “Heading 2” style)

This project is funded by the “Development and Validation of Information Perception and Environmental Control System
for Livestock (Pigs and Cows)” (No. 2016YFD0700204) . This project is also supported by the Foundation of Key
Laboratory of Modern Agricultural Equipment and Technology, Ministry of Education. At the same time, it was supported
by Hebei Province Science and Technology Project “Research on Target Recognition and Tracking Based on Video
Surveillance” (No. 15226633) from Hebei Agricultural University. And College of Information Science and Technology,
Agricultural University of Hebei participated in program debugging.

References:
Duan, Y., L. Ma, and G. Liu. (2015). Target Tracking and Behavior Detection Method in Piggery Scenarios. Transactions
of the Chinese Society for Agricultural Machinery (S1):187-193.
Gronskyte, R., L. H. Clemmensen, M. S. Hviid, and M. Kulahci. (2015). Pig herd monitoring and undesirable tripping and
stepping prevention. Computers and Electronics in Agriculture 119:51-60.
ASABE 2017 Annual International Meeting Page 4
Guo, Y., W. Zhu, P. Jiao, C. Ma, and J. Yang. (2015). Multi-object extraction from topview group-housed pig images based
on adaptive partitioning and multilevel thresholding segmentation. Biosystems Engineering 135:54-60.
Kaixuan, Z., L. Guoqiang, and H. Dongjian. (2017). Automatic recognition of cows’ body parts in depth images using
Automatic recognition of cows’ body parts in depth images using machine learning. Transactions of the Chinese Society
for Agricultural Machinery (04):1-9.
Ma, L., Y. Duan, Z. Zong, and G. Liu. (2015). Segmentation of Thermal Infrared Image for Sow Based on Improved Convex
Active Contours. Transactions of the Chinese Society for Agricultural Machinery (S1):180-186.
Xiu-Wei, Z., Z. Yan-Ning, G. Zhe, Z. Jing, and T. Xiao-Min. 2011. Advances and perspective on motion detection fusion
in visual and thermal framework. Journal of Infrared Millimeter Waves (04):354-360.
Zhang, Y. Z., C. Z. Zeng, H. Liang, J. Luo, and F. Xu. (2016). A visual target tracking algorithm based on Improved
Kernelized Correlation Filters. IEEE International Conference on Mechatronics and Automation:199-204.
Li, Q., L. Li, and Q. Huang. (2016). Scale adaptive tracker based on kernelized correlation filtering. Journal of Computer
Applications (12):3385-3388.
Hare, S., S. Golodetz, A. Saffari, V. Vineet, M. Cheng, S. L. Hicks, and P. H. S. Torr. (2016). Struck: Structured Output
Tracking with Kernels. IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (10):2096-2109.
Kalal, Z., K. Mikolajczyk, and J. Matas. (2012). Tracking-Learning-Detection. IEEE Trans Pattern Anal Mach Intell 34
(7):1409-1422.
Boris, B., M. Yang, and B. Serge. (2011). Robust Object Tracking with Online Multiple Instance Learning. IEEE
Transactions on Pattern Analysis and Machine Intelligence (VOL. 23):263-270.
Zhang, K., L. Zhang, and M. Yang. (2012). Real-Time Compressive Tracking. European Conference on Computer Vision
(Vol.7574):864-877.
Bolme, D. S., J. R. Beveridge, B. A. Draper, and Y. M. Lui. (2010). Visual Object Tracking using Adaptive Correlation
Filters, 2544-2550.
Li, Z., G. Wang, Y. Liu, and H. Li. (2017). Adaptive Visual Tracking Based on the Correlation Filter. JOURNAL OF
GRAPHICS (02):198-203.
Henriques, J. F., R. Caseiro, P. Martins, and J. Batista. (2015). High-Speed Tracking with Kernelized Correlation Filters.
IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (3):583-596.
Henriques, J. A. F., R. Caseiro, P. Martins, and J. Batista. (2012). Exploiting the Circulant Structure of Tracking-by-
detection with Kernels. European Conference on Computer Vision Vol.7575:702-715.
Bibi, A., and B. Ghanem. (2015). Multi-Template Scale-Adaptive Kernelized Correlation Filters. International Conference
on Computer Vision Workshops.
Liyang, Y., F. Chunxiao, and M. Yue. (2015). Improved target tracking algorithm based on kernelized correlation filter.
Journal of Computer Applications (12):3550-3554.

ASABE 2017 Annual International Meeting Page 5