Abstract

While color information is known to provide rich discriminative clues for visual inference, most modern visual trackers limit themselves to the grayscale realm. Despite recent efforts to integrate color in tracking, there is a lack of comprehensive understanding of the role color information can play. In this paper, we attack this problem by conducting a systematic study from both the algorithm and benchmark perspectives. On the algorithm side, we comprehensively encode 10 chromatic models into 16 carefully selected state-of-the-art visual trackers. On the benchmark side, we compile a large set of 128 color sequences with ground truth and challenge factor annotations (e.g., occlusion). A thorough evaluation is conducted by running all the color-encoded trackers, together with two recently proposed color trackers. A further validation is conducted on a RGBD tracking benchmark. The results clearly show the benefit of encoding color information for tracking. We also perform detailed analysis on several issues including the behavior of various combinations between color model and visual tracker, the hardness of each sequence for tracking, and how different challenge factors affect the tracking performance. We expect the study to provide the guidance, motivation and benchmark for future work on encoding color in visual tracking.

The dataset and source code of some color trackers are made available for reference.

Reference

Encoding Color Information for Visual Tracking: Algorithms and Benchmark P. Liang, E. Blasch, and H. Ling IEEE Trans. on Image Processing (T-IP), 24(12): 5630-5644, 2015. PDF

Dataset

How to download the benchmark?

All the 128 sequences of Temple Color 128 including ground truth and attribute annotation can be downloaded entirely Temple-color-128.zip, or downloaded separately by clicking the name of each sequence below.

How to use the data?

The folder of each sequence contains the following files:

• a directory includes the original image sequence.
• a [name]_frames.txt file indicates the first frame and last frame used for evaluation.
• a [name]_gt.txt contains the groud truth. The format of each bounding box is [target_top_left_x,target_top_left_y,target_width,target_height].
• a [name]_att.txt contains the challenge factors of the sequence.

Note: The sequence name of which ending with "ce" are newly collected from the Interent (78 in total); others are collected from previous studies (50 in total). There are two targets in the sequence "Jogging".

Airport_ce SV,OCC	Baby_ce SV,OCC,DEF,OPR	Badminton_ce1 DEF,MB,OPR	Badminton_ce2 DEF,MB,OPR	Ball_ce1 OCC,MB,IPR,OPR...	Ball_ce2 IV,OCC,MB,OV...	Ball_ce3 OV,SV,FM	Ball_ce4 FM,OCC,OPR,OV
Basketball IV,OPR,OCC,DEF...	Basketball_ce1 IV,SV,OCC,DEF...	Basketball_ce2 IV,OCC,DEF,OPR...	Basketball_ce3 SV,OCC,DEF,MB...	Bee_ce BC,LR	Bicycle IV,SV,BC	Bike_ce1 IV,SV,FM,IPR...	Bike_ce2 SV,IPR,LR
Biker SV,MB,FM,OPR	Bikeshow_ce SV,DEF,FM,IPR...	Bird OCC,FM,OPR	Board SV,MB,FM,OV...	Boat_ce1 SV,OCC,IPR,OPR	Boat_ce2 SV,OCC,FM,IPR...	Bolt OPR,OCC,DEF,IPR	Boy OPR,SV,MB,FM...
Busstation_ce1 OCC,BC	Busstation_ce2 OCC,IPR,OPR,BC	CarDark IV,BC	CarScale OPR,SV,OCC,FM...	Carchasing_ce1 IV,SV,OCC,FM...	Carchasing_ce3	Carchasing_ce4 SV	Charger_ce IV,OCC,MB,IPR...
Coke IV,OPR,OCC,FM...	Couple OPR,SV,DEF,FM...	Crossing SV,DEF,BC	Cup BC	Cup_ce MB,IPR,OPR,FM	David IV,OPR,SV,OCC...	David3 OPR,OCC,DEF,BC	Deer MB,FM,IPR,BC
Diving SV,DEF,MB	Doll IV,OPR,SV,OCC...	Eagle_ce SV,IPR,OPR,BC	Electricalbike_ce SV,OCC	FaceOcc1 OCC	Face_ce SV,OCC,IPR,OPR...	Face_ce2 IV,OCC,MB,IPR...	Fish_ce1 OCC,IPR,OPR,SV
Fish_ce2 OCC,IPR,OPR,SV	Football1 OPR,IPR,BC	Girl OPR,SV,OCC,IPR	Girlmov OCC,MB,OPR	Guitar_ce1 FM	Guitar_ce2 IV,FM,IPR,OPR	Gym SV,DEF,IPR,OPR	Hand IV,SV
Hand_ce1 FM,DEF,MB,BC	Hand_ce2 MB,OV,FM	Hurdle_ce1 DEF,MB,BC	Hurdle_ce2 DEF,FM,BC	Iceskater SV,IPR,OPR	Ironman IV,OPR,OCC,MB...	Jogging [1] [2] OPR,OCC,DEF	Juice SV
Kite_ce1 OCC,IPR,BC	Kite_ce2 SV,OCC,IPR,BC	Kite_ce3 FM,IPR,BC	Kobe_ce SV,OCC,DEF,MB...	Lemming IV,OPR,SV,OCC...	Liquor IV,OPR,SV,OCC...	Logo_ce SV,IPR	Matrix IV,OPR,SV,OCC...
Messi_ce SV,OCC,DEF,MB...	Michaeljackson_ce IV,DEF,FM,IPR...	Microphone_ce1 OCC,FM	Microphone_ce2 OPR,LR	MotorRolling IV,SV,MB,FM...	Motorbike_ce IV,OCC,BC	MountainBike OPR,IPR,BC	Panda SV,OCC,IPR,LR
Plane_ce2 SV,FM,IPR,OPR	Plate_ce1 SV,LR	Plate_ce2 LR	Pool_ce1 LR	Pool_ce2 LR	Pool_ce3 MB,BC,LR	Railwaystation_ce OCC,IPR,BC	Ring_ce BC,LR
Sailor_ce FM	Shaking IV,OPR,IPR,BC	Singer1 IV,OPR,SV,OCC	Singer2 IV,OPR,DEF,IPR...	Singer_ce1 IV,DEF,FM,BC	Singer_ce2 IV,SV,DEF,OPR...	Skating1 IV,OPR,SV,OCC...	Skating2 SV,DEF,FM,OPR
Skating_ce1 SV,OCC,DEF,FM...	Skating_ce2 SV,MB,FM,IPR...	Skiing IV,OPR,SV,DEF...	Skiing_ce SV,OCC,FM	Skyjumping_ce IV,SV,DEF,FM...	Soccer IV,OPR,SV,OCC...	Spiderman_ce OCC,SV,FM,IPR...	Subway OCC,DEF,BC
Suitcase_ce IV,OCC,BC	Sunshade IV	SuperMario_ce OV,LR	Surf_ce1 OCC,SV,FM,IPR...	Surf_ce2 OCC,SV,FM,IPR...	Surf_ce3 OCC,FM,IPR,OPR	Surf_ce4 FM,IPR,OPR	TableTennis_ce MB,BC,LR
TennisBall_ce OCC,MB,FM,LR	Tennis_ce1 SV,MB,IPR,OPR	Tennis_ce2 MB,FM,IPR,OPR	Tennis_ce3 OPR	Thunder_ce	Tiger1 IV,OPR,OCC,DEF...	Tiger2 IV,OPR,OCC,DEF...	Torus OPR
Toyplane_ce SV,OCC,FM,OPR	Trellis IV,OPR,SV,IPR...	Walking SV,OCC,DEF,LR	Walking2 SV,OCC	Woman IV,OPR,SV,OCC...	Yo-yos_ce1 MB,OV,LR,FM	Yo-yos_ce2 MB,OV,LR,FM	Yo-yos_ce3 MB,FM,OV,SV

Evaluation

All the evaluation resulsts (bounding box in each frame) and the code for generating the success and precision plots in the paper can be downloaded here.

	Avg. rank	ASLA	CPF	CSK	CT	DFT	FCT	Frag	IVT	KCF	L1APG	LOT	MEEM	MIL	OAB	SemiT	Struck
OPP	1.94	0.395(2)	0.303(5)	0.350(1)	0.336(2)	0.313(3)	0.357(1)	0.375(2)	0.304(1)	0.418(2)	0.376(1)	0.358(1)	0.483(3)	0.387(2)	0.368(2)	0.365(1)	0.462(2)
HSV	2.63	0.375(4)	0.342(1)	0.340(3)	0.345(1)	0.314(2)	0.349(2)	0.408(1)	0.281(4)	0.405(3)	0.320(6)	0.350(2)	0.489(2)	0.374(3)	0.343(4)	0.354(3)	0.464(1)
LAB	2.75	0.406(1)	0.308(4)	0.346(2)	0.334(3)	0.303(6)	0.347(3)	0.364(4)	0.275(5)	0.418(1)	0.360(2)	0.298(5)	0.500(1)	0.393(1)	0.389(1)	0.357(2)	0.459(3)
RGB	4.31	0.380(3)	0.314(3)	0.307(5)	0.322(4)	0.312(4)	0.315(5)	0.374(3)	0.289(2)	0.384(8)	0.325(4)	0.330(3)	0.459(5)	0.334(5)	0.316(7)	0.312(4)	0.441(4)
TRGB	4.63	0.366(5)	0.315(2)	0.309(4)	0.320(5)	0.316(1)	0.317(4)	0.044(9)	0.289(3)	0.385(7)	0.325(5)	0.330(4)	0.473(4)	0.347(4)	0.318(6)	0.307(5)	0.405(6)
C-OPP	6.06	0.328(7)	0.253(8)	0.265(7)	0.297(6)	0.224(7)	0.294(7)	0.264(6)	0.244(7)	0.394(4)	0.354(3)	0.286(6)	0.448(6)	0.313(7)	0.344(3)	0.304(6)	0.396(7)
Intensity	6.94	0.350(6)	0.260(6)	0.297(6)	0.285(9)	0.308(5)	0.291(8)	0.324(5)	0.254(6)	0.386(6)	0.318(7)	0.250(8)	0.431(9)	0.291(9)	0.300(9)	0.303(7)	0.409(5)
N-OPP	7.75	0.175(9)	0.249(9)	0.243(8)	0.290(8)	0.183(9)	0.296(6)	0.264(7)	0.182(8)	0.388(5)	0.302(9)	0.246(9)	0.437(7)	0.314(6)	0.309(8)	0.300(8)	0.368(8)
rg	8.00	0.327(8)	0.256(7)	0.231(9)	0.295(7)	0.197(8)	0.283(9)	0.250(8)	0.169(9)	0.375(9)	0.304(8)	0.250(7)	0.435(8)	0.305(8)	0.318(5)	0.293(9)	0.251(9)
Hue	10.00	0.118(10)	0.100(10)	0.016(10)	0.122(10)	0.047(10)	0.113(10)	0.043(10)	0.039(10)	0.325(10)	0.160(10)	0.071(10)	0.327(10)	0.112(10)	0.175(10)	0.154(10)	0.183(10)

Code

Following are the source code of two color-enhanced trackers.

Base tracker	Code enhanced
KCF: J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” PAMI, 2015.	Download
Struck: S. Hare, A. Saffari, and P. H. Torr, “Struck: Structured output tracking with kernels,” ICCV, 2011	Download

Contact

If you have any questions, please contact Pengpeng Liang at pliang AT temple.edu or Haibin Ling at hbling AT temple.edu.