Skip to content

tsenst/CrowdFlow

Branches Tags

Folders and files

Name Name
Last commit message
Last commit date

Latest commit

 

History

40 Commits
doc
doc
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
Supplemental_material.pdf
Supplemental_material.pdf
 
 
file_parser.py
file_parser.py
 
 
opticalflow_estimate.py
opticalflow_estimate.py
 
 
opticalflow_evaluate.py
opticalflow_evaluate.py
 
 
trajectory_evaluate.py
trajectory_evaluate.py
 
 
util.py
util.py
 
 

Repository files navigation

Dataset samples

TUB CrowdFlow Dataset

Optical Flow Dataset and Evaluation Kit for Visual Crowd Analysis developed at Communication Systems Group at TU-Berlin desciribed in the AVSS 2018 paper Optical Flow Dataset and Benchmark for Visual Crowd Analysis or TUBCrowdFlow@arxiv.org .

The Dataset contains 10 sequences showing 5 scenes. Each scene is rendered twice: with a static point of view and a dynamic camera to simulate drone/UAV based surveillance. We render at HD resolution (1280x720) at 25 fps, which is typical for current commercial CCTV surveillance systems. The total number of frames 3200.

For each sequence we provide the following ground-truth data:

  • Optical flow fields
  • Person trajectories (up to 1451)
  • Dense pixel trajectories

This evaluation framework is released under the MIT License (details in LICENSE ). If you use the dataset or evaluation kit or think our work is useful in your research, please consider citing:

 @INPROCEEDINGS{TUBCrowdFlow2018, AUTHOR = {Gregory Schr{\"o}der and Tobias Senst and Erik Bochinski and Thomas Sikora}, TITLE = {Optical Flow Dataset and Benchmark for Visual Crowd Analysis}, BOOKTITLE = {IEEE International Conference on Advanced Video and Signals-based Surveillance}, YEAR = {2018}, }

Download the dataset via the following direct link

The password is the case sensitive name of the repository.

Unpack the dataset by:

 sudo apt-get install unrar unrar x TUBCrowdFlow

The TUB CrowdFlow dataset is made available for academic use only. If you wish to use this dataset commercially please contact sikora@nue.tu-berlin.de .

Contact

If you have any questions or encounter problems regarding the method/code or want to send us your optical flow benchmark results feel free to contact me at tobias.senst@gmail.com

Installation

Minimum required python version: 3.5

Install dependencies on Ubuntu:

 sudo apt-get install python3-dev python3-virtualenv virtualenv

Create a virtual environment and install python requirements:

 virtualenv -p python3 crowdflow_env source crowdflow_env/bin/activate pip3 install numpy progressbar2 opencv-contrib-python

Evaluation Framework

To evaluate an optical flow method with the providen framework perform these step:

  • create a new directory in the /TUBCrowdFlow/estimate directory.
  • compute flow fields and save them in .flo fileformat with the structure given in by the /TUBCrowdFlow/images directory. For example optical flow results from the image pair /TUBCrowdFlow/images/IM01/frame_0000.png and /TUBCrowdFlow/images/IM01/frame_0001.png must be stored as `/estimate/[mymethod]/images/IM01/frame_0000.flo
  • run opticalflow_evaluate.py to compute EPE and R2 short-term metrics.
  • run trajectory_evaluate.py to compute tracking accuracy long-term metrics.

Optical Flow Samples

opticalflow_estimate.py <dataset_root_path> <flow_method_name_1> <flow_method_name_2> ...

With the following program optical flow fields for the TUB CrowdFlow dataset will be estimated with Dual-TVL1.

 source crowdflow_env/bin/activate python3 opticalflow_estimate.py TUBCrowdFlow/ dual farneback plk

The optical flow files will be stored in the directory /estimate/dual/ .

Short-Term Evaluation

Short-term evaluation performs classical approch for optical flow evaluation, i.e. measures based on ground-truth optical flow fields (e.g. end-point error, RX measures)

opticalflow_evaluate.py <dataset_root_path> <dir_name_method_1> <dir_name_method_2> ... <dir_name_method_n>

Example:

 source crowdflow_env/bin/activate python3 opticalflow_evaluate.py TUBCrowdFlow/ dual plk farneback

After execution the file short_term_results.tex will contain the evaluation results ( method 1 - method n ) in form of a latex table. short_term_results.pb will contain the evaluation results stored with pickle.

Long-Term Evaluation

Long-term evaluation performs evaluation based on ground-truth trajectories, i.e. person trajectories and dense pixel trajectories (see paper).

trajectory_evaluate.py <dataset_root_path> <dir_name_method_1> <dir_name_method_2> ... <dir_name_method_n>

Example:

 source crowdflow_env/bin/activate python3 trajectory_evaluate.py TUBCrowdFlow/ dual plk farneback

After execution the file long_term_results.tex will contain the evaluation results ( method 1 - method n ) in form of a latex table. long_term_results.pb will contain the evaluation results stored with pickle.

Results

To assess the quality of the optical flow we propose to use two types of metrics: i) common optical flow metrics, i.e. average endpoint error (EPE) and percentage of erroneous pixel (RX) and ii) long-term motion metrics based on trajectories. An detailed overview of the optical flow parameters can be found in the document: Supplemental_materials.pdf ).

Common optical flow metrics (short-term)

FG (Static) FG (Static) BG (Static) BG (Static) FG (Dynamic) FG (Dynamic) BG (Dynamic) BG (Dynamic) FG(Avg.) FG(Avg.) BG(Avg.) BG(Avg.) Avg. Avg.
EPE R2[%] EPE R2[%] EPE R2[%] EPE R2[%] EPE R2[%] EPE R2[%] EPE R2[%] t[sec]
FlowFields (Bailer2015) zero point seven five six eight point two seven zero point two one three two point seven nine one point zero six nine fourteen point nine two two point five seven one fifty-one point four two zero point nine one three eleven point five nine five one point three nine two twenty-seven point one zero zero point nine one five eleven point seven four forty-three point five three
RIC (Hu2017) zero point eight five nine eight point six four zero point two four three three point three one one point one six six fifteen point six nine two point six two three fifty-three point five eight one point zero one three twelve point one six four one point four three three twenty-eight point four five one point zero one five twelve point three two eight point three zero
CPM (Li2018) zero point seven zero one seven point zero nine zero point two four seven three point six three one point zero two six thirteen point nine four two point five eight five fifty-one point seven eight zero point eight six four ten point five one seven one point four one six twenty-seven point seven one zero point eight six eight ten point six nine fourteen point seven four
DeepFlow (Weinzaepfel2013) zero point six two nine six point one nine zero point two three seven three point six seven one point zero zero five thirteen point nine five two point five nine four fifty-one point six seven zero point eight one seven ten point zero six nine one point four one six twenty-seven point six seven zero point eight two two ten point two five thirty-nine point six three
RLOF6 (Geistert2016) zero point seven five three eight point six one zero point three one five five one point zero eight eight fifteen point six one two point six five five fifty-three point four seven zero point nine two one twelve point one one two one point four eight five twenty-nine point two three zero point nine two four twelve point two seven one point four nine
RLOF10 (Geistert2016) zero point seven seven two eight point eight zero zero point three two four five point one zero one point one zero four fifteen point eight zero two point six five eight fifty-three point six zero zero point nine three eight twelve point three zero three one point four nine one twenty-nine point three five zero point nine four one twelve point four six zero point eight zero
DIS4 (Kroeger2016) zero point six two seven five point seven two zero point three five six five point eight five zero point nine two eight eleven point eight six two point six six five fifty-three point six seven zero point seven seven seven eight point seven nine zero one point five one one twenty-nine point seven six zero point seven eight four nine point zero one one point seven zero
DIS2 (Kroeger2016) one point four four one twenty point four zero zero point five two eight eight point two four one point seven two six twenty-seven point four one three point zero zero one sixty-four point zero one one point five eight three twenty-three point nine zero three one point seven six five thirty-six point one three one point five seven nine twenty-three point nine two zero point two eight
Farneback (Farneback2003) zero point seven three seven seven point two one zero point four four one seven point three zero zero point nine nine six twelve point six seven two point four nine one fifty point six zero zero point eight six seven nine point nine four zero one point four six six twenty-eight point nine five zero point eight seven two ten point one three
Sparse to Dense PLK (Bouguet2000) zero point seven nine three eight point zero seven zero point five six three nine point one two one point zero four one thirteen point two four two point eight seven five fifty-six point two nine zero point nine one seven ten point six five three one point seven one nine thirty-two point seven one zero point nine two five ten point eight eight

Tracking Accuracy (long-term)

Dense Trajectories

IM01 (Dyn) IM02 (Dyn) IM03 (Dyn) IM04 (Dyn) IM05 (Dyn) Avg.
FlowFields (Bailer2015) seventy point six three sixty-one point seven nine fifty-six point six nine forty-five point nine three seventy-one point four six sixty-eight point three five forty-two point two seven thirty-seven point six three sixty-five point one five fifty-nine point six one fifty-seven point nine five
RIC (Hu2017) seventy-four point three nine sixty-nine point four one fifty-eight point seven two fifty point three three fifty-four point one eight seventy-three point eight zero forty-four point two one thirty-nine point five two sixty point two three sixty point two eight fifty-eight point five one
CPM (Li2018) seventy-three point four one sixty-five point one six fifty-eight point three one forty-seven point five seven seventy-four point four one seventy-one point one three forty-six point two three forty-one point one five sixty-seven point nine seven sixty-one point six eight sixty point seven zero
DeepFlow (Weinzaepfel2013) eighty-three point eight four eighty-one point nine zero sixty-three point three three fifty-five point five two eighty-three point three eight eighty point eight seven fifty-seven point zero eight fifty-six point six five seventy-one point two five sixty-four point six seven sixty-nine point eight five
RLOF6 (Geistert2016) eighty-two point eight zero seventy-eight point three one sixty-three point one six fifty-seven point six eight eighty-seven point four six eighty-six point seven six fifty point five six fifty point five three sixty-nine point eight six sixty-eight point seven three sixty-nine point five nine
RLOF10 (Geistert2016) eighty point one four seventy-three point nine five sixty-two point zero five fifty-five point five four eighty-five point four four eighty-four point three nine forty-eight point eight zero forty-seven point eight four sixty-seven point five three sixty-seven point four one sixty-seven point three one
DIS4 (Kroeger2016) eighty point four four seventy-six point one nine sixty-four point one one fifty-six point nine nine eighty-two point eight nine eighty-two point two four fifty-three point nine one fifty-two point seven five seventy-two point one one seventy point seven one sixty-nine point two three
DIS2 (Kroeger2016) forty-seven point five five thirty-three point zero three thirty-six point five two twenty-five point three two twenty-two point five nine nineteen point seven six twenty-six point seven nine twenty point eight nine twenty-seven point six three twenty-seven point nine one twenty-eight point eight zero
Farneback (Farneback2003) seventy-eight point six nine seventy-four point two four sixty-five point two two fifty-nine point four three eighty-six point eight nine eighty-seven point one seven fifty-two point eight five fifty-five point two nine seventy point two two sixty-eight point nine four sixty-nine point eight nine
Sparse to Dense PLK (Bouguet2000) seventy-five point one five sixty-eight point five four sixty-four point seven one fifty-seven point eight eight eighty-four point seven one eighty-four point one one fifty point zero eight forty-nine point two six sixty-eight point four five sixty-nine point seven five sixty-seven point two six

Person Trajectories

IM01 (Dyn) IM02 (Dyn) IM03 (Dyn) IM04 (Dyn) IM05 (Dyn) Avg.
FlowFields (Bailer2015) seventy-seven point nine four sixty-two point six eight fifty-two point three five thirty-eight point two two sixty-six point seven six sixty-three point one seven thirty point zero nine twenty-five point two four sixty-five point six seven sixty-eight point two zero fifty-five point zero three
RIC (Hu2017) eighty-seven point eight eight eighty point eight seven fifty-six point five six forty-eight point one four forty-three point four nine seventy point nine eight thirty-two point four eight twenty-seven point eight one fifty-seven point four seven sixty-eight point five six fifty-seven point four two
CPM (Li2018) eighty-two point one seven sixty-eight point eight two fifty-four point five six forty point nine nine seventy point three seven sixty-six point six nine thirty-five point nine eight thirty sixty-nine point six four seventy-one point five eight fifty-nine point zero eight
DeepFlow (Weinzaepfel2013) ninety-nine point one nine ninety-five point three two sixty-eight point six zero sixty-three point zero four eighty-three point one eight eighty-one point two zero fifty-three point eight two fifty-two point two two seventy-six point three two seventy-nine point one five seventy-five point two zero
RLOF6 (Geistert2016) ninety-seven point seven zero ninety-two point three seven sixty-six point seven zero sixty-five point zero eight eighty-eight point seven three ninety point two two forty-three point five six forty-six point four seven seventy-two point six zero eighty point one two seventy-four point three six
RLOF10 (Geistert2016) ninety-six eighty-five point zero two sixty-three point zero eight fifty-nine point seven seven eighty-five point nine seven eighty-six point six nine thirty-nine point four one forty point four eight sixty-nine point zero nine seventy-eight point seven zero seventy point four two
DIS4 (Kroeger2016) ninety-two point two two eighty-five point nine eight sixty-three point nine seven fifty-six point three five eighty-one point five nine eighty-one point six one forty-four point five eight forty-two point six four seventy-four point nine five eighty-two point zero nine seventy point six zero
DIS2 (Kroeger2016) forty point eight one twenty-two point three nine twenty-two point eight six fifteen point three seven nine point zero five six point seven two thirteen point six three nine point seven two seventeen point eight six eighteen point one zero seventeen point six five
Farneback (Farneback2003) eighty-eight point seven five eighty-one point three three sixty-four point six nine fifty-nine point zero five eighty-five point nine two eighty-seven point four four forty-two point four two forty-five point three five seventy-one point five one seventy-nine point six three seventy point six one
Sparse to Dense PLK (Bouguet2000) seventy-nine point three one sixty-six point eight three sixty-one point zero five fifty-two point four one eighty-two point six three eighty-three point one one thirty-seven point nine two thirty-six point eight one sixty-seven point five three seventy-six point one eight sixty-four point three eight
NMC (IDREES2014) ninety-six point nine six ninety point three three seventy-two point one eight seventy-one point four four ninety-two point two eight twenty point seven zero thirty-two point seven two forty-two point three eight sixty point one five fifty-six point zero two sixty-three point five two

References - Optical Flow Algorithm

 @inproceedings {Bailer2015, title = {Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation}, author={Bailer,  C. and Taetz, B. and Stricker, D.}, booktitle = {International Conference on Computer Vision}, pages={4015--4023}, year = {2015} }
 @inproceedings{Hu2017, title={Robust interpolation of correspondences for large displacement optical flow}, author={Hu,  Y. and Li, Y. and Song, R.}, booktitle={Conference on Computer Vision and Pattern Recognition}, pages={4791--4799}, year={2017}, }
 @article{Li2018,  author={Y. Li and Y. Hu and R. Song and P. Rao and Y. Wang},  journal={IEEE Transactions on Circuits and Systems for Video Technology},  title={Coarse-to-Fine PatchMatch for Dense Correspondence},  year={2018},  volume={28},  number={9},  pages={2233-2245},  }
 @inproceedings{Weinzaepfel2013, AUTHOR = {Weinzaepfel,  Philippe and Revaud, Jerome and Harchaoui, Zaid and Schmid, Cordelia}, TITLE = {{DeepFlow: Large displacement optical flow with deep matching}}, BOOKTITLE = {{Intenational Conference on Computer Vision }}, YEAR = {2013}, }
 @inproceedings{Geistert2016, AUTHOR = {Jonas Geistert and Tobias Senst and Thomas Sikora}, TITLE = {Robust Local Optical Flow: Dense Motion Vector Field Interpolation}, BOOKTITLE = {Picture Coding Symposium}, YEAR = {2016}, PAGES = {1--5}, }
 @inproceedings{Kroeger2016,  Author = {Till Kroeger and Radu Timofte and Dengxin Dai and Luc Van Gool},  Title = {Fast Optical Flow using Dense Inverse Search},  Booktitle = {European Conference on Computer Vision },  Year = {2016} }
 @inproceedings{Farneback2003, Author = 	 {Gunnar Farneb{\"a}ck}, Title = 	 {Two-Frame Motion Estimation Based on Polynomial Expansion}, Booktitle = 	 {Proceedings of the 13th Scandinavian Conference on Image Analysis}, Pages = 	 {363--370}, Year = 	 {2003}, }
 @TECHREPORT{Bouguet2000, author = {J.-Y. Bouguet}, title = {Pyramidal Implementation of the Lucas Kanade Feature Tracker}, institution = {Intel Corporation Microprocessor Research Lab}, year = {2000}, type = {Technical  {R}eport }, publisher = {Intel Corporation Microprocessor Research Labs}, timestamp = {2013.04.03} }

References - Person Tracking Algorithm

 @article{IDREES2014, title = "Tracking in dense crowds using prominence and neighborhood motion concurrence", journal = "Image and Vision Computing", volume = "32", number = "1", pages = "14 - 26", year = "2014", author = "Haroon Idrees and Nolan Warner and Mubarak Shah", }

About

Optical Flow Dataset and Benchmark for Visual Crowd Analysis

Topics

tracking computer-vision dataset video-processing synthetic-images video-surveillance optical-flow benchmark-suite motion-estimation multi-object-tracking evaluation-framework trajectories video-analytics crowd-counting crowd-analysis tracking-by-detection tub-crowdflow-dataset

Resources

Readme

License

GPL-3.0 license
Activity

Stars

one hundred and four stars

Watchers

seven watching

Forks

twenty-two forks
Report repository

Releases one

CrowdFlow 1.0.0 Latest
Dec 18, 2018

Packages

No packages published

Contributors two

  •  
  •  

Languages