Single-particle-tracking is a fundamental prerequisite for studying biological processes in time-lapse microscopy. However, it remains a challenging task in many applications where numerous particles are driven by fast and complex motion patterns. To anticipate the motion of particles most tracking algorithms usually assume near constant position, velocity or acceleration over consecutive frames. However, such assumptions are not robust to large and sudden velocity changes that typically occur in in vivo imaging. In this paper, we exploit optical flow to directly measure the velocity of particles in a Kalman filtering context. The resulting method shows improved robustness to correctly predict particles positions, even with sudden motions. We validate our method on simulated images with high particle density and fast elastic motion patterns. Quantitative results show a decrease of tracking errors by a factor of two, when compared to other tracking algorithms, while preserving fast computational time.