We propose Video Driven Traffic Modelling (VDTM) for accurate simulation of real-world traffic behaviours with detailed information and low-cost model development and maintenance. Computer vision techniques are employed to estimate traffic parameters. These parameters are used to build and update a traffic system model. The model is simulated using the Paramics traffic simulation platform. Based on the simulation techniques, effects of traffic interventions can be evaluated in order to achieve better decision makings for traffic management authorities. In this paper, traffic parameters such as vehicle types, times of starting trips and corresponding origin-destinations are extracted from a video. A road network is manually defined according to the traffic composition in the video, and individual vehicles associated with extracted properties are modelled and simulated within the defined road network using Paramics. VDTM has widespread potential applications in supporting traffic decision-makings. To demonstrate the effectiveness, we apply it in optimizing a traffic signal control system, which adaptively adjusts green times of signals at an intersection to reduce traffic congestion.