A quadtree-polygon scaled boundary finite element-based approach for image-based modelling of concrete fracture at the mesoscale is developed. Digital images representing the two-phase mesostructure of concrete, which comprises of coarse aggregates and mortar are either generated using a take-and-place algorithm with a user-defined aggregate volume ratio or obtained from X-ray computed tomography as an input. The digital images are automatically discretised for analysis by applying a balanced quadtree decomposition in combination with a smoothing operation. The scaled boundary finite element method is applied to model the constituents in the concrete mesostructure. A quadtree formulation within the framework of the scaled boundary finite element method is advantageous in that the displacement compatibility between the cells are automatically preserved even in the presence of hanging nodes. Moreover, the geometric flexibility of the scaled boundary finite element method facilitates the use of arbitrary sided polygons, allowing better representation of the aggregate boundaries. The computational burden is significantly reduced as there are only finite number of cell types in a balanced quadtree mesh. The cells in the mesh are connected to each other using cohesive interface elements with appropriate softening laws to model the fracture of the mesostructure. Parametric studies are carried out on concrete specimens subjected to uniaxial tension to investigate the effects of various parameters e.g. aggregate size distribution, porosity and aggregate volume ratio on the fracture of concrete at the meso-scale. Mesoscale fracture of concrete specimens obtained from X-ray computed tomography scans are carried out to demonstrate its feasibility.