This manuscript presents an extension of the recently-developed hybrid polygon-quadtree-based scaled boundary finite element method to model crack propagation in concrete. This hybrid approach combines the use of quadtree cells with arbitrary sided polygons for domain discretization. The scaled boundary finite element formulation does not distinguish between quadtree cells and arbitrary sided polygons in the mesh. A single formulation is applicable to all types of cells and polygons in the mesh. This eliminates the need to develop transitional elements to bridge the cells belonging to different levels in the quadtree hierarchy. Further to this, the use of arbitrary sided polygons facilitate the accurate discretization of curved boundaries that may result during crack propagation. The fracture process zone that is characteristic in concrete fracture is modelled using zero-thickness interface elements that are coupled to the scaled boundary finite element method using a shadow domain procedure. The scaled boundary finite element method can accurately model the asymptotic stress field in the vicinity of the crack tip with cohesive tractions. This leads to the accurate computation of the stress intensity factors, which is used to determine the condition for crack propagation and the resulting direction. Crack growth can be efficiently resolved using an efficient remeshing algorithm that employs a combination of quadtree decomposition functions and simple Booleans operations. The flexibility of the scaled boundary finite element method to be formulated on arbitrary sided polygons also result in a flexible remeshing algorithm for modelling crack propagation. The developed method is validated using three laboratory experiments of notched concrete beams subjected to different loading conditions.