Finite element analysis incorporating the shear strength reduction method was applied to study the west slope stability of an open cut mine in Australia using Mohr–Coulomb and generalized Hoek–Brown criteria. The pit of the mine had multiphase excavations and reached 180 m in depth. The study investigated three slope configurations, namely, Stage 1 inter ramp slope 43°, Stage 2 inter ramp slope 49° and optimized Stage 2 slope 54°. When implementing the generalized Hoek–Brown failure criterion, the equivalent factor of safety was 1.96, 1.87 and 1.40 under dry slope for the three configurations, respectively. However, under partly saturated conditions, the optimised slope would have a factor of safety 1.16. Furthermore, the generalised Hoek–Brown criterion generated lower factors of safety than the Mohr–Coulomb failure criterion. The difference is related to an overestimation of the shear strength parameters by the linear Mohr–Coulomb criterion under low confining stresses compared with the non-linear Hoek–Brown.