Transformation of strain energy increment in catastrophe model and its application to stability analysis of host rock in nuclear waste disposal caverns

• Introduces a novel energy-driven instability criterion based on cusp catastrophe theory, reducing subjectivity in deep rock engineering stability assessments. • Derives an explicit algebraic transformation linking quartic energy potential to standard cusp form, enabling the mutation eigenvalue Δ as a physically interpretable instability measure. • Validates the framework through scaled physical model tests and simulations, achieving conservative safety factors (K=2.33 and 2.73) with deviations under 6% from experimental results. • Demonstrates that the improved nonlinear H-B strength reduction method yields more conservative safety factors than equivalent Mohr-Coulomb approach, better matching observed failure processes.