A cusp catastrophic model of square pillar instability was developed under uniaxial stress conditions, based on the assumptive softrening constitutive relations under static mechanics. The static criterion of rock pillar instability is given: only when the stiffness of upper rock was smaller than the rock pillars’ stiffness at the inflexion of the descending weaken segment, the rock pillar would lose its instability. The dynamical nonlinear differential equation under nonequilibrium state based on the cusp catastrophic model were introduced. The characteristics of rock pillar’s chaotic evaluation were studied under the change of stiffness. The results show that the rock pillar’s motion is chaotic when stiffness were in a certain interval, which were proved by the calculated max Lyapunov exponent and the Poincare section.