Recent experiments in quantum optics provide an ideal platform to study non-equilibrium phase transitions. In particular, these experiments provided the first realization of a fundamental model of statistical mechanics, the Dicke model, and demonstrated its second-order phase transition. A detailed analysis of the experimental observations, however, reveals important discrepancies from the Dicke model. This is due to decay and dephasing processes, which cannot be taken into account in the framework of the common mean-field approximation. We include these effects by an appropriate cumulant expansion of the Lindblad master equations and find significant modifications to the stability diagram of the system. We find that in addition to the known Dicke phase transition, the model can undergo a distinctive non-equilibrium transition. This instability is analogous to a lasing transition, but is induced by counter-rotating terms. This novel effect is demonstrated by recent experiments on trapped atoms in optical cavities.