Particles held in optical tweezers are commonly thought to be at thermodynamic equilibrium with their environment. Under this assumption the elastic energy of the trap is equal to the thermal energy. As a result the variance of the particle position is completely independent of viscosity and inversely proportional to the optical power in the trap. Here we show that these conditions only hold for very high symmetry cases e.g. perfectly spherical particles in unaberrated, linearly polarized Gaussian traps. Here we show that any reduction in symmetry leads to asymmetrically coupled degrees of freedom. The associated force field is linearly non-conservative and the tweezer is no longer at equilibrium. In overdamped systems the effect is a underlying systematic bias to the Brownian motion. In underdamped systems, this systematic component can accumulate momentum, eventually destabilizing the trap. We illustrate this latter effect with reference to two systems, (i) an isotropic sphere in a circularly polarized trap, and (ii) a birefringent sphere in a linearly polarized trap. In both cases the instability can be approached either by decreasing air pressure or by increasing optical power. Close to instability, the trapped particle executes increasingly coherent motion that is highly sensitive to external perturbations. Potential applications to weak force sensing are discussed.