Low-Entropy Latent Variables Harm Out-of-Distribution Performance