For a while now I had an issue where the playback from Spotify was starting stuttering if it wasn't the active window. For no apparent reason. CPU and RAM where both ok and had a lot of free capacity.
Apparently it seems to be connected to App Nap. When disabling it for the Spotify app, everything was back to normal. To disable it use the following command line:
$ defaults write com.spotify.client NSAppSleepDisabled -bool YES