It's affection always.

So what does it mean if I tell you to go fuck yourself?

Or if I say that you're beautiful to me.