How can I not be seen as so vulnerable?

Just say NO. Seriously, it's OK to pick and choose what you do and don't want to do. It's your life and I get that you don't want to sweat the small stuff but there's a fine line to managing this and I don't think you've found it yet. I mean really, do you think that someone's going to be devastated if you don't join them on some occasions. It does seem a bit arrogant of you, especially if you think that someone is wanting to spend time with you just to "take up my time" and want to be "seen" with you. The only way I can think of to not come across as vulnerable is to be more decisive and take a stand once in a while.