Hi @billyjoe, welcome to the forum. I grew up with very similar fears. I always found a way to convince myself not to even talk to them sometimes let alone, accept invitations, so I suppose me fear were stronger than yours. I'm in my 50s now and don't have any friends I'd class a really close. I've done exceedingly well at keeping everyone at a distance it seems.
So having dealt with this and tried to figure it out for over half a century... with limited success, but I have learnt a few things. The main one is (and I know exactly how difficult it is) is to have a bit more confidence in yourself. I know it sounds obvious, but the core of it is to accept that some people may not like you, some people will like you a bit, but some people will really like you. So if we assume everyone is one of the first two types then we missed out on the third.
Some of the may think you're weird and/or annoying, but there's only one way to find out. Some people may like weird. Chances are though, they dont! Why would they invite you out if they didn't like you? I know this from any years of personal experience... the vast majority of the bad things we think about ourselves... other people dont see. What they do see, is a person who keeps refusing invitations or not getting more involved and they then start to think that person isn't interested and so the invites stop. Not because of what they think about you, but because they think you're not interested.
I still dread going to things where there are other people there, especially people I don't know! Yeah, its gotten easier but it still makes me very uncomfortable. I know I usually have a good time, but I still worry about what people might think of me. So we may not know if people like us or not, but if people really don't like you... you will know. And that's the thing I try to always keep in mind... if they do end up not liking me, they'll make it known. It's the people im you're not sure about, but never seem to keep in touch, that I still struggle with. I've been told I do tend to overthink things, and that's probably true. I've been told this by psychologists so it probably is. I think about that as well.
So if they're still your friends, they still invite you out, they still hang out with you, they still talk to you... they like you. I know it's hard, but sometimes in life we just need to accept the truth that some people will like us. Plus I would add, mainly because it makes me feel better about myself as well, that if you don't think a lot about "am I annoying?..." then I think we constantly look for cues from people to find out and try to adjust. Most people wont care and just assume they're not like that and that their friends really like them. In the vast majority of cases that is true, but a some from this group tend to be the ones who everyone finds annoying but no one tells them. Why would someone change if they see no problem?
God knows if I've just made things better or worse for you. I hope the first one. You seem like a good person to me, but like me, you need to stop convincing yourself all your fears are true.