Feeling confused, diagnosis has suddenly changed?

Note: not asking for medical advice here, just trying to see if anyone can relate/give support

I've been told for 3 years that my diagnosis has been bipolar. It's been somthing that I agreed with, connected with and have been medicated for for a long time.

I saw my psychiatrist like normal last week. I was hypomanic at the time according to my psychologist. My psychiatrist started off like normal, mentioning I was elevated due to my bipolar, but then he suddenly said halfway through my session that he now thinks I have BPD. He started saying that he thinks it's that because my relationships are unstable (I've recently fell out of touch from some uni friends because I've changed my studies), that I have mood swings (once every few months/weeks my mood will change but I thought this was due to bipolar) and that I have identity issues due to doubting my career. I really don't connect with the BPD diagnosis or feel that it fits, and honestly I feel that I do have bipolar, but stupid me at the time was hypomanic and when he asked if I thought I had bipolar I said no because I always refuse my diagnosis when I'm elevated.

Then I asked him if I could stop my meds since I don't have bipolar, now I ALWAYS want to stop my meds when I'm elevated. He told me sure, and when I asked him how he told me to "do whatever I want because I'll do it anyway". Like what?! So that night I was so excited to not take any meds, then my brother called who is a doctor. He told me that I do have bipolar and not to stop my meds under any circumstances, and that we need to wait for a second opinion. Somehow I managed to listen to him, and not only did I not stop my meds but I started taking some PRN medication for a few days to calm the mania.

I'm feeling really confused now though. I think the medicaiton has settled my mood and I'm feeling much more euthymic. I hate taking medicaiton so I still really want to stop them, it's such a battle to take them every day and I feel like now that he said I could stop them that it'll be fine if I do. I don't know if I should believe my brother or my psychiarist. This is such a hard choice to make!