Hi @MintyX,
I can absolutely relate to what you are saying. Years ago, I was in the same position as you. I was very lonely in my life, so I would go and visit my parents for a few days at a time, just for some company and something to do.
The thing is, my sister would bring her kids over all the time for my parents to mind! I would find that they would be very pre-occupied with the kids and I would feel that I was more alone than ever. Also, the children were only toddlers and they seemed to be always crying and my mother seemed to be always yelling. Just stress, chaos and noise, constantly. I believe this situation contributed to my depression, I'm sorry to say.
Unless you love kids, then being around them constantly is very tiring; it's not soothing.
The other thing is that for people like me, who suffer from anxiety and depression, you can actually feel better in the evenings anyway, after a full day, so that my partly explain why you feel better alone at night.
It could be worth going for a long walk by yourself, when the children are over, to get some alone time. That's what I used to do. Or go to the library and sit and read. Or go to the movies. Either way, it's good that you recognise that being in a busy environment with lots of people is not for you right now.
When you begin to heal from depression, you may again love being in the company of lots of people. Or you may be more of a loner. We all have different needs when it comes to socializing and solitude.