Two separate issues
i) First Christmas overnighter is huge ... but the courts hold sway .. give kids some comfort objects for settling ... and breathe and be calm and punctual and civil, if cool, with a pounding head and heart.
I wouldnt jump to the Rose Batty scenario immediately ... those events are among the great minority, though we see them in the news. We cant know for sure so I wont say never.
Whether he is a ticking time bomb and you are on hyper hyper alert ... try and proceed as if things will turn out for the best. Dont be falsely happy or make promises about how it will or wont be ... I just followed the line that I wanted kids to have a father (or my eldest to have contact with her natural mother) but if there were problems .. be there for them.
I remember our worry over our "first" child in 1986 .. she went to her grandmothers when she was 9 months for Christmas .. and it has dominated every christmas since ... so now I keep some detachment about the silly season. We were terrified, but everything did turn out ok. The biological mother was only awarded supervised access.
To my thinking, the best outcome is that the dad gets a wake up call about dealing with small children ... men often need that ... and sometimes do rise to the challenge ...
ii) Full time WORK is huge .. it can be better to get straight back in .. you have had a few years off ... but you will probably get your confidence back once you are in a routine ... and the distraction of work can be a relief ... I am surprised you could nt find a part-time option but as that seems not possible ..
I hear your doubt and anxiety but would not want to discourage you from going back. The majority of woman find part-time more viable ... but you may be better off at work. ... sorry to equivocate ... I couldnt have done it ... but thats just me.