this worries me - this sounds so much like the ''victim mentality'' that my ex screams about whenever he gets the opportunity. If you apply for residence and get it, it''s out of his hands, yes. But you''re the bad guy. When it''s ''why don''t we see you as much as we used to daddy'', he can reply ''well, your mum thought it best that you spent more time with her''. As they get older, he''s able to tell them that you took him to court so he had to abide by a court order (and that''s assuming you get one) so again, he''s the victim, he wanted to spend more time with his children but the ex wife and the courts conspired against him....
I would personally suggest removing yourself from the family home as best you can so that he can''t drop back when it suits him, not picking up your phone and letting him leave a message so you can call back if it''s some kind of emergency rather than him wanting to cut his contact time short. I''d do that before applying to the courts for a Residence Order which will cost you both money and who knows how much emotionally, not to mention the impact on the children. Even
mediation might help if he''s willing. But I''d personally avoid court unless you''re backed into a corner. And even then, I''d do what I could to avoid it!