I agree with Fiona in that sometimes there is a valid reason and forcing contact is counter-productive.
However if there have been no real issues in the past (eg violence) and the younger child is happy to go. I think without realising it you are blocking contact by allowing the child to have a ''''choice''''.
Time for a bit of tough love I am afraid. You have to be firm and say sorry but your going.
I agree on this - 7 is too young to make that choice. My youngest is six and has said a couple fo times she doesnt want to go to dads, mainly cos she wants to make cakes with me, or dad told her off last time and she hasnt forgotten it!
She gets put in the car and after ten mins I usually get a text off my ex saying she is fine.
You wouldnt allow a 7 yr old to not go to school or the doctors or dentist - not that I am saying going to their dads is like the dentist btw, just making the comparison!
At 7 I feel he is too young to be honest to make up his own mind.