With children as young as yours there does need to be a routine taking into account the shift pattern of your ex.
Like I say my ex works a 16 week rolling rota and his days off are never the same.
I offered him lots of contact but said that it needed to be structured in some way so that the children knew when they were seeing him. Eventually he only felt able to commit to what works out to 12 hours a month.
As far as the weekends are concerned, he is already stating that he doesn't want the children every weekend he isn't working. In my opinion you need to listen to this, not because he has a point because as PWC we don't have a choice when we have the children but because contact will not work if he feels like he doesn't have a life.
For ages I was pushing for my ex to have reasonable contact but eventually realised that he wanted minimal hours so that he could see the OW, play golf, go fishing, go out drinking etc etc. At our only mediation session (he walked out half way through not to return) he stated that he was entitled to a social life.
Working it out the other day, my ex sees our children less than 1/5 of the time that he doesn't work (not including the days he is on early shift etc).
As for relocating, this could be raised at mediation and something written into the agreement as a base to work from if and when the need arises.
Like I say, you need to decide what you are happy to go along with and what you feel will not work for you as the PWC (not being obstructive but you and the children are entitled to a life as well).