Oh that is sad. I think children of that age should not have a choice until they are maybe about 12 or so.
I learned in the for the sake of the children course that children often say one thing to one parent and another to the other. So perhaps your son is saying this because he doesn't want to hurt your feelings?
Why doesn't he want to go? Does he rationalize a reason or is it that there are no toys and stuff at his dads? I think your son deserves to have a relationship with his father. My X let my kids NOT come to see me a few times and it just about breaks my heart, and then when I do have them and got ot ake them home they cry and say they don't want to go home and that just about breaks my heart. Anyway my opinion is that you should sent your son to spend time with his father until he is of an older age.