I know men are going to hate this response, but you asked, so here goes.
Honesty is the best policy....but NOT with this man!
Don't ever tell him that his size affects how you feel about him.
But do be honest with yourself and decide if this is a deal breaker for you and if it is, spare the both of you future problems and stop seeing him now while the relationship is still new.
I know you can feel shallow for this, but try to look at it as something else. Say, a pot belly or heavy breathing that gets on your nerves or anything else that might affect an initial attraction.
Attractiion is a tricky thing and as the sitcom "Seinfeld" reveals...even small issues can cause a problem.
So....this is your particular problem. And I use the word "problem" here because you're writing about it and are questioning whether you can deal with it. I'm certainly not saying lack of size is a problem.
I'm sure this man is very nice. And if he's quality stuff and you are attracted to him in every other regard, you might try sticking with him a bit longer because I'm telling you, a good man is hard to find.
But as I said before, if your feelings are fading and the relationship is fairly new, it might be best to move on.
He won't be alone. Some lucky woman who is fine with all of him will snatch him up and have a great life with him.
Maybe he's just not for you.
And look at men! They brag about being "breast men" or "leg men" or "ass men" and don't make any apologies for it! Women are ALWAYS judged on their physical attributes. Ugh...don't get me started!
so...just be kind...keep your mouth shut and decide what this man is worth to you and DO be honest with yourself. don't try to force yourself to stay with someone just because he's nice and you don't want to hurt his feelings. You won't be doing him any favors.
Eh...I'm not sure I expressed myself very well.
Just be nice and best of luck.