I haven't read every response so I may be repeating someone. I work in pre-school and kindergarten (speech pathologist) and I've had some experience observing kids going through that transition. First of all, full-day kindergarten sounds scary, but they really adjust to it very nicely. There are kids who I see who seem very young starting kindergarten and by the end of the year it is amazing how much they have learned and grown up! It really is a matter of readiness. In kindergarten they do expect you to sit and listen for much longer than in pre-school. I know you probably had a kindergarten screening, so I would look at how that went as well, even though some kids are very shy on those days and don't show what they really know.
One option is to try it and if it really is just not working, you can pull him out. That is not ideal. But chances are he would end up doing just fine. ANother scenario is that he could do kindergarten twice. That actually happens quite a bit and just gives them that extra year of skills to get ready for first grade.
If you do decide to hold him back, I would make sure that he is in a pre-school program, rather than a daycare program. Someplace where they have educators who really focus on the readiness skills.
Don't worry about what anyone else says in the long run. All in all, it won't make or break him no matter what people say!
