Both of my dss were young for their grade. Oldest ds started K in one state. We moved the summer between K and 1st grade and the new district's cut off was 1 month BEFORE ds's b-day. He is and has always been the youngest in his class. He also had fine motor difficulties and was small for age but academically bright. The first few years were rough socially but once he found his niche, he did really well. He's now in HS, all honors classes, doing well. The only issue now is that some of his friends who are freshmen will actually receive their drivers license at the same time as or even right before ds. It doesn't seem to bother him.
Youngest ds's b-day was 2 weeks before the cut off date. This ds has had more health problems than older ds and is smaller (5%) than older ds was. He is always the smallest and one of the youngest, if not youngest. Like older ds, he, too had some fine motor difficulties but was bright. He's very social and has not encountered any problems at school (2nd grade.) He's in the gifted pull out class but is bored to tears with the math curriculum (no acceleration in math at our school.) Like oldest ds, I couldn't imagine him being in the lower grade without significant problems.
Funny thing, my dd's b-day falls right in the middle of the school calendar. I thought we'd be in great shape with her but I actually worry MORE about her than her brothers. Like both of them, she too, has fine motor and even some mild gross motor difficulties. Although she's doing well in school, she's not at the same level her brothers were at that age. If her b-day were close to the cut-off like the boys, I am sure I would have held her back. Size wise she's right on target with the other kids.
My suggestion is to look at the whole picture. Forget about just the size because someone has to be the smallest... (Yes, that's easier said than done, I know.) If your ds is academically ready and is a fairly social kid, he'll probably be ok. If, however, he's on the edge of being ready in terms of social readiness AND academic readiness, I'd consider holding him back. Also, can you make a decision closer to the fall? Children change so much so quickly, he may well be ready by then. Good luck!