- Population growth is not solely dependent on food production.
- Transportation matters. Especially in the early stage, food produced far away simply can't reach the city. Inventing the wheel would greatly enlarge that reach, for example.
- There will be other factors that make ridiculously large cities undesireable, like crime and health.
Thanks for the feedback so far. I hope you might also have to interesting ideas on how to work this out, instead of just saying what would be wrong.