The limit for attached units is how many your military buildings have slots for. Your regular military buildings start with 2 slots available. 2 more can be unlocked for coins and supplies. 1 can be unlocked with diamonds. Each unlocked slot supports 1 attached unit. So 1 building can have up to 5 units attached to it. I don't unlock the last slot with diamonds, because when I advanced to the next age I delete the building (and replace it with a military building for the new age) - so the cost in diamonds isn't worth it.
You can add additional military buildings for more slots and can have multiples of each type. So you could have 2 spear fighter camps for example (or more).
Special buildings like the Rogue Hideout and Drummers camp have different slot setups, but they work the same way when unlocked.