As you grow and get better housing options, use that to your advantage by building ones that give you more population per unit size, and selling off the old ones. Gradually replace with the new. It gives you more room to do other things and still have the population growth. Don't be afraid of change.