The problem is that all of your available population are taken up by working at your goods producing buildings. If you add all that are required for each goods buildings, you do not have enough left over, spare population, to ditch the house you are trying to ditch.
The number at top, with the people is your spare population, if the building you are trying to ditch has over that number, then you would not have enough to run your goods buildings. So, you would have to build the larger building, wait till that population is added, then ditch the old building. ok?