You actually can have too many friends in the game if too many of yours are inactive, barely active, or simply not Aiding/Visiting enough. Always be ready to cycle through the list and keep the good ones while discarding the bad ones.
Go down the whole friends list and Aid everyone for a couple days. Then, start Aiding from Town Hall only. After a week (or whatever) of that, right after Aiding from TH, go back to the whole Friends list and every person you seen that still has orange on their Aid tab you can delete and make room for better, unless they do something else worth keeping them around. I have a SAAB player on mine who doesn't Aid, he does Visit, and he's my emergency Goods Dealer for the rare times I can't get them from my Guild and another who has a massive Arc and takes first place on my GB's for over 1.9.
Think of the friends list as a garden. You have to keep it maintained and pull a few weeds every now and then.