How many bath towels, hand towels, washcloths do you need for each person in your home? One in use and one in the wash or in the linen closet? More than that? It all depends on your comfort level.
Do you need towels for guests?
Also, consider how many old, worn-out towels you need to keep on hand for major spills, cleaning, minor floods. When you sorted through your bath towels and washcloths did you set older ones?
Check with local animal shelters or wildlife rehabilitation centers and see if they could use old towels. Or, check to see if your recycling and disposal service collects old clothing and fabric items for recycling.