In my opinion, humans beings have really screwed up with nature. The ratio of guy to girls should be something like 1:10 or 1:4 Ever observed the primary predator like Lion, what is the ratio of males to females there? Besides, the primary psychology as well as biological difference between male and female also makes it necessary that there are more females and few males.
You have all the right to be concerned but don't just be concerned ... do somethin'...let few of your concubines go...
As I said above, there is a significant difference in psychology of man and woman. Even if "I let them go" they'll go to the one they want to be with (which is usually one in hundreds or thousands or more realistically one in ten). So they will flock to someone who already has a horde. Not the same with guys. If they are let loose, they can go to anyone.