The problem is that once you entrust growth of the population and future direction to powerful individuals, whether in the government or in business, the ordinary people lose their right and ability to secure their own future. Having a system whereby a nation is subject to arbitrary population policy is a crime against that nation of people. Period. Policy which may influence birth rates through incentives or disincentives is different, as population in those cases is still determined by people’s individual reproductive choices. The future of a population of people is no longer in the hands of the people who share its future, but in those who may only see the people as a resource, or worse still, only see the state as a resource and people as merely human assets. People then become interchangeable and replaceable. A subtle, but very significant shift happens here. The identity of the people, the nation itself, the nation of people no longer exists in a political sense. The country has changed from being a tool of a people to organise themselves and to serve them, to a resource controlled by a few, whose make up is arbitrary and can be changed at will. All that matters now is numbers, and some will argue the population should be higher, others lower, but this is now all done from the perspective of the needs of the economy and the needs of the system. The people essentially hand themselves over and are now owned by state and government interests. Whether the state is a democracy or not, whether it has been popularly voted it or not doesn't matter. No individual should have the right to decide the fate of a people.
This has been happening for some time in the West and is so obvious, that no one sees it any more. But if, for example, you took control of Nepal and convinced them to have a population policy, and that the primary political endeavour would be to expand "Nepal" economically and structurally, then you would have essentially changed the nation to one which exists for its own sake because it is a nation of people, to one which is working for defined objectives (Growth, etc.). The people are now a means to an ends, rather than the ends in themselves. Now having established that, you can implement any legal population policy you like, even if it is one which results in rapid growth and in the original population becoming a minority or even being assimilated into the population you constructed and defined. You have also severely undermined the moral authority of the people to oppose you. You can even go further, and accuse those who object of undermining the country’s future.
This is an act of hostility against the nation, but by getting people to accept the premise that "Nepal" can and should be run as an enterprise, rather than be acknowledged as a nation of people, they've tacitly accepted this fate. If my goal was to engineer, modify the nation, or even completely change it demographically, I have targeted them, their way of life and identity for destruction. If one’s vision of 'progress' however involves destruction, destruction of suburbs, of parks, of heritage buildings and open spaces which people enjoy, why wouldn't it extend to include people? But there is a significant difference. One can own a building, and while it is a shame to allow people to destroy a building of beauty for gaudy apartment blocks, their ownership fits in with our morality.
But can one own a people? Does someone have the same right to engineer or change a population) the way they do their possessions? I argue they do not, and that being democratically elected does not grant one that right. Elected or not, no one has a right to decide for a people what their future is. Countries don't create nations. Nations of people create countries. Therefore the government which runs a country has no right to engineer a nation. Whether it is the Japanese nation, German or Kenyan, their respective government cannot assume the right to change the make up of the nations they are responsible for. The people of those nations must therefore call their governments out when they assume powers that aren't theirs.
We, however, are discouraged from looking at the issue in this manner. "Practicality" we are told, must trump all. But does economic benefit, growth or the housing industry justify this? Is it acceptable to reshape humanity, or sections of it for these purposes, or allow this to happen because of your policy?
We can see a more recent, clear example here.
This conference believes that the destiny of the natives of Aboriginal origin, but not of the full blood, lies in their ultimate absorption by the people of the Commonwealth and it therefore recommends that all efforts be directed to that end. Nobody who knows about these groups could deny that their members are socially and culturally deprived. We must improve their lot so that they can take their place economically and socially in the general community. Once this is done, the breakup of such groups will be rapid.  
This was viewed by some as genocide, but not through killing or displacement, but by engineering the population. This was a deliberate attempt, but what if it was just promoted as an ideal? What if the government gave incentives for people to move to make this happen? What if the government marginalised Aboriginals who objected as being backward and intolerant of change? Are there any circumstances by which this goal could be achieved, and this be considered acceptable? I would argue not, and most people would likely agree.
We must be aware of the potential problems that can occur through poor 'population policy'. We should analyse it not just on economic and practical terms, but make those who promote policy accountable for the cultural and national impact that such policies have. If those policies pose threats to an identifiable group of people, then we must hold people who propose such policies morally accountable. This seems odd, as in the West we have been conditioned not to view ourselves this way. Populations always change over time, naturally, this is to be expected. But there is a smooth continuum from natural change, to forced change, from the migrations of people westwards into Europe prior to the establishment of civilisation, to forced assimilation to “breed out the black”. Between these two, there is a vast grey area, an area where it easy to move from acceptable to unacceptable. Where change can turn from natural to forced. Where demographic change can turn from natural evolution to forced engineering, as is happening in the West.
The idea of 'population policy', the tool that governments use to move masses of people can easily swing us from acceptable to unacceptable. It is this we must be wary of, and be willing to call our elected leaders out on.