Exactly. When farmers outnumber you, guess who's winning any land disputes? Even if land changes hands peacefully, a farmer can give you more food for your land than you could possibly gather on it. As Diamond points out, today's gatherers live on land that's not good for farming.