Do the poor have any right to live in expensive areas?

Lots of areas that are now expensive were not in recent history. So every time the wealthy move in to poorer areas to get cheap property, do the places up until they're expensive we then kick out the original inhabitants do we? Might just as well reintroduce the Enclosure Act and put the poor firmly back in their place. If this happens I hope every lowly paid worker refuses to work as cheap labour, then see how the richest cope.