Ellerton abbey: How slave owners created one of the North's most romantic ruins

Yet its estate was probably bought with the proceeds of slavery in the 1690s, and the mansion itself was built in the 1830s when its owners were claiming a large amount of compensation from the British government for freeing the slaves on their sugar plantations in Barbados.

The owners’ names can still be seen on boundary stones that still stand in Swaledale: “ED”, say the stones, marking out the territory of the Erle-Drax family.

A small priory at Ellerton, which is between Richmond and Reeth and clearly visible from the main road which runs along the dale's bottom, was founded about 1200. It enabled a maximum of 13 nuns to live in a cloister around their church with a little village nearby. Although the nuns were never rich, they did rig up an elaborate water supply fed from a reservoir dug high on the south side of the dale.

Henry VIII took control of the priory in 1536, and the prioress, Johanna Herkay, went off “without demur” to live in Richmond. The medieval village was deserted and the priory buildings fell into wrack and ruin.

The Ellerton estate went through several private owners until the 1690s when Colonel Henry Drax of Drax Hall in Barbados was looking to buy an English estate that would provide him with an income of £10,000 a year. He bought Ellerton.

The colonel’s father was James Drax (1609-61), originally of Warwickshire, who was one of the first immigrants to land in Barbados. Initially, James lived in a cave while he cleared the land to grow first tobacco but then sugar.

James – from 1658, Sir James – was renowned for his ingenuity and became the first successful sugar grower in the Caribbean, sending his first sugar back to London in 1643. Using the first slaves, he quickly became phenomenally wealthy, and built Drax Hall which is now one of the oldest buildings on Barbados.

The colonel, who owned the biggest plantation on Barbados, began re-shoring some of that fortune by investing in Ellerton, and then the his son, Henry (1693-1755) married his first cousin, Elizabeth Ernle. She was the heiress of the Erle family of Charnborough Park and so the surname became Erle-Drax, as can still be seen on the Swaledale boundary stones (the family liked acquiring surnames and the current resident of Charnborough Park, who is a Conservative MP, is a Grosvenor-Plunkett-Ernle-Erle-Drax).

This Henry was also the first of five generations of his family to be an MP while connected to Ellerton. They usually represented seats in Dorset and Kent and, like the Lascelles family of Northallerton who featured here recently, liked to be in the Commons to keep an eye on legislation affecting the slave trade.

They failed to prevent it from being abolished in 1807 and slaves from being emancipated in 1833. The British government allocated £20m to compensate slave-owners, and a seventh generation of Draxes applied for £4,293 12s 6d for the 189 slaves they liberated at the Drax Hall estate (more than half-a-million pounds in today’s values, according to the Bank of England Inflation Calculator).

Many slave-owners found ways to bring their compensation back to the UK, investing in property or industrial companies.

It might therefore be more than coincidence that in the 1830s, John Samuel Wanley Swabridge Erle-Drax spent a considerable sum of money building for his wife, Jane Frances, through whom he had acquired the Ellerton estate, a Georgian villa shooting lodge on the edge of the Swale.

Perhaps it was to appeal to her romantic nature that the remains of the priory’s 15th Century church were stabilised so that the tall tower survives with specimen trees growing in the roofless nave.

It is a beautiful spot, ultimately still owned by the Drax family, but it shows how the proceeds of slavery seeped into so many corners of our countryside in centuries gone by.

• We are hugely grateful to Alan Gibson for his help with this article