Enemy Lord cannot be removed from conquered castle.

As the title says, If I take a castle and there's an enemy lord garrisoned at that castle then basically I take him, I ransom or execute him, and then *poof* I go back to the castle and NOW he's in the Lord's Hall and I cannot ransom him, I can't talk to him and he won't leave. No big deal right....WRONG! Here's what happens say you declare peace between his faction and yours and pay a tidy sum to get that peace because it's the Western Empire. Well after a day because that Lord is still in your castle you AUTOMATICALLY GO BACK TO WAR WITH HIS FACTION BECAUSE HE'S CONSIDERED A PRISONER! Even though he's not in the prison he's just sitting in the lord's hall and it does this for every single castle that has an enemy lord in it so take a few castles and you're permanently at war with half the world.

I haven't found any fix for this in the community or mods on nexus. Am I the only one encountering this bug?! Is there a fix?!?