Brutal difficulty

I’m playing a private world with 4 friends, and we’ve been playing on brutal difficulty and we are up to Tristan. If we turn the difficulty down to normal, will the bosses just start to melt or will it still be difficult? Because we’ve ended up skipping the boss in the left of the forest because the amount of shit she summons is just horrible. I fear for our sanity when the bosses actually get very challenging.