Isn't it crazy how luffy didn't care much when Ussop left the crew after 2-3 days but decided to stop being a pirate without Sanji in his crew?

Is it possible that the SHs could somehow manage to counter Perona and Sugar without Ussop? This is the protagonist's crew we are talking about they could have found a way. Ussop is never beating the allegations is he?

Edit: Guys pls don't down vote this is just a late night thought that struck me I am not trying to be offensive towards Ussop