Why are days with no structure so difficult and overwhelming??

Let me start this off by saying I do not want comments such as "it's good to give yourself down time" or "you need to rest and time to recharge".This is not what this post is about.

I really struggle on days where I've not planned anything for the day meaning my time goes to waste. I might not have energy until like 5pm and by then everything is winding down, especially on weekends. How do I structure the days I don't have work with my ADHD?? I've wasted my whole Sunday.

I'm so annoyed at myself. I have so many days like this where I feel so fed up with myself especially when I spent the day bed rotting.