What do you usually do during the model training?

Hey everyone! Just curious—what do you usually do during those long, long training runs?

For me, I end up double-checking my entire code to make sure there are no mistakes, then reading papers while I wait. I also check my training curve every 2–3 hours to see if everything’s on track.

I feel like this whole monitoring process could be automated. Has anyone tried something similar? Or do you just write your own scripts for it?