jasonjiang

yes, just stream it, it's a no brainer in your case which won't fill your disk at all just make sure your networking infra is fast enough

steaming is always the way to go since generally neural nets trainings are stateful any way🚀