The fix for this is supposed to be go to into your Android settings and turn off battery optimization for Google Home and each app you cast.

It didn't work for me but loads of people reported success with it. It's a shame because I loved the Chromecast but had the exact same issue as you describe.