Well, I tried a new HDMI topology and that didn't improve things. I tried intentionally mis-configuring the TV to break ARC but still have CEC power and volume control of my sound bar, but the sound bar still defaults to ARC on power up rather than SPDIF.

I've given up on ARC completely in my setup. The Nvidia Shield remote can control sound bar volume via IR. I just successfully watched a full movie without any drop outs. Also, I may be imagining it, but the Shield feels more responsive. It seems like it must be the TV's fault. Don't buy a Sony I guess.