Although I am a heavy VirtualBox-user (vagrant is just too good), I can relate to this.

The only reason why my machine ever panicks is exactly the one cited in this mail: vboxdrv.