that can easily get pretty tricky if you don't want to depend a lot on libraries, which the last time I checked weren't great for golang in this area. I have worked with ALSA (advanced linux sound architecture) a bit, implementing basic midi input from a keyboard + sound generators + plugins with go interfaces and pkg/plugin + stream to audio output... and it's a mess of devices, buffers and formats. and that's only for one OS. and if you start doing anything non-trivial, making the programs work well in real-time is not for the faint of heart.
if there was a single format supported everywhere, minimum buffer sizes and a common API for all OSes, it would be a whole another, much more pleasant story