Author here. I actually came up with the idea a long time ago, I first experimented with variants of this in Caffe (before Tensorflow was a thing).