As long as CLI programs stick to the 8 or 16 standard colors and refrain from setting background colors (inverse mode is fine), as well as from explicitly setting white or black as text color, everyone can reasonably configure their terminal colors so that everything is readable.
When going beyond that, the colors really need to be configurable on the application.
Sticking to the \x1b[4X background color is probably safe as that can be tweaked by terminal color palettes. It’s when you use the 256 or RGB \x1b codes that it becomes an issue. Ok for foreground.
> constraining to invert is not sufficient in my opinion
Eh. Doing green/red text color with default background, maybe inverted works amazing for me. In fact, I'd say that every sensible colour scheme for a terminal should have as the default foreground/background colours something that is more or less contrasting against every other explicitly named colour, including black and white (I personally have #212121 on #EEEEEE).
When going beyond that, the colors really need to be configurable on the application.