There's nothing scary about clinical decision support systems. We have had those in use for years prior to the advent of LLMs. None of them have ever been 100% accurate. If they meet the criteria to be classed as regulated medical devices then they have to pass FDA certification testing regardless of the algorithm used. And ultimately the licensed human clinician is still legally and professionally accountable for the diagnosis regardless of which tools they might have used in the process.
The medical diagnosis example was just what I used to use with my ex-PhD supervisor cos he was doing medical based machine learning. Was just the first example that came to mind (after having to regurgitate it repeatedly over 3 years).
That shopping list will result in something user eats. Even that can be dangerous. Now imagine the users asking if the recipe is safe give their allergies, even banal scenarios like can get out of hand quickly.
> I don't know why it's so important to have puritan output from LLMs …
These are small, toy examples demonstrating a wider, well established problem with all machine learning models.
If you take an ML model and put it in a position to do something safety and security critical — it can be made to do very bad things.
The current use case of LLMs right now is fairly benign, as you point out. I understand the perspective you’re coming from.
But if you change the use case from
To then it gets a lot more scary and important.