"It doesn't" depends on specific implementation. "It can't" is wrong. https://arxiv.org/abs/2404.15993 "Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach (...) our method is easy to implement and adaptable to different levels of model accessibility including black box, grey box, and white box. "
It's trained on labelled data - to figure out how to interpret the LLM. But the external system is used only to interpret the hidden states already present in the analysed network. That means the original LLM already contains the "knows/doesn't" signal. It's just not output by default.
"By looking at different wrong answers generated by the LLM, we note that although our approach sometimes gives a high confidence score on a wrong answer generated by the LLM, at other times it shows desirable properties such as giving higher uncertainty scores to better answers, and giving low confidence score when LLM does not know the answer."
