You will often read statements to the effect “LLMs do not understand what they produce as output”. My controversial view is that, in a way, we do not understand what we are saying. Consider this. When we speak words flow from us in a stream without ratiocination, that is, straight from our neural network. If stopped and asked to justify what we have said we are able to create a plausible answer. We believe we understand in the sense of always being able to justify our statements and answer questions about the topic and we can test if someone really understands a subject using a viva voce.
We believe we have a logical conceptual model of the world from which our statements flow but we actually have an impenetrable neural network. I maintain that our accessible mental models are a thin veneer of conceptual deep structures but the real work is done by an inaccessible neural network. This means that at some level our explanations and justifications are illusory, they are constructed post hoc. In the same way, LLMs can, and will become better at, explaining their output and they thus demonstrate their understanding. If an LLM can pass a doctorate level viva voce then can we plausibly argue that it does not really understand the subject?
I am not arguing that LLMs are a model of our neural structures, far from it. They are potentially much more powerful as we have many well-known limitations, our short-term memory is tiny, our ability to analyse deeply structured logic is very limited and our speed of thought is slow. I believe it is a mistake to argue that LLMs are not conscious, are not sentient and do not have feelings and emotions. All this may or may not be true but it is irrelevant to everyone except philosophers. The key questions is are they useful and beyond that can they perform at a level that exceeds any human. If so, we have a very useful tool, to say the least, that will amplify all that is good and all that is bad about our human species.