Does ChatGPT Have Something Up It's Sleave?
Micah Beck
Let's apply information theory to generative AI. My hypothesis is that LLMs like ChatGPT contain within their data structures the information equivalent of a (possibly compressed) version of portions of their training sets. I'm not saying that there is a bitwise copy of any particular portion of the training set. Just that the information content of the parts of the the training set on which there is statistical agreement is present in some form.
This hypothesis implies that LLMs are answering questions about this subset of the training set in a manner that is equivalent to choosing a good answer from its training set and then rewriting it to hide its source. If course, what LLMs actually do is different - they decompose the source into tokens and probabilities and then reconstruct answers in a randomized fashion. So they never actually "look at" an element of the training set in writing their answer. The suggestion is that for this subset, the decomposition and reconstruction is equivalent to looking at the source and rewriting it.
The suggestion is that LLMs are doing sometime akin to card counting in Blackjack. The card counter looks for situations where their statistical methods enable them to play very well, and is otherwise an ordinary player. LLMs may be dazzling us when they detect situations in which they "know the right answer" because they have its information content stored, and otherwise are passable at bein g coherent and generally saying relevant things.
Unfortunately, we cannot analyze the stored information in ChatGPT to see if it contains the information equivalent of copies of its training set because we are not allowed to see either the stored data structure or the training set. "Move along, there's nothing to see here!"