|
Does ChatGPT Have Something Up It's Sleave?Let's apply information theory to generative AI. My hypothesis is that LLMs like ChatGPT contain within their data structures the information equivalent of a (possibly compressed) version of portions of their training sets. I'm not saying that there is a bitwise copy of any particular portion of the training set. Just that the information content of the parts of the the training set on which there is statistical agreement is present in some form.This hypothesis implies that LLMs are answering questions about this subset of the training set in a manner that is equivalent to choosing a good answer from its training set and then rewriting it to hide its source. If course, what LLMs actually do is different - they decompose the source into tokens and probabilities and then reconstruct answers in a randomized fashion. So they never actually "look at" an element of the training set in writing their answer. The suggestion is that for this subset, the decomposition and reconstruction is equivalent to looking at the source and rewriting it. The suggestion is that LLMs are doing sometime akin to card counting in Blackjack. The card counter looks for situations where their statistical methods enable them to play very well, and is otherwise an ordinary player. LLMs may be dazzling us when they detect situations in which they "know the right answer" because they have its information content stored, and otherwise are passable at being coherent and generally saying relevant things. Unfortunately, we cannot analyze the stored information in ChatGPT to see if it contains the information equivalent of copies of its training set because we are not allowed to see either the stored data structure or the training set. "Move along, there's nothing to see here!" |
How We Ruined The InternetIn this paper we examine an assumption that underpinned the development of the Internet architecture, namely that a loosely synchronous point-to-point datagram delivery service could adequately meet the needs of all network applications, including those which deliver content and services to a mass audience at global scale. We examine how the inability of the Networking community to provide a public and affordable mechanism to support such asynchronous point-to-multipoint applications led to the development of private overlay infrastructure, namely CDNs and Cloud networks, whose architecture stands at odds with the Open Data Networking goals of the early Internet advocates. We argue that the contradiction between those initial goals and the monopolistic commercial imperatives of hypergiant overlay infrastructure operators is an important reason for the apparent contradiction posed by the negative impact of their most profitable applications (e.g., social media) and strategies (e.g., targeted advertisement).How We Ruined The Internet Micah Beck, Terry Moore arXiv:2209.03482306.01101, June 2023 Submitted to Communications of the ACM |
Breaking Up A Digital Monopoly
Breaking Up A Digital Monopoly |
The Hedge Podcast Episode 150: Universal Broadband
A discussion of whether a less synchronous form of broadband connectivity be more cheaply and easily deployed to the entire world.
Deployment Scalability in Exposed Buffer Processing Micah Beck 17th IEEE International Conference on Mobile Ad-Hoc and Smart Systems (MASS 2020) Delhi NCR, India, December 10-13, 2020 (Virtual conference) |
IEEE MASS 2020 presentation, December 2020 |
"On The Hourglass Model" Micah Beck Communications of the ACM, July 2019, Vol. 62 No. 7, Pages 48-57. |
Communications of the ACM, July 2019 |
Exposed Buffer Architecture for Continuum Convergence Micah Beck & Terry Moore arXiv:2008.00989, Aug 2020 |
![]() |
https://rule11.tech/the-hedge-podcast-episode-27-new-directions-in-network-and-computing-systems