ChatGPT is not built on scientific advances but on scaling of existing approaches using gigantic resources of labor, money, hardware, electricity, and capital. OpenAI’s CEO Sam Altman suggests that it is now at the end of its development. The warning against global extinction from AI issued by Altman and other AI luminaries points the discussion in the wrong direction. Warning for the future superhuman intelligence may attract smart employees and hype-sensitive investors with deep pockets, but it deflects attention from the slowly unfloding epistemic disaster that ChatGPT contributes to
To understand this disaster, and ChatGPT’s contribution it, let’s look at two elements of OpenAI’s business model: its value proposition and its value network.
ChatGPT’s value proposition
The value proposition that OpenAI offers with ChatGPT is to answer a question with well-structured, grammatically correct text based on anything that can be found on the Internet till 2021. There is no guarantee that the text is true or false, so ChatGPT is a bullshit generator. It produces texts only with regard to its structure, but without regard for their truth value.[i]
ChatGPT’s many uses
Marketers dream of using ChatGPT to discover new customer segments by analyzing customer reviews, discovering customer preferences, creating personalized messages, and much more. Students use it to summarize information and create a first version of a report. Job seekers may use it to create a CV. ChatGPT can propose code to solve a problem. It has been used to guide the design of a tomato-harvesting robot.
It may also be put to malicious uses, such as gathering information about the target of an attack, creating phishing emails, or generating malicious code.
Unless you don’t mind acting on false statements, you need to check the truth value ChatGPT’s output. The English-language CV it created for me contained six errors in the first six lines. The Dutch-language CV contained only 3 errors in the first six lines ¾ different errors, in a CV that stated different facts about me than the English-language one
Which brings me to the core epistemic problem of using ChatGPT: If you want to use the text only if it is true, you need to check it. You need to find its sources and check the reliability of those sources. This means that you need to embed it in a network of knowledge sources. The network of contract workers, researchers, hardware vendors, energy providers, software developers, data providers, text providers and investors that contributed to ChatGPT as it is used, must be extended by a network of knowledge sources needed to test the reliability of ChatGPT’s output. Let’s zoom in on this part of its value network.
ChatGPT’s value network
ChatGPT’s value network consists of its users, plus all stakeholders whose products or services are needed to produce its value proposition.[ii]
The an important set of stakeholders in its value network are the publishers whose texts were used by OpenAI to train the language model of ChatGPT. It was trained on all text OpenAI could access on the Internet, without paying the publishers. Its sources include individual CV’s, income tax statements, professional news, even if hidden behind a paywall[iii], sites of conspiracy theorists, pages from the Wikipedia, and a lot more. Training ChatGPT was a Big Steal. It is the Real Big Steal.
ChatGPT contributes to the current epistemic disaster because it sucks up text regardless of its truth value. It uses these texts to compute the probability of the next word in an answer, given the previous words in the answer, the query and the training data. Its answers may be false or nonsensical, but these are not mistakes. It does exactly what its neural network has been trained to do: produce grammatically correct well-structured text.
This contributes to a slowly unfolding epistemic disaster not by automatic the production of bullshit, but by nibbling away at our capability to produce true statements. We can see this by considering the rules of behavior in two kinds of networks in which ChatGPT can be used.
In reality-based networks, members of the network aim at true statements by sticking to some rules. These are taken from the two books by Jonathan Rauch (who talks of reality-based communities instead of reality-based networks).[iv]
The first rule is that network members start from the assumption that all truth claims, without exception, are fallible, and hence must be checked on their truth value.
This implies that statements must be falsifiable, meaning that there must be a way to test them, and that the test may show the statement to be false. It cannot show that the statement is definitely, true. The second rule of reality-based networks is that network members produce falsifiable truth claims.
The assumption of fallibility also implies that the test must be repeatable by different people, regardless of their identity. The third rule in reality-based networks is that no one has final authority.
Examples of reality-based networks are scientists, journalists, engineers, health professionals, legal professionals, teachers, statistical bureaus, and the intelligence community.[v] Attackers, thieves, and military people also form reality-based communities. Basing your journalistic report, your medical treatment, or your attack on a false idea of the real world is a bad strategy.
No one is able to test all statements they are confronted with, and so we all depend on a network of trusted network members who we know follow the rules of falsifiability and repeatability. Knowledge ends up in textbooks and is verified by thousands of students every year. Most reality-based networks have a professional code of conduct that enforces trustworthiness. It takes years of training and practice to become a member of the network. Violation of the code of conduct destroys reputation and may result in excommunication.
This makes it impossible for members of a reality-based community to trust ChatGPT. Its answers are well-structured boring texts that contain no reference to sources. Their truth value must be checked by the reader. But ChatGPT gives no hint where to find its sources.
The epistemic risk this creates is that aspiring members of a reality-based network, who have not yet internalized the professional code of conduct about truth claims, use a text without checking it. Students may use it to overcome a writer’s block, improve text or clarify concepts.
Even if a student can’t write (and therefore uses ChatGPT), as an aspiring members of a reality-based network he or she should be able to support truth claims by evidence, logic and reliable sources. Using ChatGPT as source for a truth claim will not teach students any of these competencies. This nibbles away, student by student , at the size and strength of a reality-based network.
All human networks are based on resonance, because members of the network share a background, expertise, feeling, goal, or anything else which makes them communicate and cooperate. Members of a sports club resonate in their love of a sport; members of a political party resonate in their convictions about the ideal society or in their hate of the opposing party.
However, realization-based networks use resonance not just to energize cooperation and communication, they use it as a truth criterion. Examples are flat-earth theorists, climate change deniers, and conspiracy theorists. Members of the network feel that they have swallowed the red pill. They realize what is really going on, even if others don’t see it. Hence, I call it a realization-based network. They know something outsiders don’t know.
Lee McIntyre identifies a few shared characteristics of realization-based networks.[vi] Realization-based communities assume total certainty of what they believe to be true. Evidence is treated with confirmation bias, and logic is twisted so it supports their conclusions. Experts to rely on are other members of the network, not because they are held to the reality-based rules of falsifiability and repeatability, but because they are members of the network and resonate on their shared belief.
Reinforcement learning through human feedback has eliminated the well-known conspiracy theories from ChatGPT’s language model. However, in a test, ChatGPT elaborated on 100 false stories. Asking it to write a story from the point of view of the conspiracy theorist Alex Jones about a high school mass shooting produced Jones’ false conspiracy theory.
This should not surprise us, as the web contains a vast amount of conspiracy text, and ChatGPT is trained on it. Reinforcement learning is a labor-intensive activity that can eliminate some, but not all of it from ChatGPT’s answers.
The contribution of ChatGPT to our epistemic disaster here is that it is a helpful tool to reinforce and spread beliefs that resonate with the false beliefs that a realization-based community already is convinced of. This will help to grow these communities.
What to do
Accepting fallibility and using repeatable falsifiability as knowledge criterion is a few hundred years old. Using resonance in a community as knowledge criterion may be as old as humankind. We did not need ChatGPT to lure us into resonance-based epistemics.
Answers produced by ChatGPT ring true but are so boring that I do not expect them to play a significant role in the growth of resonance-based knowledge networks. These networks thrive on the infrastructure of social media, TV and radio. ChatGPT is a minor addition to the soil on which conspiracy theories grow. It can be tricked into racial slurs and hate speech, but this is because these texts are already available on the Web.
A bigger risk is its impact on reality-based knowledge networks. Professionals who are unaware of the fact that ChatGPTR generates well-formulated, boring bullshit without a claim to truth, learn this lesson very quickly when they start using it in their practice. But aspiring candidate members of a reality-based network may be slow or unwilling to learn this lesson.
Students and schoolchildren, who should be trained in the values and norms of reality-based truth claims, have started using ChatGPT to produce summaries, write reports, and answer research questions. But ChatGPT is not part of any reality-based network. Using its output as if ChatGPT can be trusted to make reliable truth claims disconnects the user from any reality-based network.
Without the influx of young members, these networks would become extinct. The institutions who train young people to join a reality-based network must take action to prevent young community members from using resonance rather than reality as a knowledge criterion.
There is only one action that does this: demand that authors who use a ChatGPT-generated text support the text logical argument, references to by evidence, and reference to trustworthy members of a reality-based network.
[i] Harry Frankfurt. On Bullshit. Princeton University Press, 2005.
[ii] Roel Wieringa & Jaap Gordijn. Digital Business Ecosystems. How to Create, Deliver and Capture Value in Business Networks. TVE Press, to appear, 2023.
[iii] This page is in Dutch. But you can use Google Translate to understand it, or ChatGPT to summarize it.
[iv] Jonathan Rauch. Kindly Inquisitors. The New Attacks on Free Thought. The University of Chicago Press. Expanded edition, 2013.
[v] This list extends the one given by Jonathan Rauch. The Constitution of Knowledge. A Defense of Truth. The Brookings Institution Press, 2021.
[vi] Lee McIntyre. How to Talk to a Science Denier. Conversations with Flat Earthers, Climate Deniers, and Others Who Defy Reason. MIT Press 2021.