The internet is buzzing this week with various use cases, positive and negative, for OpenAI’s ChatGPT tool, which has produced some interesting results. In a Facebook post that went viral, Furman Professor, Darren Hudson Hick, talked about using ChatGPT’s counter AI tool to root out plagiarism. Professor Hick writes:
“Today, I turned in the first plagiarist I’ve caught using A.I. software… and I thought some people might be curious about the details. The student used ChatGPT … an advanced chatbot that produces human-like responses to user-generated prompts…. The essay confidently and thoroughly described Hume’s views on the paradox of horror in a way that were thoroughly wrong. It did say some true things about Hume, and it knew what the paradox of horror was, but it was just bullshitting after that. To someone who didn’t know what Hume would say about the paradox, it was perfectly readable—even compelling. To someone familiar with the material, it raised any number of flags. ChatGPT also sucks at citing, another flag. This is good news for upper-level courses in philosophy, where the material is pretty complex and obscure. But for freshman-level classes (to say nothing of assignments in other disciplines, where one might be asked to explain the dominant themes of Moby Dick, or the causes of the war in Ukraine—both prompts I tested), this is a game-changer. ChatGPT uses a neural network, a kind of artificial intelligence that is trained on a large set of data… the “programmers” won’t really know what’s going on inside it: the neural network takes in a whole mess of data, where it’s added to a soup, with data points connected in any number of ways. The more it trains, the better it gets. Essentially, ChatGPT is learning, and ChatGPT is an infant. In a month, it will be smarter. Happily, the same team who developed ChatGPT also developed a GPT Detector (https://huggingface.co/openai-detector/), which uses the same methods that ChatGPT uses to produce responses to analyze text to determine the likelihood that it was produced using GPT technology. Happily, I knew about the GPT Detector and used it to analyze samples of the student’s essay, and compared it with other student responses to the same essay prompt. The Detector spits out a likelihood that the text is “Fake” or “Real”. Any random chunk of the student’s essay came back around 99.9% Fake, versus any random chunk of any other student’s writing, which would come around 99.9% Real. This gave me some confidence in my hypothesis. The problem is that, unlike plagiarism detecting software like TurnItIn, the GPT Detector can’t point at something on the Internet that one might use to independently verify plagiarism…. Administrations are going to have to develop standards for dealing with these kinds of cases, and they’re going to have to do it FAST….In future, I expect I’m going to institute a policy stating that if I believe material submitted by a student was produced by A.I., I will throw it out and give the student an impromptu oral exam on the same material.”