AI bot ChatGPT has been put to the test on a number of tasks in recent weeks, and its latest challenge comes courtesy of computer science researchers from Johannes Gutenberg University and University College London, who find(Opens in a new window) that ChatGPT can weed out errors with sample code and fix it better than existing programs designed to do the same.
Researchers gave 40 pieces of buggy code to four different code-fixing systems: ChatGPT, Codex, CoCoNut, and Standard APR. Essentially, they asked ChatGPT: “What’s wrong with this code?” and then copy and pasted it into the chat function.
On the first pass, ChatGPT performed about as well as the other systems. ChatGPT solved 19 problems, Codex solved 21, CoCoNut solved 19, and standard APR methods figured out seven. The researchers found its answers to be most similar to Codex, which was “not surprising, as ChatGPT and Codex are from the same family of language models.”
However, the ability to, well, chat with ChatGPT after receiving the initial answer made the difference, ultimately leading to ChatGPT solving 31 questions, and easily outperforming the others, which provided more static answers.
[…]
They found that ChatGPT was able to solve some problems quickly, while others took more back and forth. “ChatGPT seems to have a relatively high variance when fixing bugs,” the study says. “For an end-user, however, this means that it can be helpful to execute requests multiple times.”
For example, when the researchers asked the question pictured below, they expected ChatGPT to recommend replacing n^=n-1 with n&=n-1, but the first thing ChatGPT said was, “I’m unable to tell if the program has a bug without more information on the expected behavior.” On ChatGPT’s third response, after more prompting from researchers, it found the problem.
(Credit: Dominik Sobania, Martin Briesch, Carol Hanna, Justyna Petke)However, when PCMag entered the same question into ChatGPT, it answered differently. Rather than needing to tell it what the expected behavior is, it guessed what it was.
[…]
Source: Watch Out, Software Engineers: ChatGPT Is Now Finding, Fixing Bugs in Code
Robin Edgar
Organisational Structures | Technology and Science | Military, IT and Lifestyle consultancy | Social, Broadcast & Cross Media | Flying aircraft