Coders paired with bot buddies work fast, but take too many shortcuts

Developers who "pair code" with an AI assistant stand to learn as much as they do in traditional human-human pairings, but also show a less critical attitude toward their silicon-based partner's output, academics have found.

Pair programming is a common practice in developer circles, though it did not become a formal "pattern" until the turn of this century.

The practice is credited with producing better quality code, savings in development time, and knowledge transfer. And, done right and with the right pairing, should generally make for a more pleasant experience.

Yet increasingly, developers are working with code assistants, rather than other walking, talking coders. So, researchers at Saarland University in Germany sought to "analyze knowledge transfer in both human-human and human-AI settings."

One group of human-human pairs tackled a programming task, while another group of individual developers tackled the task with the assistance of GitHub Copilot.

The task involved implementing features within an existing codebase of approximately 400 lines including both Python code and comments, distributed across 5 files.

The researchers sought to answer two questions. Firstly, "To what extent do the frequency, length, and depth of knowledge transfer episodes differ between human-human pair programming and human-AI pair programming?" And secondly, "How do the quality and diversity of knowledge transfer episodes, including topic types and finish types, vary between human-human pair programming and human-AI pair programming?"

The academics then tracked conversational "episodes" between the meat sack duos using a speech recognition tool and used screen recordings to track interactions within the human and Copilot pairs.

Those conversations were analyzed for "contribution to knowledge transfer" with the researchers noting: "In most cases, utterances related to knowledge transfer contain an exchange of information between the two humans or between the human and GITHUB COPILOT."

They found that the human-human pairings generated 210 episodes, compared to 126 episodes in human-AI pair programming sessions.

"Code" conversations were more frequent in the human-machine pairing, whereas "lost sight" outcomes - ie the conversation got sidetracked - was more common in the human pairings.

They found "A high level of TRUST episodes in human-AI pair programming sessions. If this pattern were to generalize beyond our setup, this would carry important real-world implications, warranting further investigation. These frequent TRUST episodes can reduce opportunities for deeper learning."

Other, broader but still on topic conversations were more likely to occur in the human-human pairings.

The researchers concluded that while the use of AI might increase efficiency, it could also "reduce the broader knowledge exchange that arises from side discussions in human-human pair programming, potentially decreasing long-term efficiency."

This could mean that while "AI is useful for simple, repetitive tasks where side discussions are less valuable... when it comes to building deeper knowledge it must be treated with care, especially for students." And the researchers added: "We observe that in many GITHUB COPILOT sessions, programmers tend to accept the assistant's suggestions with minimal scrutiny, relying on the assumption that the code will perform as intended."

They suggested, "Human-human pair programming enables spontaneous interactions but also increases the risk of distraction. In contrast, knowledge transfer with GITHUB COPILOT is less likely to be aborted, yet suggestions are often accepted with less scrutiny."

However, AI assistants were good at reminding humans of key details, "such as committing database changes, that might otherwise be overlooked."

That could, arguably should, be ringing alarm bells for development leaders. It's easy to focus on the increasing efficiency AI generated code can bring. But that code still needs to be reviewed and tested before being put into production, otherwise bad things can happen.

GitHub happily trumpeted the uptake of CoPilot in its latest Octoverse report last week, with 80 percent of new users diving into the technology. The use of CoPilot, and other code assistants, is even shaping the languages developers use, with a shift to more strongly typed languages which lend themselves to code generation platforms. But generating code is just part of the pipeline.

Research by Cloudsmith earlier this year highlighted how coders are acutely aware of the perils of LLM generated code, for instance by recommending non-existent or even malicious packages. At the same time, a third of developers were deploying AI generated code without review. ®