A bust-up between mathematicians is fuelling fears that AI could undermine science

Two boxes containing barcoded glass tubes, holding one hundred billion genes, may possess ten times more information than all the text available on the internet, according to Dr Oliver Vince, co-founder of Basecamp Research. His firm aims to gather extensive DNA data to build an artificial intelligence capable of understanding biology's language, mirroring how US and Chinese tech firms amassed online text for Large Language Models (LLMs).

While many in academia and business anticipate AI will greatly accelerate scientific inquiry, this potential is increasingly shadowed by controversy. Recently, OpenAI, the creator of ChatGPT, claimed its AI agents solved a major unsolved mathematical problem concerning the Navier–Stokes equations in just 88 hours. The body overseeing the prize noted it had "apparently been settled," though a review would take time.

However, before this announcement, mathematician Tristan Buckmaster of New York University publicly voiced concerns. He collaborated with Levent Alpöge, a mathematician from Anthropic, and both teams used OpenAI's Codex coding tool. Buckmaster mentioned his co-researcher was informed that news of their progress had "reached" OpenAI, questioning if his Codex research could have influenced OpenAI's work, while stressing he was "not accusing anyone of anything."

OpenAI initially stated that neither human nor AI at the company saw the other teams' work before publication. Later, it asserted that Buckmaster's Codex prompts from the two months preceding the announcement "could not have influenced the system in any way." This situation underscores a growing concern: AI's application in science risks eroding faith in the scientific process itself, questioning research accuracy and novelty.

Beyond the risk of AI "hallucinating" untrue information, a more subtle danger looms: the potential pollution of scientific journals. Dalmeet Singh Chawla, a science journalist, stated, "Generative AI has basically disrupted the whole scientific process," warning that publishers' integrity staff will need to "drastically increase [in number and effort] if they're anywhere near the fight," adding, "Right now, it seems like they're not."

The power of AI is undeniable; Google DeepMind’s AlphaFold model, for instance, accurately predicted 200 million protein folds in one year, a feat that would have taken hundreds of millions of years in labs. Dr Pushmeet Kohli, who leads DeepMind’s science work, called AlphaFold a "watershed moment." Yet, this success has sparked debates, such as when DeepMind released over two million new crystal structures, which two UC chemists found contained "yet to find any strikingly novel compounds."

The risk of AI "slop"—research written by computers rather than human scientists, leading to fake citations—is also significant. In 2024, Prof Dmitry Kobak of Ghent University found that nearly 90% of over a million biomedical papers contained AI fingerprints by December of that year. Kobak warned Nature that "whatever bias the LLM may have will just suddenly permeate the (scientific) literature."

Compromises are also evident in the peer review system. A Stanford University team found up to 17% of referee reports for four major computer science conferences had been substantially modified by a language model. Furthermore, Cornell University researchers discovered that AI was being used to compile citations, inventing approximately 147,000 pieces of phantom research last year.

To counter these issues, some publishers are taking action. Elsevier's integrity team has created 100 signals to flag papers for closer review, and Nature's journals prohibit AI-generated images. Sir Demis Hassabis, DeepMind's chief executive, supports responsible integration, stating, "What we're calling research-grade AI would focus on ensuring the researcher is in the driving seat."