Science Is Being Corrupted By Fake Research (And No, It’s Not Just About AI)

Back in 2020, we ran a short experiment called This Research Does Not Exist. Inspired by AI-powered face generator This Person Does Not Exist, we used machine learning tools to help generate abstracts for fake scientific articles that looked, at least superficially, like genuine published research. In a time before ChatGPT had taken over the internet, the goal was to showcase not only the rapidly advancing capabilities of machine learning, but also the ease with which it could be used to generate misinformation. Sure enough, science is now under serious threat from fictitious AI-generated research papers, but many are unaware that fake research has been a growing problem for a decade.

Not Just An AI Problem

As with many modern problems commonly blamed on AI technology, AI did not create this problem – though it’s going to make it a whole lot worse. Just as students were paying online services to write their essays for them long before ChatGPT would do it for free, so too were ‘scientists’ submitting phoney research papers to scientific journals. Regrettably, some of these papers are published, even after having undergone peer review. It’s evident that the scale of this problem has skyrocketed in recent years – according to Nature, there were just over 1,000 retractions of scientific papers in 2013, but over 4000 in 2022 and over 10,000 in 2023.

Though no country is free from such practices, they seem to have originated in China, where young doctors and scientists were are required to have published scientific papers in order to be promoted in their field. This led to the rise of ‘paper mills’ – organisations that exist to churn out fabricated research. Over time, these paper mills have spread to many other countries.

Until now, the methods of paper mills have been relatively primitive. Their products often take the form of a template article with the names of random genes and diseases slotted in. You might think that such fabrications would not be that hard to protect against, and you’d be right. The publication of these articles, it seems, is down to lack of diligence on the part of editors and peer-reviewers (or bribery in some cases), not due to the papers being convincing in nature. In one particularly amusing example, Computational and Mathematical Methods in Medicine accepted a paper about Marxist ideology, which has since been retracted for, among other things, ‘Incoherent, meaningless and/or irrelevant content’.

Needless to say that the acceptance of fraudulent science by scientific journals is extremely damaging. By means of peer-review and the editorial process, journals are supposed to ensure a minimum standard of scientific research that other scientists can rely on. Scientists use existing research and scientific databases to inform their own research, including in areas such as drug discovery, so the fact that these databases are being ‘poisoned’ is concerning.

The Threat Of AI

AI tools not only make fake research easier to generate, but also harder to detect. The power of machine learning is the ability to create content that looks extremely convincing but offers no guarantee of truthfulness. When faced with a fake article that has been made by ‘slotting in random genes and diseases’, a competent peer-reviewer should be able to spot when the paper doesn’t make sense, or when the presented data and images don’t line up with the rest of the article. AI, on the other hand, has the capability to generate much more convincing research, complete with generated text, microscopy images and other data that are actually consistent with one-another. When directed by a human that knows what they are doing, AI tools will soon (and perhaps already can) generate papers that are capable of fooling peer-reviewers consistently.

What Are The Solutions?

Given the accessibility of AI tools, it would be naïve to think that AI-generated papers are not already making their way into scientific journals. So, how can they be stopped? More specifically, how can science defend against fake studies without penalising scientists who use AI to assist their research? Clearly, there needs to be a more thorough verification that the research being presented has actually been done. One proposition is to require the submission of raw data, potentially with digital watermarks. This will make AI-generated research and fake research in general harder to produce. Journals can apply heavier scrutiny of the authors and their affiliations, as authors may provide fake credentials when submitting fake research. Journals can also use AI tools of their own to detect whether components of a submitted paper are AI-generated, though again, we do not want to penalise researchers who use AI in innovative and positive ways.

Unfortunately, it could take years to develop truly effective countermeasures against fraudulent research, while it takes a matter of seconds for AI to generate a fake research article. As time passes, these articles will only get harder to detect.