How Hundreds Of Studies Measured The Wrong Protein By Mistake

Getting your Trinity Audio player ready...

Have you ever ordered something online, only to later discover you bought something slightly different (but in a really important way) to what you intended? Maybe the product was confusingly labelled, or maybe you just ordered the first item that came up in the search results without properly reading the description. I think we’ve all been there. But imagine that instead of buying cheap goods of dubious manufacturing quality, you were a scientist shopping for antibodies in order to measure the levels a specific protein implicated in cancer and senescence (a state in which cells stop dividing, believed to be an important driver of ageing). Imagine you accidentally bought a different antibody that is used to measure a completely different, unrelated protein that has nothing to do with what you are trying to study. Then imagine you didn’t realise your mistake, meaning that your conclusions concerning that protein were completely incorrect, and that you then published those conclusions in a peer-reviewed scientific journal. Just imagine how embarrassing that would be for everyone involved!

But what if I told you that this exact mistake has been made, with the same antibody, in over 300 scientific papers? Well it has – it turns out that in over 300 studies in which researchers thought they were measuring a tumour-suppressing protein called p16-INK4a, they were actually measuring a component of the cell’s ‘skeleton’ called ARPC5. More eyebrow-raising is the fact that many of these studies produced positive findings, despite measuring the wrong thing. We’re here to discuss how this happened and what it means for the study of senescence. This widespread recurring error was first noticed and reported by molecular biologist and scientific fraud-hunter Sholto David, whose post on the subject you can find here. It’s worth a read, especially if you want a more detailed breakdown of some of the specific studies involved.

Setting The Scene

How do hundreds of researchers manage to make exactly the same mistake independently? Before getting into the details of what happened, it’s worth taking a little bit of time to explain what antibodies are and how they are used in research. Most people have heard of antibodies in the context of infections – they’re the Y-shaped proteins released by B cells that attach themselves to specific proteins (antigens) on the surface of pathogens or infected human cells, helping other immune cells to clear them up. The fact that antibodies can be highly specific (they only bind to a specific protein and not similar-looking proteins) makes them very useful in research for measuring protein levels. For example, say you want to measure the level of protein X. You can introduce anti-X antibody into your sample. You can then introduce a second, fluorescently-tagged antibody targeting the first antibody. Now you can measure the level of protein X in your sample based on the level of fluorescence.

A very common technique that applies this principal is called western blotting. This is where an electric charge is used to separate out different proteins in a sample according to their molecular weight. Antibodies are then used to visualise the proteins of interest, allowing the levels of multiple proteins in different samples to be compared simultaneously.

When researchers need a specific antibody, they will purchase one that suits their needs from a vendor. Ideally, researchers should then validate the antibody, meaning they should perform tests to confirm that it accurately measures the protein they want to study before they start using it in their research. This is because antibodies are notoriously unreliable – they can vary batch-to-batch, have cross-reactivity (bind to proteins other than those advertised) or may be specific to a certain isoform (a protein encoded by the same gene, but with a different structure). These intricacies may be unspecified or unknown to the manufacturer, so it’s up to the buyer to confirm that the antibody is fit for purpose.

Introducing The Characters

This brings us to the antibodies and proteins in question. The story begins with a protein called p16-INK4a, often abbreviated to p16. p16 is actually the least important part of the name, and simply refers to the protein’s molecular mass of around 16 kilodaltons (kDa). INK4a is short for inhibitor of cyclin-dependent kinase 4a. The role of this protein is to slow down cell division. Without p16 (for example, if it loses function due to a genetic mutation) the cell divides too quickly, which is one step on the way to becoming cancerous. On the other end of the spectrum, elevated p16 levels are also linked to the complete cessation of cell division, or cellular senescence. Senescence occurs when cells have divided too many times or have sustained enough damage that further division might trigger cancer, at which point senescence kicks in as a protective mechanism. Unfortunately, as we age, senescent cells start to accumulate in sufficient numbers to become a problem. Besides being ‘dead weight’ within tissues, senescent cells also release harmful inflammatory signals into their surroundings that contribute to age-related diseases.

Because of the importance of p16 for cell division, studies investigating cancer and senescence often measure p16 levels alongside other proteins. p16 can be used as a marker of senescence – for example, lower levels of p16 in a sample is one indicator that there are fewer senescent cells. As already discussed, researchers can compare the levels of a protein like p16 in a western blot, using an antibody that targets p16.

Our second character is a protein called ARPC5, which stands for actin-related protein 2/3 complex subunit 5. It’s a relatively obscure protein involved in the extension of actin filaments – the protein ‘skeleton’ of the cell. The precise function of ARPC5 is unknown and it is not a well studied protein – fewer than 10 studies a year mention ARPC5 in their titles, compared to around a thousand per year for p16. So why is ARPC5 important to this story? Well it just so happens that ARPC5 has a molecular mass of around 16kDa. Because of this, ARPC5 has an alternative but less-used name: p16-ARC. You can see where this is going.

The Mix-Up

p16-ARC is not a commonly used alternative to ARPC5, but several vendors include both names in the product title for antibodies targeting this protein. For example, the major vendor Abcam labelled it as ‘anti-ARPC5/p16 ARC antibody’. Due to its relative obscurity, no one would use ‘p16’ to abbreviate p16-ARC. However, if you simply enter ‘p16’ into the search bar, p16-ARC actually comes up before p16-INK4a, presumably because results are in alphabetical order. But surely no one would just buy the first antibody in the search list without double checking it was the right one… right?

Well, studies are obliged to include the product codes of everything they used so that they are traceable, and so it is always possible to confirm which antibody was actually used. It turns out that out of just over 400 publications that listed product codes for p16-ARC antibodies, 95% of those that were accessible thought they were using p16-INK4a antibody. Sholto David put together a spreadsheet listing all of the the studies using p16-ARC antibody and whether they got it right, which you can find here. This amounted to at least 317 studies that described using p16-INK4a antibody and presented data for p16-INK4a protein, but included product codes indicating they had actually used p16-ARC antibody and had therefore measured the wrong thing.

These are of course fairly egregious errors, and also suggesting that labs were not properly validating their antibodies before using them – had they tested the antibodies on samples lacking p16-INK4a, they would have noticed something was wrong when the results turned up positive. You might therefore be forgiven for thinking that these weren’t particularly high-impact studies coming from well-regarded labs. The guilty studies certainly include their fair share of ‘stinkers’, but there are unfortunately some respected names as well. The most cited study that got it wrong was published in Nature and has 423 citations.

The Implications?

For those studies involved, it’s pretty bad. p16-ARC has, at least as far as we know, no relationship with p16-INK4a, cancer or senescence whatsoever. This means that any data showing any changes in supposed ‘p16-INK4a’ levels in these studies has no relation to anything meaningful that they were trying to investigate. In fact, ‘p16-INK4a’ shouldn’t really change at all in most cases, since the protein that’s actually being measured (p16-ARC) is related to actin and correlates with actin levels, and actin is used as the control in the western blot to verify that the total quantity of protein in each sample is similar (known as a loading control). In other words, since ‘p16-INK4a’ is actually p16-ARC, it should look exactly like the control in terms of how it varies from one sample to the next. Here’s an example:

This image shows a western blot for 3 different senescence-related proteins (p53, p21 and p16) and beta-actin, the loading control. The experiment was looking at how these proteins varied in response to different doses of gamma radiation (0-10 Gy). You can see that bands for p53 and p21 appear to increase from left to right, indicating that higher doses of radiation correlated with higher levels of these proteins. By contrast, p16 correlates with the beta-actin control (it doesn’t change significantly). This is of course because unbeknownst to the authors, the p16 that is being measured is actually the actin-related protein p16-ARC, so the fact that it correlates with actin is hardly surprising.

But here’s where things take an interesting turn: not all of the studies presented data like the western blot above. Somehow, a number of studies managed to produce a change in p16 that aligned with their hypothesis. There are only two plausible ways in which this could happen: either the data is fake, or researchers did actually use the correct antibody, but noted down the wrong product ID. Given the track record of some of the labs involved, it’s probably a mix of both. The authors of the aforementioned Nature paper, for example, responded that they found the receipt showing that they did in fact buy the correct antibody, and have published a correction. This probably did happen multiple times, but it’s still not a great sign – best practice is to record product codes the moment said product is used. If the wrong product code was entered, this means that someone entered the product code from the website rather than recording what they actually used. The point of keeping a lab record is to provide a complete and accurate account of what was done in an experiment that researchers can later go back to if necessary, for example to spot if a mistake was made. A retrospective account of what you think you did isn’t particularly helpful.

Some Good News

You may be wondering by now what this means for cancer and senescence research. The good news is that the answer is ‘not a lot’. While 300+ studies might sound like a large number, thousands of cellular senescence studies and hundreds of thousands of cancer studies are published per year. Much of what we know about cancer and senescence has been verified through multiple different approaches that all point the same way and have been replicated, so the impact of a few studies using the wrong antibody is negligible to these fields as a whole. The bigger issue here is with trust in the science – one has to wonder what other, easily avoidable mistakes are being made. Worse still, it’s really unlikely for a keen-eyed reader to spot these kinds of mistakes. A study may use hundreds of reagents – when the methods section states that they used reagent X, the average reader (or even a busy peer-reviewer) is not going to look up the product code.