If I were to summarize the ideal scientific article in four sentences, it would look like this:
- Look at this cool thing we did.
- This is how we did the cool.
- This is the cool thing.
- Wasn’t that cool?
We like to think that the Standard Format (not to be confused with the Standard Model) was followed beautifully in days gone by. Today, of course, that is not the case. Because it always gets worse, right? In reality, scientific papers have always looked like this:
- Look at this cool thing we did, IT’S REALLY COOL, BE INTERESTING.
- Here’s how we did the cool (apart from this bit we “forgot” to mention, the other bit we didn’t think was important, and that bit a company contributed and wants to keep secret. Have fun replicating the results!) .
- This is the cool thing.
- This thing that we did is not only cool, but is going to completely cure cancer, even though we’ve never talked about cancer and are basically studying the ecology of the less-spotted physicist.
Call me cynical, but missing information in the method section, as described in parentheses in point two, really bothers me. I think it bothers me more now than I did ten years ago, even though I’m no longer the student in the lab stuck with filling in the missing methods myself.
Perhaps I’m angrier than I used to be, because as I’ve gained experience it’s become easier to spot the carefully camouflaged holes.
Let’s play the blame game
I blame publishers. In the good old days of printed magazines, each edition contained only a finite amount of information, so paper length was limited. While you might need 30 pages of closely spaced text to describe how you accomplished an arcane scientific achievement, some journals only gave you half a column. Any scientific results that could not be properly communicated in a short format ended up in another journal that could accommodate them.
To keep the papers short, many journals emphasize results and conclusions at the expense of methods.
This sometimes led to duplicate publications: one magazine contained a short description of your results, while the detailed explanation of what you did appeared in a more technical publication.
Over time, short, direct articles have become more prestigious. Because prestige matters to university administrators, scientists are under increasing pressure to publish condensed forms of their research. The publishers, many of whom benefit from this pressure, are happy to respond to this.
To keep the papers short, many journals emphasize results and conclusions at the expense of methods, often moving them to the end and printing them in a font that requires a microscope. When I tried to report on an article on adiabatic quantum computing recently published in NatureI was stunned to discover that all useful information about methods was not in it Nature not at all, but in a separate document called additional information.
Additionally
Additional information doesn’t come in the print version of journals, so good luck understanding a paper if you enjoy reading the paper version. It’s also not included with the newspaper if you download it to read later – additional information is usually a separate download, sometimes much larger than the newspaper itself, and often behind a paywall. So if you want to download a study’s methods, you’ll need to be on a campus with access to the journal, use your institutional proxy, or jump through any hoop.
At this point you might be saying to yourself, “Come on, Lee, you must have noticed that you needed the additional information when you first flipped through that quantum computer paper in Nature.” But that’s not how I work. When I browse, I focus on the introduction and the results. Essentially, I want to know two things at my first reading: is this article relevant to my work? And/or is it interesting to read about If a paper doesn’t cross that threshold, I don’t care how it works.
For my day job, most of the magazines I read do not publish articles with additional information. When I write to Ars, I often work about a month behind the publication date so that I have time to read and possibly download the additional information. But most of my paper reading takes place on trains where internet access is spotty. I left more than one story because I found I needed additional information that I couldn’t download. (If you didn’t know it was possible to stop writing in anger, you do now.)
Papers must not be crippled to meet arbitrary page limits.
The article on adiabatic quantum computing was even worse. The newspaper had not yet been published at the time, so I only had the press kit. Additional information is often provided to the press, but Nature apparently didn’t care this time. Yes, the results looked nice, but I couldn’t even tell if they were experimental results or just calculations. Even when I decided it had to be experimental results, the one photo of the researchers’ real set of qubits showed something that was nothing like what the results pointed to. The press kit made extensive reference to additional information that would have clarified all this, but of course I had no access to that. When I contacted the authors, they were kind enough to provide the additional information. Suddenly I had twelve pages left and their paper finally made sense.
This is not how science works. Papers must not be crippled to meet arbitrary page limits. And even with the additional information, I don’t believe a competing group of researchers could reproduce the work described in this article. There simply isn’t enough information to make that possible.
Unnecessary speed bumps
There is no reason why an online research paper should be accompanied by a named document additional information. Everything should be in the main document, even if some stuff is relegated to an attachment. Just because publishers have chosen to make their print versions increasingly cryptic doesn’t mean they should cripple their online versions as well.
But more generally, methods should not be part of some appendix – they are a central pillar of any research report.
Think of the general reader. After all, public journals want to publish results that are attractive to a wide range of scientific specialists. How exactly Nature quantum computing paper, and others like it, help? By itself, it is illegible and meaningless (not the fault of the authors – if you read the supplementary document before looking at the numbers in the main article, you will have a clear, well-written article).
The authors… were kind enough to provide the additional information. Suddenly I had twelve pages left and their paper finally made sense.
Shouldn’t a “public interest” paper please interested parties, including physicists from further afield than me? And what about chemists, biologists, social scientists and all those other scientific specialties? How do they benefit from such an incomplete document? Apparently magazines think public interest doesn’t mean “write papers so that people will be satisfied both inside and outside the field”. They think public interest means “remove context so everything looks like gibberish”. If you’re an expert in the field, the article itself isn’t relevant – the only things worth knowing are in the additional information.
The solution here is obvious: limit additional information to things that really can’t be included in a .pdf (e.g. video content) and make sure that a paper’s .pdf contains all relevant information. Or at least provide a one-click link that downloads all relevant content. This will still confuse print readers, but I am willing to make some sacrifices for the greater good.