The Sagan standard is the adage that “extraordinary claims require extraordinary evidence” (a concept abbreviated as ECREE). This signifies that if a claim seems extremely unlikely to be true, usually because of relevant past evidence, then it should be supported by extremely strong evidence before it’s accepted.
For example, based on the Sagan standard, if someone claims that they saw a unicorn on their way to work, then they would be expected to produce extremely strong evidence in order to verify this claim, since there’s currently no substantial preexisting evidence showing that unicorns even exist. Conversely, if someone claimed that they saw a horse on the way to work, then they would usually be expected to provide less evidence in order to verify their claim, since there’s substantial preexisting evidence that horses do exist.
This concept is important to understand, because it can help you identify situations where you need to expect stronger proof for claims, both from others and from yourself. But, it’s also important to know how to avoid overapplying this concept, since doing so can cause you to incorrectly dismiss valuable claims. As such, in the following article you will learn more about the Sagan standard, understand the rationale behind it, and see how you can implement it yourself in the best way possible.
Why extraordinary claims require extraordinary evidence
Examples of ordinary and extraordinary claims
Though it can be difficult, in some cases, to determine whether a certain claim is ordinary or extraordinary, there are situations where this distinction is fairly intuitive and uncontroversial.
For example, in most cases, saying that I saw a herd of horses standing around in a field on my way to work would be a fairly ordinary claim, that wouldn’t require much evidence, since there is no notable reason to doubt it. Conversely, saying that I saw a herd of unicorns would be considered an extraordinary claim, which requires much more substantial evidence in order to prove, since there is presently no credible evidence showing that unicorns exist.
Similarly, saying that I ate breakfast this morning would, for the most part, be considered a relatively ordinary claim, while saying that I flew to the moon and back would be relatively extraordinary.
Understanding the concept of extraordinary claims
When it comes to extraordinary claims, it’s generally preferable to view ‘extraordinariness’ as a spectrum, rather than as a binary status. This means that instead of viewing claims as either ordinary or extraordinary, it’s better to view them as ranging between these two ends of the spectrum, based on how likely they are given everything that is known on the subject.
For example, based on this conceptualization, we have claims that are clearly ordinary on one end (e.g. people need water in order to survive in the long term), and claims that are clearly extraordinary on the other (e.g. people can talk to fish using their mind), with more moderate claims somewhere in the middle.
Because it can be difficult to define the exact threshold on the ordinary-extraordinary spectrum that a certain claim needs to cross before it’s considered extraordinary, it’s generally preferable to focus on how extraordinary a claim is instead, and to expect a stand of proof that matches that degree of extraordinariness.
This concept can be further operationalized when it comes to statistical calculations. For example, when using Bayesian inference, it’s possible to modify the prior, which allows you to take preexisting knowledge and beliefs into account when assessing new claims and evidence.
However, it’s important to keep in mind the limitations of such tools, and namely the fact that even when these measures are based on objective information, such as the findings of scientific research on the topic, the data that it’s based on is often imperfect, as is the way in which this information is chosen and weighted, which is always somewhat subjective, even if it’s well-justified.
Finally, note that a claim should generally not be viewed as extraordinary simply because it’s novel, but rather because it contradicts existing evidence. This is because a claim might be novel but not extraordinary, as in the case of studies which have new findings that are in line with the rest of the scientific literature on the topic, just as a claim might not be novel but still be extraordinary, as in the case of reports of various paranormal phenomena.
Understanding the concept of extraordinary evidence
The burden of proof is the obligation to provide supporting evidence for claims that you make. The Sagan standard suggests that the more extraordinary a claim is, the greater the burden of proof that is associated with it. This means that those who make extraordinary claims are expected to produce supporting evidence that is more substantial than what they would be expected to produce if they had made less extraordinary claims.
As in the case of extraordinary claims, there is no clear distinction between what constitutes ‘ordinary’ evidence and what constitutes ‘extraordinary’ evidence. Furthermore, when it comes to determining what constitutes extraordinary evidence, the answer is always going to be subjective, though it should nevertheless be based on sound reasoning.
This means that even though there is no definitive agreement regarding what constitutes ‘extraordinary’ evidence, it’s possible to argue in favor or against the extraordinariness of evidence, based on its quantity and quality, and based on the relevant standards that apply to the subject being discussed.
For example, if I say that I saw a horse on my way to work today, I would be expected to produce less extraordinary evidence than if I said that I saw a unicorn. This means, for instance, that a simple picture or video might be enough to convince most people that I saw a horse, but much more substantial evidence would be required to convince them that I saw a unicorn.
Caveats about ECREE
The Sagan standard and the concept of ECREE do not mean that you should simply ignore any claims that appear to be extraordinary. This is important to keep in mind, since ECREE is often unintentionally misapplied, and has also been intentionally used as a rhetorical tool in order to discredit and suppress valid theories.
For example, when asked about his alleged use of illegal performance-enhancing drugs, cyclist Lance Armstrong is reported to have said that “extraordinary allegations require extraordinary evidence”, in an attempt to discredit those allegations, even though they later turned out to be true.
The use of ECREE in this case is hard to justify, since the idea of a top cyclist using illegal performance-enhancing drugs isn’t particularly unlikely, given what is known about doping by others in his field, that it would require “extraordinary” evidence. Of course, those who accused him still needed to support their allegations with concrete evidence, but the application of ECREE here was fairly unjustified, and based primarily on his desire to casually dismiss the stance of those who accused him.
Similarly, in the scientific context, ECREE doesn’t mean that we should hold studies that support the scientific consensus to a lower standard than we do studies that contradict it, since both types of studies should still be held to the same rigorous standard of scientific conduct. Rather, it means that in order to reliably convince the scientific community of something that contradicts the current scientific consensus, a more substantial body of evidence would generally be expected than it would take to convince them of something that does not contradict the consensus.
For example, if someone wanted to prove that astrology is a valid way to predict people’s personality, they would need to present an extraordinary amount of evidence, given that this contradicts current scientific theories. In contrast, if someone wanted to develop a psychometric test which predicts people’s personality, and based this test on existing theories and pre-validated psychological tools, they would still be expected to provide significant and credible evidence in support of their test, but their findings would likely be accepted by the scientific community based on less evidence.
Overall, the concept of ECREE does not mean that claims that contradict the current scientific consensus should always be ignored, or that claims that support the consensus should be automatically accepted. Rather, ECREE simply suggests that the likelihood that a certain claim is true, based on preexisting knowledge, should be taken into account when determining how much evidence is needed in order to verify it.
Criticism of ECREE
The main criticism of ECREE is that the definition of ‘extraordinary’ is arbitrary, both when it comes to determining what extraordinary claims are, as well as when it comes to assessing the evidence used to support those claims. This criticism is valid, since the quality of ‘extraordinariness’ will always have a degree of subjectivity involved, which opens it to various issues.
However, this criticism doesn’t invalidate the usefulness of ECREE as a general tool for thinking. This is because, as we saw above, it’s possible to operationalize the concept of extraordinariness in a way that is relatively well-defined and well-supported, even if it does contain a degree of subjectivity that often appears in other, similar tools.
Another notable criticism of ECREE is that it can cause people to ignore claims that contradict the current consensus, while automatically accepting claims that support it. This is certainly an issue that could occur if ECREE is applied incorrectly, as is the case with many other similar tools that can be overapplied (e.g. parsimony).
However, the fact that this concept can be misapplied doesn’t mean that it shouldn’t be used at all, since the proper application of ECREE can be beneficial when it comes to assessing the validity of claims.
How to implement the Sagan standard
Implementing the Sagan standard—and therefore the concept of ECREE —means that, when presented with a certain claim, you should expect the strength of evidence that is used to support it to be proportional to how unlikely that claim is, based on prior information. As such, the more unlikely a claim is, the more supporting substantial the evidence that you should require before accepting it as true.
For example, let’s say that you’re feeling sick, and a friend gives you advice on what you should do.
If they recommend that you go see a doctor, it would be relatively reasonable to accept this advice, since it’s in line with general healthcare recommendations made by professionals. Conversely, if your friend recommends that you place magic crystals under your pillow in order to heal yourself, it would be relatively reasonable to ask them for substantial evidence in support of their suggestion before you act on it, because this course of treatment is less supported by prior evidence than the option of going to the doctor.
However, as we saw above, when implementing the Sagan standard, it’s important to avoid unnecessarily rejecting claims that are perceived as ‘extraordinary’ without justification, just as it’s important to avoid automatically accepting concepts that are perceived as ‘ordinary’.
Note: for more information that can help you implement the Sagan standard properly, see the sections preceding this one, which will help you understand what it means for claims and evidence to be extraordinary and what are the common pitfalls that you’re likely to encounter when implementing this principle.
History of the Sagan standard and ECREE
The following section describes the history behind the Sagan standard, and behind the idea that extraordinary claims require extraordinary evidence. This information can be interesting to those who want to learn more about this concept, but it’s not crucial if you’re only interested in its practical applications.
The Sagan standard, which has been called a “fundamental principle of scientific skepticism”, was popularized by astronomer Carl Sagan, who said that:
“What counts is not what sounds plausible, not what we would like to believe, not what one or two witnesses claim, but only what is supported by hard evidence, rigorously and skeptically examined. Extraordinary claims require extraordinary evidence.”
— From the 1980 TV series ‘Cosmos’, episode 12 (titled ‘Encyclopedia Galactica’)
However, the concept of ECREE has also been mentioned by various other scientists, at earlier points within the same general time period.
For example, Marcello Truzzi, founding co-chairman of the Committee for the Scientific Investigation of Claims of the Paranormal, is reported to have said that “an extraordinary claim requires extraordinary proof”, in a 1975 letter to the editor of Parapsychology Review, as well as in a 1978 article published in the Zetetic Scholar.
Similarly, a 1978 report credits Philip H. Ableson, then editor of Science, as having said that “these extraordinary claims require extraordinary evidence”, with regard to parapsychological research, which deals with topics such as telekinesis, telepathy, and clairvoyance. He further went on to say that:
“Findings that question the basic laws of nature must be subjected to rigorous scientific scrutiny, and must be able to be duplicated by impartial investigators. Until then, many scientists will remain unconvinced”.
Furthermore, the underlying idea behind ECREE has been discussed by others in various formulations during earlier periods of history.
For example, Thomas Jefferson, who was skeptical about a report of a meteor, stated in an 1808 letter that:
“… we certainly are not to deny whatever we cannot account for. A thousand phenomena present themselves daily which we cannot explain. But where facts are suggested, bearing no analogy with the laws of nature as yet known to us, their verity needs proofs proportioned to their difficulty.
A cautious mind will weigh well the opposition of the phenomenon to every thing hitherto observed, the strength of the testimony by which it is supported, and the errors & misconceptions to which even our senses are liable.”
Similarly, Scottish philosopher David Hume said the following, with regard to the idea of miracles:
“All effects follow not with like certainty from their supposed causes. Some events are found, in all countries and all ages, to have been constantly conjoined together: Others are found to have been more variable, and sometimes to disappoint our expectations; so that, in our reasonings concerning matter of fact, there are all imaginable degrees of assurance, from the highest certainty to the lowest species of moral evidence.
A wise man, therefore, proportions his belief to the evidence. In such conclusions as are founded on an infallible experience, he expects the event with the last degree of assurance, and regards his past experience as a full proof of the future existence of that event. In other cases, he proceeds with more caution: He weighs the opposite experiments: He considers which side is supported by the greater number of experiments: To that side he inclines, with doubt and hesitation; and when at last he fixes his judgment, the evidence exceeds not what we properly call probability. All probability, then, supposes an opposition of experiments and observations, where the one side is found to overbalance the other, and to produce a degree of evidence, proportioned to the superiority. A hundred instances or experiments on one side, and fifty on another, afford a doubtful expectation of any event; though a hundred uniform experiments, with only one that is contradictory, reasonably beget a pretty strong degree of assurance. In all cases, we must balance the opposite experiments, where they are opposite, and deduct the smaller number from the greater, in order to know the exact force of the superior evidence.”
— In “An Enquiry concerning Human Understanding”, section 10 (1748)
Finally, French scholar Pierre-Simon Laplace wrote about this concept in his highly influential book on mathematical probability theory:
“The more extraordinary a fact is, the more it needs to be supported by strong evidence.” [Originally in French: “plus un fait est extraordinaire, plus il a besoin d’être appuyé de fortes preuves.”]
— From “Théorie analytique des probabilités” (1812)
Note that in some places, the following statement is attributed to Laplace instead:
“The weight of evidence for an extraordinary claim must be proportioned to its strangeness”.
This could stem from a misquotation of a book written by Theodore Flournoy, where the author of the book discusses Laplace’s thoughts on the notion of ECREE, and then formulates what he calls the ‘Principle of Laplace’ in the following manner, while acknowledging that it may be expressed in various ways:
“The weight of the evidence ought to be proportioned to the strangeness of the facts.”
– “From India to the Planet Mars: A Study of a Case of Somnambulism” (1900)
Summary and conclusions
- The Sagan standard is the adage that “extraordinary claims require extraordinary evidence” (a concept abbreviated as ECREE).
- This signifies that if a claim seems extremely unlikely to be true, usually because of relevant past evidence, then it should be supported by extremely strong evidence before it’s accepted.
- For example, based on the Sagan standard, if someone claims that they saw a unicorn on their way to work, then they would be expected to produce extremely strong evidence in order to verify this claim, compared to if they made a less extraordinary claim, such as that they saw a horse.
- The definition of “extraordinary” is subjective, both when it comes to claims and when it comes to evidence, but it can still be justified based on factors like the level of preexisting evidence.
- When implementing the Sagan standard, it’s important to avoid using it as justification to automatically dismiss all information that contradicts the current consensus.