Framework for claim-level reviews

BACKGROUND

Based on Climate Feedback article reviews, the dimensions of information credibility scientists have identified in climate change reporting cluster around the following categories:

• whether facts are accurate,
• whether explanations convey correct interpretation and understanding of the facts with sufficient context, and
• whether the conclusions reached follow sound logic

Informative reporting also relies on:

• the use of clear and precise language, and not ambiguous or vague claims,
• the use of adequate sources to back up important claims (“experts”, references), and
• following an unbiased approach to guarantee objectivity.

[see Climate Feedback’s framework to evaluate the scientific credibility of information at the article level.]

VERDICTS

We rate the scientific credibility of claims following our scale of scientific credibility. In our claim reviews, the background color of the tag indicates the scientific credibility level according to this color code.

Our “verdict tags” also feature a word summarizing the reason that the claim earned its credibility rating. Here is how we rate claims:

• Cases where a claim is of “Very High” credibility:

If the claim is a statement of fact, it is tagged as “Accurate” when it describes an observation in a way that is consistent with available data and does not leave out any relevant element of context. If the claim is an explanation of the causes of an observation (aka a “theory” or an “hypothesis” in science), it is deemed “Correct” when it has been well tested in scientific studies and generates expected observations that are confirmed by actual observations.

Tags used in this category: 

[Note that in science a theory cannot be “proved” to be correct, in the mathematical sense of the term. The words “Correct” and “Accurate” are often both used to qualify an explanation, but for the sake of avoiding ambiguity we reserve the use of the words ”(in)accurate” for statements of facts and ”(in)correct” for explanations, hypotheses, and theories.]

• Cases where a claim is of “High” credibility:

A statement of fact is deemed “Mostly Accurate” if it needs some clarification or additional information to be fully accurate. An explanation is deemed ‘Mostly Correct’ if it presents a theory that is well tested in scientific studies, but its formulation in the claim might overstate the confidence scientists actually have in the theory or slightly distort what can be predicted based on the theory.

Tags used in this category: 

• Cases where a claim is of “Neutral” credibility:

A claim is characterized as “Neutral” if it leaves out important information or is made out of context. For instance, a claim would be tagged “Correct but…” if it significantly overstates scientific confidence in a theory, or “Imprecise” if it uses ill-defined terms or lacks specifics so that one cannot unambiguously know what is meant without making additional unstated assumptions.

Tags used in this category: 

• Cases where a claim is of “Low” credibility:

A claim is deemed of “Low” credibility when it is made without backing from an adequate reference or if the available evidence does not support the statement (tagged as “Unsupported”). If a claim contains an element of truth but leaves the reader with a false understanding of reality, for instance by omitting critical background context, it would be tagged as “Misleading”.

Tags used in this category: 

• Cases where a claim is of “Very Low” credibility:

A claim is deemed of “Very Low” credibility when it is clearly wrong—for instance, if it makes a statement of fact in direct contradiction with available data (tagged as “Inaccurate”), or if it provides an explanation or a theory whose predictions have been invalidated (tagged as “Incorrect”).

Tags used in this category: 

Process for deciding on a verdict

The final ruling regarding the verdict attributed to the claim is made by a Climate Feedback editor based on suggestions by the scientists contributing to the review.

DETAILS

The “verdict details” section provides an explanation of the issue(s) raised by scientists—notably when a statement is implicitly or explicitly composed of several sub-claims. Below is a list of issues that have been raised in reviews along with a brief definitions.

Issues related to factual accuracy

• Factually inaccurate: A statement of fact in direct contradiction with available observations/data.
• Conflates factual statement and opinion: Presents opinion as fact or fact as opinion.

Issues related to explanations

• Correct: The theory/hypothesis is consistent with available data and has not been disproven.
• Misleading: Leaves the reader with a false or poor understanding of how things work.
• Misrepresents a complex reality: Fails to recognize that an observation can be influenced by more than one factor.
• Fails to grasp significance of observation: Uses an observation in support of a conclusion that it does not support.

Issues related to logic

• Flawed reasoning: Reasoning is flawed if conclusions do not follow from the premises.

Issues related to context

• Lack of context: The claim lacks elements of context (observations or explanations) that would change the reader’s takeaway.
• Cherry-picking: The claim depends on highlighting only a subset of all the available relevant evidence.

Issues related to representation of the scientific process

• Overstates scientific confidence: Presents a conclusion as conclusive while the hypothesis is still being investigated and there remains genuine scientific uncertainty about it.
• Overstates the scientific impact of a finding: For instance, claims that a new scientific study overturns all previous knowledge when, in reality, it is just an incremental update.

Issues related to sources

• Inadequate support: Reference used to support the claim is non-existent, of low scientific credibility or insufficient (according to the principle that extraordinary claims require extraordinary evidence).
• Misrepresents source (Strawman): Substitutes a misrepresentation of a source’s conclusion for its actual conclusion, often in order to make it easier to discredit the idea of an “opponent”.

Issues related to precision or clarity of language

• Lacks specifics: The claim is too broad or vague, or lacking supporting details, to be clearly verifiable.
• Imprecise: Uses a scientific term in a way that does not refer to the concept in the same way it is used in science.

Note: This typology is based on concrete examples that have been observed in Climate Feedback reviews so far. It will be updated as we encounter new examples.