Our Workflow at a glance:

 

  1. Select an article for assessment, according to subject matter, relevance in current climate change discussion, and potential digital influence
  2. Call on scientists with relevant expertise to participate in article analysis
  3. Annotate selected article, with a focus on assessing (claimed) fact-based assertions and scientific reasoning
  4. Evaluate the article’s overall credibility against current state-of-the-art knowledge and thinking in climate science
  5. Summarize the main findings
  6. Provide Feedback + Publicize our analysis


cf_photoicons-08

1. Select

Our selection process is based on a variety of factors, including the quantity or degree of claimed scientific evidence within the reporting, the potential relevance of the article in shaping public discussion, and the size of the source’s existing readership.

We strive for our reviews to be representative of the spectrum of influential climate discussion in the media. We review articles and claims in a variety of media, without a priori perspective, regardless of whether they insightfully report on, exaggerate or downplay the consequences of climate change.

If you think we missed an important claim or article, you can suggest it for review here.


cf_photoicons-14-01

2. Invite reviewers

Once an article has been chosen, an editor is in charge of overseeing the review process from inception to completion. The editor invites scientists with relevant expertise -both inside and outside of our network- to contribute to the review.


cf_photoicons-12-01

3. Annotate

Scientists are asked to comment on the article to indicate whether the facts underlying the reasoning are consistent with up-to-date scientific knowledge.
The generally proposed format for comments is:

  • State the scientific fact in its correct form, citing the most relevant supporting reference(s);
  • Include a figure if applicable;
  • Explain briefly why the annotated information is (or is not) consistent with the reference;
  • Provide a reference* and a link to the source.
    *When an expert speaks within their own domain of expertise, including a reference may not be needed; however, if challenged on their claim, the expert should provide additional references to support the annotation

Comments should be concise, accessible to general audiences as much as possible, and adhere to proper argumentation structure. Reviewers should represent the state of knowledge in the scientific literature, using strongly supported scientific theories and observations as references, and refrain from pointing to partial/isolated/weakly supported findings. Reviewers are expected to refrain from sarcastic comments, maintain a respectful tone and engage in serious, evidence-based conversation.

The primary objective of scientists should be to check the scientific accuracy of the facts and assumptions on which the article is based. However, their contributions need not be limited to fact-checking and could extend to:

  1. Adding Relevant Information: Signaling additional resources or information related to the text, providing context, or indicating a perspective based on a scientific publication can also be the purpose of an annotation. Scientists should not focus on proving the author “right” or “wrong”. The main objective should be to base thinking on scientific facts and provide valuable resources to journalists and readers.
  2. Evaluating Scientific Reasoning: Misleading arguments are often based not on  erroneous factual propositions, but on flawed/misleading reasoning or hidden and weakly supported assumptions. Scientists are also invited to reflect on this aspect: promoting logic, scientific reasoning, and the fact that conclusions should be supported by the argument made… in both annotations and in the evaluation phase.


cf_photoicons-11-01

4. Evaluate

To help readers better understand the comments made, we invite scientists to provide an assessment regarding the overall scientific credibility of the article. The following criteria are used to help guide the evaluation:

  1. Factual Accuracy. Does the article build on scientifically accurate information and solid evidence?
  2. Scientific understanding. Beyond accurately describing basic observations, does the article provide a correct or a mistaken understanding of how the climate system works? Does it understand or misinterpret the significance of the observations?
  3. Logic/Reasoning. Do the conclusions follow from the evidence? Are there incoherencies in the reasoning?
  4. Precision/Clarity. Does the article precisely refer to concepts as they are used by the scientific community? Or is there some confusion which might mislead the reader?
  5. Sources Quality. Does the article rely on adequate and credible sources (experts, studies) to back up important claims?
  6. Fairness/Objectivity. Does the author depict a complete or biased view of the relevant science? Does he hold for ‘right’ certain sources without proper justification? Would inclusion of other information change the main conclusion?

 

Suggested guidelines for the overall scientific credibility rating

Remember that we do not evaluate the opinion of the author, but instead the scientific accuracy of facts contained within the text, and the scientific quality of reasoning used.

  • +2 = Very High: No inaccuracies, fairly represents the state of scientific knowledge, well argumented and documented, references are provided for key elements. The article provides insights to the reader about climate change mechanisms and implications.
  • +1 = High: The article does not contain major scientific inaccuracies and its conclusion follows from the evidence provided.
  • 0 = Neutral: No major inaccuracies, but no important insight to better explain implications of the science.
  • -1 = Low: The article contains significant scientific inaccuracies or misleading statements.
  • -2 = Very Low: The article contains major scientific inaccuracies for key facts supporting the author’s argumentation and/or omits to mention important information and/or presents logical flaws in using information to reach his or her conclusion.
  • n/a = Not Applicable: The article does not build on scientifically verifiable information (e.g., it is mostly about politics or opinions).

The final Climate Feedback rating is the average of all the reviewers’ ratings; in cases when reviewers largely disagree no rating is issued and the review is left as ‘debated’.

Definition of the article-level tags/keywords

  • Accurate: free from factual errors, describes reality in a way that is consistent with available data/observations.
  • Inaccurate: contains assertions that do not correspond to observations.
  • Insightful: offers a deep understanding of the issue based on accurate information and proper context that clarifies the implications of observations.
  • Misleading: offers an incorrect impression on some aspect(s) of the science, leaves the reader with false understanding of how things work, for instance by omitting necessary background context.
  • Biased: holds some ideas (persons) as true (right) without proper justification, lack of objectivity, ideological.
  • Unbiased: not biased, impartial, weights evidence for/against ideas
  • Sound reasoning: conclusion follows from the evidence presented.
  • Flawed reasoning: conclusion does not follow from the evidence presented.
  • Cherry-picking: highlights only a subset of all the available relevant evidence that seem to confirm a particular conclusion, ignoring a significant portion of evidence that would contradict it.
  • Alarmist: overstates / exaggerates the risks of climate change.
  • Derogatory: contains ad hominem attacks on scientists or scientific institutions.
  • Inappropriate backing: relies on low credibility sources, provides insufficient evidence in support of claims made.
  • Imprecise/Unclear: uses ill-defined terms or lacks specifics so that one cannot unambiguously know what is meant without making additional unstated assumptions.

 

Methodology adapted from recommendations by the Foundation for Critical Thinking


cf_photoicons-09

5. Summarize

After the review process is complete, the editor writes a summary of the most salient points brought by the scientists.


cf_photoicons-13-01

6. Provide Feedback & Publicize

We provide feedback to journalists and/or editors about the accuracy of the content published in their outlets. All ‘Feedbacks’ are promoted on our website, across our social platforms, and shared with key media and scientific partners.

You can help us amplify the voice of science by following us on social media, sharing our content and signing-up to our email notifications!


Corrections

We aim for our “feedbacks” to be as accurate and up-to-date as possible. If we discover a mistake has been made, we will correct it as soon as possible and a note will be added on the original item. If you think we’ve made an error or missed some relevant information, contact us.


Suggest an item to review

If you wish to submit a suggestion of an article or claim to review, please use this online form. Please note that we focus on reviewing claims and articles that are scientifically verifiable and that reach large audiences.