Our Workflow at a glance:

 

  1. Select an article for assessment, according to subject matter, relevance in current climate change discussion, and potential digital influence
  2. Call on scientists with relevant expertise to participate in article analysis
  3. Annotate selected article, with a focus on assessing (claimed) fact-based assertions and scientific reasoning
  4. Evaluate the article’s overall credibility against current state-of-the-art knowledge and thinking in climate science
  5. Summarize the main findings
  6. Provide Feedback + Publicize our analysis to traditional and digital media outlets, with help from fellow partners in academia, media, and journalism


cf_photoicons-08

1. Select

Our selection process is based on a variety of factors, including the quantity or degree of claimed scientific evidence within the reporting, the potential relevance of the article in shaping public discussion, and the size of the source’s existing digital readership.


cf_photoicons-14-01

2. Call

Once an article has been chosen, a coordinator is in charge of overseeing the annotation process from inception to completion. The coordinator is tasked with inviting scientists both inside and outside of our community with relevant expertise to contribute to the evaluation.


cf_photoicons-12-01

3. Annotate

Scientists are asked to annotate the article to indicate whether the facts underlying the reasoning are consistent with up-to-date scientific knowledge.
The generally proposed format for annotations is:

  • State the scientific fact in its correct form, citing the most relevant supporting reference(s);
  • Include a figure if applicable;
  • Explain briefly why the annotated information is (or is not consistent) with the reference;
  • Provide a reference* and a link to the source.
    *When an expert speaks within his own domain of expertise, including a reference may not be needed; however, if challenged on his claim, the expert should provide additional references to support his annotation

Annotations should be concise, accessible to general audiences as much as possible, and adhere to proper argumentation structure. Annotators should represent the state of knowledge in the scientific literature, using strongly supported scientific theories and observations as references, and refrain from pointing to partial/isolated/weakly supported findings. Annotators are expected to refrain from sarcastic or whimsical comments, and engage in serious, evidence-based conversation.

The primary objective of scientists should be to check the scientific accuracy of the facts and assumptions on which the article is based. However, their contribution need not be limited to fact checking and could extend to:

  1. Adding Relevant Information: Signaling additional resources or information related to the text, providing context, or indicating a perspective based on a scientific publication can also be the purpose of an annotation. Scientists should not focus on proving the author “right” or “wrong”. The main objective should be to base thinking in scientific facts and provide valuable resources to journalists and readers.
  2. Evaluating Scientific Reasoning: Oftentimes, misleading arguments are not about building on erroneous factual propositions but about combining them with a flawed/misleading reasoning or about building on hidden and weakly supported assumptions. Scientists are also invited to reflect on this aspect: promoting logic, scientific reasoning, the fact that conclusions should be supported by the argument made… in both annotations and in the evaluation phase.


cf_photoicons-11-01

4. Evaluate

To help readers better understand the annotations made, we invite scientists to provide an assessment regarding the overall scientific credibility of the article. The following criteria are used to help guide the evaluation:

  1. Accuracy. Does the article build on scientifically accurate information and solid evidence?
  2. Logic/Reasoning. Do the conclusions follow from the evidence? Are there incoherencies in the reasoning?
  3. Fairness/Objectivity. Does the author depict a complete or biased view of the relevant science? Does he hold for ‘right’ certain sources without proper justification? Would inclusion of other information change the main conclusion?
  4. Precision. Does the article precisely refer to concepts as they are used by the scientific community? Or is there some confusion which might mislead the reader?

 

Suggested guidelines for the overall scientific credibility rating

Remember that we do not evaluate the opinion of the author, but instead the scientific accuracy of facts contained within the text, and the scientific quality of reasoning used

  • +2 = Very High: no inaccuracies, fairly represents the state of scientific knowledge, well argumented and documented, references are provided for key elements. The article provides insights to the reader about climate change mechanisms and implications.
  • +1 = High: the article does not contain major scientific inaccuracies and its conclusion follows from the evidences.
  • 0 = Neutral: no major inaccuracies, but no important insight either that would have helped the reader understand the implications of the science.
  • -1 = Low: the article contains significant scientific inaccuracies or misleading statements.
  • -2 = Very Low: the article contains major scientific inaccuracies for key facts supporting his argumentation and/or omits to mention important information and/or presents logical flaws in using information to reach his conclusion.
  • n/a = Not Applicable: the article does not build on verifiable information (eg it is purely about politics).


 

Definition of the article-level tags/keywords

  • Accurate: free from factual errors
  • Inaccurate: contains assertions that are not supported by the scientific literature
  • Insightful: offers a deep understanding of the issue, offers perspective
  • Misleading: offers a wrong impression on some aspects of the science
  • Biased: holds some ideas (persons) as true (right) without proper justification, lack of objectivity, ideological
  • Unbiased: not biased, impartial, weights evidence for/against ideas
  • Sound reasoning: conclusion follows from the evidence presented
  • Flawed reasoning: conclusion does not follow from the evidence presented
  • Cherry-picking: pointing to individual cases or data that seem to confirm a particular position, while ignoring a significant portion of related cases or data that may contradict that position
  • Alarmist: overstates / exaggerates the risks of climate change
  • Derogatory: ad hominem attacks on scientists or scientific institution

 

Methodology adapted from recommendations by the Foundation for Critical Thinking


cf_photoicons-09

5. Summarize

After the annotation / evaluation processes are complete, a summary of the most salient points brought by the scientists is drafted by our team. This summary is crafted and reviewed by editors and members of our scientific community before being posted to the Climate Feedback website and made available to the general public.

Corrections: If we discover a mistake has been made, we will correct it as soon as possible.


cf_photoicons-13-01

6. Provide Feedback & Publicize

We inform authors and editors about the accuracy of their content. All ‘Feedbacks’ are promoted on our website, across our social platforms, and shared with key media and scientific partners. We’re always looking to connect with new outlets and fellow citizens eager to promote the efforts of the scientific community.