What we learned assessing 10,000 COP30 articles
A methodological dry run: what a large-scale assessment can show, what it cannot, and why the denominator matters.
This article describes a planned study design and uses illustrative figures. We have not completed a 10,000-article assessment, and no finding below should be cited as a measured result. We are publishing the method early so it can be criticised before we run it.
Large-scale content studies are seductive because they produce memorable numbers. “We assessed 10,000 articles” sounds like evidence. Sometimes it is. Often it is a denominator in search of a question.
This piece explains how we would run a 10,000-article assessment of COP30 coverage, what we would expect to learn, and where the method could mislead us. The point is not the number. The point is whether the number can bear the conclusion placed on it.
Why COP30 coverage is a useful test set
Climate coverage combines everything that makes quality assessment hard. It is technical, politically contested, geographically uneven, and full of numbers that are easy to quote and hard to check. A single article can mix peer-reviewed findings, advocacy claims, government projections, and breaking news within a few paragraphs.
That makes it a good stress test for a seven-dimension framework. Accuracy is not a property of the whole article; it is a property of individual claims. Evidence varies by paragraph. Context depends on what a reader already knows. Fairness is not the same as false balance.
COP coverage also has a clear comparison point: the underlying documents. Nationally determined contributions, IPCC summaries, and official statements are public. If a claim cannot be traced to one of them, that is a finding — but not necessarily a failure. Journalists legitimately rely on interviews, observations, and leaks.
The study design we would use
A credible design needs four things: a defined corpus, a sampling frame, a coding protocol, and a pre-registered analysis plan.
1. The corpus
We would define the corpus by publication date, language, and outlet type. “COP30 coverage” is not a self-selecting category. We would include articles that mention the conference in the headline or first three paragraphs, published between two weeks before the summit and two weeks after it closes.
We would exclude opinion, sponsored content, and live blogs, not because they are unimportant, but because their quality criteria differ. Mixing them into one dataset produces averages that describe nothing.
2. The sample
A random sample stratified by outlet type and language would be better than a convenience sample of familiar outlets. If we want to say something about “coverage,” the sample has to look like coverage, not like our reading list.
Ten thousand articles is a lot until it is divided by language, outlet type, and content type. At that point some cells are thin. We would publish cell sizes alongside every result, because an average across six languages can hide the fact that one language has twelve articles.
3. The coding protocol
Each article would be assessed on the seven dimensions, with evidence attached to each score. Two trained coders would independently rate a subsample. We would report agreement — not as a formality, but because low agreement means the dimension is not yet measurable.
Where coders disagree, we would keep the disagreement in the dataset. Collapsing it into a majority vote makes the method look cleaner and the result less honest.
4. The analysis plan
Before running the study, we would specify the primary outcomes, the subgroups, and the correction for multiple comparisons. Exploratory findings would be labelled as exploratory. This is basic, and it is routinely skipped in content studies that need a headline.
What we would expect to find (illustratively)
Based on smaller dry runs and interviews, we would expect three patterns. These are hypotheses, not results.
- Evidence scores would vary more within outlets than between them. Desk, format, and reporter matter more than masthead.
- Context would be the weakest dimension in breaking coverage and the strongest in explainers, which is almost tautological but still worth measuring.
- Headline-body alignment would be a persistent problem, especially where SEO pressure shapes the headline after the body is filed.
We would also expect to find things we did not anticipate. That is the point of running the study rather than asserting the conclusion.
Where the method can mislead
False precision
A score of 71.4 looks more scientific than “mixed.” It is not. If the confidence interval is ±12, the decimal is noise. We would report intervals and avoid ranking outlets by fractions of a point.
Construct validity
Does “Evidence” measure evidence, or does it measure how easy the article is to parse? If the system rewards linked sources, it may penalise valuable reporting that protects sources. That is a design choice with ethical consequences, not a neutral measurement.
Language and context
Standards differ across press traditions. A framework built on English-language wire copy will misfire elsewhere. We would validate per language and publish where we cannot.
The denominator problem
Ten thousand articles is not “all coverage.” It is a sample, and the sample defines the claim. If we say “climate coverage struggles with context,” we need to be precise about which coverage, in which languages, over which window.
What this means for newsrooms
A large study is useful for identifying patterns, not for grading individual journalists. The same distinction applies inside a newsroom. Desk-level trends can inform training and process. An individual score attached to a byline invites gaming and defensiveness without improving the work.
We would use the study to ask better questions: Which formats consistently miss context? Where does evidence break down? Are corrections being made quickly enough? Those questions are actionable. “Our average is 68” is not.
What we would publish
If we run the study, we will publish the corpus definition, sampling code, coding protocol, agreement statistics, per-cell sample sizes, and limitations. We will also publish the cases where the method failed. A study that reports only its cleanest findings is marketing with a methods section.
Until then, treat this as a design document. The number 10,000 is a commitment to scale, not a result.