How RatedNews scores the news
Every article on RatedNews is scored automatically by AI, then blended with community ratings over time. This page explains the full process — including the exact prompt we send to the AI, the weighting formula, and where the approach falls short. Nothing is hidden.
What gets scored
RatedNews scores two things: individual articles and outlets.
Articles are scored as they are ingested — every 30 minutes, across 100+ syndicated feeds. Each article receives an accuracy score, bias direction, partisan intensity, headline verdict, category, geographic scope, and region tag. These are generated in a single AI call per article.
Outlet scores are rolling 30-day averages of their articles' scores, recalculated every hour. They are then blended with community ratings using a tiered weighting system explained below.
How AI scoring works
Each article is sent to Claude Haiku (Anthropic's fastest model) with a fixed system prompt. The model receives three inputs: the article title, the outlet name, and the syndicated summary — typically 1–3 sentences.
The model returns a structured JSON response with all score fields in a single pass. We use prompt caching so the system instructions are only processed once per session, reducing cost and latency.
This prompt is updated periodically as we refine our definitions. Changes are logged in the changelog at the bottom of this page.
How outlet scores are calculated
An outlet's overall score blends three signals. The balance shifts as community ratings accumulate — early on the AI leads, but a well-rated outlet's community voice eventually carries the most weight.
| Community ratings | AI score | Editorial baseline | Community score |
|---|---|---|---|
| 0 ratings | 70% | 30% | — |
| 1–4 ratings | 50% | 30% | 20% |
| 5–19 ratings | 40% | 25% | 35% |
| 20+ ratings | 35% | 25% | 40% |
The editorial baseline is fixed at 50 (neutral) until we introduce manual editorial review. It acts as a stabilising anchor — preventing a handful of extreme article scores from producing a wildly unrepresentative outlet score.
The community score converts star ratings (1–5 stars) to a 0–100 scale. Community ratings only appear on an outlet's score once at least one rating exists.
Known limitations
Accuracy and bias scores are derived from the headline and syndicated summary — the content publishers explicitly provide for distribution. We don't fetch or process full article HTML. A misleading article with a factual-sounding headline can score well. A nuanced long-read might score differently in full. Treat scores as signals, not verdicts.
"Accuracy score: 82" does not mean 82% of the claims in the article are true. It means the writing pattern matches what well-sourced, factual journalism tends to look like. We are pattern-matching on language, not fact-checking claims. No automated system can do that reliably at this scale.
An outlet with 8 articles this week has a much less reliable score than one with 300. A single unusual article can swing a small outlet's score significantly. We display article counts on outlet pages so you can judge the sample size yourself.
Claude (like all large language models) has been trained on a large corpus of text that reflects particular distributions of language, perspective, and framing. We have tuned the prompt carefully, but the model's underlying tendencies are not fully transparent even to us. We change the model or prompt, we log it.
Detecting political lean from a 12-word headline is an inherently imprecise task. Short headlines lack the context that makes bias legible. Bias direction scores are more reliable for long-form pieces with clear editorial framing than for breaking news headlines.
Changelog
Every significant change to the scoring prompt, model, or weighting formula is logged here. If you notice a score shift that seems unexplained, check this first.
Questions about the methodology or spotted something wrong? Get in touch. We update this page whenever the scoring system changes.