Europolls

A Wikipedia-direct database of opinion polls covering 30 European countries — the EU-27 minus Belgium, plus the UK, Norway, Iceland, and Switzerland — with France split into separate presidential and legislative cycles (31 country-codes in total).

Europolls scrapes the per-country Opinion polling for the X election articles from English Wikipedia, parses the polling tables into long and wide formats, joins to a stable party identifier (partyfacts_id), and ships the result as a public, reproducible dataset.

When a dedicated polling article doesn't exist for a given cycle (common for smaller countries and pre-2010 cycles), the fetcher falls back to the election article itself and parses any wikitable under its Opinion polls / Polling section. This recovers ~12k poll-party observations across 39 of 51 attempted fallback cycles.

The pipeline is deterministic: every row carries (i) the source URL of the original poll publication as cited in Wikipedia and (ii) the Wikipedia article's permanent ?oldid=<revid> permalink, so revisions can't silently drift the dataset.

Inspired by Pitas (2023) Europepolls, which is no longer available; rebuilt from scratch with reproducible API-based scraping and several improvements (long format, source-URL provenance, partyfacts harmonization, election-article fallback).

Coverage

31 country-codes, ~33k unique polls, ~298k poll-party rows in the canonical long file (~278k after dropping non-voting-intention rows).

Country	Polls	Span	Top parties
UK (United Kingdom)	5,246	2005–2026	Con, Lab, LD, Grn
IT (Italy)	5,018	2005–2026	PD, Lega, M5S, FdI
DE (Germany)	4,442	2009–2026	SPD, Grüne, FDP, Linke, Union
ES (Spain)	2,915	2004–2026	PSOE, PP, PNV, Vox, Cs
PL (Poland)	1,714	2005–2026	PiS, KO, Konfederacja, Lewica
GR (Greece)	1,645	2007–2026	SYRIZA, ND, KKE, PASOK
NO (Norway)	1,464	2009–2026	FrP, KrF, H, SV, Sp
SE (Sweden)	1,182	2010–2024	S, M, SD, C, V
AT (Austria)	987	2012–2026	ÖVP, SPÖ, FPÖ, Grüne, NEOS
DK (Denmark)	847	2007–2031	A, V, O, B, Ø
FR (France) — presidential	847	2002–2022	Le Pen, Sarkozy, Macron, Mélenchon
PT (Portugal)	834	2005–2026	PSD, PS, BE, CDU, CH
CZ (Czechia)	695	2006–2026	ANO, ODS, SPOLU, Piráti, STAN
EE (Estonia)	690	2010–2026	KE, RE, EKRE, SDE, E200
NL (Netherlands)	650	2010–2026	VVD, PVV, GL–PvdA, CDA, D66
HU (Hungary)	575	2009–2026	Fidesz, Tisza, DK, Jobbik
SI (Slovenia)	463	2011–2026	SDS, SD, GS, Levica
IE (Ireland)	462	2011–2026	FF, FG, SF, Lab, GP
SK (Slovakia)	397	2012–2026	SMER-SD, PS, Hlas-SD, OĽaNO
FI (Finland)	378	2011–2026	KESK, KOK, SDP, PS, VIHR
IS (Iceland)	377	2007–2024	D, B, S, V, P
BG (Bulgaria)	283	2013–2024	GERB, PP–DB, DPS, BSP, ITN
FR_LEG (France) — legislative	253	2002–2024	RN, LFI, NFP, ENS, LR
LT (Lithuania)	246	2016–2024	LSDP, TS-LKD, LVŽS, LRLS
RO (Romania)	173	2012–2026	PSD, PNL, USR, AUR
LV (Latvia)	146	2014–2022	JV, ZZS, Saskaņa, NA
MT (Malta)	111	2012–2022	PL, PN, ADPD
CY (Cyprus)	72	2016–2023	(presidential candidates)
CH (Switzerland)	61	2011–2023	SVP, SP, FDP, Die Mitte
HR (Croatia)	49	2015–2016	HDZ, SDP, Most
LU (Luxembourg)	21	2013–2023	CSV, LSAP, DP

Belgium is not included. BE polling articles publish separate Flemish and Walloon subtables that the wikitable parser cannot disentangle cleanly — each party's series would alternate between its home-region value (~25%) and its non-home-region value (~2%). Region-aware parsing is a known TODO.

A small number of older cycles (LV 2011/2014, NO 2005, NL 2010, SK 2012, etc.) produce no rows because the election article has no polling section, uses a chart-only / transposed layout, or otherwise can't be parsed by the wikitable-based pipeline. See docs/ADDING_A_COUNTRY.md#known-patterns-and-pitfalls.

Schema

Long format — data/processed/polls_long.csv

column	type	description
country	str	ISO2 country code
cycle	str	electoral cycle identifier (e.g. 2022, 2019a, current)
polldate_start, polldate_end, polldate_mid	date	fieldwork dates
pollster	str	polling firm name
source_url	str	URL of the original poll publication (from Wikipedia cell)
sample_size	int (nullable)	reported sample size
party_short	str	party abbreviation as shown in Wikipedia
is_coalition	bool	true if the column is a coalition aggregate
vote_share	float (0–100)	reported vote share
wiki_article	str	source Wikipedia article title (either the dedicated polling article or the election article when fallback fired)
wiki_revid	int	pinned revision id
wiki_url	str	?oldid=<revid> permalink

The provenance of each row's article (polling article vs election-article fallback) is recorded per-cycle in data/raw/{COUNTRY}/{CYCLE}/source.json as source_kind: polling_article|election_article.

Cross-cycle and within-cycle duplicate rows — same (country, polldate_mid, pollster, party_short) appearing twice because Wikipedia editors copy prior election results into new polling articles, or list the same poll in multiple wikitables — are collapsed during the concat step. The surviving row is the one whose cycle's election year is closest to the poll's own year, with ties broken by larger sample size then by newer revid.

Wide format — data/processed/{COUNTRY}_polls_wide.csv

One row per poll. Meta columns first, then one column per party_short, sorted by observation count. Coalition aggregates prefixed coal_.

Harmonized format — data/interim/harmonized/{COUNTRY}_long_harmonized.csv

Long format augmented with partyfacts_id from Party Facts — enables joins to ParlGov, MARPOR, PopuList, CHES, and similar datasets. Covers all 31 country-codes via config/party_mappings/{CC}.yaml; 97.1% of voting-intention poll rows carry a partyfacts_id, with provenance in partyfacts_id_source / partyfacts_id_confidence / partyfacts_id_notes. The same four columns are also joined onto the canonical data/processed/polls_long.csv so downstream consumers don't need to re-apply the mapping. See scripts/PARTY_CROSSWALK.md for how the per-country YAMLs were built.

Two further columns mark non-voting-intention rows that survive the canonical long file:

• is_dropped_meta (0/1) — set on rows whose party_short is a meta-label (Don't know, Others, Abstention, Undecided buckets) per config/party_mappings/_meta.yaml. Filter is_dropped_meta == 0 to keep only real-party voting-intention rows.
• dropped_meta_reason — the bucket (others / dont_know / undecided / abstention / approval / chancellor_preference / total / lead / response_rate) when flagged.

Approval-rating columns (Approve / Disapprove / Net / Lead), leader-preference items (DE Merkel / Steinbrück Kanzlerfrage columns), survey-meta stats (Resp.|Response rate), and redundant coalition totals (BG Total, IT Coalitions) are pruned entirely at the concat step — they are not voting-intention signal. Source-URL-keyed drops also remove non-vote-share publications such as election.de's seat-forecast map. Coalition aggregate rows (IT CDX/CSX, DK Blocs, DE multi-party blocs flagged at parse time) are collapsed when the same poll already lists the component parties; bloc labels that are reported as single observables (IT AVS, PL UnitedRight) are preserved.

partyfacts_id_source distinguishes confirmed-absences from uncovered cases: partyfacts_external / whogov_minister_match / hand_curated / manual_review / gemini_* for matches; verified_not_in_pf when a hand check confirms Party Facts has no record (e.g. ES Sumar, IT AVS); gemini_no_match for auto-build no-matches that warrant periodic re-verification.

pandas gotcha. polls_long.csv contains real party labels that pandas' default read_csv interprets as NaN (notably GR NA = New Left, 418 rows). Read with:

pd.read_csv("polls_long.csv", keep_default_na=False, na_values=[""])

R / Stata / DuckDB don't have this issue out of the box.

partyfacts_name + party_canonical + europolls_party_id give a stable cross-row aggregation key:

• partyfacts_name is PF's canonical English name (falls back to native / short) for rows that carry a partyfacts_id. Looked up from config/partyfacts_names.yaml — a slim repo-committed PF id → name lookup that you regenerate with python scripts/build_partyfacts_names.py whenever a fresh PF snapshot lands.
• party_canonical = partyfacts_name when present, otherwise the row's party_short. This is the column to group by when you want IT Lega + LN (different shorts, same partyfacts_id=1221) to collapse to one series — ES Sumar and IT AVS (no PF id, real party) stay as themselves rather than disappearing.
• europolls_party_id is the stable cross-poll, cross-country identifier: str(partyfacts_id) when present (e.g. "1221"), otherwise "EU:CC:party_short" (e.g. "EU:IT:AVS"). Never collides with PF's namespace and never ambiguous across countries — every country has parties called Lab or PSD, so a bare party_short is not enough as a join key for multi-country panels. Use this column when joining the dataset to other multi-country sources or merging across cycles.

Validation

The Italian scrape was validated against the official Italian government poll register (via ondata's liberiamoli-tutti mirror of sondaggipoliticoelettorali.it):

• 2,523 overlapping (year-month × pollster × party) cells
• 92.9% agree within ±0.5 pp
• 96.9% agree within ±1.0 pp
• 98.9% agree within ±2.0 pp
• Mean absolute discrepancy: 0.21 pp; median 0.05 pp.

The largest outliers (~10 cells) trace to coalition-vs-individual-party reporting splits, not scraping bugs.

Usage

Consume pre-built data

Pre-built CSVs are published as GitHub Release assets (the repo itself ships only the code and config — data is reproducible). Download the latest release from https://github.com/gfrt0/europolls/releases.

The CI build also uploads data/processed/polls_long.csv as a per-commit artifact (30-day retention) — downloadable from the Actions tab without needing to re-run the pipeline locally.

Reproduce locally

git clone https://github.com/gfrt0/europolls.git
cd europolls
python -m venv ~/venvs/europolls
~/venvs/europolls/bin/pip install -e .

# Full pipeline (fetches all wikitext, parses, pivots, dedups, concatenates, harmonizes):
python scripts/build_all.py

# Or a single country:
python -m europolls.fetch IT 2006 2008 2013 2018 2022 current
python -m europolls.parse IT
python -m europolls.pivot_wide IT
python -m europolls.harmonize IT

Outputs land under:

• data/raw/{COUNTRY}/{CYCLE}/article.wikitext — Wikipedia snapshots
• data/raw/{COUNTRY}/{CYCLE}/source.json — title, revid, fetched_iso, source_kind
• data/interim/{COUNTRY}_{CYCLE}.csv — long format per (country, cycle)
• data/processed/{COUNTRY}_polls_wide.csv — wide pivot per country
• data/processed/polls_long.csv — single concatenated + deduped long file
• data/interim/harmonized/{COUNTRY}_long_harmonized.csv — partyfacts_id-augmented

The fetcher rate-limits itself (1.5s between requests) and retries on HTTP 429 / 503 with exponential backoff; a full sweep runs in ~6 min.

Extending coverage

Adding a new country takes ~50 lines of YAML + one JS constant. See docs/ADDING_A_COUNTRY.md for a step-by-step recipe with concrete templates and known pitfalls. Candidates beyond the current 32 are listed at Wikipedia's opinion-polling-by-country category.

The presentation layer (chart and tables on the live site) draws on a small family of normalization config files:

• config/pollster_aliases.yaml — global pollster-name canonicalization (Techne → Tecnè, Kantar Public / TNS Sofres → Kantar, etc.).
• config/party_aliases/{CC}.yaml — per-country party-short canonicalization, optional drop: list for leader-approval or cabinet-rating columns, and drop_regex: for pattern matches. _meta.yaml folds Don't-know / Abstain / Others / Neither across all countries.
• config/party_names.yaml — full party names used for the chart's hover tooltips.

Roadmap

• <ref>-footnote URL extraction. The current source_url column captures the poll's external link only when Wikipedia writes inline [url Pollster] markup (~69% of rows). Articles that cite via <ref name="...">{{cite web|url=...}}</ref> footnotes — notably ES, LV, IE (0%), and partially GR/NL/FI — lose their citations because the parser strips <ref> blocks as HTML tags before URL extraction. Adding a pre-strip pass that finds <ref> adjacent to the pollster cell and pulls URLs from the citation template would meaningfully bump coverage.
• Parser defensive fallback for malformed style="..."| headers. Six Wikipedia revisions identified during the party-mapping work (CH 2023 LeeWas table 28, DE current INSA tables 10–11, GR current main table, LV 2018/2022 voting-intention tables, SI 2018/2022 tables) drop ~635 poll-party rows because the column-header markup omits the closing " on a style= attribute. The cleanest fix is to add a regex fallback in europolls/lib.strip_wikitext that recovers the wikilink even from an unterminated style prefix. Alternatively, file editorial fixes to the offending Wikipedia revisions — scripts/wikipedia_feedback.py (TBD) would emit a list of the exact one-character edits.
• Upstream Wikipedia contributions. Two opportunities: (a) the malformed-style fixes above; (b) adding [[Article|short]] wikilinks to bare-text column headers we identified during the wikilink-disambiguation pass — would let our pipeline pick up future polls of those parties automatically rather than via hand mapping.
• Sub-10-row gemini_no_match tail. 530 rows / 186 pairs after the wikilink + reverse-lookup + PT/FR_LEG candidate cleanup. Mostly micro-parties below the threshold where individual investigation pays off; diminishing returns.
• europolls_party_id namespace. Stable identifier for parties not in PF (Sumar, AVS, etc.). party_canonical falls back to party_short today, which is fine for in-country joins but ambiguous across countries (every country has a Lab, PSD, etc.). A namespaced fallback (e.g. EU:IT:AVS) would make cross-country joins safer.
• Long-tail wide-parties curation across the newly-added Central/Eastern European and Baltic countries — the lists are first-pass guesses that haven't been hand-validated.
• Five fallback cycles (LT 2012, LT 2016, NL 2010, NO 2013, SK 2012) contain real polling wikitables in transposed orientation (parties as rows, dates/pollsters as columns) — see #9. The remaining seven of the twelve historical-fallback cycles have no polling section to recover (CY 2011, CZ 2025, LV 2011 / 2014, NO 2005, RO 2024, SE 2010); they need an upstream Wikipedia article.

Citation

Forte, G. (2026). Europolls: a Wikipedia-direct opinion polling dataset for 32 European countries. https://github.com/gfrt0/europolls

If you use individual polls, please also cite the original polling firm and the underlying Wikipedia article (both are referenced per-row in the dataset).

Licences

• Code: MIT (see LICENSE)
• Data: CC-BY-SA 4.0 (see LICENSE-DATA), inherited from Wikipedia