NLP · Political Text · 2024

Party lines,
from speech alone.

Abstract: A classifier trained on the speech of Spain's two largest parties, then pointed at parties it has never seen. If a model that only knows how PSOE and PP talk can still tell Podemos from Vox, then the left-right axis it learned is carrying real semantic content, and where the model fails is as informative as where it succeeds.
Data: ParlaMint's Spanish corpus: 32,551 transcribed speeches, 2015-2023, five legislative terms, 445 sessions, more than fifty parties. Filtered to interventions by sitting deputies from eight of them.
Method: Corpus consolidation → speaker-level split → chunking and giveaway removal → TF-IDF → Random Forest and XGBoost → transfer to unseen party pairs.

The design

We split the corpus into four pairs, each one a left party against a right party: PSOE-PP (about 12.5 k speeches), Unidas Podemos-Vox (4 k), ERC-JxCat (3 k) and Bildu-PNV (2.6 k). Only the first pair is ever used for training. The others are held out entirely, not as unseen speeches, but as unseen parties.

Within PSOE-PP the split is by speaker, not by speech: 70% of deputies train, 30% test. A model that has simply memorised how one deputy phrases things gets no credit, because that deputy is not in the test set.

Stylised illustration of a parliamentary chamber — Fig. 1 · Eight parties, five legislatures, one axis.

The cleaning is the experiment

Parliamentary transcripts are mostly not argument. They are procedure, thanks, and interruption. We dropped interventions under roughly a hundred tokens, sliced the long ones into chunks of at most three hundred, then dropped the stubs left over from slicing.

The consequential step was the list of giveaways: names of prominent deputies and of the parties themselves, removed from the vocabulary. Spanish parliamentary rhetoric is built on naming your opponent, so these terms are enormously predictive and entirely useless, because a model trained on them learns who is being attacked rather than how. Removing them cost more than five points in-sample. We kept them out anyway; the entire point was the transfer.

Models

Speeches are vectorised with TF-IDF over unigrams and bigrams, dropping terms in fewer than 1% or more than 40% of documents. On top of that, a Random Forest of a thousand trees and an XGBoost, with a random classifier as the floor.

On out-of-sample PSOE and PP speakers (5,166 chunks), the Random Forest reaches 0.83 accuracy and the XGBoost 0.82, against 0.49 for random. Their ROC-AUC is 0.90 and 0.91 respectively. The two models disagree in a useful way: the forest's probabilities cluster near 0.5, XGBoost's spread across the interval, and they still arrive at the same ranking.

Predicted-probability densities for PSOE and PP speeches — Fig. 2 · PSOE against PP, on speakers held out of training.

The transfer

Now the model is shown Podemos and Vox, whose speech it has never seen, and asked the only question it knows: does this sound more like PSOE or more like PP? It separates them at 0.86 AUC. Nothing in the model understands ideology; it is matching vocabulary. That it works implies Podemos and PSOE share enough rhetoric (and Vox and PP likewise) for one to stand in for the other.

Predicted-probability densities for Podemos and Vox speeches — Fig. 3 · Podemos against Vox. Neither party appeared in training; the separation survives anyway.

Where it breaks

On the Catalan pair the model has nothing to say. ERC and JxCat sit on opposite sides of the left-right axis in principle, but their parliamentary speech is organised around Catalan independence, an axis orthogonal to the one learned in the PSOE-PP data. The densities lie on top of each other, which is the correct answer rather than a failure of fitting.

Predicted-probability densities for ERC and JxCat speeches — Fig. 4 · ERC against JxCat. Two distributions with less to distinguish them.

What the features say

The heaviest n-grams are derecha, ciudadanía, transformación, socio and crear empleo. Much of the signal is positional rather than ideological: PSOE governs for most of the window, so terms of achievement (crear empleo, mayoría absoluta, extraordinario) sit on one side and terms of grievance (recorte, propaganda, destruir) on the other. In hindsight derecha and izquierda belonged on the giveaway list too.

Lessons

The degree of transfer is itself a measurement. A classifier that generalises to one unseen pair and collapses on another is telling you something substantive about which axis each pair is actually arguing along, and it will only tell you that if you take the easy words away first.