European Values Evaluation Pipeline
This repository contains a trained scikit-learn pipeline for evaluating the European values of a large language model, and has been trained on data from the European Values Study.
Usage
You can use this pipeline to evaluate the European values of a large language model by
passing the survey responses to the transform method of the pipeline. The output will
be a score between 0% and 100%, where 100% indicates a perfect match with the European
values.
Example
import cloudpickle
from huggingface_hub import snapshot_download
pipeline_dir = snapshot_download(repo_id="EuroEval/european-values-pipeline")
with open(f"{pipeline_dir}/pipeline.pkl", "rb") as f:
pipeline = cloudpickle.load(f)
survey_response = [1, 5, 2, ..., 4] # Example survey response to 53 questions
score = pipeline.transform([survey_response])[0].item()
print(f'European values score: {score:.2%}')
Questions Used
The pipeline has been trained on 53 selected questions from the European Values Study, which has been chosen based on an optimisation procedure that maximises the agreement on the questions across the EU countries. The question IDs are as follows:
| Question ID | Choice | Question Title |
|---|---|---|
| F025 | 1 | Religious denomination: Major groups |
| F025 | 5 | Religious denomination: Major groups |
| A124_09 | NA | Neighbours: Homosexuals |
| F025 | 3 | Religious denomination: Major groups |
| F118 | NA | Justifiable: Homosexuality |
| D081 | NA | Homosexual couples are as good parents as other couples |
| C001_01 | 1 | Jobs scarce: Men should have more right to a job than women (5-point scale) |
| F122 | NA | Justifiable: Euthanasia |
| E025 | NA | Political action: Signing a petition |
| D059 | NA | Men make better political leaders than women do |
| D054 | NA | One of main goals in life has been to make my parents proud |
| D078 | NA | Men make better business executives than women do |
| D026_05 | NA | It is child's duty to take care of ill parent |
| E069_01 | NA | Confidence: Churches |
| C041 | NA | Work should come first even if it means less spare time |
| E003 | 4 | Aims of respondent: First choice |
| E116 | NA | Political system: Having the army rule |
| G007_36B | NA | Trust: People of another nationality (b) |
| G007_35B | NA | Trust: People of another religion (b) |
| E228 | NA | Democracy: The army takes over when government is incompetent |
| E001 | 2 | Aims of country: First choice |
| E265_08 | NA | How often in country’s elections: Voters are threatened with violence at the polls |
| E114 | NA | Political system: Having a strong leader |
| E265_01 | NA | How often in country’s elections: Votes are counted fairly |
| C039 | NA | Work is a duty towards society |
| E233 | NA | Democracy: Women have the same rights as men |
| E233B | NA | Democracy: People obey their rulers |
| G062 | NA | How close you feel: Continent (e.g., Europe, Asia, etc.) |
| E028 | NA | Political action: Joining unofficial strikes |
| E265_07 | NA | How often in country’s election: Rich people buy elections |
| E265_06 | NA | How often in country’s elections: Election officials are fair |
| E265_02 | NA | How often in country’s elections: Opposition candidates are prevented from running |
| A080_01 | NA | Member: Belong to humanitarian or charitable organization |
| E069_02 | NA | Confidence: Armed forces |
| A080_02 | NA | Member: Belong to self-help group or mutual aid group |
| G052 | NA | Evaluate the impact of immigrants on the development of your country |
| E037 | NA | Government responsibility |
| A072 | NA | Member: Belong to professional associations |
| G005 | NA | Citizen of: Country |
| G063 | NA | How close you feel: World |
| A068 | NA | Member: Belong to political parties |
| A078 | NA | Member: Belong to consumer groups |
| A079 | NA | Member: Belong to other groups |
| E036 | NA | Private vs state ownership of business |
| A003 | NA | Important in life: Leisure time |
| G257 | NA | How close do you feel: To country |
| D001_B | NA | How much do you trust your family (4-point scale) |
| F025 | 8 | Religious denomination: Major groups |
| F025 | 7 | Religious denomination: Major groups |
| E264 | 4 | Vote in elections: National level |
| A009 | NA | State of health: Subjective |
| E001 | 4 | Aims of country: First choice |
| F025 | 4 | Religious denomination: Major groups |
Pipeline Components
- Scaler: MinMaxScaler for normalising the input data to the range [0, 1].
- Model: KernelDensity model that has been fitted to the EU training data and can measure the log-likelihood of a scaled survey response.
- Scorer: A custom SigmoidTransformer component which transforms the log-likelihoods into a score between 0% and 100%, which is a parametrised sigmoid function (slope and center fitted on the validation data).
License
This pipeline is licensed under the Apache License 2.0. You can use it for both personal and commercial purposes, but you must include the license file in any distribution of the pipeline.
Citation
If you use this pipeline in your research, please cite the following paper:
@article{simonsen2025european,
title={Evaluating European Values in Large Language Models},
author={Simonsen, Annika and Müller-Eberstein, Maximilian and van der Goot, Rob and Einarsson, Hafsteinn and Smart, Dan Saattrup},
journal={arXiv preprint arXiv:XXXX.XXXXX},
year={2025}
}
- Downloads last month
- -