PublicAlgoBeta
All case studies

NL · City of Amsterdam · 11 June 2025

The Limits of Ethical AI: Amsterdam's Smart Check welfare pilot

Image: Lighthouse Reports, from the original investigation. All rights reserved by the owner.

Reported by
Lighthouse Reports, MIT Technology Review, Trouw
Story language
EN · NL
Methods
  • FOI / records request
  • Code or model obtained
  • Data analysis
  • Interviews and surveys

Amsterdam did nearly everything the responsible-AI playbook asks: it opened its records, tested for bias, consulted citizens and built a model that could be explained. The journalists still found it unfair. In the live pilot the bias reversed, with Dutch applicants and women wrongly flagged more often, and the model did no better than caseworkers. The city stopped the pilot in November 2023.

lighthousereports.comThe Limits of Ethical AI: Amsterdam's Smart Check welfare pilotAn explainable machine-learning model (15 features) built by Amsterdam's welfare department to score welfare applications and decide which ones go to the fraud investigations unit, replacing the caseworker's first flagging step.Netherlands (City of Amsterdam) · Reported by Lighthouse Reports, MIT Technology Review, Trouw (opens in a new tab)

How they investigated

  1. FOI / records request

    Lighthouse filed a public records request in early 2023. The city answered quickly with documentation, code and several versions of the model, then invited the reporters to a meeting.

  2. Code or model obtained

    The city disclosed the original model and a reweighted version as model files, plus its training, scoring and bias-evaluation code. The model files only run on an old software setup, and Lighthouse published code to load them. It could not test the city's own code, because that needs individual-level data the city could not share. The reporters also built a hypothetical applicant to see how the model scored one case.

  3. Data analysis

    Data protection law barred row-level applicant data. So Lighthouse designed the tests, the city ran them on real data, and the city returned only aggregate results for eight groups, across training, test, prepilot and pilot data, before and after reweighting. Lighthouse computed several fairness measures from those tables. The method requires trusting the city's numbers. Code and aggregate data are published.

  4. Interviews and surveys

    Interviews over several months with city officials, data scientists, digital-rights advocates and welfare recipients, plus reporting on the city's welfare-recipient participation council, which opposed the project and later withdrew.

Read alongside