I'm working on this topic
Kilian Vieth-Ditlmann
Policy Director
AI systems can lead to flawed, false, and even biased results. Error-prone, often discriminatory facial recognition systems have already led to the incarceration of people who have been mistakenly identified as criminal offenders. The Dutch child care benefits scandal demonstrated that it can be flagged as a fraudster with algorithms biased towards innocent people. In such cases, the impact on people's lives is harmful. This may even be a massive effect as the system perpetuates a pattern of bias towards a particular group of people.
The EU AI Rule Book – Artificial Intelligence (AI) Act – is the first comprehensive law to reduce the red line for certain AI practices. The law prohibits AI systems that believe pose unacceptable risks to people's safety, health and fundamental rights. This protection also takes precedence when the system is operating outside the EU but generates output in the EU. Prohibited AI practices are listed in Article 5 of the Rules.
Violation of the prohibition of AI practices mentioned in Article 5 shall be subject to a maximum of 35,000,000 euros of administrative fines, or, for the company, 7% of annual revenue.
Which AI practices are (partially) prohibited?
It is prohibited:
- AI systems that change people's behavior, such as voice-activating toys that use manipulative or deceptive techniques to encourage children to act dangerously;
- AI systems that utilize vulnerabilities in people and groups;
- A particular form of social scoring is independent of the context in which the data used was collected;
- AI systems that create or extend facial recognition databases via reduction of facial images from the Internet or CCTV footage (such as ClearView AI and Pimeys),
- Live face recognition in public spaces by police (with exceptions).
Partially prohibited:
- Predictive policing is when an AI system performs risk assessments of individuals based on personality traits to predict the risk of committing a crime. What is not prohibited is the use of predictive policing systems based on “objective and verifiable facts directly linked to criminal conduct.”
- It is a classification system that seeks to infer people's race, political opinions, union membership, religious or philosophical beliefs, sexual orientation based on sex life or biometric data. This ban does not apply to law enforcement agencies.
- Educational context and workplace emotion recognition systems seek to infer distinctive traits based on emotional states (work fatigue, negative attitudes, stress, anger, or disrespectful) and biometric data (except for medical and safety reasons) such as facial expressions.
Retrospective RBI is not prohibited by this law at all. The alleged crime committed is sufficient to justify the development.
loophole
Security hardliners in the EU have successfully exempt AI law requirements when it comes to “national security.” This means that, regardless of whether this is done by a public or private company, if regulatory safeguards are used or used only when they are developed or used in AI systems. National security could be a valid reason for exceptions from AI law, but this should be assessed case by case in line with the EU's Charter of Fundamental Rights and therefore not misunderstood as a comprehensive exemption.
The AI Act prohibition applies only to systems located in or used in the EU market. This leaves a dangerous loophole. Prohibited systems that are deemed “incompatible” by EU regulators can export EU values to third countries.
Read more about Artificial Intelligence Law Policy & Advocacy Work
