AI must be designed to enhance and improve our daily life. Nevertheless, like every novel technology, its potential for good can be impacted by different factors, such as a faulty design.
If the AI is designed to influence human behaviour this could have consequences on our autonomy or otherwise, self-determination. Ultimately, we don’t want to be the product of manipulative or distorting technology which we might not understand or know how to handle.
The autonomy is a broad concept. However, we want to focus on how AI can constraint humans through manipulation, confusion, over-reliance and interference with decisions making.
Some examples
An example is an online service, powered by what we call weak AI, that manipulates users to click unwanted pages. In this case, both the product management and designers must be held accountable for dark patterns or unethical choices.
Another example is AI that creates compulsion, leading to overuse and reducing autonomy (i.e., video games, social media). The ethical design is compromised when it is coupled with content that fosters misinformation, emotional manipulation, and exploitation.
How overreliance on AI can pose a risk to human life?
In a 2016 article the Guardian was informing the public about a Tesla Model S autopilot failure to distinguish a large white 18-wheel truck and trailer crossing the highway from a bright spring sky. Unfortunately, the car drove full speed under the trailer, killing the driver.
A second example is the 2018 fatal self-driving Uber crash. The self-driving car detected the pedestrian as early as 6 seconds before the crash. However, it did not slow or stop because its emergency braking systems were deliberately disabled. As Devin Coldewey observed for TechCrunch: “This isn’t just a safety issue, it’s a failure of judgment by Uber, and one that cost a person’s life.”
The AI Act perspective
The AI Act prohibits altogether the deployment of manipulative and exploitative technology. Furthermore, it regulates the deployment of AI that could impair human autonomy.
In one of its Recitals, the AI Act references the 2019 Ethics guidelines for trustworthy AI developed by the independent AI HLEG appointed by the Commission.
In the guidelines, the AI HLEG developed seven non-binding ethical principles for AI which are intended to help ensure that AI is trustworthy and ethically sound. One of the seven principles include human agency and oversight.
The way forward
We advise organisations to use professional support to assess their AI systems in the development stage and, in any case, before any use of AI. Assessments must gauge the interaction between AI and humans, the algorithmic decision-making design, overreliance on AI, inadvertent AI behaviour, and other situations that could create risks to human agency and oversight.
At Symmetry Compliance we assist our clients to comply with the AI Act. We have solid experience in operationalising organisation’s compliance with data protection and governance legislation. For AI compliance, we set up a dedicated team ready to assist with a diversity of queries and assignments.
Contact us for an initial discussion.
Image: generated by Microsoft Bing
