The AI Revolution in Consulting Firms: How AI Act Is Changing the Game

A recent research article published in the United States drew attention to the presence of racial bias in hospital algorithms. The study revealed that, in cases where Artificial Intelligence assessed individuals’ risk levels, African American individuals were found to be in worse health compared to their white counterparts, despite having the same risk scores. Because the algorithm played a role in selecting patients for treatment, this led to a situation where African American patients received less care than others, despite having the same degree of illness.

In order to minimize the risks of such negative consequences in Europe, the European Union aims to introduce legislation that establishes a framework for AI, while aiming to minimize disruption to technological innovation and progress as much as possible. Although the Artificial Intelligence Act or “AI Act” is still under negotiations, consulting companies aiming to be leader in its integration consulting compagnies should begin considering it now and prepare for its future requirements.

What is AI ACT?

Aimed at framing the use of AI, the objective is to ensure that AI systems introduced in the European market be as transparent and ethical as possible to guarantee the fundamental rights of individuals while ensuring the development of companies.

This Regulation will apply to all AI systems used in the European market, regardless of the provider of the system. Many questions rose about competitiveness for the EU in this specific field, but it is necessary to have a clear line of conduct, as we did for the GDPR but applied here to Artificial Intelligence. Still in negotiation, the AI Act will probably come into force in 2025 or 2026.

The AI Act takes a risk-based approach, meaning that the requirements for an AI system are tailored to the level of risk that it poses. High-risk systems will be subject to the most stringent requirements such as mandatory risk assessment, conformity assessment procedure and registration in a European Union database, that need to be done before introduction to the market. The providers of such systems must put in place appropriate measures to detect, prevent and mitigate possible biases. The characteristics of high-risk AI systems will be established in an Annex to the Regulation, and the European Commission will have the possibility to update it as the technology evolves.

Here are the different risk categories presented:

• Prohibited AI systems applications: systems used for social scoring or manipulation of individuals (dark pattern AI), behavior exploitation.
• High-risk systems: using biometric data, systems used for assessing candidates or employees, etc.
• Systems that pose specific risks when interacting with humans, such as chatbots or deepfakes including emotion detection. There will be a transparency requirement, the users must be made aware they are interacting with an AI system and that the output is generated automatically.

For AI Systems that do not fall under any of the above categories (identified as low risk systems), the AI Act encourages providers to establish their own codes of conduct around the ethical development and use of AI.

The AI Act provides guidelines to prevent bias in data for high-risk algorithms. Article 10 requires that data used in training and testing be relevant, representative, complete, and error-free. The article also outlines detailed steps for data processing, including cleaning, filling, and annotation, with checks at each stage. The AI Act addresses potential sources of bias in data collection, in addition to addressing the algorithms themselves.

Another important aspect of the AI Act relies on responsibility management. In fact, responsibility is assigned to the provider for high-risk AI systems. However, if a substantial modification is made to a high-risk system, the deployer who made the modification will be held responsible for the system during its use. Additionally, if an AI system is not high-risk at first but becomes so through modifications by a deployer, then the deployer will be held responsible.

What to do now?

For now, the best way to be operational and not experience any slowdown when the AI Act comes into force is to prepare for compliance.

First step involves evaluating the risk of algorithms: When a high-risk algorithm is identified — based on the AI Act criteria — the following questions should be asked:

• Regarding the training data, what is the purpose of the collection and are the collected data necessary? Additionally, does the data collection reflect reality? Is it relevant, complete, and unbiased? Particular attention should be given to personal data. The GDPR requirements are applicable in this case.
• Next, when implementing the algorithm, is transparency and interpretability of the AI system being respected? Do we have control over the AI to modify an algorithm when needed?
• For monitoring and implementation, is there documentation in place that updates the various developments of the algorithm and its potential security risk evolution (from one level of risk to another) ?
• Finally, is there ongoing risk assessment throughout the process?

Once we have answered all these questions, it will be easy for the company to deal with the standards of this new regulation.

Impact on companies

Data consulting companies play a leading role in the AI field. They will see the services they offer directly impacted by this law. It will be the company’s duty to provide clear explications regarding the choice of algorithms produced. In this way, the explicability of models becomes the heart of the debate.

Of course, several areas of data science are involved, but the most telling example is Natural Language Processing (NLP). This sector concerns the automatic processing of text or speech, enabling computers to understand, analyze, generate and interact with human language in a more natural way. Words and sentences are intelligently transformed into a digital space, enabling them to be processed by algorithms which then use operations to extract the mathematical meaning of these vectors. These techniques can produce biased results (e.g., racism, sexism, etc.). As these algorithms are trained on millions of texts, it can be difficult to clearly establish the origins of the bias.

The paper by Caliskan et al. deals with bias in the encoding of words prior to processing by algorithms (i.e. word embedding). During this encoding, they show that the names of European-Americans such as “Brad” or “Geoffrey” are related to pleasant concepts like “joy”, “love” or “peace”, whereas the names of African Americans such as “Darnell” or “Jermaine” are associated with unpleasant concepts like “terrible”, “agony”.

How to detect bias

To avoid this kind of drift, it is important to identify the source of bias and quantify it. There are several possible explanations, and the aim of AI players, in integrating the AI Act, is to be able to explain where these biases come from and correct them.

One of the first causes may be societal bias, which is reflected in the algorithm’s training data.A concrete example of societal bias in NLP is text generation models that are trained on corpora containing data from the Internet. If the training data contain prejudices, hate speech or stereotyped representations, the model can learn to reproduce these biases when generating text. For example, a model might be more likely to generate descriptions of stereotypical professions or roles, such as assigning housework to women, due to biases present in the training data.

Another source of bias may be insufficient selection of minority training data. For example, if an emotion detection system is trained primarily on data from speakers of a single language or culture, it may not be sufficiently sensitive to the emotional nuances and expressions used by other linguistic or cultural groups, leading to inferior performance for these minorities. Other biases may also be present, such as algorithmic bias, which is a consequence of the poor architecture of the NLP models used.

Once biases have been identified, they need to be corrected. Indeed, as we all know, these biases can be the result of decades of social prejudices present on the Internet through the articles used to train the models, but the good news is that it is possible to limit and overcome them.

Several techniques can be used to achieve satisfactory results, both in terms of quality and, above all, ethics. One simple solution is to invert or delete the words connoting the bias you wish to reduce, so that the algorithm’s learning process is as neutral as possible. For example, if in all the sentences containing the first name “Greg” we create a new identical one by changing the first name, the algorithm will treat both first names in the same way and remove the bias. Many other techniques are possible, and we will go delve deeper into them in subsequent articles.

Conclusion

In conclusion, this article highlights the need for legislation to frame the use of AI and prevent discriminatory biases, such as racial bias in algorithms used by hospitals. The European Union is seeking to introduce legislation that guarantees the transparency and ethics of AI while maintaining business competitiveness.

For Sia Partners, this legislation implies developing certified tools and technologies, while ensuring the explicability of models, particularly in the field of NLP. The identification, quantification and correction of biases are essential to guarantee fair and ethical results. A forthcoming article will detail the various techniques for mitigating bias, such as interchanging or deleting connoted words, or the impact of using word vectors (embedding) in a different way, aiming to improve the neutrality of algorithms.