Mariana Lima Rodrigues Carneiro (Masters in European Union Law from the School of Law of University of Minho)

The deployment of xAI’s Grok chatbot has become a focal point of systemic risk within the European digital landscape. The European Commission first opened formal proceedings against X in December 2023.[1] In January of 2026, the scope of this regulatory oversight was significantly expanded under the Digital Services Act (DSA) to investigate Grok’s functionalities.[2] This investigation specifically targets risks such as the dissemination of non-consensual sexual deepfakes and antisemitic discourse. These controversies reveal a programmed tendency towards neutral language that masks structural biases within AI systems.[3]

This article explores how this systemic opacity creates an insurmountable barrier for individuals seeking legal redress against algorithmic discrimination. The core objective is to analyse the failure of the current reversal of the burden of proof mechanism, as provided by European anti-discrimination directives, when faced with high-dimensional mathematical optimisation. Ultimately, this text examines the necessity of technical solutions to harmonise automated processing with the values of justice and equality that underpin the European legal order.

1. The “black box effect”

Artificial Intelligence (AI) is divided into several subareas, among which Machine Learning (ML) represents a major one. ML focuses on developing algorithmic models capable of making predictions or decisions based on data sets. Deep Learning (DL) constitutes a specialized subset of ML based on a conceptual model of the human brain, utilizing artificial neural networks structured into multiple hidden layers, commonly referred to as Deep Neural Networks (DNNs).[4] Consequently, all DL systems are ML systems, but not all ML systems are DL systems.

The predictive process begins with an input layer, which corresponds to the raw information entering the system. This is followed by numerous hidden layers where each “neuron” encodes a mathematical function that transforms the input into an output, creating a final prediction. The term “deep” refers to the high number of these hidden layers. Crucially, the more layers a system has, the more complex and obscure the decision-making process becomes, making it difficult for humans to understand how the final result was reached.[5]

This interaction results in the “black box effect”, characterised by the lack of transparency and interpretability regarding the system’s internal processes. While present in ML, this effect is exacerbated in DL systems due to the high complexity of the DNNs. As these algorithms are increasingly used in high-risk areas, such as conducting recruitment or assessing credit eligibility, the lack of transparency may violate fundamental legal principles, including due process and equality before the law.[6]

Ultimately, there is a fundamental trade-off between precision and explainability. High-dimensional mathematical optimisation often transcends human cognitive capacity, creating an incompatibility between the system’s efficiency and human semantic interpretation.[7] As systems evolve and acquire new data, their layers and connections deepen, further intensifying their structural opacity.[8] Therefore, as accuracy increases, transparency typically decreases, leaving individuals in a state of legal defencelessness when faced with automated decisions.[9]

2. Case studies: AI biases in practice

Algorithmic systems are often implemented under the promise of efficiency, precision, and neutrality. However, despite this aura of impartiality, these systems can internalise and reproduce the prejudices and inequalities present in society. Training data is not neutral; it inevitably reflects structural asymmetries and biases. Without corrective mechanisms, algorithms tend to amplify these patterns, perpetuating discrimination against historically vulnerable groups.[10]

Biases may emerge at any stage of development, including data labelling[11] or attribute selection.[12] Even when datasets seem impartial, they can mask a history of discrimination. Ultimately, automation cloaks these decisions in an appearance of technical neutrality, making their scrutiny and legal challenge far more difficult. This opacity hinders the establishment of a clear causal link between the software functioning and unfavourable treatment, effectively rendering the right to non-discrimination unenforceable in practice.[13]

To comprehend the specific ways in which these distortions manifest, it is essential to apply the taxonomy established by Batya Friedman and Helen Nissenbaum, who categorised algorithmic bias into three distinct types: pre-existing, technical, and emergent.[14]

Pre-existing bias has its roots in social institutions, practices, and attitudes that exist prior to the creation of the system, effectively acting as a digital mirror of historical prejudices. A clear example of this is the COMPAS system used in US courts, which demonstrated a tendency to incorrectly flag Black defendants as high-risk compared to White defendants, since the training data reflected racial discriminatory patterns in the criminal justice system.[15] Technical bias arises from flaws in the design stage or the lack of representativeness in training databases. This was evident in Amazon’s discontinued recruitment tool, which penalised resumes containing the word “woman” because it was trained on historical data dominated by male applicants.[16] Finally, emergent bias occurs during real-world use as a result of interaction between the system and its users. The Tay chatbot remains a paradigm of this category, as it rapidly assimilated and reproduced offensive messages from users who manipulated its learning mechanism in real time.[17]

Building on these manifestations, the impact of algorithmic bias is further evidenced in specific industry sectors. The 2019 Apple Card case highlighted the dangers of systemic opacity in credit access. Users reported that the black box algorithm granted men significantly higher credit limits than their wives, even when they shared bank accounts. Representatives from Apple and Goldman Sachs were unable to explain the decision-making process because the model’s complexity hindered its interpretability.[18] While a subsequent investigation by the NYSDFS concluded that gender was not a direct variable, the case still raised ethical concerns.[19]

Similarly, the Optum Algorithm used in US hospitals revealed how a monetary bias can convert into a racial one. The system used healthcare costs as a proxy for health needs. Since White patients historically had higher medical expenses due to structural socioeconomic privileges, the algorithm incorrectly identified them as being “sicker” than Black patients with similar clinical conditions, who had less access to services. Consequently, only 18% of Black patients were selected for intensive care programs when the actual need was 47%.[20]

Another significant area of concern involves the use of facial recognition systems, since they present some of the most critical risks to fundamental rights. These algorithms suffer from a lack of diversity in their databases, leading to significantly higher error rates for people with darker skin tones, particularly women.[21]

A test conducted by the ACLU using Amazon’s Rekognition algorithm found that the software incorrectly matched 28 members of the US Congress with a mugshot database. Nearly 40% of these false matches involved people of colour, despite them making up only 20% of Congress.[22] Similar technical failures have already produced real-life consequences. In a well-known case, Robert Williams, a Black man, was wrongfully arrested in Detroit and held for 30 hours after a facial recognition system incorrectly identified his driver’s license photo as that of the shoplifter. These cases underscore the dangers of blind trust in automated identification systems, especially in the context of criminal justice.[23]

3. Algorithmic discrimination and the burden of proof

The principle of non-discrimination is a pillar of the European Union legal order, operationalised through a comprehensive framework that includes Directive 2000/43/EC, Directive 2000/78/EC, Directive 2004/113/EC, and Directive 2006/54/EC. These instruments aim to ensure equality of treatment by prohibiting both direct and indirect discrimination across various domains, from employment to the provision of goods and services.

Direct discrimination occurs when a person of a protected group is treated less favourably than another in a comparable situation based on a protected characteristic. In contrast, indirect discrimination arises when an apparently neutral provision or practice puts a person of a protected group at a particular disadvantage.[24] Within the digital ecosystem, algorithmic discrimination is predominantly indirect, as systems rarely rely on explicit protected traits but instead use neutral data points that function as proxies for those characteristics.

The structural opacity of the black box creates significant challenges for the traditional application of these legal concepts. When a decision is the result of high-dimensional correlations rather than human intentionality, identifying the discriminatory factor becomes an exercise in technical forensic analysis. This complexity leads to an acute information asymmetry between the technology provider and the individual, where the victim is often unable to demonstrate that they have been subject to a discriminatory outcome. This lack of transparency effectively hinders the ability to establish the necessary facts to initiate litigation.[25]

To ensure the right to effective judicial protection, EU law provides for the reversal of the burden of proof. Once a plaintiff establishes facts from which it may be presumed that there has been discrimination, it is for the respondent to prove that there has been no breach of the principal of non-discrimination.[26] However, in the context of AI use, this mechanism faces a probatio diabolica. The plaintiff struggles to provide even the initial prima facie evidence because they cannot access or interpret the algorithmic logic. This creates an impediment to access to justice where the burden of proof remains an insurmountable barrier.

Furthermore, the respondent also faces emerging challenges. Even when acting in good faith, a defendant may struggle to prove the absence of bias due to the systems complexity and its opaque nature. This creates a dual burden: while the plaintiff is trapped by the inability to see inside the black box, the defendant is often unable to provide a human-understandable explanation for the automated output.[27] Consequently, the effectiveness of the right to non-discrimination depends on evolving current evidentiary standards to account for the unique technical realities of AI.

4. The European legal architecture

The European Union has developed a multifaceted regulatory strategy to tackle some of the challenges of algorithmic bias. The General Data Protection Regulation (Regulation 2016/679) establishes the foundational layer, specifically through Article 22, which provides individuals with the right not to be subject to decisions based solely on automated processing that produce legal or significantly similar effects.[28] This provision ensures a baseline of human intervention and the right to contest automated outcomes. Although the right to an explanation is not explicitly named in Article 22, a systemic interpretation of Articles 13, 14, and 15 in conjunction with Article 22 effectively establishes such a right. This interpretation ensures that data subjects receive meaningful information about the logic involved in automated decisions, acting as a tool to bridge the knowledge gap and providing the individual with the transparency needed to scrutinize the decision.[29]

The AI Act (Regulation 2024/1689) represents the most significant advancement in this field. By adopting a risk-based classification, it mandates rigorous compliance for high-risk systems, particularly those used in employment and essential services. Article 11 requires technical documentation and Article 13 requires transparency, while Article 86 creates a specific right to an explanation for individuals.[30] This right is essential to mitigate the information asymmetry between the technology provider and the person affected, allowing for a better understanding of the logic behind the automated decision.

Regarding liability, the Product Liability Directive (Directive 2024/2853) has been modernised to include software, ensuring that discriminatory algorithmic outputs can be classified as defective products.[31] Alongside this, the Proposal for an AI Liability Directive sought to introduce rebuttable presumptions of causality to ease the burden of proof.[32] However, the 2025 Commission Work Programme confirmed the abandonment of this specific legislative path, marking a significant setback in the creation of a specialised liability regime for AI.[33] Despite this, the Representative Actions Directive (Directive 2020/1828) empowers consumer organisations to challenge systemic biases, moving the defence of non-discrimination from the individual to the collective sphere.[34]

5. Technical solutions for algorithmic fairness: The CDD metric

The inherent opacity of black box systems constitutes a significant technical and legal obstacle to the reversal of the burden of proof, a cornerstone of EU anti-discrimination law. In response to this probatory impasse, data science has developed various fairness metrics aimed at mitigating bias and ensuring equitable decisions. However, a conceptual gap remains between the contextual justice required by European jurisprudence and the rigid mathematical rules of technical metrics.[35]

Most traditional metrics, such as Predictive Parity or Equalized Odds,[36] often rely on a logic of formal equality. As Sandra Wachter, Brent Mittelstadt, and Chris Russell argue, these can be classified as bias preserving metrics because they often perpetuate the status quo by ignoring historical and structural disadvantages present in the training data. In contrast, bias transforming metrics align more closely with the EU concept of substantive equality by actively seeking to correct social inequalities.[37]

In this context, the Conditional Demographic Disparity (CDD) metric emerges as an insightful tool that warrants further exploration. This metric works by identifying residual statistical differences between social groups only after accounting for relevant, legitimate and determining variables.[38] By doing so, it evaluates whether individuals in similar conditions of merit still experience systematic disparities. This approach aligns with the gold standard established by the CJEU in the Seymour-Smith case, which demands a comprehensive comparison to evaluate discriminatory impacts.[39]

By producing objective summary statistics, CDD enables claimants to gather the necessary evidence to establish a prima facie case, effectively overcoming the black box effect.[40] Furthermore, the metric functions as a preventive instrument, allowing developers to detect and mitigate biases during the design phase.

Ultimately, the implementation of such metrics depends on a socio-technical approach that seeks to integrate technological innovation with legal, social, and ethical principles.[41] This synergy allows for a more robust protection of fundamental rights, ensuring that algorithmic processing is harmonised with the values of justice and equality that underpin the European legal order.

6. Final considerations

The increasing automation of high-impact decisions risks rendering the reversal of the burden of proof ineffective. The black box effect and information asymmetry create an almost insurmountable evidentiary barrier, making the task of establishing a prima facie case nearly impossible. While the AI Act provides a right to an explanation, this remains informative rather than evidentiary. The abandonment of the AI Liability Directive leaves a significant gap, as the Product Liability Directive and Representative Actions Directive were not specifically designed for the complexity of AI. To address these shortcomings, a socio-technical approach is required to operationalise evidentiary means.

Cases such as Albania’s digital minister, Diella, reveal the fallacy of technical neutrality.[42] We must reject the notion that algorithms are purely objective and impartial tools, recognising that they can reproduce or even amplify pre-existing human biases.[43] As Cathy O’Neil warns, the “privileged, we’ll see time and time again, are processed more by people, the masses by machines.”[44] Ultimately, the safeguard of fundamental rights depends on demanding from algorithms the same scrutiny and responsibility expected from human decisions.

---

[1] European Commission, “Commission opens formal proceedings against X under the Digital Services Act”, Brussels, 18 December 2023, https://ec.europa.eu/commission/presscorner/detail/en/ip_23_6709.

[2] European Commission, “Commission investigates Grok and X’s recommender systems under the Digital Services Act”, Brussels, 26 January 2026, https://ec.europa.eu/commission/presscorner/detail/en/ip_26_203.

[3] Janga Bussaja, “Analyzing Grok 4’s engagement with racism: a case study in AI fragility and deception”, 11 July 2025, http://dx.doi.org/10.2139/ssrn.5348379.

[4] Maciej Satkiewicz, “Towards white box deep learning”, 2024. Warsaw: [s.n.] (Report no. 2403.09863), https://arxiv.org/pdf/2403.09863.

[5] Fabio Gagliardi Cozman & Dora Kaufman, “Viés no aprendizado de máquina em sistemas de inteligência artificial: a diversidade de origens e os caminhos de mitigação”, Revista USP. São Paulo, no. 135 (2022): 195-210.

[6] Mônia Clarissa Hennig Leal & Lucas Moreschi Paulo, “Algorítmos discriminatórios e jurisdição constitucional: os riscos jurídicos e sociais do impacto dos vieses nas plataformas de inteligência artificial de amplo acesso”, Revista de Direitos e Garantias Fundamentais. ISSN 2175-6058, vol. 24, no. 3 (2023), doi: 10.18759/rdgf.v24i3.2311.

[7] Dora Kaufman, Tainá Junquilho, Priscila Reis, “Externalidades negativas da inteligência artificial: conflitos entre limites da técnica e direitos humanos”, Revista de Direitos e Garantias Fundamentais. ISSN 2175-6058, vol. 24, no. 3 (2023), doi: 10.18759/rdgf.v24i3.2198.

[8] Dora Kaufman, Tainá Junquilho, Priscila Reis, “Externalidades negativas da inteligência artificial…”, 52.

[9] Alessandra Silveira, “Automated individual decision-making and profiling [on case C-634/21 – SCHUFA (Scoring)]”, UNIO – EU Law Journal, vol. 8, no. 2 (2023): 74–85, https://doi.org/10.21814/unio.8.2.4. 

[10] Sandra Wachter, Brent Mittelstadt, Chris Russell, “Bias preservation in machine learning: the legality of fairness metrics under EU non-discrimination law”, West Virginia Law Review, vol. 123, no. 3 (2021), https://researchrepository.wvu.edu/wvlr/vol123/iss3/4/.

[11] Fabio Gagliardi Cozman & Dora Kaufman, “Viés no aprendizado…”.

[12] Jeffrey Dastin, “Insight – Amazon scraps secret AI recruiting tool that showed bias against women”, Reuters, 11 October 2018, https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G/.

[13] Sandra Wachter, Brent Mittelstadt, Chris Russell, “Why fairness cannot be automated: bridging the gap between EU non-discrimination law and AI”, Computer Law & Security Review. ISSN 0267-3649, vol. 41 (2021): 105567, http://dx.doi.org/10.2139/ssrn.3547922.

[14] Batya Friedman, Helen Nissenbaum, “Bias in computer systems”, ACM Transactions on Information Systems, vol. 14, no. 3 (1996): 330–347, http://doi.org/10.4324/9781315259697-23.

[15] Jeff Larson, Surya Mattu, Lauren Kirchner and Julia Angwin, “How we analyzed the COMPAS recidivism algorithm”, ProPublica, 23 May 2016, https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm.

[16] Jeffrey Dastin, “Insight – Amazon scraps…”.

[17] Dave Lee, “Tay: Microsoft issues apology over racist chatbot fiasco”, BBC News, 25 March 2016, https://www.bbc.com/news/technology-35902104.

[18] Chris Wiltz, “The Apple card is the most high-profile case of AI bias yet”, Design News, 13 November 2019, https://www.designnews.com/artificial-intelligence/the-apple-card-is-the-most-high-profile-case-of-ai-bias-yet. 

[19] Ian Carlos Campbell, “The Apple card doesn’t actually discriminate against women, investigators say”, The Verge, 24 March 2021, https://www.theverge.com/2021/3/23/22347127/goldman-sachs-apple-card-no-gender-discrimination.

[20] Ziad Obermeyer, et al., “Dissecting racial bias in an algorithm used to manage the health of populations”, Science. ISSN 0036-8075, vol. 366, no. 464 (2019), Doi.org/10.1126/science.aax2342.

[21] Patrick Grother, Mei Ngan, Kayee Hanaoka, “Face recognition vendor test part 3: demographic effects”. NISTIR 8280, December 2019, https://doi.org/10.6028/NIST.IR.8280.

[22] Jacob Snow, “Amazon’s face recognition falsely matched 28 members of Congress with mugshots”, ACLU News & Commentary, 26 July 2018, https://www.aclu.org/news/privacy-technology/amazons-face-recognition-falsely-matched-28.

[23] Kashmir Hill, “Facial recognition led to wrongful arrests. So Detroit is making changes”, The New York Times, 29 June 2024, https://www.nytimes.com/2024/06/29/technology/detroit-facial-recognition-false-arrests.html.

[24] European Union Agency for Fundamental Rights (FRA), “Handbook on European non-discrimination law”, 21 March 2018, https://fra.europa.eu/en/publication/2018/handbook-european-non-discrimination-law-2018-edition.

[25] Sandra Wachter, Brent Mittelstadt, Chris Russell, “Why fairness…”.

[26] Council Directive 2000/43/EC of 29 June 2000 implementing the principle of equal treatment between persons irrespective of racial or ethnic origin.

[27] Alessandra Silveira, “Automated individual…”.

[28] Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation).

[29] Tiago Sérgio Cabral, “AI and the right to explanation: three legal bases under the GDPR”, in Data protection and privacy: enforcing rights in a changing world, ed. Dara Hallinan, Ronald Leenes and Paul De Hert (Hart Publishing, 2021), 29-56, https://doi.org/10.5040/9781509941780.ch-002.

[30] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act).

[31] Directive (EU) 2024/2853 of the European Parliament and of the Council of 23 October 2024 on liability for defective products.

[32] Proposal for a Directive of the European Parliament and of the Council on adapting non-contractual civil liability rules to artificial intelligence (AI Liability Directive).

[33] Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, Commission work programme 2025 – Moving forward together: a bolder, simpler, faster Union, Strasbourg, 11.2.2025, COM(2025) 45 final, https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52025DC0045&qid=1761126737792.

[34] Directive (EU) 2020/1828 of the European Parliament and of the Council of 25 November 2020 on representative actions for the protection of the collective interests of consumers.

[35] Sandra Wachter, Brent Mittelstadt, Chris Russell, “Why fairness…”.

[36] Alessandro Castelnovo, et al., A clarification of the nuances in the fairness metrics landscape. Scientific Reports, vol. 12, no. 1 (2022), doi: 10.1038/s41598-022-07939-1.

[37] Sandra Wachter, Brent Mittelstadt, Chris Russell, “Bias preservation…”.

[38] Sandra Wachter, Brent Mittelstadt, Chris Russell, “Why fairness…”.

[39] Judgment of the Court Seymour-Smith and Perez, 9 February 1999, case C-167/97, ECLI:EU:C:1999:60.

[40] Sandra Wachter, Brent Mittelstadt, Chris Russell, “Why fairness…”.

[41] Reva Schwartz, et al., “Towards a standard for identifying and managing bias in artificial intelligence”, Special Publication (NIST SP), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.SP.1270.

[42] Pascale Davies, “Albania appoints world’s first AI government ‘minister’ to root out corruption”, Euronews, 12 September 2025, https://www.euronews.com/next/2025/09/12/albania-appoints-worlds-first-ai-government-minister-to-root-out-corruption.

[43] Mônia Clarissa Hennig Leal & Lucas Moreschi Paulo, “Algorítmos discriminatórios e jurisdição constitucional…”.

[44] Cathy O’Neil, Weapons of math destruction: how big data increases inequality and threatens democracy (New York: Broadway Books, 2017), 6.

---

Picture credit: by UMA media on pexels.com.