A Comparison of Statistical and Counterfactual Fairness Metrics for Explainable AI

Problem Statement

This thesis comparatively analyzes two families of fairness metrics, group-based and counterfactual-based, to investigate their properties and potential contradictions in high-stakes domains. Using Logistic Regression and Random Forest models trained on the COMPAS. Traditional group fairness measures such as Statistical Parity and Equal Opportunity are compared against the individual, counterfactual-based metrics AMPD, MBEandmodified variants of CFlips (GlobalCflips) and nDCCF (GlobalnDCCF) derived from the field of XAI. For generating the counterfactuals, a new multi-instance adaption of the NICE algorithm with ranking (rankedMINICE) is proposed. The analysis reveals a critical divergence: while group fairness metrics showed significant unfairness in both models, counterfactual metrics uncovered an even more pronounced bias across the dataset. The primary conclusion is that the choice of fairness metric fundamentally determines the perception of a model’s fairness. It has been found that group statistics can indicate fairness. However, a comprehensive fairness audit requires a multi-metric approach, with the complementary perspectives of both group and individual measures being leveraged for a holistic and reliable assessment.

Tasks

• Perform a literature review. Identify key challenges and gaps in current approaches to metrics for fairness in machine learning models
• Implement a pipeline for these fairness metrics
• Do statistical evaluations
• Write a thesis and present your findings in the Disputationsseminar
• (Optional:) co-write a research paper