Stereotype Detection in Indian Languages

Gokul S Krishnan , Sameer Deshpande , Chaithanya V , Balaraman Ravindran , Anindita Sahoo

Description:

our study explores how fairness research in NLP—traditionally focused on Western contexts—can be adapted to evaluate fairness in Indian settings. Our work focuses on stereotypes across India-specific axes of disparity such as religion and caste, as well as Indian notions of global disparities like gender, age, physical appearance, and disability. We propose IndicFRAME (Indic Fairness Review and Assessment for Multilingual Evaluation), a framework to make stereotype benchmarking datasets available in Indian languages and to examine the fairness of Indian LLMs across prominent and low-resource Indian languages. Using a fill-in-the-blank multiple-choice approach (IndiFIB-MCQ) and a novel Bias Retention Score (BRS) metric, we extensively analyze Indian stereotypes and quantify stereotypical or anti-stereotypical tendencies in these models, ultimately providing benchmarks for safe deployability.