Research Article
SafeText: A Unified Approach for Detecting and Mitigating Toxicity and Bias in Textual Data
@INPROCEEDINGS{10.1007/978-3-032-05832-4_5, author={Aanchal Agarwal and Anup Bera and Tanusree De}, title={SafeText: A Unified Approach for Detecting and Mitigating Toxicity and Bias in Textual Data}, proceedings={Safe, Secure, Ethical, Responsible Technologies and Emerging Applications. Second EAI International Conference, SAFER-TEA 2024, Dubai, United Arab Emirates, December 9--10, 2024, Proceedings}, proceedings_a={SAFER-TEA}, year={2026}, month={11}, keywords={Content Moderation Societal Toxicity and Bias Detection Toxicity and Bias Mitigation Synthetic Toxic and Bias Data Generation Data Quality improvement Responsible AI}, doi={10.1007/978-3-032-05832-4_5} }- Aanchal Agarwal
Anup Bera
Tanusree De
Year: 2026
SafeText: A Unified Approach for Detecting and Mitigating Toxicity and Bias in Textual Data
SAFER-TEA
Springer
DOI: 10.1007/978-3-032-05832-4_5
Abstract
This paper introduces an innovative approach to AI-powered content moderation, in the context of any textual content; either human-written or AI/Large Language Model generated. The first and foremost in content moderation is detection of bias and toxic language in the content. Once the biased or toxic sentences are identified, we need to debias or detoxify the sentences by rewording or rephrasing, keeping the contextual meaning of the content intact. To achieve this, we fine-tuned an mT5-small model for detection and a T5-base model for mitigation. Our training process incorporated a broad range of datasets, including open-source data and synthetic data generated using the advanced GPT-4 model. A pivotal element in our methodology is the utilization of synthetic data generation, significantly augmenting the diversity and realism of the training dataset. Contrastive learning is employed to further enhance the model’s capability to discern subtle nuances in content. Noteworthy is our detection model, which not only identifies toxicity and bias but also offers reasoning, thereby introducing a layer of explainability to the content moderation process. Our results illustrate that our comprehensive approach, integrating sophisticated techniques and diverse data sources, surpasses existing solutions. This achievement provides scalable and effective tools for advancing digital safety and fostering inclusivity.