Fair Recommender Systems

In this project, we are investigating several questions of fairness and bias in recommender systems:

What does it mean for a recommender to be fair, unfair, or biased?
What potentially discriminatory biases are present in the recommender’s input data, algorithmic structure, or output?
How do these biases change over time through the recommender-user feedback loop?

This is a part of our overall, ongoing goal to help make recommenders (and other AI systems) better for the people they affect. The key starting point to read about this research is our monograph.

Blog Posts and Other Coverage

Author Gender in Book Recommendations (30 Jul 2018)
Presenting at FAT* (1 Feb 2018)

Funding

2018–2023: NSF award 1751278, $482,081: CAREER: User-Based Simulation Methods for Quantifying Sources of Error and Bias in Recommender Systems

Workshops and Meetings

I have been involved with several workshops, sessions, etc. related to fair recommendation.

Fair Recommendation & Retrieval tutorial
Fairness track at TREC 2019
FATREC2 at RecSys 2018
Recommenders Session at FAT* 2018
FATREC at RecSys 2017, the Workshop on Responsible Recommendation (proceedings)

Publications

Jacy Reese Anthis, Kristian Lum, Michael Ekstrand, Avi Feller, and Chenhao Tan. 2025. Dubious Debiasing: The Intractability of Fair General-Purpose LLMs. To appear in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), Jul 27–01, 2025. arXiv:2406.03198. Cited 20 times. Cited 14 times.

Andrés Ferraro, Michael D. Ekstrand, and Christine Bauer. 2024. It’s Not You, It’s Me: The Impact of Choice Models and Ranking Strategies on Gender Imbalance in Music Recommendation. Short paper in Proceedings of the 18th ACM Conference on Recommender Systems (RecSys ’24), Oct 14, 2024. ACM, pp. 884–889. DOI 10.1145/3640457.3688163. arXiv:2409.03781 [cs.IR]. NSF PAR 10568004. Acceptance rate: 22%. Cited 3 times. Cited 2 times.

Michael D. Ekstrand, Lex Beattie, Maria Soledad Pera, and Henriette Cramer. 2024. Not Just Algorithms: Strategically Addressing Consumer Impacts in Information Retrieval. In Proceedings of the 46th European Conference on Information Retrieval (ECIR ’24, IR for Good track), Mar 24–28, 2024. Lecture Notes in Computer Science 14611:314–335. DOI 10.1007/978-3-031-56066-8_25. NSF PAR 10497110. Acceptance rate: 35.9%. Cited 12 times. Cited 5 times.

Amifa Raj and Michael D. Ekstrand. 2024. Towards Optimizing Ranking in Grid-Layout for Provider-side Fairness. In Proceedings of the 46th European Conference on Information Retrieval (ECIR ’24, IR for Good track), Mar 24–28, 2024. Lecture Notes in Computer Science 14612:90–105. DOI 10.1007/978-3-031-56069-9_7. NSF PAR 10497109. Acceptance rate: 35.9%. Cited 3 times. Cited 2 times.

Jonathan Stray, Alon Halevy, Parisa Assar, Dylan Hadfield-Menell, Chloe Bakalar, Craig Boutilier, Amar Ashar, Lex Beattie, Michael Ekstrand, Claire Leibowicz, Connie Moon Sehat, Sara Johansen, Lianne Kerlin, David Vickrey, Spandana Singh, Sanne Vrijenhoek, Amy Zhang, McKane Andrus, Natali Helberger, Polina Proutskova, Tanushree Mitra, and Nina Vasan. 2024. Building Human Values into Recommender Systems: An Interdisciplinary Synthesis. Transactions on Recommender Systems 2(3) (June 2024; online Nov 12, 2023), 20:1–57. DOI 10.1145/3632297. arXiv:2207.10192 [cs.IR]. Cited 101 times. Cited 69 times.

Michael D. Ekstrand, Ben Carterette, and Fernando Diaz. 2024. Distributionally-Informed Recommender System Evaluation. Transactions on Recommender Systems 2(1) (March 2024; online Aug 4, 2023), 6:1–27. DOI 10.1145/3613455. arXiv:2309.05892 [cs.IR]. NSF PAR 10461937. Cited 18 times. Cited 11 times.

Amifa Raj and Michael D. Ekstrand. 2023. Towards Measuring Fairness in Grid Layout in Recommender Systems. Presented at the 6th FAccTrec Workshop on Responsible Recommendation at RecSys 2023 (peer-reviewed but not archived). arXiv:2309.10271 [cs.IR]. Cited 1 time.

Amifa Raj, Bhaskar Mitra, Michael D. Ekstrand, and Nick Craswell. 2023. Patterns of Gender-Specializing Query Reformulation. Short paper in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23), Jul 23, 2023. pp. 2241–2245. DOI 10.1145/3539618.3592034. arXiv:2304.13129. NSF PAR 10423689. Acceptance rate: 25.1%. Cited 4 times. Cited 1 time.

Christine Pinney, Amifa Raj, Alex Hanna, and Michael D. Ekstrand. 2023. Much Ado About Gender: Current Practices and Future Recommendations for Appropriate Gender-Aware Information Access. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval (CHIIR ’23), Mar 19, 2023. pp. 269–279. DOI 10.1145/3576840.3578316. arXiv:2301.04780. NSF PAR 10423693. Acceptance rate: 39.4%. Cited 25 times. Cited 13 times.

Michael D. Ekstrand and Maria Soledad Pera. 2022. Matching Consumer Fairness Objectives & Strategies for RecSys. Presented at the 5th FAccTrec Workshop on Responsible Recommendation at RecSys 2022 (peer-reviewed but not archived). arXiv:2209.02662 [cs.IR]. Cited 5 times. Cited 3 times.

Amifa Raj and Michael D. Ekstrand. 2022. Fire Dragon and Unicorn Princess: Gender Stereotypes and Children’s Products in Search Engine Responses. In SIGIR eCom ’22, Jul 15, 2022. 9 pp. DOI 10.48550/arXiv.2206.13747. arXiv:2206.13747 [cs.IR]. Cited 14 times. Cited 6 times.

Amifa Raj and Michael D. Ekstrand. 2022. Measuring Fairness in Ranked Results: An Analytical and Empirical Comparison. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22), Jul 11, 2022. pp. 726–736. DOI 10.1145/3477495.3532018. NSF PAR 10329880. Acceptance rate: 20%. Cited 79 times. Cited 54 times.

Amifa Raj, Ashlee Milton, and Michael D. Ekstrand. 2021. Pink for Princesses, Blue for Superheroes: The Need to Examine Gender Stereotypes in Kids’ Products in Search and Recommendations. In Proceedings of the 5th International and Interdisciplinary Workshop on Children & Recommender Systems (KidRec ’21), at IDC 2021, Jun 27, 2021. arXiv:2105.09296. NSF PAR 10335669. Cited 10 times. Cited 6 times.

Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2022. Fairness in Information Access Systems. Foundations and Trends® in Information Retrieval 16(1–2) (July 2022), 1–177. DOI 10.1561/1500000079. arXiv:2105.05779 [cs.IR]. NSF PAR 10347630. Impact factor: 8. Cited 219 times. Cited 103 times.

Nasim Sonboli, Robin Burke, Michael Ekstrand, and Rishabh Mehrotra. 2022. The Multisided Complexity of Fairness in Recommender Systems. AI Magazine 43(2) (June 2022), 164–176. DOI 10.1002/aaai.12054. NSF PAR 10334796. Cited 40 times. Cited 22 times.

Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. 2022. Fairness in Recommender Systems. In Recommender Systems Handbook (3rd edition). Francesco Ricci, Lior Roach, and Bracha Shapira, eds. Springer-Verlagpp. 679–707. DOI 10.1007/978-1-0716-2197-4_18. ISBN 978-1-0716-2196-7. Cited 49 times. Cited 22 times.

Michael D. Ekstrand and Daniel Kluver. 2021. Exploring Author Gender in Book Rating and Recommendation. User Modeling and User-Adapted Interaction 31(3) (February 2021), 377–420. DOI 10.1007/s11257-020-09284-2. arXiv:1808.07586v2. NSF PAR 10218853. Impact factor: 4.412. Cited 213 times (shared with RecSys18. Cited 115 times (shared with RecSys18.

Ömer Kırnap, Fernando Diaz, Asia J. Biega, Michael D. Ekstrand, Ben Carterette, and Emine Yılmaz. 2021. Estimation of Fair Ranking Metrics with Incomplete Judgments. In Proceedings of The Web Conference 2021 (TheWebConf 2021), Apr 19, 2021. ACM, pp. 1065–1075. DOI 10.1145/3442381.3450080. arXiv:2108.05152. NSF PAR 10237411. Acceptance rate: 21%. Cited 55 times. Cited 38 times.

Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, and Ben Carterette. 2020. Evaluating Stochastic Rankings with Expected Exposure. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM ’20), Oct 21, 2020. ACM, pp. 275–284. DOI 10.1145/3340531.3411962. arXiv:2004.13157 [cs.IR]. NSF PAR 10199451. Acceptance rate: 20%. Nominated for Best Long Paper. Cited 208 times. Cited 179 times.

Amifa Raj, Connor Wood, Ananda Montoly, and Michael D. Ekstrand. 2020. Comparing Fair Ranking Metrics. Presented at the 3rd FAccTrec Workshop on Responsible Recommendation at RecSys 2020 (peer-reviewed but not archived). arXiv:2009.01311 [cs.IR]. Cited 40 times. Cited 30 times.

Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, and Sebastian Kohlmeier. 2020. Overview of the TREC 2019 Fair Ranking Track. Meeting summary in The Twenty-Eighth Text REtrieval Conference (TREC 2019) Proceedings (TREC 2019), Mar 25, 2020. arXiv:2003.11650. Cited 51 times. Cited 15 times.

Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand, Adam Roegiest, Aldo Lipani, Alex Beutel, Ana Lucic, Ana-Andreea Stoica, Anubrata Das, Asia Biega, Bart Voorn, Claudia Hauff, Damiano Spina, David Lewis, Douglas W Oard, Emine Yilmaz, Faegheh Hasibi, Gabriella Kazai, Graham McDonald, Hinda Haned, Iadh Ounis, Ilse van der Linden, Joris Baan, Kamuela N Lau, Krisztian Balog, Mahmoud Sayed, Maria Panteli, Mark Sanderson, Matthew Lease, Preethi Lahoti, and Toshihiro Kamishima. 2019. FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval. SIGIR Forum 53(2) (December 2019), 20–43. DOI 10.1145/3458553.3458556. Cited 53 times. Cited 19 times.

Michael D. Ekstrand, Mucun Tian, Mohammed R. Imran Kazi, Hoda Mehrpouyan, and Daniel Kluver. 2018. Exploring Author Gender in Book Rating and Recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18), Oct 3, 2018. ACM, pp. 242–250. DOI 10.1145/3240323.3240373. arXiv:1808.07586v1 [cs.IR]. Acceptance rate: 17.5%. Citations reported under UMUAI21. Citations reported under UMUAI21.

Michael D. Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D. Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All The Cool Kids, How Do They Fit In?: Popularity and Demographic Biases in Recommender Evaluation and Effectiveness. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (FAT* 2018), Feb 23, 2018. PMLR, Proceedings of Machine Learning Research 81:172–186. Acceptance rate: 24%. Cited 324 times. Cited 226 times.

Michael D. Ekstrand and Maria Soledad Pera. 2017. The Demographics of Cool: Popularity and Recommender Performance for Different Groups of Users. In RecSys 2017 Poster Proceedings, Aug 28, 2017. CEUR, Workshop Proceedings 1905. Cited 17 times. Cited 6 times.

Workshop Summaries

Michael D. Ekstrand, Jean Garcia-Gathright, Nasim Sonboli, Amifa Raj, and Karlijn Dinnissen. 2023. FAccTRec 2023: The 6th Workshop on Responsible Recommendation. Meeting summary in Proceedings of the 17th ACM Conference on Recommender Systems (RecSys ’23), Sep 24, 2023. ACM. DOI 10.1145/3604915.3608761. Cited 1 time. Cited 1 time.

Michael D. Ekstrand, Pierre-Nicolas Schwab, Toshihiro Kamishima, and Nasim Sonboli. 2021. FAccTRec 2021: The 4th Workshop on Responsible Recommendation. Meeting summary in Proceedings of the 15th ACM Conference on Recommender Systems (RecSys ’21), Sep 24, 2021. ACM. DOI 10.1145/3460231.3470932. Cited 2 times. Cited 2 times.

Michael D. Ekstrand, Pierre-Nicolas Schwab, Jean Garcia-Gathright, Toshihiro Kamishima, and Nasim Sonboli. 2020. 3rd FATREC Workshop: Responsible Recommendation. Meeting summary in Proceedings of the 14th ACM Conference on Recommender Systems (RecSys ’20), Sep 15, 2020. ACM. DOI 10.1145/3383313.3411538. Cited 6 times. Cited 6 times.

Bamshad Mobasher, Stylani Kleanthous, Michael D. Ekstrand, Bettina Berendt, Janna Otterbacher, and Avital Schulner Tal. 2020. FairUMAP 2020: The 3rd Workshop on Fairness in User Modeling, Adaptation and Personalization. Meeting summary in Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization (UMAP ’20), Jul 1, 2020. ACM. DOI 10.1145/3340631.3398671. Cited 5 times. Cited 2 times.

Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, and Michael D. Ekstrand. 2019. Workshop on Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval (FACTS-IR). Meeting summary in Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’19), Jul 21, 2019. ACM. DOI 10.1145/3331184.3331644. Cited 6 times.

Bettina Berendt, Veronika Bogina, Robin Burke, Michael D. Ekstrand, Alan Hartman, Stylani Kleanthous, Tsvi Kuflik, Bamshad Mobasher, and Janna Otterbacher. 2019. FairUMAP 2019 Chairs’ Welcome Overview. Meeting summary in Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization (UMAP ’19), Jun 9, 2019. ACM. DOI 10.1145/3314183.3323842.

Toshihiro Kamishima, Pierre-Nicolas Schwab, and Michael D. Ekstrand. 2018. 2nd FATREC Workshop: Responsible Recommendation. Meeting summary in Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18), Oct 7, 2018. ACM. DOI 10.1145/3240323.3240335. Cited 13 times. Cited 11 times.

Bamshad Mobasher, Robin Burke, Michael D. Ekstrand, and Bettina Berendt. 2018. UMAP 2018 Fairness in User Modeling, Adaptation and Personalization (FairUMAP 2018) Chairs’ Welcome & Organization. Meeting summary in Adjunct Publication of the 26th Conference on User Modeling, Adaptation, and Personalization (UMAP ’18), Jul 8, 2018. ACM. DOI 10.1145/3213586.3226200.

Michael D. Ekstrand and Amit Sharma. 2017. The FATREC Workshop on Responsible Recommendation. Meeting summary in Proceedings of the 11th ACM Conference on Recommender Systems (RecSys ’17), Aug 28, 2017. ACM. DOI 10.1145/3109859.3109960. Cited 6 times. Cited 14 times.