- Ph.D (2014)
- Computer Science, University of Minnesota. Thesis: Towards Recommender Engineering: Tools and Experiments for Identifying Recommender Differences.
Advisers: John T. Riedl and Joseph A. Konstan
- B.S. (2007)
- Computer Engineering (With Distinction), Iowa State University.
- Assistant Professor, Dept. of Computer Science, Boise State University
- Co-founder, People and Information Research Team (PIReT)
- Assistant Professor, Dept. of Computer Science, Texas State University
- Graduate Research Assistant, GroupLens Research, Dept. of Computer Science, University of Minnesota
- Su 2012, F 2013
- Instructor, University of Minnesota
- Summer 2010
- Research Intern, Autodesk Research, Toronto, CA
- 2007–2008, S 2011
- Teaching Assistant, University of Minnesota
- Undergraduate Research Assistant, Scalable Computing Laboratory, Ames Lab, Iowa State University
Boise State University
|Su18||CS 310-HU||Intro to Databases||6|
|F17||CS 533||Intro to Data Science||22|
|S17||CS 597||Recommender Systems||13|
Texas State University
|S16||CS 3320||Internet Software Development||48|
|S16||CS 5369Q/4379Q||Recommender Systems||26|
|F15||CS 4332||Intro to Database Systems||39|
|S15||CS 5369Q/4379Q||Recommender Systems||28|
|S15||CS 4350||Unix Systems Programming||32|
|F14||CS 4332||Intro to Database Systems||50|
I co-created the Recommender Systems specialization on Coursera, along with its two previous single-class versions, with Joseph A. Konstan.
University of Minnesota
|F13||CSCI 5980-1||Intro to Recommender Systems||Inst|
|S12||CSCI 1902||Structure of Computer Programming II||Inst|
|S11||CSCI 5125||Collaborative and Social Computing (TA)||TA|
|Su08||CSCI 1902||Structure of Computer Programming II||TA|
|S08||CSCI 1902||Structure of Computer Programming II||TA|
|F07||CSCI 1902||Structure of Computer Programming II||TA|
- Amifa Raj (started Fall 2018)
- Mucun Tian (M.S. expected 2019)
- Vaibhav Mahant (M.S. 2016, Texas State University; thesis: Improving Top-N Evaluation of Recommender Systems)
- Sushma Channamsetty (M.S. 2016, Texas State University; thesis: Recommender Response to User Profile Diversity and Popularity Bias)
- Mohammed Imran R Kazi (M.S. 2016, Texas State University; thesis: Exploring Potentially Discriminatory Biases in Book Recommendation)
- Shuvabrata Saha (M.S. 2016, Texas State University; co-advised with Dr. Apan Qasem; thesis: A Multi-objective Autotuning Framework For The Java Virtual Machine)
Teaching Professional Development
- Boise State University Center for Teaching and Learning Course Design Institute, a one-week intensive session in Summer 2017.
- CTL workshops on service learning, mastery-based grading, and other topics.
- Texas State University’s Program for Excellence in Teaching and Learning (2014–2015).
- Preparing Future Faculty at the University of Minnesota.
Author formatting key:, , . Citation counts from Microsoft Academic via Microsoft Cognitive Services.
2018. Rating-Based Collaborative Filtering: Algorithms and Evaluation. In Social Information Access. Peter Brusilovsky and Daqing He, eds. Springer-Verlag, Lecture Notes in Computer Science vol. 10100, pp. 344–390. . . Cited 8 times., , and .
2016. Dependency Injection with Static Analysis and Context-Aware Policy. Journal of Object Technology 15(1) (February 2016), 1:1–31. . Cited 1 times.and .
2015. Teaching Recommender Systems at Large Scale: Evaluation and Lessons Learned from a Hybrid MOOC. Transactions on Computer-Human Interaction 22(2) (April 2015). . Cited 13 times., , , , and .
2011. RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures. Proceedings of the VLDB Endowment 4(11) (August 2011), 911–920. Acceptance rate: 18%. Cited 7 times., , , , , and .
These papers have been published in peer-reviewed conference proceedings.
2018. Exploring Author Gender in Book Rating and Recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18). ACM, pp. 242–250. . . Acceptance rate: 17.5%. Cited 1 times., , , , and .
2018. All The Cool Kids, How Do They Fit In?: Popularity and Demographic Biases in Recommender Evaluation and Effectiveness. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (FAT* 2018). PMLR, Proceedings of Machine Learning Research 81:172–186. Acceptance rate: 24%. Cited 2 times., , , , , , and .
2018. Privacy for All: Ensuring Fair and Equitable Privacy Protections. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (FAT* 2018). PMLR, Proceedings of Machine Learning Research 81:35–47. Acceptance rate: 24%. Cited 2 times., , and .
2017. Sturgeon and the Cool Kids: Problems with Random Decoys for Top-N Recommender Evaluation. In Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference. AAAI, pp. 639–644.and .
2017. Recommender Response to Diversity and Popularity Bias in User Profiles. Short paper in Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference. AAAI, pp. 657–660. Cited 1 times.and .
2015. Letting Users Choose Recommender Algorithms: An Experimental Study. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys ’15). ACM. . Acceptance rate: 21%. Cited 32 times., , , and .
2014. User Perception of Differences in Recommender Algorithms. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys ’14). ACM. . Acceptance rate: 23%. Cited 54 times., , , and .
2014. Teaching Recommender Systems at Large Scale: Evaluation and Lessons Learned from a Hybrid MOOC. In Proceedings of the First ACM Conference on Learning @ Scale (L@S ’14). ACM. . Acceptance rate: 37%. Cited 25 times., , , , and .
2013. Rating Support Interfaces to Improve User Experience and Recommender Accuracy. In Proceedings of the 7th ACM Conference on Recommender Systems (RecSys ’13). ACM. . Acceptance rate: 24%. Cited 26 times., , , , , , and .
2012. When Recommenders Fail: Predicting Recommender Failure for Algorithm Selection and Combination. Short paper in Proceedings of the Sixth ACM Conference on Recommender Systems (RecSys ’12). ACM, pp. 233–236. . Acceptance rate: 32%. Cited 29 times.and .
2012. RecStore: An Extensible And Adaptive Framework for Online Recommender Queries Inside the Database Engine. In Proceedings of the 15th International Conference on Extending Database Technology (EDBT ’12). ACM, pp. 86–96. . Acceptance rate: 23%. Cited 11 times., , , and .
2011. Rethinking The Recommender Research Ecosystem: Reproducibility, Openness, and LensKit. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys ’11). ACM, pp. 133–140. . Acceptance rate: 27% (20% for oral presentation, which this received). Cited 99 times., , , and .
2011. Searching for Software Learning Resources Using Application Context. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, pp. 195–204. . Acceptance rate: 25%. Cited 25 times., , , , and .
2009. rv you’re dumb: Identifying Discarded Work in Wiki Article History. In Proceedings of the 5th International Symposium on Wikis and Open Collaboration (WikiSym ’09). ACM, 10 pp.. . Acceptance rate: 36% (Selected as Best Paper). Cited 23 times.and .
Workshops, Seminars, Posters, Etc.
These papers have undergone some form of peer review, and are published in workshops, poster proceedings, and similar venues.
2018. Retrieving and Recommending for the Classroom: Stakeholders, Objectives, Resources, and Users. In Proceedings of the ComplexRec 2018 Second Workshop on Recommendation in Complex Scenarios (ComplexRec ’18), at RecSys 2018., , , and .
2018. Monte Carlo Estimates of Evaluation Metric Error and Bias. Computer Science Faculty Publications and Presentations 148. Boise State University. Presented at the REVEAL 2018 Workshop on Offline Evaluation for Recommender Systems, a workshop at RecSys 2018. . .and .
2018. The LKPY Package for Recommender Systems Experiments: Next-Generation Tools and Lessons Learned from the LensKit Project. Computer Science Faculty Publications and Presentations 147. Boise State University. Presented at the REVEAL 2018 Workshop on Offline Evaluation for Recommender Systems, a workshop at RecSys 2018. . ..
2018. Recommending Texts to Children with an Expert in the Loop. In Proceedings of the 2nd International Workshop on Children & Recommender Systems (KidRec ’18), at IDC 2018. ., , and .
2018. Do Different Groups Have Comparable Privacy Tradeoffs?. At Moving Beyond a ‘One-Size Fits All’ Approach: Exploring Individual Differences in Privacy, a workshop at CHI 2018., , , and .
2017. The Demographics of Cool: Popularity and Recommender Performance for Different Groups of Users. In RecSys 2017 Poster Proceedings. CEUR, Workshop Proceedings 1905.and .
2017. Challenges in Evaluating Recommendations for Children. In Proceedings of the International Workshop on Children & Recommender Systems (KidRec), at RecSys 2017..
2016. Behaviorism is Not Enough: Better Recommendations through Listening to Users. In Proceedings of the Tenth ACM Conference on Recommender Systems (RecSys ’16). ACM. . Acceptance rate: 36% (Past, Present, and Future track). Cited 9 times.and .
2016. First Do No Harm: Considering and Minimizing Harm in Recommender Systems Designed for Engendering Health. In Proceedings of the Workshop on Recommender Systems for Health at RecSys ’16. Cited 4 times.and .
2014. Building Open-Source Tools for Reproducible Research and Education. At Sharing, Re-use, and Circulation of Resources in Cooperative Scientific Work, a workshop at CSCW 2014..
Other Publications and Presentations
These publications are unreviewed reports, preprints, etc.
2019. Supplementing Classroom Texts with Online Resources. At 2019 American Educational Research Association Conference., , , and .
2019. Recommender Systems Notation: Proposed Common Notation for Teaching and Research. Computer Science Faculty Publications and Presentations 177. Boise State University. . .and .
2018. From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442). Dagstuhl Manifestos 7(1) (November 2018), 96–139. ., , , , , , , , , , , , , , , , , , , , and .
2018. Supplementing Classroom Texts with Online Resources. At 2018 Annual Meeting of the Northwest Rocky Mountain Educational Research Association., , and .
2018. 2nd FATREC Workshop: Responsible Recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18). ACM. ., , and .
2018. UMAP 2018 Fairness in User Modeling, Adaptation and Personalization (FairUMAP 2018) Chairs’ Welcome & Organization. In Adjunct Publication of the 26th Conference on User Modeling, Adaptation, and Personalization (UMAP ’18). ACM. ., , , and .
2018. The Dagstuhl Perspectives Workshop on Performance Modeling and Prediction. SIGIR Forum 52(1) (June 2018), 91–101. . Cited 2 times., , , , , , , , , , , , , , , , , , , , and .
2014. Towards Recommender Engineering: Tools and Experiments in Recommender Differences. Ph.D thesis, University of Minnesota. . Cited 3 times..
2011. UCERSTI 2: Second Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces. In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys ’11). ACM, pp. 395–396. . Cited 4 times., , and .
- 2018–2023: NSF award CHS 17-51278, $482,081: CAREER: User-Based Simulation Methods for Quantifying Sources of Error and Bias in Recommender Systems (PI)
- 2017: $19K Boise State College of Education Empathy Grant LITERATE: Locating Informational Texts for Engaging Readers And Teaching Equitably (co-PI; with PI Katherine Wright & co-PI Sole Pera)
- 2014: $8K Texas State University Research Enhancement Program (competitive internal research grant) Temporal Analysis of Recommender Systems (PI)
- December 7, 2018: ‘User, Agent, Subject, Spy’ seminar at Clemson University
- November 9, 2018: ‘User, Agent, Subject, Spy’ seminar at Carnegie Mellon University Human-Computer Interaction Institute
- November 16, 2017: ‘Making Information Systems Good for People’ at Whitman College (Walla Walla, WA)
- June 26, 2017: ‘Recommending for People’ seminar at RecSysNL at TU Delft
- June 20, 2017: ‘Recommending for People’ seminar at Jheronimus Academy of Data Science
- June 19, 2017: ‘Recommending for People’ seminar at UCL Mons
- June 15, 2017: ‘Responsible Recommendation’ at the Brussels Big Data and Ethics Meetup, the inaugural event of the DigitYser Big Data community
- November 21, 2016: ‘Recommending for People’ colloquium at the University at Albany Dept. of Computer Science
- October 27, 2016: ‘Introduction to Recommender Systems’ at the Clearwater Developer Conference
- September 20, 2015: ‘Challenges in Scaling Recommender Systems Research’ at the Workshop on Large-Scale Recommender Systems at RecSys ’15 in Vienna, Austria
- September 19, 2015: ‘Levelling Up your Academic Career’ at the Doctoral Symposium at RecSys ’15 in Vienna, Austria
- 2012: ‘Flexible Recommender Experiments with LensKit’ at the RecSys Challenge Workshop at RecSys ’12 in Dublin, Ireland
- 2012: ‘The MovieLens Data Set’ (invited talk) at the RecSys Challenge Workshop at RecSys ’12 in Dublin, Ireland
I have built several open-source software packages in the course of my research and other work. Open-source software distribution is a key piece of my research dissemination strategy. My more significant development efforts include:
- LensKit, a toolkit for building, researching, and studying recommender systems, used in many published papers. http://lenskit.org
- Grapht, a dependency injection framework for Java with novel configuration and static analysis capabilities. http://grapht.grouplens.org
- Goanna (now defunct), a graphical tool for visualizing InfiniBand networks and compute clusters. Written while at the Scalable Computing Laboratory.
Ongoing Professional Service and Memberships
- Steering committee, ACM Conference on Recommender Systems (RecSys), 2017–present
- Steering committee, ACM Conference on Fairness, Accountability, and Transparency (FAT*), 2017–present
- Distinguished Reviewer, ACM Transactions on Interactive Intelligent Systems (TiiS), 2017–present
- Member of the ACM (professional member 2014–present)
- Guest editor, 2020 special issue of User Modeling and User-Adapted Interaction on fairness in user modeling.
- Co-organizer, TREC 2019 Track on Fairness in Information Retrieval
- Senior program committee, ACM RecSys 2019
- Program committee, ACM UMAP 2019
- PR & Publicity Co-chair, 2nd Confernece on Fairness, Accountability, and Transparency (ACM FAT* 2018)
- General co-chair, ACM RecSys 2018
- Program committee, ACM WWW Track on Behavior Analysis and Personalization (2016—2019)
- Co-organizer, FATREC Workshop on Responsible Recommendation at RecSys 2017 & 2018
- Co-organizer, FairUMAP workshop at UMAP 2018 & 2019
- Track co-chair, 2018 Conference on Fairness, Accountability, and Transparency Systems track
- Participant in Dagstuhl Perspectives Workshop Towards Cross-Domain Performance Modeling and Prediction: IR/RecSys/NLP
- Program committee, ACM RecSys (2014–2017) and poster session (2016)
- Program committee, FLAIRS Special Track on Recommender Systems (2015–2017)
- Publicity co-chair, ACM RecSys 2016
- External advisor, CrowdRec (EU Framework Programme collaborative research project, 2014–2016)
- Proceedings co-chair, ACM CHI 2012–2013
- Demos co-chair, ACM RecSys 2012
- Reviewer for numerous conferences and journals, including:
- ACM conferences CHI (2015, 2013, 2016, 2017), CSCW (2015, 2017), IUI (2017), SAC Recommender Systems track (2013, 2017), UIST (2012, 2016, 2017), WikiSym (2012), UMAP (2018)
- Workshops at ACM RecSys: IntRS (2016–2017), LSRS (2016), KidRec (2017–2018), HealthRecSys (2018)
- FATML 2017
- Communications of the ACM
- International Journal of Artificial Intelligence Tools (2016)
- JMLR Open Source (2016)
- IBM Journal of Research and Development (2016)
- ICWSM 2012
- ACM journals TIST, TOIS, TWEB, TKDD, and TIIS
- IEEE journals TDSC, TKDE
- Interacting with Computers
- International Conference on Service-Oriented Computing (2016)
- PLOS ONE (2016)
- User Modeling and User-Adapted Interaction
- Information Retrieval Journal
- ACM Computing Surveys (2014, 2015)
- User Modeling
- Artificial Intelligence Review (Springer)
- Hindawi journals Advances in Multimedia, Advances in Artificial Intelligence
Department and University Service
- Boise State Ph.D in Computing Steering Committee (2017–present)
- Boise State CS Dept. Curriculum Committee (2017–present)
- Boise State CS Dept. Graduate Recruiting Committee (2017)
- Texas State CS Dept. Undergraduate Committee (2014–2016)
- Texas State CS Dept. Written Comp Exam Grading (2014–2016)
- UMN CS Graduate Student Association secretary (2009–2010)
- February 2019 — testified before Idaho State House Judiciary Committee on H.B. 118, regulating pretrial risk assessment algorithms.
- December 2017 — Boise Public Library panel on preparing for a career in computer science
- Judge, 2015 — Travis Elementary School Science Fair