In episode 21 of Recsperts, we welcome Martijn Willemsen, Associate Professor at the Jheronimus Academy of Data Science and Eindhoven University of Technology. Martijn's researches on interactive recommender systems which includes aspects of decision psychology and user-centric evaluation. We discuss how users gain control over recommendations, how to support their goals and needs as well as how the user-centric evaluation framework fits into all of this.

In our interview, Martijn outlines the reasons for providing users control over recommendations and how to holistically evaluate the satisfaction and usefulness of recommendations for users goals and needs. We discuss the psychology of decision making with respect to how well or not recommender systems support it. We also dive into music recommender systems and discuss how nudging users to explore new genres can work as well as how longitudinal studies in recommender systems research can advance insights.

Towards the end of the episode, Martijn and I also discuss some examples and the usefulness of enabling users to provide negative explicit feedback to the system.

Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.
Don't forget to follow the podcast and please leave a review

• (00:00) - Introduction
• (03:03) - About Martijn Willemsen
• (15:14) - Waves of User-Centric Evaluation in RecSys
• (19:35) - Behaviorism is not Enough
• (46:21) - User-Centric Evaluation Framework
• (01:05:38) - Genre Exploration and Longitudinal Studies in Music RecSys
• (01:20:59) - User Control and Negative Explicit Feedback
• (01:31:50) - Closing Remarks

• Martijn Willemsen on LinkedIn
• Martijn Willemsen's Website
• User-centric Evaluation Framework
• Behaviorism is not Enough (Talk at RecSys 2016)
• Neil Hunt: Quantifying the Value of Better Recommendations (Keynote at RecSys 2014)
• What recommender systems can learn from decision psychology about preference elicitation and behavioral change (Talk at Boise State (Idaho) and Grouplens at University of Minnesota)
• Eric J. Johnson: The Elements of Choice
• Rasch Model
• Spotify Web API

Papers:

• Ekstrand et al. (2016): Behaviorism is not Enough: Better Recommendations Through Listening to Users
• Knijenburg et al. (2012): Explaining the user experience of recommender systems
• Ekstrand et al. (2014): User perception of differences in recommender algorithms
• Liang et al. (2022): Exploring the longitudinal effects of nudging on users’ music genre exploration behavior and listening preferences
• McNee et al. (2006): Being accurate is not enough: how accuracy metrics have hurt recommender systems

General Links:

• Follow me on LinkedIn
• Follow me on X
• Send me your comments, questions and suggestions to marcel.kurovski@gmail.com
• Recsperts Website

In episode 20 of Recsperts, we welcome Bram van den Akker, Senior Machine Learning Scientist at Booking.com. Bram's work focuses on bandit algorithms and counterfactual learning. He was one of the creators of the Practical Bandits tutorial at the World Wide Web conference. We talk about the role of bandit feedback in decision making systems and in specific for recommendations in the travel industry.

In our interview, Bram elaborates on bandit feedback and how it is used in practice. We discuss off-policy- and on-policy-bandits, and we learn that counterfactual evaluation is right for selecting the best model candidates for downstream A/B-testing, but not a replacement. We hear more about the practical challenges of bandit feedback, for example the difference between model scores and propensities, the role of stochasticity or the nitty-gritty details of reward signals. Bram also shares with us the challenges of recommendations in the travel domain, where he points out the sparsity of signals or the feedback delay.

At the end of the episode, we can both agree on a good example for a clickbait-heavy news service in our phones.

Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.
Don't forget to follow the podcast and please leave a review

• (00:00) - Introduction
• (02:58) - About Bram van den Akker
• (09:16) - Motivation for Practical Bandits Tutorial
• (16:53) - Specifics and Challenges of Travel Recommendations
• (26:19) - Role of Bandit Feedback in Practice
• (49:13) - Motivation for Bandit Feedback
• (01:00:54) - Practical Start for Counterfactual Evaluation
• (01:06:33) - Role of Business Rules
• (01:11:26) - better cut this section coherently
• (01:17:48) - Rewards and More
• (01:32:45) - Closing Remarks

• Bram van den Akker on LinkedIn
• Practical Bandits: An Industry Perspective (Website)
• Practical Bandits: An Industry Perspective (Recording)
• Tutorial at The Web Conference 2020: Unbiased Learning to Rank: Counterfactual and Online Approaches
• Tutorial at RecSys 2021: Counterfactual Learning and Evaluation for Recommender Systems: Foundations, Implementations, and Recent Advances
• GitHub: Open Bandit Pipeline

Papers:

• van den Akker et al. (2023): Practical Bandits: An Industry Perspective
• van den Akker et al. (2022): Extending Open Bandit Pipeline to Simulate Industry Challenges
• van den Akker et al. (2019): ViTOR: Learning to Rank Webpages Based on Visual Features

General Links:

• Follow me on LinkedIn
• Follow me on X
• Send me your comments, questions and suggestions to marcel.kurovski@gmail.com
• Recsperts Website

In episode 19 of Recsperts, we welcome Himan Abdollahpouri who is an Applied Research Scientist for Personalization & Machine Learning at Spotify. We discuss the role of popularity bias in recommender systems which was the dissertation topic of Himan. We talk about multi-objective and multi-stakeholder recommender systems as well as the challenges of music and podcast streaming personalization at Spotify.

In our interview, Himan walks us through popularity bias as the main cause of unfair recommendations for multiple stakeholders. We discuss the consumer- and provider-side implications and how to evaluate popularity bias. Not the sheer existence of popularity bias is the major problem, but its propagation in various collaborative filtering algorithms. But we also learn how to counteract by debiasing the data, the model itself, or it's output. We also hear more about the relationship between multi-objective and multi-stakeholder recommender systems.

At the end of the episode, Himan also shares the influence of popularity bias in music and podcast streaming at Spotify as well as how calibration helps to better cater content to users' preferences.

Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.
Don't forget to follow the podcast and please leave a review

• (00:00) - Introduction
• (04:43) - About Himan Abdollahpouri
• (15:23) - What is Popularity Bias and why is it important?
• (25:05) - Effect of Popularity Bias in Collaborative Filtering
• (30:30) - Individual Sensitivity towards Popularity
• (36:25) - Introduction to Bias Mitigation
• (53:16) - Content for Bias Mitigation
• (56:53) - Evaluating Popularity Bias
• (01:05:01) - Popularity Bias in Music and Podcast Streaming
• (01:08:04) - Multi-Objective Recommender Systems
• (01:16:13) - Multi-Stakeholder Recommender Systems
• (01:18:38) - Recommendation Challenges at Spotify
• (01:35:16) - Closing Remarks

• Himan Abdollahpouri on LinkedIn
• Himan Abdollahpouri on X
• Himan's Website
• Himan's PhD Thesis on "Popularity Bias in Recommendation: A Multi-stakeholder Perspective"
• 2nd Workshop on Multi-Objective Recommender Systems (MORS @ RecSys 2022)

Papers:

• Su et al. (2009): A Survey on Collaborative Filtering Techniques
• Mehrotra et al. (2018): Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommender Systems
• Abdollahpouri et al. (2021): User-centered Evaluation of Popularity Bias in Recommender Systems
• Abdollahpouri et al. (2019): The Unfairness of Popularity Bias in Recommendation
• Abdollahpouri et al. (2017): Controlling Popularity Bias in Learning-to-Rank Recommendation
• Wasilewsi et al. (2016): Incorporating Diversity in a Learning to Rank Recommender System
• Oh et al. (2011): Novel Recommendation Based on Personal Popularity Tendency
• Steck (2018): Calibrated Recommendations
• Abdollahpouri et al. (2023): Calibrated Recommendations as a Minimum-Cost Flow Problem
• Seymen et al. (2022): Making smart recommendations for perishable and stockout products

General Links:

• Follow me on LinkedIn
• Follow me on X
• Send me your comments, questions and suggestions to marcel@recsperts.com
• Recsperts Website