Letting Users Choose Recommenders
This is a paper about a drop-down menu. I’ll be presenting on it in the first session on Monday at RecSys 2015; the work is joint work with Daniel Kluver, Max Harper, and Joe Konstan at GroupLens.
In our previous paper, we examined what algorithms users say they prefer and the differences they perceive between those algorithms. This paper asks the follow-up question: when users are allowed to select the algorithm they actually use, what do they do?
The new version of MovieLens is now the generally-available MovieLens, rather than a beta. When Max and his team rolled out this version, we added the switch that we promised users in the list comparison study: a menu to pick the recommender that they would use. We logged users’ actions on this menu, and looked for a few things:
- Do users take advantage of the option, and if so, how?
- What algorithm(s) do they tend to prefer?
- Can we predict this with characteristics of either the algorithm’s output or the user’s profile?
The paper has the details, and my talk will have quite a few of them. But here’s the executive summary:
- About 1/4 of the users tried other recommenders. Most users played with the menu once or twice and then left it alone.
- Users preferred personalized algorithms, and more users preferred SVD than preferred item-item. This latter difference is small, but present; it also provides an interesting contrast with our previous study, where we did not find a preference between these two algorithms.
- We have not yet been able to isolate algorithm properties or user properties that predict the user’s choice of algorithms.
There were 2 other interesting results we observed. First, users who experimented with the menu were more likely to come back to the site. It is important to note that we do not have evidence that this is causal — it is quite likely that more active users are more prone to experiment with the menu.
Second, there were objectively measurable differences in the output of the different algorithms.
Read the paper for more details, and I hope to see you in Vienna.