Why use k = 1,000 for MAP@k evaluation?

Although we have argued against cut-off-measures like precision@k (often in recommender literature precision@5 or precsion@10) it is reasonable to cut off lists at some point. Compared to many other areas where recommendations are used (e.g., people, movie, book or website recommenders) the time needed to evaluate a recommendation is very short (if you like a name, just click it) and the cost in terms of money or time spent for following a recommendation that turns out bad, is very low. At the same time, finding the perfect name for a child is often a process of months rather than minutes (like for finding the next book to read or deciding which movie to watch) or seconds (deciding which tag to use or which website to visit on the net). Thus it is reasonable to assume that parents-to-be are willing to scroll through lists of names longer than the usual top 10. Additionally, consider that one of the traditional ways of searching for names is to go through first names dictionaries where names are listed unpersonalized, in alphabetical order. In such books usually there are a lot more than 1,000 names that have to be read and therefore it seems reasonable that readers of such books won’t mind studying longer name lists on the web.

Comments are closed.