Melody extraction has been an active topic of research in Music Information Retrieval for decades now. And yet - what is a melody? As a community we still (mostly) shy away from this question, resorting to definition-by-annotation. How well do past/present/future algorithms perform? Despite known limitations with existing datasets and metrics, we still (mostly) stick to the same ones. And last but not least, why do melody extraction at all? Despite great promise (e.g. query-by-humming, large-scale musicological analyses, etc.), melody extraction has seen limited application outside of MIR research. In this talk I will present three problems that are common to several of research area in music informatics: the challenge of trying to model ambiguous musical concepts by training models with somewhat arbitrary reference annotations, the lack of model generalization in the face of small, low-variance training sets, and the possible disconnect between parts of the music informatics research community and the potential users of the technologies it produces.