Modern large language models (LLMs) have demonstrated cooperative synergy on complex task-solving, and collective decision-making (CDM) is a pivotal component in LLM-based multi-agent collaboration frameworks.Surprisingly, our survey on 52 recent such systems reveals a severe lack of diversity and heavy reliance on dictatorial and plurality voting for CDM. Using social choice theory as a lens, we critically examine widely-adopted CDM methods and identify their limitations. To enrich current monotonous and limited landscape of LLM-based CDM, we introduce 8 ordinal preferential voting mechanisms that can be easily integrated with various multi-agent frameworks. Our empirical case study on MMLU benchmark demonstrates that incorporating certain CDM methods alone can enhance the reasoning performance and robustness of some state-of-the-art LLMs, without any complex system designs. Furthermore, some CDM mechanisms generate positive synergies with as few as three agents, foreshadowing a profitable computation trade-off.