Timezone: »

On the Relation between Quality-Diversity Evaluation and Distribution-Fitting Goal in Text Generation
Jianing Li · Yanyan Lan · Jiafeng Guo · Xueqi Cheng

Tue Jul 14 06:00 PM -- 06:45 PM & Wed Jul 15 04:00 AM -- 04:45 AM (PDT) @

The goal of text generation models is to fit the underlying real probability distribution of text. For performance evaluation, quality and diversity metrics are usually applied. However, it is still not clear to what extend can the quality-diversity evaluation reflect the distribution-fitting goal. In this paper, we try to reveal such relation in a theoretical approach. We prove that under certain conditions, a linear combination of quality and diversity constitutes a divergence metric between the generated distribution and the real distribution. We also show that the commonly used BLEU/Self-BLEU metric pair fails to match any divergence metric, thus propose CR/NRR as a substitute for quality/diversity metric pair.

Author Information

Jianing Li (Institute of Computing Technology, Chinese Academy of Sciences)
Yanyan Lan (Institute of Computing Technology)
Jiafeng Guo (Institute of Computing Technology, Chinese Academy of Sciences)
Xueqi Cheng (Institute of Computing Technology, CAS, China)

More from the Same Authors