The rapid development of generative AI models has gained groundbreaking attention. This article looks into the practical challenges of generative model-based synthetic datasets with an intersection of ethical considerations inherent to this field. These challenges include privacy attacks and limitations in existing privacy-preserving approaches. We also highlight future research directions that foster fair and responsible use of synthetic data while ensuring ethical oversight in the landscape of generative AI.