Seamlessly communicating with machines has always been the ultimate goal of artificial intelligence. This talk addresses the two key milestones towards general intelligence: how to effectively track infinite long history and improve the generation content's factuality. Specifically, we will talk about a stateful transformer architecture that can achieve effective memory read and write. We will also address how to use reinforcement learning with human feedback to improve generation content's faithfulness.