We introduce Astroconformer, a Transformer-based model to analyze stellar light curves from the Kepler mission. Astrconformer embeds light curves to low-dimension representation, and we demonstrate that the model can robustly infer the stellar surface gravity as a downstream task. Importantly, as Transformer captures long-range information in the time series, it outperforms the state-of-the-art data-driven method in the field, and the critical role of self-attention is proved through ablation experiments. Futhermore, the attention map from Astroconformer exemplifies the long-range correlation information learned by the model, leading to a more interpretable deep learning approach for asteroseismology. Besides data from Kepler, we also show that the method can generalize to sparse cadence light curves from the Rubin Observatory, paving the way for the new era of asteroseismology, harnessing information from long-cadence ground-based observations.