We tackle two key hurdles in building foundation models for time series data: (1) a lack of readily available, large public datasets, and (2) the absence of experimental benchmarks to evaluate these models, particularly for situations with limited resources and supervision. To overcome these challenges, we introduce the Time Series Pile, a comprehensive collection of public time series data. Additionally, we expand on existing work by creating a benchmark that assesses time series foundation models on various tasks and datasets, even with limited supervision. We conclude the paper by outlining key areas for future research in developing datasets and benchmarks for time series foundation models. Time Series Pile is available on Huggingface.