While educational achievement surveys revolutionised the research possibilities for answeringpolicy relevant education questions in an international context, the surveys have been repeatedly subject of heated debate since first results were published. This paper discusses the soundness of survey methodology of large scale achievement surveys aiming to summarise results of the author’s and other up to date research. Thereby, different components of the ‘total survey error’will be discussed covering among others validity, non-response errors and the choice of item response models on results. Findings indicate that there are legitimate concerns about what educational achievement surveys measure and how. Survey organisers should be more transparent in the documentation of the methodological choices that underlie the creation of the data and more explicit about the impact of these choices on the results.