12  Conclusion

Future Directions

This book is a living resource and it will add new content with each version. Versions will be released in the Github repository as on the Open Science Framework (DOI here when created). In future releases, there are a number of features that are planned to be added. Here is a list of those features:

• Interactive visualizations of artifacts and bias using RShinyLive
• More effect size types with corrections: Repeated measures SMDs, Variability ratios, and unstandardized effect sizes (mean differences and regression coefficients).
• A chapter on the artifact of treatment non-compliance.
• Z-transformations for corrected correlations.
• Practice problems.
• Small simulations for assessing robustness of corrections in non-normal distributions (e.g., heavy-tails, skew)

Suggestions for added features are welcome. Suggestions can be made by submitting an issue to the Github repository.

Limitations

Applying artifact corrections has trade-offs that should be considered before putting them into use. Obtaining more accurate estimates of our target effect sizes is the obvious major benefit of applying corrections, however it is also true that these artifact corrections can impose assumptions that may not be met in practice. The assumptions were laid out in each of the chapters, but it is important for the researcher to consider how reasonable these assumptions are for their research context. If the assumption violations are extreme, than the correction can potentially over- or under-correct the effect size estimates.

Defining the scientific estimand is the most important step before proceeding with artifact corrections. Each correction procedure is meant to estimate very specific quantity. If a researcher’s target quantity is different from the one that the correction procedure is trying to estimate, then the correction may further bias the effect size. A good example of this is in college admissions testing, where it can be quite tempting to correct the correlation between test scores and college grade-point average (GPA) for measurement error in the test scores. However, the practical utility of test scores is limited to the prediction capability of observed test scores since admission committees do not have access to true scores. Therefore if our quantity of interest is the on-the-ground predictive utility of test scores on GPA, then correcting the correlation for measurement errors in test scores will over-estimate the predictive utility. Researchers should ensure that the ultimate research goals align with the correction procedure.

Concluding Remarks

Imperfections in studies can obscure our view of reality, inhibit us from making sense of our observations, and they can mislead us. Corrections to these imperfections can allow us to see what findings may look like in an ideal study with perfect conditions. Meta-analyses will inevitably incorporate imperfect studies that vary in design and methodology that may restrict our ability to estimate a specific estimand. However, this should not limit the goal of meta-analysis to be merely a description of the literature. Corrections can allow meta-analysts to estimate specific scientifically-relevant quantities. Donald Rubin (1992) discusses the importance of estimating effect sizes in theoretically perfect studies:

Literature synthesis is fine, but, before scientifically valid statistical inference can take place, scientifically relevant quantities (i.e., estimands) must be defined, and the population estimands in the traditional view may be of limited scientific interest. The scientifically most interesting estimands involve the hypothetical results of technically perfect studies, rather than the average of results from some population of fallible studies.

Most artifact correction meta-analyses have been conducted almost exclusively in industrial and organizational psychology (e.g., personnel selection) and education assessment research (e.g., college admissions testing). However, the concepts in this book apply to many fields including biomedical, social science, clinical research, among others. No discipline is immune to measurement error and selection effects, therefore it is important to address these artifacts in every field of research.

This book could not have been accomplished without the pioneering work of John Hunter and Frank Schmidt. This book stands as a testament to their foundational contributions to artifact corrections and meta-analysis. The goal of this current book was to expand the methodologies and applications presented in their book (Hunter and Schmidt 2015). It also aims to provide a comprehensive guide, drawing upon previous scholarship while adapting to the evolving landscape of research methodologies.

I hope reading this book will provide invaluable insights and tools that empower people to address the bias in our studies. May it encourage a deeper awareness of these issues when confronted with them.

We want our research findings to accurately describe reality. Artifact corrections help us get closer to that goal.

Acknowledgements

Thank you to Dr. Blair T. Johnson, Dr. Christopher Rhoads, and Dr. Elizabeth Schifano for taking the time to review and provide extremely valuable feedback over the course of writing this book.

Matthew B. Jané

@MatthewBJane

Hunter, John E., and Frank L. Schmidt. 2015. Methods of meta-analysis: correcting error and bias in research findings (third). Third. Thousand Oaks, California: Sage Publications.

Rubin, Donald B. 1992. “Meta-Analysis: Literature Synthesis or Effect-Size Surface Estimation?” Journal of Educational Statistics 17 (4): 363–74. https://doi.org/10.3102/10769986017004363.