Recent advances in machine learning, data availability, and computational resources have given rise to complex nonlinear models of psychological processes. Though theory can help guide model selection, there is no widely agreed-upon framework for identifying models that are appropriately complex and accurate for examining the phenomenon of interest. This thesis develops a framework for selecting statistical models for intensive longitudinal data in a principled manner. Drawing on literature from both social science and machine learning, I argue that neither qualitative model assessment nor automated accuracy testing is sufficient for selecting an appropriate longitudinal model.
I instead offer a structured model selection framework that integrates three model properties – complexity, efficacy, and interpretability – to provide clarity on which models are best aligned with a researcher’s analysis goals. I outline how to measure the three properties through both existing and newly proposed metrics. I illustrate how complexity and efficacy can be comprehensively measured using well-known metrics, like training time and prediction error. For interpretability, I argue that existing measurement methods are insufficient. Drawing on the perspectives of modeling experts, I present a new definition of interpretability and break it down into measurable facets. To illustrate the practical utility of the model selection framework, I apply it to two example research inquiries. The first example focuses on the time-oriented process of cognitive skill acquisition and uses simulated data to allow for testing of model assumptions. The second example examines an empirical dataset of intensive longitudinal psychophysiological data and explores the intricacies of model selection when there is no clearly defined data-generating function.
After walking through these examples, I set forth practical guidelines to aid researchers in applying principled model selection to their own work. These guidelines include documenting the model selection process and letting one’s research question guide the interpretation of model evaluation results. I also identify general trends indicative of theoretical relationships among complexity, efficacy, and interpretability.
