Abstract

Background: Machine learning techniques have started to be used in various healthcare datasets to identify frail persons who may benefit from interventions. However, evidence on the performance of machine learning techniques in comparison to conventional statistical methods and existing clinical scales is mixed. It is also unclear what methodological and database factors are associated with the performance.

Objectives: In this study, we aimed to compare the classification accuracies of various machine learning classifiers in identifying frailty under different scenarios.

Methods: We used anonymized data collected from older adults (aged 65+) who were assessed with interRAI-Home Care (interRAI-HC) in New Zealand between January 1st, 2012, to December 31st, 2016. A total of 138 interRAI assessment items were employed to predict six-month mortality, using three machine learning classifiers (including random forest [RF], extreme gradient boosting [XGBoost], and multilayer perceptron [MLP]) and logistic regression. We conducted a simulation study to compare the performance of machine learning models with validated clinical scales, including the Simple Frailty Scale, Clinical Frailty Scale, and interRAI Home Care Frailty Scale. The effects of sample sizes, number of features and train-test split ratios were also examined.

Results: A total of 95,042 older adults (median age 82·66 years; 39·42% male) receiving home care were analyzed in this study. The six-month mortality predictive performance of logistic regression exceeded the benchmark frailty scales by using features equal to or larger to 4,000 and sample size equal or larger to 20 according to average AUC-ROCs and F1-Scores. In terms of average AUC-ROCs, compared to XGBoost, MLP and RF, logistic regression had better performance by using limited number of features (80) and limited sample size (6,000); and XGBoost outperformed logistic regression as number of features and sample sizes increased. Nevertheless, in terms of average F1-Scores, logistic regression had greater advantages in predicting six-month mortality over XGBoost, MLP and RF in every scenario.

Conclusions: In situations where the number of features and sample sizes were small, the performance of conventional logistic regression is sufficiently good for identifying frail older adults in long-term care. Machine learning classifiers outperform conventional method only when the number of features and sample sizes were large.