What are the downfalls of selecting most parsimonious model using AIC?

Jbel

New Member
#1
Hi all, I am rather new to statistics, but I am reviewing a paper that applies the most parsimonious model for logistic regression using Akaike’s Information Criterion. It is a clinical study in ~200 patients and covariates assessed are mostly categorical variables.

Results are opposite of what one would expect on a biological basis and I believe it is due to a number of confounding variables that were not included in the final model using the AIC method and were deemed not significant. However, although these variables are excluded statistically, there is a strong theoretical basis to have included at least some of them. For example, as the outcome is disease activity and one group had more patients on high efficacy therapy to prevent disease activity it seems logical to include this variable despite its statistical significance.

Is this method appropriate with a small sample size and categorical variables (with some that are broken into up to 6 categories)? And are there any other downfalls based on this method, which might explain these opposite results? I appreciate any feedback, Thanks!
 

staassis

Active Member
#2
When the sample size converges to infinity (10,000s in practice), the verdict of AIC converges to the truth. So AIC is the right tool to use in medium-sized and large samples. In small samples, however, AIC tends to bee too generous. It chooses models which are somewhat bigger than the optimum. The models contain predictors they should not contain.

In small samples, BIC (Bayesian Information Criterion) works better than AIC (on average). In your case it is likely to suggest removing even more variables that you love so much.

Removing variables which deserve a place in the model from the theoretical point of view is fine. You have to do it as long as there is not enough statistical power in the data set to accurately estimate the biggest theoretically sound model. For example, if you have only 5 observations in your data set, who cares what other papers have published?
 

hlsmith

Not a robit
#3
What was the prevalence of the outcome and how many variables were examined and kept?

AICc is a reason approach for comparing models based on out of sample deviance. Issues, no model knows the underlying context as you mention. So it could miss clinically important variables. Also when finding the best model, estimates should ideally come from a new or held out data, not the set the covariates were determined from, in order to increase external validity.
 

Jbel

New Member
#4
Thanks for your input! I think this is quite helpful.

What was the prevalence of the outcome and how many variables were examined and kept?

AICc is a reason approach for comparing models based on out of sample deviance. Issues, no model knows the underlying context as you mention. So it could miss clinically important variables. Also when finding the best model, estimates should ideally come from a new or held out data, not the set the covariates were determined from, in order to increase external validity.
The outcome occurred in 37% of patients. 13 variables were assessed and 4 were kept. Type of therapy was examined using 6 categories, some with only 5 patients, but was excluded.