Of course logistic regression is regression (and other fake categories)

I recently came across this article “Is logistic regression regression?” by data scientist Richard Vale. The post is a response to an apparent view in the machine learning community that “logistic regression” is a misnomer, because it is used for classification problems (with categorical outcomes), not regression problems (with real-valued outcomes).

As someone who learned statistics before machine learning, by way of biology and then epidemiology, this question would never have occurred to me. I was taught to group models by outcome type—logistic regression for binary outcomes, linear regression for continuous outcomes, multinomial regression for unordered categorical outcomes, Cox regression for time-to-event outcomes, and so on. Of course logistic regression is regression. It is regression with a binary outcome and a logit link. And besides (as Vale points out), logistic regression does not output a category label. Its raw output is log-odds, which may be transformed into a probability. “Classification” is a separate step layered on top, usually by imposing a threshold of 0.5.

This point goes to the statisticians.

I was not taught to think in terms of two types of predictive modelling: regression and classification. In fact, I’m not sure I ever came across the term “classification” until I picked up The Elements of Statistical Learning (okay, it was actually An Introduction to Statistical Learning). Rather, I was taught to approach modelling based on what I was trying to achieve: the type of research question. In this framework, there are three basic types of questions: descriptive (how are things distributed?), predictive (what outcome can we expect?), and causal (what difference would an exposure/intervention make?).

But as statistician Andrew Gelman points out:

I kinda think that descriptive, predictive, and causal are all the same thing–or, more precisely, that “descriptive” and “causal” are special cases of “predictive,” under different conditions. But if you want to divide them into three tasks, sure, go for it. Personally, I’d rather divide statistics into the goals of exploration, estimation, and discrimination, but I think that’s because I’m thinking in a more general “data science” perspective

So maybe the machine learning folks get a point here. Not the point that logistic regression is badly named, but the broader instinct that prediction is a bigger category than those of us trained in epidemiology sometimes allow.

This line of thinking—that neat categories may be less distinct than they appear—applies to statistical tests, too. Normality, linearity, equal variances, independence, differences between groups, associations between variables, treatment effects: almost anything you can think to test can be approached the same basic way. As we’ve covered on this blog before, there is only one statistical test. No matter which statistician’s name adorns the procedure, you are still doing the same thing: computing an observed statistic, generating its null distribution, and asking how often the null produces something at least as extreme.

I guess the trouble starts when useful teaching shortcuts get mistaken for the thing itself.