What do statistical tests really do

Anything over 20 posts goes in circles....:p

The only idea that is forever cemented in my head is that every time I assume there is general agreement on any methods issue I am wrong. Over the six years I have come here I have ended up abandoning much of what I learned in my graduate (as in college not high school) statistics classes and my many hours of reading stats to be sure I understand what I do at work.

I now am not convinced anything is really agreed on in statistics at least for more than 5 year. I have lost track of the times I have read an article saying some well known statistical method is in error and should not be used :p Which is not very different than academics, but somehow I assumed statistics was different.
Its not so much that things are not agreed upon, its just that statistics is an exceptionally wide field (that is becoming wider as more areas adopt strong statistical methodological approaches). Regression is the cornerstone and then it gets tweaked to handle different nuances of different fields. Take the difference between an education statistician and a neuro-psychological statistician. If a person is an educational statistician, there is a good chance they will be primarily interested in large-scale longitudinal data sets. Data is likely to be hierarchical (nested, clustered, panel, whatever your field refers to high intraclass correlated data) and there could be missing data due to redaction. Choices largely center upon correct model specification due to massive number of available covariates (google the public education data for finance in your state to get a good idea of how many potential variables). Conversely, the psychologist working in an MRI lab also has to deal with hierarchical data. The functional brain scans are longitudinal (like the education data) but the voxels (cubic millimeters of brain mass) are correlated with each other due to brain matter, blood vessels, etc. This statistician is worried about issues like psuedoreplication and correctly adjusting p-values due to inflation as well as correcting data for abnormalities.

In short, both statisticians, both dealing with clustered longitudinal data BUT very different approaches in terms of nuance.

So, the TLDR of my post is that statisticians arnt agreeing because the field has exploded and everyone is working on their own little niches. And just because your solution works for your niche does not mean its going to be able to appreciate all the nuances of another statisticians field.