Confidence-building is an under-researched area in statistics. Some pieces of confidence-building:
– Software validation (also see here)
– Convergence of numerical algorithms
– Checking fit of model to data (see chapter 6 of BDA)
– Cross-validation
– External validation
Various things have been written on all these topics, but I haven’t seen them put into a coherent framework with the larger goal of building confidence in a model, which I think is a limiting factor in a lot of statistical analyses.
Also not much discussed is “model understanding”–that is, tools for following the path from data to inferences, questions such as, “what aspects of the data allow us to estimate alpha,” “what part of the data tells us about beta,” and so forth. There’s a lot of discussion out there of identifiability and partial identifiability, and this seems relevant to model understanding, but I haven’t really seen a good set of tools for doing this–beyond simple plots of fitted regression lines and raw data.
As we say to the funding agencies, further work is needed.
Colin Mallows' Fisher Memorial Lecture – The Zeroth Problem – dealt with some of this.
"Unfortunately" there has been a custom of just stating your model and getting on with the "real statistical" work (i.e. such as identifiability)
The "why" the model was chosen for "what purpose" and with what "transparencies" and "trade-offs" gets set aside.
Not much math involved (though Peter McCullagh has done some of that) and it might not look like statistics.
Keith