Contributions are welcome. ]]>

Yes they can. I use cross-validation for GP survival models, but WAIC could be used, too.

]]>You mentioned Survival data, can these criterions be used under censoring? ]]>

You’re correct; he leaves it on that scale in his paper. What can I say, I’m unfairly biased by BDA3 8-)

]]>WAIC as defined by Watanabe is not on the same scale! Watanabe multiplies by -1/n

]]>Why Akaike decided to multiply the negative loglikelihood by -2 is probably an artifact of linear regression and the chi-square value, but for better or for worse, most information criteria metrics which have followed (BIC, HQIC, QAIC, TIC, CAIC, CAICF, ICOMP, DIC, WAIC) are on the same scale.

]]>It’s too easy to make this kind of mistakes, because we are sometimes using log density and sometimes deviance…

]]>D’oh! OK, I just fixed it and reposted. Thanks for noticing.

]]>Shouldn’t it be an overestimate? As I understand it seems to say that the new dataset is expected to fit better than the data we already used.

]]>In this example x is the number of the year. x is deterministic and your goal is to predict the value of the random variable in year 11 before seeing the value of year 12.

LOO assumes both x and y to be random (or future (x,y) pair to be unknown), so it is not exactly what you want but if your model is simple it is likely that it is adequate for the model comparison.

WAIC assumes x to be fixed (e.g. if you would want to predict new observations in years 1 to 10), so it is not exactly what you want but if your model is simple it is likely that it is adequate for the model comparison.

Sequential approach would be closer to your prediction task, that is predict each observed year given the previous years.

Sequential approach wastes information, and there are ways to improve that, but they are bit more complex to explain.

If you are comparing (generalised) linear models with a number of observations much larger than the number of covariates, then it is likely that it does not matter whether you use LOO or WAIC.

]]>I forgot to mention that IS-LOO behavior is somewhere between real LOO and WAIC, because with a finite number of samples from the full posterior the samples do not cover the tails well and thus the result is biased towards the posterior (as can be seen in Fig 2 of our paper).

]]>In this case, the parameters of the distribution are hierarchical (be they mu/sigma of the lognormal, alpha/theta of the gamma, or even hyperparameters on those parameters) but there aren’t seperate observations of different classes, just the annual observations of the variables themselves.

In this case, which statistic would be preferable, or, since there isn’t a complicated hierarchical structure and subsetted observations, it wouldn’t matter?

]]>> In a nutshell, WAIC is better suited to reflect estimates based on a new observation from existing data and LOO is better suited to reflect estimates based on a new observation from new data?

You might have the right idea, but what you write is wrong.

For hierarchical models, WAIC useful to estimates predictive performance for a new observation from an existing group (group j with parameter alpha_j) and cross-validation is better suited to reflect estimates based on a new observation from a new group (group J+1 with parameter alpha_{J+1}). If each group has single observation, which is common, for example, when the group is indexed by continuous covariate values (alpha|x), then LOO is sensible. If each group has several observations, then leave-one-group-out cross-validation should be used to estimate the performance in a new group. For example in cognitive brain research, we have several observations per trial and several trials per subject and want to whether we can generalize the findings for new subjects and then leave-one-subject-out cross-validation is used. Importance sampling does not usually work well if we want to approximate effect of leaving several observations out, but as there are usually not so many subjects in these studies, we can easily compute each leave-one-subject-out posterior.

]]>Also, if this gets rolled into RStan, there are a few tweaks for optimization that could be made. Most importantly, it would probably pay to have the loglikelihood be made method compatible with the ‘logLik‘ class to allow further integration with existing R packages and functions.

Thanks.

]]>