Someone named Sean writes:
I am interested to hear your thoughts on the study Self-reported dental anxiety and injection phobia among individuals with tattoos and piercings, published in the Journal of Oral Sciences.
To summarise (very) briefly: The authors visited 10 tattoo parlours and handed out a questionnaire to tattooed and/or pierced individuals (n = 55). The questionnaire measures self-reported injection phobia and self-reported dental anxiety. The control group (n = 46) consisted of a randomly selected group from a local hair salon (!), where none of the individuals had tattoos or piercings.
Apparently the study is in part motivated by a “myth” (anecdotal evidence) in some odontology circles that tattooed/pierced people are more afraid of dental visits than the general population.
A large Norwegian (the main author’s nationality) newspaper ran a story on the study, declaring that “Study confirms that people with tattoos and piercings are most bothered by injections and dental anxiety”. The same newspaper wrote a condensed article (which I assume is primarily meant for consumption on social media) with the terrible headline “Piercings may cause injection phobia”.
The authors do note several possible sources of biases in their paper, and they diligently warn against reading too much into the results, so the newspaper headlines are not their fault. However, I do find the study rather strange.
Any thoughts on the hair salon control group, or on the study design in general?
My reply: Yeah, I have some thoughts!
Here’s the abstract of the article:
Injection phobia and dental anxiety can, in severe cases, lead to avoidance of necessary treatment. The aim of this pilot study was to investigate self-reported injection phobia and dental anxiety among individuals with tattoos and/or piercings. The Injection Phobia Scale-Anxiety (IPSA) short form and the Modified Dental Anxiety Scale (MDAS) questionnaires were applied. Both the total IPSA and MDAS scores were significantly higher for individuals with tattoos in comparison with a control group (P < 0.001), suggesting a need for anxiety-reducing measures and facilitated treatment for this group of patients.
It says right there that it’s a pilot study, in which case the convenience sampling is just fine. The purpose of a pilot study is to demonstrate the feasibility of an experiment, not to estimate the treatment effect. The mistake is to compute the significance test and, even more, that last bit on “suggesting a need for anxiety-reducing measures and facilitated treatment for this group of patients.” It’s a pilot study! The point is that you’ve demonstrated that you can collect these data. Now do a real study.
Also, later on, the authors write:
No previous report has indicated a higher degree of injection phobia and dental anxiety among tattooed and/or pierced individuals.
Not to get all Bayesian here, but when there’s no good prior theory supporting the hypothesis and no good prior evidence supporting the hypothesis, then it seems like a stretch to take this convenience sample and, from that, to draw general policy recommendations.
And, sure, the newspaper headlines are not the fault of the authors of the article—but the authors didn’t help any by drawing strong conclusions from this convenience sample.
Sean concluded his email as follows:
P.S. If you refer to this on your blog, I would appreciate it if you only used my first name.
It’s funny how many people want anonymity when posting here.
Many social studies are marked by a positive/negative dualism when in real life people have complex reactions to even simple situations. If we assume that all people can be sorted into two camps, needle accepting/needle avoiding, then it follows that tattooed people should be in the accepting camp, and their dental anxiety appears contradictory. In the real world, people sometimes seek out anxiety producing situations in order to discharge the anxiety. Various risk sports and scary movies are pretty popular. When you get a tattoo, you’re in charge of the needle which makes the situation tolerable. In the dentist’s chair, you are relatively powerless which makes it intolerable.
BTW, I’m not tatted. My tribe is against the practice; instead we amputate part of our genitalia which somehow is supposed to be sensible.
may I ask what’s wrong with this? they used a ranked sign test. Isn’t the ranked sign test recommended in these cases (small N?, etc)
Gabby:
As discussed in the post above, there’s nothing wrong with collecting data and reporting summary statistics; the problem is with making general conclusions from a pilot study. The purpose of the pilot study is to assess potential problems in data collection, not to draw inferences about the population.
Also: The N=100 sample is a convenience sample, which is fine for a pilot study, not so fine for drawing larger conclusions. N=100 is not a small sample for the purposes of the central limit theorem (so not “small N” in the sense that you are using in your comment), and in general a sign test is a bad idea because it throws away information. N=100 could well be a small sample for the purpose of estimating a small difference from noisy data, but that’s a separate question from what data summary to use.
Thanks. I thought is it N=43 for the two groups. Then they did a paired rank test.
This post makes me think about vaccine hesitancy. There’s a claim going around that some non-negligible share of vaccine hesitancy is due to fear of needles. And so, for instance, if we could develop a nasal spray vaccine, that could help increase take-up.
Tattooed people presumably have *less* of a fear of needles than otherwise similar non-tattooed people (ignoring this fake study, which posits the opposite sign; and ignoring the difficulty of defining the proper control group of “otherwise similar” people). Certainly not tattooed people who plan on getting more tattoos. So, I wonder how their vaccination rates are.
My first thought: why did the authors use the term pilot study? To indicate, that they do not have not strong prior about the effect they observed because some reviewer asked them? Obviously, they cite published observations of higher prevalence of other anxiety symptoms in tattooed individuals.. Do they really intend to replicate their finding?
My second thought: Just because you label some score “blood injury scale anxiety” does not mean, that you assess the corresponding construct.. Several years ago, my team published an evidence synthesis about cortisol responses to venepuncture. We found that the probability of cortisol responses to be larger when individuals have previously been exposed to venepuncture and argued that this is due to classical conditioning to stimulus features (another more recent review argued that hypocapnia is the corresponding unconditioned stimulus). In accordance with this line of thought, the authors find that median scores in the “1 tattoo” grp amounts to 4 whereas the “>1 tattoo” grp amounts to 14.
So to me, the result does not seem as surprising but Andrew might argue nonetheless, that these findings do not suffice to justify an advocacy prior.
Hi Andrew – I generally agree with your post but maybe you are reading too much into the abstract and its p-value. NB: I haven’t read the full article and I’m commenting only on what I see here.
> The purpose of a pilot study is to demonstrate the feasibility of an experiment, not to estimate the treatment effect.
Yes, but within reasonable limits in a pilot study you allow yourself to play with the data and the analysis, right? If you don’t get a shade of significance even under such circumstances then maybe you should invest your resources elsewhere and stamp the experiment as unfeasible.
> The mistake is to compute the significance test […] but the authors didn’t help any by drawing strong conclusions from this convenience sample.
Isn’t “mistake” a word too strong? After all a p-value is also a summary statistic of the data, I don’t see why it should be a mistake to report it. Following from the above I think it is useful to mention it to get a sense of what to expect. Also, the conclusions the authors draw don’t seem too strong to me (they use the word “suggest”) and they don’t recommend a general policy; they just imply that a proper study is warranted. I guess the main point is how meaningful it is to compare tattoo studios to hair saloons, but this is beyond statistical considerations.
—-
> It’s funny how many people want anonymity when posting here.
I don’t know… At a glance (no statistical significance!), it’s more or less the same as in other social places I look at, like the StackExchange sites or GitHub. I think many people don’t like to be under the spotlight and don’t want their name to be visible to google – I don’t think it implies fears of retaliation or an unhealthy system. Besides, the fact that posts here cannot be deleted or edited doesn’t help in the direction of openness?
Dario:
All things are possible, but the combination of an unexpected finding, an uncontrolled analysis, and a pilot study makes me doubt that this is telling us anything about the outside world. This all just seems too much like a recipe for noise mining and jumping at random patterns in nonrepresentative data.
I guess the right way to do this, if you’re interested in the topic, would be to design and construct a controlled study and then see what happens.