Wednesday, January 7, 2009

Neuropsychology Research is Lame


Admittedly, I'm biased. I've taken many cheap shots at psychiatry and, to a lesser extent, clinical psychology. However, my training is in neuropsychology, which is the study of the relations between brain function and behavior. I have yet to bite the hand that feeds me. Since it's a new year, I figured I'd begin by doing just that.

In the January 2009 issue Neuropsychology (1), an article titled "Exploring Effects of Type 2 Diabetes on Cognitive Functioning in Older Adults" was published. First, let's identify the assumption in this title: Type 2 diabetes affects cognitive functioning. The article is summarized over at Sciencedaily.com (2). As usual, the article begins with hyperbole, "Adults with diabetes experience a slowdown in several types of mental processing, which appears early in the disease and persists into old age, according to new research." Now, when I looked up the definition for the word "several," the definition I found was this, "more than two but fewer than many." In the actual study, of all the administered neuropsychological tests, only two statistically significant differences were found. According to my own new research, the opening sentence of the Sciencedaily article is nonsense.

However, there's more. The Sciencedaily article incorrectly summarizes the results, "Healthy adults performed significantly better than adults with diabetes on two of the five domains tested: executive functioning, with significant differences across four different tests, and speed, with significant differences or trends across five different tests. There were no significant differences on tests of episodic and semantic memory, verbal fluency, reaction time and perceptual speed."

Actually, out of the four tests of executive functioning, healthy controls performed significantly better than the diabetes group on one test only. Also, heed attention to this phrase, "...and speed [referring to processing speed], with significant differences or trends across five different tests. " [my emphasis]. Buyer beware when you come across the word "trend" in research articles. The term is not interchangeable with statistical significance. To say "significant differences or trends" is misleading. When a difference is statistically significant, we believe that it's likely the result didn't happen by chance. When that the difference was "close" to being statistically significant (i.e., it could have occurred by chance), we use to term trend instead. Trends do have heuristic value. They can identify areas that need further research or closer scrutiny; however, to include the phrase in a popular article is sloppy. It makes the results seem more significant than they are. It's akin to reporting response rates as opposed to remission rates in antidepressant research (e.g., 70% response rate, 25% remission rate). It's semantics, not science. In the actual study, only one test yielded significant results, not five.

What we have is a study that found two significant results (and 19 negative results), and the conclusion is that people with type 2 diabetes experience a slowdown in executive functioning and processing speed. There are a few problems that the researchers don't address. First, the more neuropsychological tests an individual is administered, the probability of having an abnormal test result increases. Second, all this study shows is that there were differences between two groups, not that the scores obtained were abnormal.

In neuropsychology, we categorize performance based on standard scores and percentiles. The line between normal and abnormal varies depending on the cut off point (typically a cut off of one standard deviation is used). Nowhere in this study do the authors state that the type 2 diabetes group had abnormal scores (i.e., below one standard deviation). They're just different from the control group. Also, the diabetes group had 41 subjects while the control group had 424 subjects. Again, we have a problem with external validity (3).

I would have an impromptu visit from a proctologist if a patient performed poorly on a single test and then I concluded that they had an impairment. The reason we give multiple tests for each domain of functioning (e.g., executive, language, visual-spatial, memory, somatosensory, processing speed, and olfaction), is to look for patterns of deficits, not just deficits. It's the patterns of deficits that lead us to a diagnosis.

Getting back to the actual study, the following domains were assessed: verbal memory (they incorrectly called this episodic memory, which is different), semantic memory (i.e., crystallized intelligence to you CHC fans out there), verbal fluency, executive functioning, and processing speed (they used the fancy term neurocognitive speed).

They improperly assessed memory, and the data they reported makes no sense. The results they listed deal with proactive and retroactive interference, which is source memory, not the ability to learn, encode, and to retain information.

They used a form of verbal fluency that isn't widely used clinically.

The tests they used to assess so-called executive functioning, the Hayling Sentence Completion Test and the Brixton Spatial Anticipation Test, don't even crack the top 40 tests used by neuropsychologists (4). The tests have only been around since 1997, and they are poorly normed.

The tests used to assess processing speed (excuse me, neurocognitive speed) are not used clinically.

I question whether these researchers know what they are doing. I understand that the more widely used neuropsychological instruments are expensive, but they still could have used tests that have clinical relevance (at least ones that crack the top 40). Secondly, identifying group differences without indicating if results are abnormal (e.g., mild deficit or moderate deficit) is clinically meaningless (in this case). I can administer IQ tests to high school graduates and college graduates. Will there be group differences. Very likely. Does that mean those differences are abnormal and require clinical intervention? NO! They're just different. Somehow, that didn't stop one of the co-authors from suggesting public health policy, "...public health programs could check the cognitive status of people with more advanced or severe cases [of diabetes]; ensure that diet and medications are effectively employed in all early diagnosed cases; and enact possible cognitive monitoring or training programs for people with diabetes." If they are normal, what's the point?

At least the last paragraph in the Sciencedaily article has shades of rationalism (I'm being ironic here), "Diabetes is a known risk factor for late-life neurodegenerative diseases such as Alzheimer's [current research suggests it doubles the risk]. Although the deficits [when were these identified as deficits? How about differences] detected in the current sample were not clinically significant [duh!], they appear (according to subsequent research by the authors [where's the reference to this research?]) to foreshadow additional deficits [differences]. Only further study would reveal whether it's possible to "connect the dots" between mild early deficits [differences] in speed and executive function, and later signs of a progressive cognitive impairment." Based on this study, many (as opposed to several) additional studies are needed.

So, getting back to the assumption implied by the article's title, does type 2 diabetes affect cognitive functioning? If this was the only article I read on the matter, I would have no idea.

2 comments:

Neuroskeptic said...

Somehow, that didn't stop one of the co-authors from suggesting public health policy, "...public health programs could check the cognitive status of people with more advanced or severe cases [of diabetes]; ensure that diet and medications are effectively employed in all early diagnosed cases; and enact possible cognitive monitoring or training programs for people with diabetes." If they are normal, what's the point?

to be fair, crap last-paragraph speculations about practical implications are a well-established part of medical research. I don't think anyone except journalists ever takes it seriously.

christine said...

I don't understand how that article was published