In 1993, Marilyn Harvey, a nurse, complained to Memorial University that her boss, world-famous Order-of-Canada scientist Dr. Ranjit Chandra, could not have done the research he claimed. A very courageous thing to do. After an investigation, Memorial did not agree. Harvey recently sued Memorial. From her complaint:

The Plaintiff [Harvey] says that the Defendant [Memorial] defamed her by taking actions which . . . caused her to be isolated, shunned, and humiliated through the following:
(a) representing to the community that her complaint was unjustified;
(b) misconducting the investigation of the complaint;
(c) misleading the research community as to the reasons for discontinuing the investigation;
(d) choosing not to conduct another investigation;
(e) misleading the Plaintiff as to the reasons for discontinuing the investigation;
(f) acquiescing in and adopting [?] the actions of Dr. Chandra when he sued her for theft of research data; and by its conduct giving the Plaintiff and the public the impression that it believed the allegations of theft to be true;
(g) treating the Plaintiff in a manner as to imply to her and the university and the healthcare communities, and the public, that her complaint was unjustified;
(h) acquiescing in and adopting statements of Dr. Chandra which impugned the Plaintiff’s motives and integrity.
The overall effect of the conduct of the Defendant was to constitute a communication to the community, and to the research and hospital community in particular, that was profoundly defamatory. . . It expose[d] her to contempt, ridicule and marginalization and [caused her] to be viewed by co-workers as a troublemaker and a pariah who could have a detrimental effect on one’s career if she were not avoided.

Memorial’s defense.

4. You write “There are multiple problems with the validity of existing EI tests that make them difficult to interpret, and make claims based on them highly suspect.” What are the main problems?

Many early tests designed to measure emotional intelligence, and some still in use today (including one developed in part by the journalist Daniel Goleman, who popularized the term emotional intelligence in his 1995 book), used self-report methods and treated the construct as a conglomerate of various desirable personality and motivational factors, such as optimism, contentiousness, happiness, and friendliness. Tests of this nature may be interesting and may predict important outcomes, but emotional intelligence, defined in this way, is really just a repackaging of old ideas. These tests are so highly correlated with traditional measures of personality as to be operationally indistinct from them. Additionally, calling this construct emotional *intelligence* is suspect: personality and intelligence are generally regarded as very different things, and assessing intelligence
through self-report is generally considered inadequate.

The MEIS and the MSCEIT, to which I referred earlier, are, to my knowledge, the only two currently published tests that assess emotional intelligence as an intelligence. These tests ask respondents to engage in a variety of tasks, such as looking at pictures of people’s faces and reading stories about human interactions, and then make judgments about the emotional content of those stimuli. These tests are a step in the right direction, but have their own problems.

The test developers have a rather odd way to score the responses people give to the stimuli on the tests. The tests were administered to a large (N=2000+) standardization sample, and the scores people are now given on the items are the percentage of people from that standardization sample who chose that alternative. In other words, if you select choice “c” on an item, and 67% of people from the standardization sample also chose “c”, then you get a .67 for that item. If you chose “d”, and only 11% of
people chose “d”, then you only get .11 for that item. Your total score is simply the sum of the weighted scores from each item.

As odd as this method of scoring may sound, it has been used in other situations where the underlying theory is not well understood (see Legree, below, for an exposition of this). However, it presents difficulties here: it defines correctness as, essentially, conformity of opinion with the standardization sample. In other words, what is actually being measured may not be “intelligence”, but rather, simply, normality or popularity of opinion: the highest-scoring respondents will simply be those who most consistently choose the responses most other people also select. Additionally, this prohibits the existence of items so difficult that most people get them wrong, such as a very subtle facial expression that only the most emotionally astute could correctly
parse: if there were any such items, the astute minority would be penalized for choosing the less-popular but more-correct alternative. This is a serious challenge to the construct validity of the test.

Additionally, the internal structure of the test itself is suspect. The test developers posit a four-factor model of emotional intelligence (the four factors being the ability to perceive emotions, the ability to allow emotions to facilitate thought, understanding emotions, and managing emotions) and have branches of the tests designed to measure all four of those factors. They have published confirmatory factor analyses that they claim support their theory; however, re-analyses of their tests, and new analyses (including one that I am conducting now) have not been able to replicate their results, calling into question the internal validity and reliability of the tests.

In my dissertation, which I can make available early next spring, I discuss all these points in greater detail.

Part 1.

References

References

Legree, P. J., Psotka, J., Tremble, T., & Bourne, D. R. (2005). Using consensus based measurement to assess emotional intelligence. In R. Schulze & R. D. Roberts (Eds.), Emotional intelligence: An international handbook (pp. 155–179). Cambridge, MA: Hogrefe & Huber.

MacCann, C., Matthews, G., Zeidner, M, & Roberts, R. D. (2003). Psychological assessment of emotional intelligence: A review of self-report and performance-based testing. International Journal of Organizational Analysis, 11, 247-274.

Mayer, J.D., Salovey, P., & Caruso, D.R. (2004). Emotional Intelligence: Theory, Findings, and Implications. Psychological Inquiry, 3, 197-215.

Mayer, J.D., Salovey, P., Caruso, D.R., & Sitarenios, G. (2001). Emotional intelligence as a standard intelligence. Emotion, 1, 232-242.

McCrae, R.R. (2000). Emotional intelligence from the perspective of the five-factor model of personality. In R. Bar-On & J.D.A. Parker (Eds.), Handbook of Emotional Intelligence (pp.92-117). San Francisco, CA: Jossey-Bass.

O’Sullivan, M. (2005) Trolling for trout, trawling for tuna: The methodological morass in measuring emotional intelligence. In press.

Roberts, R., Schulze, R., O’Brien, K., Reid, J., MacCann, C., & Maul., A. (2006). Exploring the Validity of the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT) with Established Emotions Measures. Emotion, 6(4), 663-669.

Roberts, R. D., Zeidner, M., & Matthews, G. (2001). Does emotional intelligence meet traditional standards for an intelligence? Some new data and conclusions. Emotion, 1, 196-231.

Andy Maul, who took introductory psychology with me, is a graduate student in Educational Psychology at UC Berkeley.

1. What is your research about?

I’m taking a closer look at tests recently developed to measure the construct of emotional intelligence (EI). In particular, I’m looking at the Multifactor Emotional Intelligence Scale (MEIS) and the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT), which were both developed in the past decade and evaluated using traditional methods (confirmatory factor analysis [CFA] and classical test statistics such as alpha coefficients, along with correlations with other tests and hypothesized outcomes). I’m looking at these tests again, both though the traditional lens of CFA and through the newer lens of Item Response Theory (IRT). In the end, I hope to make points both for the development of EI tests, and for psychological measurement in general, by highlighting how newer methods can improve the construct- and test-building process.

2. How did you get interested in this line of research?

I became interested in emotions by working with Professor Dacher Keltner. At some point in graduate school my interests shifted to the more quantitative side of research, and I’ve since been working with Professor Mark Wilson on test theory and statistical measurement. I thought combining the two interests, by evaluating tests of emotional intelligence through a quantitative lens, would be a good idea.

3. What’s an example of research that shows the value of measuring emotional intelligence?

The MSCEIT appears to predict some life outcomes (such as grades, prosocial behavior, and self-reported life satisfaction), even controlling for IQ and personality. Other researchers have challenged these claims as being premature and based on insufficient evidence. There are multiple problems with the validity of existing EI tests that make them difficult to interpret, and make claims based on them highly suspect.

Some researchers feel that defining and measuring emotional intelligence could clarify and expand our definitions of intelligence and cognitive abilities in general, and provide information about an area of human functioning that could predict important personal and interpersonal outcomes (such as life satisfaction and the quality of one’s relationships) above and beyond traditionally-measured intelligence and personality. In today’s era of high-stakes testing, with so much riding on what many feel to be tests with limited utility, a new, well-validated test of emotional intelligence could provide insight into what makes students successful in schools and in life.

References

Mayer, J., Salovey, P., & Caruso, D. (2002). Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT): User’s manual. Toronto, Canada: Multi-Health Systems.

Mayer, J., Salovey, P., Caruso, D., & Sitatenios, G. (2003). Measuring emotional intelligence with the MSCEIT V2.0. Emotion, 3, 97-105.

O’Sullivan, M. (2005) Trolling for trout, trawling for tuna: The methodological morass in measuring emotional intelligence. In press.

Palmer, B., Gignac, G., Manocha, R., & Stough, C. (2005). A psychometric evaluation of the Mayer-Salovey-Caruso Emotional Intelligence Test Version 2.0. Intelligence, 33, 285-305.

Roberts, R. D., Schulze, R., Zeidner, M., & Matthews, G. (2005). Understanding, measuring, and applying emotional intelligence: What have we learned? What have we missed? In R. Schulze & R. D. Roberts (Eds.), Emotional intelligence: An international handbook (311—341). Cambridge, MA: Hogrefe & Huber.