Thursday, November 4

IQ tests are flawed! But NOT due to bias...

Now don't get me wrong. IQ tests are a good measure of intelligence, and linked to many traits.

But there is a flaw with one part of some IQ tests (involving subjectivity). Because it involves subjectivity, they have no correct answer.  There may be an answer that is commonly provided by most intelligent people, but what about extremely intelligent people who may see things differently?

Have you ever seen a question ask something like: "which of the following five animals is most like a dog?".

It then lists 4 animals that don't have a tail, and one with a tail. The correct answer is deemed to be the animal with a tail.

But is the common tail the only thing in common between the animal and the dog? It's doubtful.  You can categorize an item in many ways: it's color, it's height, weight, average number of offspring, number of teeth, claws, average age of reproductive ability...there are many obscure ways to categorize something.  And who's to say which commonality is the correct one? It's subjective, you can't say.  Taking a poll might advance the process, but you can't do that while writing the test.

Perhaps the test writers do their best to minimize this problem and aim to select an animal with few obvious commonalities other than a lack of a tail.

But what if the test taker is a super genius that thinks on a completely different level and is able to determine several commonalities? (a super genius that is as rare as 1 in 10,000, not 1 in 100).  How is he to choose the correct answer? What if the test taker can think of a commonality that the test writer missed?

This is a flaw of some IQ tests.  Perhaps it's not a serious flaw.  The subjective section may still result in providing a relatively accurate score for most test takers, but it may actually reduce the score of high level geniuses who provide a correct answer that is marked incorrect because it wasn't the commonality the test writer was looking for.

15 comments:

  1. If someone is that smart then they would see both answers, see the answer that is correct by the designer of the test, answer the question correctly, and not be a d*** about it.

    ReplyDelete
  2. Actually

    if someone is that smart, it's very plausible that they would NOT see both answers. Because their mind thinks on a different level than the test writers, and such a person has little way to figure out what a less smart person would consider to be a smart answer.

    ReplyDelete
  3. Actually, someone that smart would be able to speculate upon what the test creator expects as the correct answer.
    Much as you or I or most people can.

    ReplyDelete
  4. Reader 23,

    its been speculated that someone is literally unable to understand the points made by someone who has an IQ 40+ points greater than them.

    This, among other reasons, could influence the inability of the smarter person to recognize the intended answer of the test writers. More on this below.

    Also, you have to remember that the test is a timed test.

    The quicker you move, the higher your score. One's initial inclination is to answer the question quickly.

    First, many super geniuses likely wouldn't even realize at the time that the test writer is likely to judge the correct answer to be one that is not the best available...I think most people assume that since test writers are smart, their answer is likely to be correct. I believe even super geniuses assume this out of familiarity, because through their lifetime they've likely taken hundreds of tests that seemed fair, all of which had no subjectivity (or had subjectivity that they could at least account for and get marked on, like written answers). The IQ test is different in that when you choose a subjective answer from the list of choices, you can't write the reasons why you selected it.

    Second, even if a super genius does realize that the test likely accepts a poorer answer as the correct one, the super genius is likely to become aware of this AFTER their initial inclination (which is to think of what they consider to the most logical answer).

    Because they spend the time to think of both their own answer and the correct but poorer answer, they take more time than they otherwise would, and hence it unfairly lowers their score, even if they choose the answer that's considered correct.

    And perhaps the main problem is this: Even IF the super genius is quickly aware that the test likely wants an answer that is poorer than the best answer he/she can think of, how does the super genius figure it out? This is the biggest barrier of all!

    For one, a subjective question would have SEVERAL different poorer answers than the one the super genius had in mind. How would the super genius know which to pick? (If the multiple choice question has 5 selections to choose from, the super genius could probably think of at least one way in which each item is different the the item it's being compared to).

    And remember, the super genius doesn't think on the level of others. That even further limits their ability to choose the correct answer from a list of poorer answers.

    And a super genius's mind is set up so that he/she acts very logically very quickly...it would be a complete change in the structure of thinking to try to think more illogically in order to come up with the poorer answer! I don't know that that could easily occur.

    ReplyDelete
  5. 100% agree with ya on this one. I am always over thinking stuff. I hate that term "over thinking". Your teacher say before a test, "don't over think". Is a test not a "test" on your knowledge of a topic, and the higher your thought process, the higher your score. Although this way of thinking would make it impossible to build an appropriate test....

    ReplyDelete
  6. I just looked over your IQ page and you are way over analyzing without data. I think we call that rambling. You say that mostly smart people are on facebook. Wrong, most losers without anything better to do are on facebook. You say that the smarter facebook fans are likly to take the test, wrong, my not so bright little sisters and there retarded friends are the one that take those test. The rest of use see how unimportant and a waste of time and do things like work.

    74 is 74 and that's not bad. Bad to be boosting on this blog about it is a sad way of life.

    ReplyDelete
  7. Lastly you are correct in one thing. Nobody will ever beat you in argument. You ramble on so much and over analyze everything to the extent were you confuse you opponents. At some point they have no idea what you are talking about because all the BS that is coming out.

    Of course you will just say we are on a different level than you and this is why we cant understand. Its a good tactic, my parents used it on me until I figured out they were bone heads and could easily be outsmarted.

    One thing I have learned about the art of SMARTNESS. Confusion makes you appear much smarter than you are.

    ReplyDelete
  8. Anonymous,

    You said, "You say that mostly smart people are on facebook." He was stating that the smart people on facebook were more likely to take the exam, not use facebook.

    NoSuch, has this: "its been speculated that someone is literally unable to understand the points made by someone who has an IQ 40+ points greater than them" been speculated to work backwards as well? As in, someone can't understand points made by someone 40+ points below them?

    ReplyDelete
  9. Not anonymous whatsoever

    Sun, 19 Dec 2010 4:59:00 pm

    anonymous,

    First of all, if you think you're so rational, then I suppose you would know that the term "retarded" is censored now on many networks because of the vulgarity of it. Yet you persist in your own ramblings, engaging in prolific use of the CAPS LOCK.

    Secondly, You're clearly ignorant of the sorts of people that make up the higher echelons of IQ scores. Many have some form of autism. Obviously, this would make it extremely difficult, given the deficits in Theory of Mind, to even begin to fathom what anyone other than themselves would believe the correct answer would be. (Well, I say obviously meaning it's obvious to anyone apart from you apparently)


    In other news not directed toward anonymous directly...

    Despite having an IQ north of 160, I find myself baffled by
    the very idea that one SHOULD have to guess what the answer is supposed to be, and not what it is (even though it's supposed to be a test of the taker's intellect, and not what they posit the general populace's intellect is).

    How is one expected to come up with the "intelligent" answer, when they're ALL unintelligent??


    shoe is to horn, as ankle is to:
    A)sleeve
    B)dirt
    C)neptune
    D)ankle

    Now, which one is the one that most people would think is correct?

    That's a bit harder now, isn't it?

    ReplyDelete
  10. I probably should mention that the psychologist that administered these IQ tests (all umpteen of them) actually DID take the time to ask why I would answer some a certain way or would leave them blank. I wouldn't fill in the "Not sure" dot, because I WAS sure, it just wasn't a choice.

    He adjusted accordingly. There's an instance when subjection was important, it just needed a scoring done by something CAPABLE of subjecting.

    ReplyDelete
  11. Reader 101,

    well said. More than once, I've answered similar questions as part of an IQ test, felt sure that I knew the answer, only to be shocked to see that my score was less than anticipated. It's one thing to score less than anticipated, it's another thing to score less than anticipated when you're expecting a score that's near perfect!

    I'm glad to see that a psychologist's test asked for an account of your reasoning. I have read that an IQ test administered by a Psych. is considered the mast valid IQ test, and I now see why.

    ReplyDelete
  12. Reader 100,

    the person that made the comment about the communication problems between two people with an IQ difference of 40+ points) was the smartest man in the USA. As I recall, he simply said that there was some type of problem related to difficulty understanding.

    However, I suspect that the problem is mostly a function of the less intelligent person being unable to comprehend what the more intelligent person is thinking. Myself, I often find that I'm able to quickly comprehend what the less intelligent person is getting at, as frustrating as it is sometimes (and I suppose it would be frustrating for a less intelligent person to often find themself corrected).

    What I find, myself, is that people who presumably score lower in IQ often tend to make requests or statements that don't provide complete instructions about how to handle, in addition to the basic request, a variety of related scenarios that could occur.

    I believe it's a result of the people who score lower 1) having fewer memories to recall 2) being slower in recalling the memories that they do have 3)being unable to analyse, well enough, the issues that are potentially related. For example, a boss might ask an employee to go do something and not provide information for that person to handle potential anticipatory situations..

    This is a SERIOUS problem. Yes, the more intelligent employee can ask the boss for a more comprehensive answer.

    However, I believe the main problem with this perception gap is this: One person could look at a political issue and see the basic issue itself but very few related externalities...a second person could see the issue and MANY related externalities...this can result in a situation where the less intelligent person makes a sound judgment based on the limited information they have, feeling that they have enough info, and actually believing that the more intelligent person is the one that must be illogical for believing what they do! (That is, until the more intelligent person explains it, and even then the lower IQ person might not understand fully).

    As a result, you have people absorbing the same information yet making different judgments about the information for no other reason then their minds' different levels of access to information.

    ReplyDelete
  13. So you argue that the test taker can notice something that the test writer would not. hence, the argument can be made that the test taker is smarter than the test writer. therefor, for an iq test to be successful, it would have to be written by the smartest person ever. Then you have a pandora's box of who the smartest person ever. We would need to test everyone in the world, but the test would be inaccurate because measuring intelligence based on a flawed test would be futile. An un-flawed test would have to pre-exist in order to accurately measure intelligence to find a person to write an un-flawed test. That there is the catch 22 with this argument. Then people would have to be tested constantly in order to ensure that a new smartest person hasn't come along. But this can be circumvented by having a test written by computer. I apologize for rambling

    ReplyDelete
  14. Mike,

    you are a very intelligent person, judging by the thought sequences that you created starting with the initial thought.

    An interesting series of thoughts flower from your argument that:

    "therefor, for an iq test to be successful, it would have to be written by the smartest person ever."

    In terms of the ramifications of the test flaws that I pointed out, I would argue that perhaps we should consider eliminating the subjective section completely, or at least weigh the benefits vs the costs of (assumed) underreporting of very intelligent people and more fair reporting of all others' scores.

    If one decides to do away with the subjective portion of the test, one could simply use the test to more ably locate the very smartest people...but then those same smartest people are no longer needed to write the test, since there is no longer a subjective section!

    It's interesting to follow sequences.

    What exactly are the advantages that you claim a computer would enjoy?

    ReplyDelete
  15. IQ is not much more than a measure of how well someone can take an IQ test. Generally all IQ tests ask the same types of questions, but each time with different pronouns or numbers or patterns. It's like calculating simple arithmetics--the first time, you might struggle with the symbols, and you'll make lots of errors, but eventually you'll figure out how it all works, and you'll be able to do calculations more accurately. This is not intelligence; it's skill, a product of practice.

    ReplyDelete