Myths
Products - Psychological testing - Myths
Psychometric testing – the myths and realities |
| Author: Dr Paul Englert, OPRA consulting NZ |
| Various companies who distribute psychometric tools have used a range of 'sales techniques' to improve the perceived value of their products. Many of these techniques omit or misrepresent the true issues that end-users, such as HR professionals, should consider when deciding on one test suite over another. This article uncovers the sales techniques, excessive costs and faulty logic that are all 'tricks-of-the trade' in the assessment testing industry. It provides HR professionals with information needed to make sound decisions on the use of assessment tests in their organisation. In exposing myths, unfortunately one cannot escape some critical comment. We hope that these are balanced, however, by the positive suggestions we advocate for ourselves and other providers of psychometric tools. Read on and form your own views as to which of these myths you are prepared to challenge by wider research of your own. We can but raise the issues for you. As a test provider ourselves, we cannot claim independence. Our sales technique is sharing knowledge and education. |
| Myth 1: Good quality psychometric tools must be expensive. |
| Reality: Psychometric tools are expensive due to the absence of competition in the marketplace. This myth resides in monopoly history. As is the case whenever there is a monopoly, a false baseline of cost is created in the absence of competition. This is evident in Australia's history prior to deregulation in sectors such as energy or telecommunications. The monopoly player sets a price claiming this to be fair and a measure of quality in the absence of any true benchmark for their commodity. Consumers regard this as the standard price. There is no reason why psychometric tools need to be expensive. Once developed, the overhead for assessment testing is not large and the cost of ongoing development requires only time and the employment of suitably skilled individuals to continue research. There are options in this area to create huge or modest overhead cost structures. If we look beyond Australia at more competitive markets, the costs for developing psychometric tools are recouped primarily by longevity and usage volumes, not high price structures. Any company whose prices are well and truly above that of the competition is likely to have been a monopoly player in the past. It will continue to play from this position until competition slowly erodes the market advantage it had from being first off the block. |
| Harbour Future Leaders' (HFL's) position: We aim to make our tools among the most cost-effective on the market. We believe there is no reason why clients should pay high set-up costs, high annual licence fees or excessive on-going costs. |
| Myth 2: Being a dominant test provider means they are the best. |
| Reality: Dominance has come from being first. Dominance does not in itself guarantee the best quality. Dominance again usually arises from one player being the first to the market. This advantage means that one could swamp the market and subsequent competitors have to overcome two interrelated obstacles to consumer behaviour: inertia against change and market share. Psychological testing is like banking, people change their provider only rarely. The customer will complain about the costs, level of service and hidden fees but in reality it just seems too much trouble to change. Potential new customers, however, do not have to overcome a resistance to change and are more open to decisions based on merit and value. Market share is always a ploy to sell a product and works on the philosophy of "If it is good enough for them then it is good enough for me." We are social beings, and people inherently have difficulty making decisions on their own! In this way an early test provider becomes synonymous with psychometric testing, further ingraining the myth of 'first is best'. |
| HFL's position: HFL believes that resistance to change will diminish over time as people become more aware of the issues in psychometric testing and the options that are available to them. This is evident in our own growth. |
| Myth 3: Predictive validity studies demonstrate the usefulness of a tool for all organisations. |
| Reality: Predictive validity studies primarily demonstrate the usefulness of a tool for the particular organisation in which they were conducted. Meta-analyses are the most powerful demonstration of general usefulness. Some test providers have played on this by giving people only enough knowledge to make poor decisions. Nowhere is this more prevalent than in the issue of predictive validity. Predictive validity refers to the ability of a psychometric tool to predict some future measure such as job performance. The question is: "How do we use predictive validity to assess the usefulness of a tool for our organisation?" Test providers usually brush over that question. Predictive validity is often defined by a correlation (relationship) between one variable, say a score on a verbal reasoning test and another variable, say job performance. Some consumers know that a coefficient of say, 0.25 and above is deemed useful. How they intend to use this magical figure to improve decision-making in their organisation is less certain. In reality, a personality measure may account for nine per cent of the difference between individuals' job performance. However, before you start throwing out psychometric testing, remember that interviews predict roughly the same amount of variance of job performance. The following quote from an informed test user demonstrates what predictive validity should be used for: "Predictive validities shown by test producers are only good for demonstrating that the test has some utility for predicting work in general. It is a starting point not an end point for HR professionals. It provides the necessary support to then trial the tool in your organisation to demonstrate its predictive validity for you". To demonstrate the usefulness of a test, it is important to evaluate it against your own criteria, rather than blindly accept the validity studies produced inside other organisations that may have little similarity to your own. Some test producers use dubious predictive statistics primarily because the people they are selling to are not statisticians and therefore have difficulty determining the value of the information they are reading. Issues with respect to sampling error, the application of significance testing and the promulgation of misleading correlation data all contribute to the weakening of predictive data for any one organisation. To ensure predictive validity studies apply to your organisation, you need to: 1. Check that a test has some relationship to performance. This is a starting point, like a tick on a checklist. To be of value, a test used for selection must predict the world of work. Also, check any meta-analysis research that combines single tests into joint studies. Does your test relate to the types of tests that were used in these studies? IF SO WE CAN INFER A DEGREE of predictive validity for the tool, given it has been constructed well. 2. Develop your own inhouse study. This can be done by using a testing system that stores your data and enables it to be correlated with job performance. 3. Develop a multiple regression equation. This will give you weightings to apply to test scores so that you can make predictions of performance. 4. Plug the test scores for one individual into the regression equations that come out and you will have a useful guide for your selection. All of this, by the way, can be done inhouse in less that five minutes with the appropriate assessment software that allows you to make strategic rather than transactional decisions on the basis of assessment data. |
| HFL's position: As a standard part of the GeneSys software package distributed by HFL, clients automatically have access to the capability to hold assessment data so that inhouse validity studies can be completed. |
| Myth 4: Psychometric testing is a transactional service. |
| Reality: Psychometric testing is a strategic initiative. The value of psychometric testing comes primarily from having collected a lot of data and then using that data to model performance. Test providers have done a disservice to HR professionals by not giving them the capability to collect their own candidate data for later use. In real terms, this means that most organisations have little empirical data about what predicts performance inside their organisation – despite paying thousands of dollars for assessment testing services over many years. This is an injustice. It is equivalent to having a cash register that only holds the last transaction and does not feed a database of sales over the last month, or year etc. How useful is that? |
| HFL's position: We have built our business on making psychometric testing non-transactional and giving the client the capability to collect assessment data on potential as well as existing staff. |
| Myth 5: Psychometric testing is the domain of psychologists whose main interest is in furthering the discipline of psychology. |
| Reality: Psychometric testing is too often the domain of non-psychologist business people interested solely in making a profit. In recent years, we have seen an infiltration of some profiteers into the testing industry. Their mission is simple – "How can we make as much money as possible out of psychometric testing?" They may be funded not by psychologists but by capital venture firms and large corporate backers. These stakeholders know little about testing, and while they may get a few quick wins on the board, they tend not to translate their investment into long-term returns. Indeed, the psychologists in those teams may even be the casualties. Issues of psychometric robustness are slowly being eroded as some people strive for any way possible to make a dollar. This has resulted in tests being supplied over the Internet to anyone, without suitable training, and tests being produced without any validation data available. |
| HFL's position: We are a business owned by psychologists who seek to bring you robust solutions that conform to the global body of research into psychometric assessment. |
| Myth 6: Good psychometric tests are made by psychologists. Reality: Good psychometric tests are made by psychometricians. Only the term 'registered psychologist' indicates a person whose actions are governed by the psychologist act. Moreover, despite the fact that there are many types of psychologists all with their different skill sets, psychologists will often jump field. Thus, psychologists qualified in one field can switch to occupational psychology and HR without retraining or any indepth induction to fully understand the scope of the occupational field. They can do this because the legislation still does not differentiate between types of psychologists. It is outside of the Australian Psychology Society ethics but not all so-called 'psychologists' are members of the society or are registered! The people best qualified to make psychometric tools are psychometricians (i.e. those people whose skill set is around the measurement of psychological phenomena). Psychologists use the tools but the making of them belongs in the hands of psychometricians, as they have the necessary skills to make meaningful measurement possible. |
| HFL's position: We source and supply psychometric assessment tools developed by psychometricians. Our project team always includes a registered psychologist as a final point of accountability. |
| Myth 7: Putting a test on the Internet is difficult and that is why few people offer it. |
| Reality: Putting a test up on the Internet is easy. People restrict its use for ethical and reliability reasons. The Internet provides convenience in many situations and has obvious popular appeal today. Some Internet test providers have prospered in the e-commerce environment and some have not. A warning close to home is the example of an New Zealand company whose share price has plummeted from around 50c at listing to as low as 1c. Indeed in its last half-yearly report it indicated a loss of $1.02 million. Psychometric test interpretation relies on standardised testing conditions. Instruments used for the purpose of self-assessment by the individual have been relatively easy to mount on the Internet (e.g. career or well-being assessments). In staff selection, however, standardisation means that a more controlled testing environment is required and a range of other factors need to be addressed, such as the identity of the respondent and the like. Other issues such as restricted access and data storage also need to be addressed. If one wishes to lift testing from a transactional to a strategic level, authorised access to test data for statistical analysis is a priority. These are but a few of the issues that need to be addressed for reliable, valid, ethical psychometric testing over the Internet. That is why discerning providers have been careful and deliberate in their preparations and processes before making psychometric tests available on the Internet. In this field, there has been a benefit in learning from the misfortune of those perhaps too quick off the mark. |
| HFL's position: Psytech International (developers of the GeneSys range of tests) has had the capability of delivering Internet-based tools for some time, but have held back general release until all the key issues were uncovered and solutions developed. Our position is that the Internet is merely another delivery tool and we will offer Internet-based 'testing' only in circumstances where one can define the value of the data coming back. |
| Myth 8: People have a 'work personality'. |
| Reality: A mainstream personality questionnaire is just as valid. The concept of personality transcends such artificial boundaries as our work or home life. The flexibility to adapt one's behaviour in different situations or not is merely one aspect of personality, and it is not necessary to create a whole new personality questionnaire to cover it. That is done more for plausibility rather than reliability. Meta-analysis is the standard in independent research activity and meta-analysis has found that a test which provides good measures of the 'big five' personality traits does predict performance regardless of the setting. This finding is based on the analysis of patterns in the published research in a whole field, rather than on the research of one entity. |
| HFL's position: Tests distributed by HFL are derived from psychological models that are well supported by rigorous, independent research. We offer the choice of personality tools that are underpinned by the 'big 5' model of personality as well as a 'work personality' profile. |
| Myth 9: It doesn't matter how a tool is constructed. |
| Reality: The effectiveness of a tool depends primarily on how well it has been constructed. Factor analysis is generally regarded as the most robust statistical process for ensuring the rigour of a psychometric tool. In particular, it allows us to find dimensions of personality that are distinct from each other and to identify questionnaire items that do not overlap. A fundamental flaw with tests which are not factor analysed is that they tend to be excessively long and claim to measure multiple, independent, dimensions of personality – sometimes as many as 20 - 30 traits. In reality, there is likely to be a high degree of overlap between the scales inside these assessment tools. |
| HFL's position: HFL tests are built around broadly accepted psychometric theory. This means that there is a strong reliance on the underlying factor structure that supports the tool. |
| Myth 10: Research material should only be given to current test users. |
| Reality: Research material should be made available to everyone to both further the worldwide knowledge base and to allow for informed consumer decisions. One of the strongest indicators of a test publisher's lack of commitment to psychology is a closed-door policy with regards to accessing technical manuals and/or independent research that supports their tools. If this is made available at all, it tends to be to clients who have already bought into their marketing hype. In this way, technical information is not available for people to make informed decisions, but rather is used to preach to the converted. Moreover, it contravenes the international call for open-access to research. You can read more about this topic at: http://www.soros.org/openaccess. |
| HFL's position: HFL does, and will continue to make the research for their tools publicly available. |
| Myth 11: Ipsative tests are good for making selection decisions. |
| Reality: Ipsative tests have been criticised by psychometricians as being inappropriate for use in selection. Ipsative tests are tools that use a forced choice option and require a person to select from a range of traits the one that is most and least like themselves. Non-Ipsative tests require a person to respond to a question by indicating their preference on say, a five or three-point scale. Put simply, Ipsative tests allow the comparison of relative strengths of personality traits within an individual but do not allow comparison between people. The dilemma here of course is that selection is all about making comparisons between people. Ipsative tests have numerous other problems including: |
|
| As a result of these and other issues, the use of Ipsative tools for selection has been discouraged by the British Psychological Society. Instead, Ipsative tests are of more benefit in the area of individual counselling such as career guidance. The second problem is that Ipsative tests force the spread of a person's scores and while this may make interpretation easier it does not make it any more valid. It is a classic case of people being misled by false logic and simplistic marketing. Indeed, as discussed above, it is the lack of validity with respect to making comparative judgement that is the key problem with Ipsative tools. The public misperception about the merits of Ipsative tools in selection was fostered by publishers some time ago and it has not yet left the marketplace. I quote from Johnson, Wood and Blinkhorn (1988) "…publishers and promoters of these (Ipsative tests) are either unaware of, or do not understand, or choose to ignore their limitations". For those seriously interested in exploring the misuse of Ipsative personality tools we strongly recommended one of the most prominent articles in the area (Blinkhorn, S.F., Johnson, C.E., & Wood, R.F. (1988). Spuriouser and Spuriouser: The use of Ipsative personality tests. Journal of Occupational Psychology, 61, 153-162). |
| HFL's position: HFL does not distribute Ipsative tools for selection. Moreover, HFL's training courses cover issues related to Ipsative tools explaining the relative strengths and weakness related to each type of test construction. |
| Myth 12: Psychometric tools should only be interpreted by a psychologist. |
| Reality: Psychometric tools can be interpreted by anyone who has had the relevant training. Psychometric tools are built to be interpreted in a standardised way. This is why computer-based reporting is not only valid but may be less prone to error than human generated reports. A computer report will always provide a similar reading based on a person's scores and is not impacted by any potential errors such as political bias or stereotyping. Computer programmes are not, however, sophisticated enough to make all the required integrations between individual scales but provide a fair first draft of a report. Human input is required in order to make some of the more subtle scale interactions. The irony is that psychometric tools are easier to interpret than an employment interview. Moreover, just because someone is a psychologist it does not mean that they have had the training in psychometric tools required to utilise them to their potential. |
| HFL's position: HFL does not believe that psychometric tools are solely the domain of psychologists, yet will only distribute psychometric tests to people who have the necessary training and/or academic qualifications to ensure proper interpretation. |
| Myth 13: If tests are objective anyone can interpret them and therefore training is unnecessary. |
| Reality: You need to be trained to make psychometric tools really useful. The rationale for training or ensuring certain skills is summarised as follows: 1. Ethical: Psychometric tools are used on real people to make decisions that effect their lives. People who complete these tests provide a lot of information about themselves and they are entitled to receive feedback on their results. Skills are needed to do this constructively. 2. Standardisation: As discussed earlier, the usefulness of psychometric tools is that they are administered in a standardised manner so that valid comparisons can be made. This requires some education and training. 3. Legal: Selection can be a litigious activity. It is therefore vital that best practice protocol is rigorously followed in the administration and interpretation of psychological tests as well as the delivery of candidate feedback. 4. Utility: Training teaches people how to maximise the usefulness of test data. In particular, training teaches people how to relate the information from each tool to competency models, selection decisions and people development. 5. Psychological and human resource guidelines: Given points 1-4, is it understandable why the Australia Psychological Society stipulates that tests should only be made available to trained users. While training is of value, it should not be training simply for training's sake, nor a mechanism for extracting excessive revenue. It is a matter of providing sufficient training to use the tools provided properly, without overkill. |
| HFL's position: HFL provides comprehensive training for all test users that has been prepared along British Psychological Society guidelines with the material adapted for the Australia market. HFL recognises relevant qualifications and the training offered by other test providers and will accept this on a case-by-case basis. HFL offers registration training which covers both the general principles of psychometrics as well as interpretation training in each of our major test products. Advanced training is available for individuals who want to learn more about a specific product or to develop their knowledge beyond that of the average user. The latter is not compulsory. |
| Myth 14: The size of a norm group is often promoted as the most important norm criteria. |
| Reality: The relevance and distribution is often the most important norm criteria. When a person is compared to a norm group, the size of that group has little relevance to the comparisons that can be made. Instead, it is the relevance of a norm group to your own organisation and the distribution (or spread) of scores inside that norm that is of critical importance. If a norm group is not reflective of the people you are testing and does not differentiate people well, there is NO benefit in knowing that a test publisher has collected norm groups of several thousand people. If a norm group is well built and is found to be reflective of the population of interest, a norm grouping of 200 people may be sufficient! One of the key issues with norm grouping is that we must be comparing like with like. Thus, the people inside a norm group must have sat the test in the same conditions and come from like ethnic and educational backgrounds. Only if the conditions of Internet testing are sufficiently controlled do they achieve sufficient standardisation to allow the building of robust norm categories. Many Internet tools do not. Beyond these criteria, a test publisher should also be able to confirm the gender mix of participants and what roles people were tested for. All of these factors will help define the suitability of one particular comparison group over another. As a final note, the most relevant norm comparison group will often be a representative cross section of in-house staff who perform each of the role(s) under review. |
| HFL's position: HFL updates GeneSys norms regularly. These are provided free to all users and can be broken down according to the bio data that is collected on each candidate who completes the assessment tools. The GeneSys system also allows test users to create in-house norms in their own system and to examine the make-up of each norm group so as to make an informed decision about its appropriateness to them. |
| Myth 15: There needs to be an additional charge for reporting. |
| Reality: You need only be charged once for testing. Test producers have looked at various means of extracting additional money from client organisations. This includes everything from having an organisation pay each time a candidate sits a test to every time a report is generated or both. In reality, once test data is inputted into a scoring system, no additional time commitment is required for a report to be automatically generated. If clients do not wish to incur the often significant reporting charges, they will opt to rely solely on verbal feedback to a candidate or recruiting manager (using just a profile chart) or they may prepare short-form, less comprehensive reports by hand for distribution. |
| HFL's position: HFL users are only charged ONCE to put a person's data into the GeneSys system. Once the data has been entered or imported, there is no charge to produce a report, analyse the data, or develop in-house norms. We do not subscribe to the practice of charging at each step of the testing process, relying instead on the increased usage that comes from the distribution of tests at a very cost-effective price. For information on the GeneSys software mentioned in this article please click on the following links: |
| www.genesys3.com.au and www.psytech.co.uk |
