Article

“Testing the Test” A Consumers’ Guide to Internet-based Employment Testing
by Dr. Larry Craft

No matter where you look today, whether it’s the business journals or magazines, you’ll notice the proliferation of articles relating to employment tests. It’s a hot topic that’s been fueled by a significant increase in testing that began in the early 1990s. There are two primary reasons for the new interest in an old idea.

Dr. Larry Craft has provided his employment tests to business clients for over 30 years. You can try his 3rd generation test, the CraftMetrics Personality Inventory (CMPI) for free. Call Dr. Craft personally at (941) 405-2893 or email drllcraft@craftmetrics.com, or fill in our online form to receive your free trial of the CMPI.

First, employers are realizing that empirically normed, objective assessments protect them from EEOC litigation. It is now generally accepted that managers who fly by the seat of their pants or use gut feelings to make their hiring decisions are placing themselves in harm’s way, especially in today’s litigious climate. A few years ago, a manager from MetLife came up to me after a training workshop and thanked me enthusiastically for developing their “MetSelect” program, which included both testing and structured interview questions. A short time after he had begun using our system, one of his applicants filed charges against him with the EEOC for discrimination. In the preliminary request for information, he simply presented his guide notes and empirically based test validity, and the case was dismissed within a few weeks.
 
Second, the Internet has automated the application and testing process to the point that managers can make a hiring decision in half the time with less than half the effort. Applicants see a job opening on an Internet job board (CareerBuilder or Monster) and within 30 minutes are able to fill out the application for employment and take two or three employment tests that stack-rank them for immediate review by hiring managers. The managers simply go into their special “administrator’s portal” and then contact the top-ranked candidates for their structured interview. Managers appreciate the system because it saves them countless hours reviewing résumés and interviewing unqualified candidates.
 
Last, but not least, Internet-based employment testing is now so convenient and inexpensive that employers are using it to more effectively motivate and develop existing employees. Whether it’s 3600 feedback or Internet-based attitude surveys, employment testing personalizes the management process and allows managers to target specific needs of individual employees.
 
With the blessing comes the curse. Employers are often overwhelmed with the possible choices. In the 1970s there were only a few personality tests that dominated the industry. Today, there are literally hundreds to choose from. But how do you know which is best? Before you buy any employment test, ask yourself the following questions.
 
What dimensions of success do I need to measure?
 
You can define every job description by the importance of one or more of the three dimensions of success: the Can Do, the Will Do, and the Follow-through. Remember that each of these dimensions requires a different assessment tool.
 
To measure your candidates’ Can Do (skills, knowledge, and experience), you’ll need to rely on an industry-specific aptitude test. If your culture is unique, you may even need to customize it and add a few questions of your own in the form of a pre-screener. Job-related skills, job experience, previous income, and learning aptitude (intelligence) are objectively assessed in this type of questionnaire. These tests are most applicable when job experience and measurable job skills are essential for success.
 
CraftMetrics’ Bio Data pre-screener has been used thousands of times by employers who have too many applicants and too little time. Prior to meeting the manager or formally applying for the job, the applicant is directed from the job board or local office to go to a special company Web site that asks 20-30 job-related questions such as the following:
 

When our company contacts colleges, universities and schools you have attended, they will verify that you have:
 
( ) A formal Graduate Degree (completed/awarded or actively enrolled)
( ) A formal Bachelor’s degree (completed/awarded or actively enrolled)
( ) A formal Professional Degree or certification completed/awarded
( ) A formal High School Diploma or equivalency degree completed/awarded
 
Presently, you are mostly motivated by your need for (select only one of the following):
 
( ) Service: Helping others to reach their goals.
( ) Independence: Being master of my own destiny; setting my own work schedule.
( ) Security: Having a position/job upon which I can depend in the future.
( ) Money: Earning the income I need so I can become financially independent as soon as possible.
( ) Relationships: Being/Sharing with people I enjoy and respect.
( ) Mastering a Skill: Being a professional who is highly respected in my community.
 
You have had full-time employment with the following number of organizations over the past ten years:
 
( ) One
( ) Two
( ) Three
( ) Four or more
( ) I have never had full-time employment
 
In the past five years, you have participated in the following formal sales education/training (courses, seminars, schooling, on-the-job, etc):
 
( ) None, I have not participated in any
( ) 1 to 3
( ) 4 to 6
( ) 7 or more
 
Our Sales Representatives typically spend time at home each evening to organize, plan and complete business related work. Would you be willing to commit to that schedule?
 
( ) Yes, I can commit to that type of work schedule.
( ) No, I would find it difficult to commit to that type of schedule.
( ) I am not sure if I could commit.

 

You can imagine the amount of time that can be saved by a well-constructed bio-data pre-screener. Only Can Do qualified candidates will be invited to the initial interview. In addition, employers can quantify the assessment by awarding points for the best responses. At the end, the total points determine if the applicant proceeds. This helps to eliminate the “halo effect” that negatively impacts the interviewer’s objectivity.

To measure your candidates’ Will Do (personality and motivation), you’ll need a personality test. Since you can’t customize a personality test, select the test that has the most experience (and research and validity) for your specific industry. CraftMetrics’ CMPI has the most research and experience in sales positions, but its core personality traits allow it to be used in all positions in which personality is related to employee productivity. Traits such as drive, confidence, sociability, trust, intuition, the need to serve, and optimism are important not only in sales, but most other positions also.
 
Just as valuable are the interpretive reports that are based on the results of the test. Most assessment companies offer instantly available Internet-based reports that are used for personalized motivation, training, and development. CraftMetrics offers the following categories of reports:
 
CMPI/CLAS Hiring and Promotion Charts/Reports
CMPI Coaching/Training Reports
CMPI Seminar/Workshop Hand-outs
CMPI CultureStyle Group/Team Analysis Report

 
To measure your candidates’ Follow-through (work ethic/self-discipline), you’ll need to rely on a reliability test, a job-reference survey, or structured interview questions. All of these are designed to discreetly uncover problems that can negatively impact the employee’s reliability. Reliability tests measure the candidate’s potential for absenteeism, drug use on the job, and honesty. Due to the intrusive nature of these tests, they are controversial and more open to litigation. Since this dimension of success is so socially desirable, be sure to ask the vendor for any research that shows the items are fake-proof. A “safer” approach may be a good structured interview that more discreetly addresses the candidate’s reliability. CraftMetrics has developed dozens of powerful interview questions to measure each of the three dimensions of success. Other options include due diligence background checks, or even Internet surveys completed by candidate-selected references.
 
How much can I afford to spend?
 
Can Do Aptitude tests: These tests are relatively inexpensive, ranging from $5 to $30 per administration, depending upon their validity and reliability and the protection they provide you from EEOC/ADA litigation. Tests that measure many different skills can be more expensive. Without much effort, you can find a host of nonscientific skills/aptitude tests on the Internet that have no validity and reliability, so beware.
 
Will Do Personality tests: Depending on the class of test, you can expect to pay anywhere from $10 to $250 per administration. Most personality tests can be categorized as either: Class A, Class B, or Class C tests.
 

Class A tests range from $75 to $250, depending on the volume of use and the amount of your consultant’s time is involved. Because of their high cost, they are used primarily for employee selection and promotion (rather than training or development). They are developed by psychologists and typically provide telephone support and sophisticated research and validity reports from on-staff masters’ level or PhD psychologists. These tests are designed to minimize applicant faking and maximize test reliability, and they meet all American Psychological Association (APA) and EEOC standards. Interpretive reports are personalized but not typically “user friendly.”

 

Class B tests range from $25 to $75, depending on volume of use, and can be used for both hiring and development. They are typically developed by psychologists and may even have a psychologist or two on staff to conduct ongoing research. They typically meet most, if not all, APA/EEOC standards. They are designed to minimize faking and maximize test reliability. Most offer telephone support, some without cost. Interpretive or development reports are computer-generated and personalized.

 

Class C tests are from $10 to $20 per administration. They can be recognized by their format in which candidates are asked to describe themselves by selecting the best adjectives. These tests are excellent in stress-free situations such as training or development. However, when used in stressful situations such as hiring or promotion, they can easily be faked by test-wise applicants who select only the socially desirable responses. Because they are usually developed and supported by mass-marketing test distributors, APA/EEOC standards are seldom achieved by Class C tests. Most of them are derivatives of other tests and make unsubstantiated marketing claims (“95 percent accurate”), rather than providing real validity and research. Telephone support is minimal or nonexistent. Interpretive/development reports rely on templated descriptions from a limited number of paragraphs and personality types. Feel free to use these tests for training or development. However, avoid their use in employee selection or promotion, since the publisher and distributor are not prepared to protect you from EEOC/ADA litigation. Remember, especially when it comes to personality testing, you get what you pay for.

 
Follow-through Reliability tests: Depending upon volume purchased, you can expect to pay anywhere from $20 to $50 per administration. The more established test distributors (over 20 years old) provide relatively sophisticated reliability tests that have the validity and reliability to minimize your chances of litigation. Before you buy, check your state laws to make sure they are legal and then select the company that has the best validity research and most user-friendly interpretive reports.
 
What problem do I wish to solve?
 
Retention problems: There are two types of turnover: voluntary (“I quit!”) and involuntary (“You’re fired!”). If your new employees are coming in the front door and walking out the back within a few months or so, then you have a voluntary turnover problem. In this case, you’ll need to use an attitude survey rather than a test that measures the three dimensions of success. In other words, the issue here is not success or failure of your employees; it is the culture in which your employees find themselves. Instead of a single snapshot of your company, we recommend an ongoing analysis. Administer a culture/attitude survey to the entire company to establish a benchmark, and then have your new employees take the survey every 90 days for the first two years of their employment. Statistical analyses will soon determine the problems and even point-out the individual employees with changing or negative attitudes. This is a surefire way to predict turnover and initiate intervention before it’s too late, both on a corporate and an individual employee level.
 
Involuntary turnover, on the other hand, demands a different solution. If you have to terminate employees due to compatibility or productivity problems, you’ll need to look closely at your selection system and make sure you’re using the proper test(s) and interview questions. The most commonly overlooked areas are intelligence and drive. Make sure your new employee has the mental aptitude to handle the job and the drive/motivation that is compatible to the job description. Don’t put a “Plow Horse” in a position that requires a “Racehorse” and vice versa. Use an appropriate intelligence or learning aptitude test and a personality test and make sure they are job related, predictive, and avoid adverse impact.
 
Productivity problems: If you’ve decided that turnover is not a problem and your selection process is adequate, you may want to use employment testing to increase the productivity of your work force. Provide your employees with periodic workshops and seminars in which the first step is for them to complete a personality test for training purposes. They’ll typically be quite honest when they’re responding to the questionnaire if you present the workshop as an improvement opportunity, rather than a test.
 
You’ll be amazed how much your employees will enjoy the user-friendly interpretive reports that are geared specifically to certain training needs. For example, one personality questionnaire’s interpretive report for sales reps describes the rep’s personality in terms of selling styles and then describes the prospective customers’ buying styles. Sales reps learn how to flex their style to honor the customers’ buying styles. Another profile report takes the same approach to help managers communicate with employees who have a different communication style.
 
If you’ve administered the same personality test in the selection process before the employee was hired (and there are anti-faking systems to elicit honest responses), you can use that original administration for the workshops and not have to re-administer the test. If you are using the test distributor’s Internet testing Web site, the test results are most likely captured so that you can use those same scores in the future to generate different reports. For example, you might use an employee’s hiring report to help you make a hiring decision, his coaching report to help you motivate or supervise him, his training report for workshops, and his leadership report for purposes of promoting him to a leadership position within your company.
 
All of these reports are generated from the same personality test and the same scores you received when you used the test in your selection process. Ask your test distributor to provide you with samples of all of the reports that are available from the same administration of their personality test. You may be pleasantly surprised at the increase in employee productivity from the information you glean from your employees’ personality profile results.
 
How much time will the testing take and what are the hassles?
 
The employment tests you choose will be dependent upon 1) the amount of time you want your candidate or employee to spend responding to the testing process; 2) the amount of inconvenience they can tolerate; and 3) the delays both of you can tolerate before a final decision is made.
 
Time of administration: Some employment tests created by ivory tower psychologists take 2 – 3 hours to administer. Their predictive validity is typically no better than the shorter internet-based tests that take 20 – 30 minutes. In fact, one highly predictive Internet-based system that includes the pre-screener, intelligence/aptitude test, and personality test takes less than 45 minutes. Don’t settle for a test that takes too much of your time when there are plenty of highly efficient testing systems that take less than an hour and yield comprehensive and accurate results.
 
Inconvenience: The inconvenience of the wrong employment-testing system can also take its toll. One particular test requires you to send your applicant across the city or county to their testing center or to one of their affiliates. Just setting up the appointment can take more of your time than most Internet-based test administrations.
 
Delays in scoring: Many companies require that the test forms be sent or faxed to a scoring center. Results can take anywhere from 24 hours to a week. Once again, the validity and reliability of these tests are typically no better than the ones scored in your office or over the Internet. By the time you finally get back the test results from the scoring center, the best candidates are already hired by your competition.
 
Do I want a selection system or just a test?
 
Smaller companies that interview less than 20 – 30 applicants per year will most likely need only the appropriate test(s) and a structured job-related interview, not a complete system that includes applicant tracking and EEOC/ADA compliance. Make sure you measure the appropriate dimension of success (Can Do, Will Do, or Follow-through) that best defines productivity for a given job description. For a CPA, for example, it will most likely be a skills or aptitude assessment (Can Do). For a commissioned sales rep, on the other hand, it will most likely be a Will Do personality and motivation assessment.
 
On the other hand, if you are hiring for a larger company, we recommend a formal, structured selection system that includes an Internet-based 1) application for employment; 2) skills or biodata pre-screener; 3) aptitude/intelligence test; and 4) personality test. You’ll want the system to have different levels of entry so that upper-level management will be able to predict manpower from the data being captured by hiring managers in different offices. A quick glance and the manager will know real-time status of every applicant in the system, including how many are in the initial interview, the in-depth interview, or how many have completed which phase of the testing process. Drop-out rates, pass-through rates, test norms, the most productive recruiting sources, due diligence processing, and other essential data can be collected and reported instantly. You’ll also want the system to provide you with the flexibility of assessing different positions and different levels of experience. A highly qualified, experienced candidate will not appreciate going through the same system as a college graduate.
 
You can also choose to seamlessly link your applicant recruiting Web site to your selection system before your manager invites the candidate into the initial interview. That way, your manager is armed to the teeth to conduct the most effective and efficient interview. Precious time is also saved, placing your managers in the best position to make the earliest offers to qualified candidates.
 
If your selection system is ever challenged by an EEOC or ADA complaint, you have objective tests that have been normed to avoid adverse impact and you have times, dates, and actions your manager tracked throughout the selection process. Data can be instantly accessed and studied for ongoing compliance purposes. Over time, you’ll even be able to modify the items in your questionnaires to make them more predictive or more compliant to legal requirements.
 
The testing used in your system can capture and retain scores and also include all of the interpretive reports necessary for hiring, motivating, training, and promoting new employees as they proceed through their career. Once the test is administered, future administrations may not be necessary unless there is a two or more year time lapse after the initial administration.
 
Of course, if you want a sophisticated Web-based system similar to the ones I’ve helped develop for Wells Fargo, MetLife, Xerox, and Hilton, there will be a significant up-front cost depending upon the number of tests and the sophistication/levels of your data entry and access. However, even that amount shrinks to insignificance when you understand the true cost of making a single “bad hire.” If the right employment tests are used in the right way, there most likely is not a better value in the workplace than employment testing.
 
What questions can I ask the test distributor?
 
When I developed my first employment test in 1978, there were only a few dozen test publishers who marketed directly to organizations. Choosing the best test was relatively easy. Today, there are over 1,000 tests and quizzes from which to choose, making the decision much more difficult. If you are seeking employment tests that minimize your chances for litigation and maximize employee productivity, ask test publishers/distributors the following questions:
 
Can you provide my company with:
 

1. Documented validity evidence written in accordance with the Uniform Guidelines on Employee Selection Procedures (41CFR90-3; 1978)?
2. Evidence that they honor the Principles for the Validation and Use of Personnel Selection Procedures in the development and implementation of its assessments?
3. Access to certified, doctoral-level consultants to assist with implementation and research opportunities?
4. Evidence of a track record of being an active provider in the assessment solutions industry?
5. Access to an in-house dedicated information technology (IT) team to implement customized solutions for the client company?
6. Internet access to real-time data-tracking systems for recruiting, selection, and reporting purposes?
7. Evidence that their work has been examined and reviewed in the Mental Measurements Yearbook?
8. The safety and convenience of a password-protected testing Web site where users at different levels of the organization can utilize the system while overall activity can be monitored by management?
9. Adequate training and resources in the use of assessments without additional cost?
10. Evidence of fairness in testing as required by the EEOC and other federal and state agencies?
11. Adequate post-sale support to the client company for ongoing service, training, and improvement?
12. Assessments that have accuracy or validity systems that determine when candidates have attempted to fake or distort their responses to leave an unrealistically favorable impression?
13. Assessments that have a proven method of reducing applicant faking during the assessment process?
14. Assessments that have a one-page, easy-to-read candidate summary for selection purposes?
15. Assessments that have comprehensive selection reports for managers, providing in-depth information regarding candidates’ strengths, improvement opportunities, and compatibility to the position?
16. Assessments that provide personalized, detailed, multi-page, reports for managers to use in developing/coaching employees and managers after they are hired?

 
How Do You Know if a Test is Valid?
 
It is not uncommon to have even seasoned professions ask this question. Most managers and even HR professionals haven’t taken a statistics course that focused on employment testing validity and reliability. Here is how Candidate Resources explains this complex subject to its corporate clients:
 

CONSTRUCT VALIDITY refers to the extent in which dimensions with similar names on different tests relate to one another. Two things that correlate highly are not necessarily identical, but do provide reassurance that they are related and are a “construct” or part of the makeup (like honesty, dependability, sociability, etc.) of an individual as related to actual job performance.
CONCURRENT VALIDITY is that approach whereby people who are successful within a given job within a given company or industry are evaluated and generally grouped TOP THIRD, MIDDLE THIRD, BOTTOM THIRD. The assessment scores of the people who fit each of these ranges are then compiled and Job Benchmark Standards of the TOP THIRD are used to hire, train or manage.
PREDICTIVE VALIDITY occurs when the employer hires people for a job based on normal hiring procedures (interviewing, reference checks, education/experience, etc.) and at the same time has them complete the assessment, but does not utilize any data from it in the hiring decision. Within six months, or any appropriate period of time later, the assessment is scored, and benchmarks established of the people who were hired in the new jobs who are still with the employer and whom the employer considers successful. Job Benchmark Standards are thus established through the Predictive approach.
CONTENT VALIDITY represents job function testing, i.e., typing, mathematics, design, CPA exams, physical work endurance, etc. Content validation is not the method utilized by Candidate Resources, Inc., since we do not provide content assessments to the marketplace.
We recommend that an organization establish and utilize a consistent standard hiring process when making hiring decisions. Information should be gathered in each step of the standard hiring process to have specific and measurable data to utilize in making a final hiring decision. The assessment used should count no more than one-third of the hiring decisions. The preliminary interview, job history check, in depth interview results and evaluation of education, experience and other pertinent factors should be considered as well.
Under the Uniform Federal Guidelines adopted in the 1970’s, validation of any part of the hiring process (assessments included) was no longer deemed necessary unless a company was not meeting the 4/5ths1 Rule in either hiring or promotional practices. Consequently, there are three optional approaches to using assessments:
1. Establish your own successful employee Benchmark by conducting a concurrent validation by job classification. By tying job-related criteria to the aptitudes and personality dimensions of the assessment, the ultimate in validation and job relativity is assured. Also, the Benchmark simplifies the interpretation and use of the assessment in the hiring process, since it establishes a model for hiring, promotion and training purposes.
2. Establish Benchmarks by job classifications by answering job-related questions on the requirements of the job.
3. Use of Benchmarks comprised of successful people in jobs across the United States. Then, after a reasonable period of time, compare the successful people selected to the Benchmark used for that job for confirmation of correctness and/or modification of the benchmark standards.
The in depth validation identified above is not necessary if you are in compliance with the 4/5th Rule described below. This rule was designated by the E.E.O.C. as a computation tool to establish a basis to show whether or not a company is having an adverse impact in their hiring practices.
EXAMPLE: Out of 120 job applicants (comprised of 80 white and 40 minority), 48 whites were hired and 12 minorities were hired.
48 out of 80 white applicants = 60%
12 out of 40 minority applicants = 30%
This hiring pattern results in adverse selection of minorities, since 1/2 as many minorities are hired as whites (or 30/60), whereas the hiring ratio must equal 4/5th as many minorities as whites.

 

Complimentary Offer
 
Dr. Larry Craft has provided his employment tests to business clients for over 30 years. You can try his 3rd generation test, the CraftMetrics Personality Inventory (CMPI) for free. Call Dr. Craft personally at (941) 405-2893 or email drllcraft@craftmetrics.com, or fill in our online form to receive your free trial of the CMPI.

 
For over 30 years, thousands of employers have been given the opportunity to take our assessment without cost to introduce its predictive validity and versatility. Not only will you receive individually personalized interpretive reports, but you will also receive an interpretation from a qualified CraftMetrics‘ consultant who has been trained in the nuanced interactions of the CMPI. This 15-20 minute questionnaire generates many valuable reports. Below are a few details on some of our most popular reports.
 
CMPI Compatibility Chart(s): This single-page chart/graph compares your applicants’ personality traits to peak performing employees from many different job descriptions. You’ll receive a Position Compatibility Score and the number of Coaching Hours suggested in order to compensate for any non-ideal personality traits. The report also provides interview questions for less compatible traits, so that you can cross-validate the CMPI results.
 
CMPI TeamBuilder or SalesBuilder Report: This 8-page report defines your communication or leadership style (for management or non-sales personnel) or your selling style (for sales personnel) and provides recommendations to improve communication between you and your team members, or between you and your prospective client/customer to help you close more sales.
 
CMPI Manager’s (Sales or NonSales) Coaching Report: This report defines your core CMPI personality and motivation traits, and then provides you with specific recommendations to help you become a more effective leader.
 
CLAS Summary Report: This report describes your applicant’s/employee’s learning aptitude/intelligence in terms of their speed AND accuracy. Specific recommendations are made as to the training modality to which the individual will respond most effectively. These results can provide a robust supplement to the Job Position Compatibility Charts (see above) in a validated hiring context.
 
CMPI CultureStyle Profile Report: This report plots all your employees on a single 4-Style grid to define and quantify your company’s unique culture. At one glance, you can see your employees’ strengths and challenges. The report makes specific recommendations to compensate for those challenges. You can also use the report interactively to pick most effective work groups.
 
About the Author
 
In the late 1960s, Larry Craft entered a management career in the financial services industry. Soon afterward, he recognized a profound need and developed one of the first personality questionnaires outside the field of psychology that could be used to hire and develop personnel. In the early 1980’s Dr. Craft developed one of the first software-based employment tests available to managers to personally administer, score, and generate reports. At the same time, he began his consulting practice that specialized in assessing motivation and increasing employee productivity.
 
In the 1980s, he returned to his education to receive a master of arts in psychology and later, his doctorate in organizational leadership. Dr. Craft’s research has been published in USA today, Selling Power Magazine, the Mental Measurements Yearbook, and the Journal of Psychology. Over the past 25 years, he and his consulting staff have trained over 10,000 managers, and his assessment tools have been administered to over 1,000,000 employees. Many of the nation’s largest home offices, such as Xerox, Wells Fargo, MetLife, Shell, Hilton, and Time Warner, have relied upon his company’s assessment tools and HR consulting services. In addition to nationally distributed HR manuals, he has authored Quick-Start Leadership, The Three Dimensions of Success, and Hiring and Developing HorsePower.
 
Larry and his wife, Carol, work together and spend their time on Anna Maria Island, Florida, during the winter and the Smoky Mountains of North Carolina during the summer. They have four children and thirteen grandchildren.