Myths and Facts Associated
with the MBE

Bar examiners across the country often encounter questions about myths relating to the Multistate Bar Examination (MBE), one of the important parts of their test of minimum competence for licensure to practice law. Over the years, significant research has been conducted that dispels these myths.

Myth 1
The MBE is a test of memory and test-taking ability, not of legal knowledge or analytical skill.

Fact
Research indicates that MBE scores are highly correlated with other measures of legal skills and knowledge, such as law school grades and scores on essay examinations. These correlations provide empirical evidence that the MBE is testing legal ability rather than general test-taking ability.

More recent research conducted in 2004 examined the cognitive processes used by examinees to answer MBE questions and found that the cognitive process of applying legal principles was most closely correlated with high performance, while irrelevant cognitive processes, such as test-taking strategies, had little impact on performance.

In research conducted in July 1986, incoming law students took the morning session of the MBE, and their scores were compared to the scores of graduates of the same law schools who had taken the same examination. The novices and graduates had virtually identical mean LSAT scores, so if the ability to take multiple-choice tests were the major factor influencing MBE scores, both groups should have had very similar MBE scores. In fact, the highest MBE score earned by the novices was lower than the lowest score earned by any of the graduates.

Myth 2
MBE questions are needlessly difficult, arcane, and tricky.

Fact
MBE questions are designed to be a fair index of whether an examinee has the ability to practice law. MBE questions are written by drafting committees composed of law teachers and practitioners. Before it is administered, every MBE question is reviewed at several levels: at least twice as it is edited by the drafting committee, by psychometric experts to ensure that it is fair and unbiased, and by the practitioner members of the MBE Policy Committee and their academic consultants. After a form of the MBE is administered, any question that performs in an unanticipated manner—is very difficult or is missed by examinees who did well on the rest of the test—is flagged by psychometric experts and reviewed again by content experts on the drafting committees to ensure that no ambiguity exists in the question and that the key (the correct answer) is unequivocally correct. Should an error be detected even after this thorough scrutiny, two or more answers may be deemed correct in order to ensure that no examinee is disadvantaged by having a particular question appear on the form of the MBE he or she took.

Myth 3
Not enough time is allotted to answer MBE questions.

Fact
Research shows that the time allotted to take the MBE is sufficient for 99 percent of examinees. The MBE is designed to be answered by a reasonably competent examinee in the amount of time available. The rate of correct responses at the end of a three-hour session is not significantly different from the rate of correct answers at other, earlier points in the test. 

A research project in which examinees were given virtually unlimited time to answer the MBE resulted in an average increase in score of about six raw (unscaled) points. Since all groups benefit from an increase in time to the same degree, and since the test is scaled to account for differences in difficulty, an increase in the average score would be offset in the scaling process and additional time would not be expected to increase examinees' scaled scores.

Myth 4

Essay examinations and performance tests are better ways to measure minimum competency to practice law.

 

Fact

While essay examinations and performance tests provide important information about examinees, there are several significant advantages to including multiple-choice tests on a bar examination. First, multiple-choice testing offers the opportunity for a breadth of coverage of subject areas which cannot be duplicated using only essay questions or performance tests. Second, multiple-choice questions can be scored objectively, and scores can be scaled to adjust for changes in difficulty from one test to the next. There are two sources of variation in difficulty in essay examinations and performance tests: variations in the difficulty of the test items themselves, and variations in how strict or lenient graders are.

In contrast, scores on the MBE are equated through a process that ensures that a new form of the MBE is no more nor less difficult than a previous form. By comparing the performance of examinees on a common set of items, raw scores on the test can be converted to adjusted, “scaled” scores that are directly comparable to one another. Because scores are equated, the MBE provides an anchor for other, more subjective test scores; the National Conference recommends that scores on essay examinations and performance tests be scaled to the MBE. And finally, this scaling of MBE scores allows direct comparisons of performance between tests. An examinee taking a current examination is on a level playing field with other examinees taking tests at other times.

Myth 5

The MBE discriminates against minority examinees.

Fact

The MBE neither widens nor narrows the gap in performance levels between minority and majority examinees. Research indicates that differences in mean scores between racial and ethnic groups correspond closely to differences in those groups' mean LSAT scores, law school grade point averages, and scores on other measures of ability to practice law, such as bar examination essay scores and performance test scores. Individual items on the MBE that are relatively difficult for one group are relatively difficult for all groups; the relative difficulty of the items within a subtest (e.g., the Constitutional Law items versus the Torts items) does not differ from group to group. Finally, total MBE scores are not higher or lower from group to group than they are on other test formats. 

All items on the MBE are reviewed for potential bias. The National Conference of Bar Examiners is committed to gender and ethnic diversity on all its drafting and policy committees. Each drafting committee is composed of members of both sexes, and members of ethnic minority groups participate in the preparation and review of items both at the drafting committee level and at the MBE policy committee level.

 

Myth 6

The MBE is getting easier; scores keep increasing while examinees are getting less able.

Fact

The MBE is a reliable measure of examinee ability. The average scaled score on the MBE has varied by less than two points from year to year, indicating that the ability level of the examinee pool has been fairly stable. Changes in MBE scores follow closely the variations in average scores on other measures of ability, such as the LSAT. This correlation between changes in MBE and LSAT scores indicates that increases in the average score mirror increases in the general ability level of the group being tested rather than a decline in the difficulty of the test.