Myths and Facts about the MBE

Bar Examiners across the country often encounter questions and myths relating to the Multistate Bar Examination (MBE), one of the important parts of their test of minimum competence for licensure to practice law. Over the years, significant research has been conducted that dispels these myths.

Myth 1

Examinees can pass the MBE by guessing. The MBE is a test of memory and test-taking ability, not of legal knowledge or analytical skill.

Fact

Research indicates that the MBE is not a “multiple-guess test” or an examination that rewards test-taking ability. In research conducted in July 1986, incoming law students took the morning session of the MBE, and their scores were compared to graduates of the same law schools who had taken the same examination. The novices and graduates had virtually identical mean LSAT scores, so if the ability to take multiple-choice tests were the major factor influencing MBE scores, both groups should have had very similar MBE scores. In fact, the highest MBE score earned by the novices was lower than the lowest score earned by any of the graduates.

Second, research on the MBE indicates that MBE scores are highly correlated with other measures of legal skills and knowledge, such as scores on state essay examinations and law school grades. These correlations provide empirical evidence that the MBE is testing legal ability rather than general test-taking ability. Similarly, a panel of experts convened in 1992 as part of a content validity study concluded that MBE items were material to the practice of law and that their emphasis was balanced between legal reasoning skills and memorization of legal principles.

Myth 2

MBE questions are needlessly difficult, arcane, and tricky.

Fact

MBE questions are designed to be a fair index of whether an applicant has the ability to practice law. MBE questions are written by drafting committees composed of men and women who are law teachers and practitioners. Before it is administered, every MBE question is reviewed at several levels: at least twice as it is edited by the drafting committee; by psychometric experts to insure that it is fair and unbiased; by the practitioner members of the MBE Policy Committee and their academic consultants; and by the members of boards of bar examiners across the country. After a form of the MBE is administered, any question that performs in an unanticipated manner—is very difficult or is missed by applicants who did well on the rest of the test—is flagged by psychometric experts and reviewed again by content experts on the drafting committees to ensure that no ambiguity exists in the question and that the key is unequivocally correct. Should an error be detected even after this thorough scrutiny, two or more answers may be deemed correct in order to ensure that no applicant is disadvantaged by having a particular question appear on the form of the MBE he or she took.

In a 1992 study, expert panelists reported that they believed MBE items were generally easy, correctly estimating that about 66 percent of candidates would select the right answer to a typical item. A study is underway to review the MBE test specifications (subject matter outlines) in order to make certain that the questions asked on the examination continue to relate to knowledge that is important to the practice of law.

Myth 3

Not enough time is allotted to answer MBE questions.

Fact

Research shows that the time allotted to take the MBE is sufficient for 99 percent of applicants. The MBE is designed to be answered by a reasonably competent applicant in the amount of time available. The rate of correct responses at the end of three-hour sessions is not significantly different than the rate of correct answers at other, earlier points in the test.

A research project in which applicants were given virtually unlimited time to answer the MBE resulted in an average increase in score of about 6 raw (unscaled) points. Since all groups benefit from an increase in time to the same degree, and since the test is scaled to account for differences in difficulty, an increase in average score would be offset in the scaling process and additional time would not increase applicants' scaled scores.

Myth 4

Essay examinations and performance tests are a better way to measure minimum competency to practice law.

Fact

While essay examinations and performance tests provide important information about candidates, there are several significant advantages to including multiple-choice tests on a bar examination. First, multiple-choice testing offers the opportunity for a breadth of coverage of subject areas which cannot be duplicated using only essay questions or performance tests. This breadth of coverage improves the reliability of the examination. Second, multiple-choice questions can be scored objectively, and scores can be scaled to adjust for changes in difficulty from one test to the next. There are two sources of variation in difficulty in essay examinations and performance tests: variations in the difficulty of the test items themselves, and variations in how strict or lenient graders are. In contrast, scores on the MBE are equated through a process that ensures that a new form of the MBE is no more nor less difficult than a previous form. By comparing the performance of applicants on a common set of items, raw scores on the test can be converted to adjusted, “scaled” scores that are directly comparable to one another. Because scores are equated, the MBE provides an anchor for other, more subjective test scores; the National Conference recommends that scores on essay examinations and performance tests be scaled to the MBE. And finally, this scaling of MBE scores allows direct comparisons of performance to be made among tests. An applicant taking a current examination is on a level playing field with other applicants taking tests at other times.

Myth 5

The MBE discriminates against minority applicants.

Fact

The MBE neither widens nor narrows the gap in performance level between minority and majority applicants. Research indicates that differences in mean scores among racial and ethnic groups correspond closely to differences in those groups' mean LSAT scores, law school grade-point averages, and scores on other measures of ability to practice law, such as bar examination essay scores or performance test scores. Individual items on the MBE that are relatively difficult for one group are relatively difficult for all groups; the relative difficulty of the items within a subtest (e.g., the Constitutional Law items versus the Torts items) does not differ from group to group. Finally, total MBE scores are not higher or lower from group to group than they are on other test formats.

All items on the MBE are reviewed for potential bias. Men and women serve on each drafting committee, and members of ethnic minority groups assist in the preparation and review of items at both the drafting committee level and at the level of MBE Committee and state Board review. The National Conference of Bar Examiners is committed to diverse representation on all its drafting and policy committees.

Myth 6

The MBE is getting easier; scores keep increasing while applicants are getting less able.

Fact

The MBE is a reliable measure of applicant ability. The average scaled score on the MBE has varied by less than 2 points from year to year, indicating that the ability level of the candidate pool has been fairly stable. Changes in MBE scores follow closely the variations in average scores on other measures of candidate ability, such as the LSAT. This correlation between changes in MBE and LSAT scores indicates that increases in the average score mirror increases in the general ability level of the group being tested rather than a decline in the difficulty of the test.