The art and science of IIT JEE evaluation

When I appeared for IIT JEE in 1975 — All India Rank (AIR) 31 — there was no objectivity or negativity. After 40 years, in the beginning of 2015, I set off preparing my daughter for the JEE Advanced. The profound sea change was expected. The evolution in the quality and the quantity of the students appearing for the JEE had necessitated the expansion of the syllabus, enhancing the complexity and the introduction of objective questions. To correct the anomalies inherent in the nature of objective evaluation, the IIT Joint Admission Committee (JAC) imposed negative marks, reaching a peak of two negative marks out of four for a set of 30 MCQ in JEE Advanced 2015.

Anomaly is a slippery creature. The more one tries to detach it, the more deeply it entrenches itself. While inflicting two negative marks on the hapless candidate ratiocinating to arrive at the correct answer, the JAC hoped to foil the alarming advice rendered by some coaching institutes and websites, alarming, because a few of the gurus encouraged the students to hone the craft of probability ‘to guess your way’ rather than master the principles of Kirchoff’s Laws, derailing the essence of evaluation of engineering proficiency: ‘tips to cheat’, ‘do a little guesswork’, ‘reverse engineer’, ‘discard highest and lowest outliers because in 60% cases they are not right’ and ‘check the dimensions to filter options’. Though the specter of 50% negative marks might have stopped the wizards of guessology in their tracks, it tripped the intelligent student to the former’s advantage. Sometimes, especially in the JEE 2015, an unprepared student, who has not comprehended the fundamentals of the problem and the topic and who cannot fathom an approach to solve it, nevertheless gains, over a student with a fair idea of tackling the challenge, by guessing or not replying at all.

Consider a typical multiple-choice question with one or more than one correct options, where options ‘A’, ‘B’ and ‘D’ are correct and ‘C’ is wrong. The clueless candidate moves on to the next question without vacillating, wasting no time and no marks, whereas the one having the grasp on the concept solves the problem, darkens the ‘A’ and ‘B’ bubbles, leaves the option ‘C’ untouched and unsure of the option ‘D’, does not mark it. The result? Five minutes wasted and two negative marks awarded. If this were the first question, the first candidate attains a pole position with respect to the later, for the remaining question paper. How this phenomenon affects the ranking in various bands — 1 to 50, 100 to 200, 400 to 500 — is analyzed and can be given in a detailed article.

While the candidates are reeling under the burden of two examinations and the humongous syllabus, befalls the third kneejerk reaction from the JAC — a third examination, a written section for JEE 2017 after the Mains and the Advanced tests, testing subjective skills, no doubt resulting in more coaching, more hours per day and more mess. An unviable attempt by the JAC to develop a machine-readable answerscript that can give credits to part answers will be futile and make the scenario murkier. To vindicate and justify the development, the JAC will force the framing of the questions to suit the half-baked and ill-conceived system, relegating the real purpose of the evaluation system to the background. The elite JEE Advanced should be demanding, but the students, because we cannot find a simple and efficient evaluation system, should not be pulverized with three tests in addition to the CBSC / A level and TOFFEL / SAT / GRE, all requiring different skill sets, all in one year, the 12th.

In mathematics, it is said, the best solution is the simplest solution. The present or the planned system, I am afraid, is an unwieldy solution. And what is the problem? To rank the students accurately. The IITs, being the sole premium institutes in India, attract the crème de la crème. She is enrolled at IIT before she is conceived. It is this student, not the IIT, who makes the IIT a prestigious institute. The continual presence of AIR 1 to 50, most of whom opt for Computer Science at IIT Bombay, can propel any decent institute into the top league. Do we deserve this constellation of top 50? As a group, I am sure, they would rank within top two or three out of the ensemble of top 50 engineering students of the batch from every country in the world, whereas IIT Bombay ranks 222nd in the World University Rankings and 58th in the World University Rankings by subject 2015 – Computer Science. The political system, though, has also contributed to this disappointing state.

In spite of the alluring prospect of absorbing the brightest students, despite compelling them to go through the toughest training regime and assessment, the simple task of ranking them precisely is a failure. When I walked into my hostel room for the first time on that eventful day in 1975, the message on the wall left by the passing out senior exhorted me: ‘keep the tradition of this room, don’t sweep.’ The question framing committee of 20 professors for each subject seems to take rather seriously the advice: ‘keep the tradition of JEE, err.’ JEE Advanced 2015 stumped the unsuspecting candidates with errors galore. The errors in physics paper alone affected nine marks, enough to compel a student yearning for Computer Science to take up Aeronautical Engineering. If a professor, with unlimited time and assistance of books, references and other professors, goofs up in answering a question he framed, how can we expect a candidate, on his own in the examination hall, to answer the same in three minutes? There is nothing worse that a student can take to the examination hall than a sense that while he prepared well and is confident of securing a top rank, the glitches in the assessment or ambiguity and errors in the questions or the answer keys, to be expected in JEE 2016 in spite of filtering by a squad of 60 professors, might unsettle him, forcing her to tick a fifth option to every confusing question: ‘error in question’.

Hence the quest for an elegant solution, lest the IIT entrance turns out to be a Barmecide feast for the deserving student. When this battle of anomalies has been fought out and won, when an elegant solution has been implemented, when the candidates have been relieved of the yoke of multiple tests and uncertainty, then it seems as though all the suspended sediments and algae of aberrations and excesses, making the evaluation system turbid, have been wiped out and the sparkling stream of the rankings rolls on. The key is to differentiate between an effort succeeding in solving the problem substantially and ‘not attempted’.

In the good old days of the 1975 JEE, the dedicated examiners went through all the steps the student had negotiated to arrive at an answer to assess the candidates accurately. Though ideal, this process is not feasible now. To ameliorate the negative consequences of the objective questions, we can reframe them in the following manner, dispensing with the proposed third test of JEE 2017.

Consider the solution to a multiple-choice question as ten steps of 10% each. In addition to the main question with four options, set another ‘branching question’ with four more options for which the solution branches at some point between step five and nine, traversing a further one to three steps. If the main question carries three marks, the branching question can reward two. If even one of the two answers is correct, negative marks will not be applicable. Otherwise, each attempted wrong answer will be penalized with one negative mark. If both the answers are correct, total six, with an extra bonus mark, will be awarded. One can also eliminate all the options and ask the candidates to write the final numerical answer, which can then be read by a machine.

If this scheme is calibrated, fine-tuned, intensified or diluted based on the question, it will achieve the desired result.

Sunil J. Kamat

(To receive our E-paper on whatsapp daily, please click here. We permit sharing of the paper's PDF on WhatsApp and other social media platforms.)

Free Press Journal