The Crown Game Affair
What constitutes evidence of cheating?
|
| src |
Faye Dunaway is an Academy Award-winning actress who co-starred with the late Steve McQueen in the 1968 movie “The Thomas Crown Affair.” She plays a freelance insurance fraud investigator, Vicki Anderson, who believes that millionaire playboy Thomas Crown is guilty of instigating a $2.6 million bank heist, but falls in love with him anyway. The most famous scene in the movie shows her both defeating and seducing Crown in a game of chess.
Today I write about the difficulty of detecting fraud at chess, and the role of statistical evidence.
The New York Times this morning joins several chess media outlets covering the allegations against a Bulgarian player who was searched during a tournament in Croatia last month. When we mentioned this case in our “Predictions and Principles” post earlier this month, I had the issue of principles regarding statistical evidence high in my mind, and this is reflected in my exemplar of a formal report. It accompanies a cover letter to the Association of Chess Professionals, raising the issue of what do you do when there is no physical or observational evidence but the statistical evidence is strong, and who should have oversight of standards and procedures for statistical tests.
Dunaway also had a small role in the 1999 remake, in which Crown again escapes uncaught but with a different endgame. Crown is played by Pierce Brosnan of James Bond fame. There is a James Bond quality to current speculation about possible cheating methods, from embedded chips to special reflective glasses. They are among items considered at the end of this 70-minute video by Bulgarian master Valeri Lilov, which was also covered by ChessBase.com. But none of this speculation is accompanied by any hard evidence. The real action may be not with the kind of gadgeteers to interact with M or Q or even Miss Moneypenny, but rather the actuaries down below who track the numbers.
Arbiter By Numbers
Cheating and gamesmanship at chess are nothing new—only the possible sources of illegal information during games have changed from ‘animal’ and ‘vegetable’ to ‘mineral.’ The following lyrics from the “Arbiter’s Song” in the 1986 musical “Chess” come from actual incidents at championship matches before then.
If you’re thinking of the kind of things
that we’ve seen in the past:
Chanting gurus, walkie-talkies, walkouts, hypnotists,
tempers, fists—
Not so fast.
Now there are calls for directors and judges and arbiters at chess tournaments and matches to take high-tech measures against possible cheating even while the games are going on. One limitation of testing moves in the manner needed by my model is that the games must be analyzed to evaluate all reasonable move choices with equal thoroughness, which takes a fast processor core more time than the duration of the game itself when it was played.
Still, the need for move-analysis tests is recognized by many. Indeed the very first comment posted in the story breaking on December 30 came from British master Leonard Barden, who has been chess columnist of The Guardian newspaper for 54 years—and the Financial Times for a mere 34 years. Barden put the present issues plainly:
Either
1 Borislav Ivanov is probably the first adult (as opposed to a junior talent) with a confirmed low rating ever to achieve a 2600+ GM norm performance in an event of nine rounds or more… or
2 [He] is the first player ever to successfully cheat at a major tournament over multiple rounds without the cheating mechanism being detected.
Here 2600 is a chess rating that usually distinguishes a “strong grandmaster,” while my own rating near 2400 is typical of the lesser title called “international master,” and Ivanov’s pre-tournament rating of 2227 is near the 2200 floor to be called any kind of master. Although Magnus Carlsen recently broke Garry Kasparov’s all-time rating record to reach 2861, my program for “Intrinsic Ratings” clocked Ivanov’s performance in the range 3089–3258 depending on which games and moves are counted according to supplementary information in the case, all higher than any by Carlsen and his two closest pursuers enumerated by me here, or anything here or here.
Barden bears with him the memory of a British prodigy born the same year as he, coincidentally named Gordon Thomas Crown, who passed away of illness in 1947 shortly after defeating Soviet grandmaster Alexander Kotov in one of two games during a “summit match” between Britain and the USSR. He continued:
There are no examples known of devices successfully transmitting chess moves in competitive play via contact lenses, the skin, the brain or other such concepts … [T]he cheating mechanism in this case remains unexplained.
That’s why it is important that somebody with access to Houdini or another top program examines the nine [games] with a program. Such a program check of the games may help to establish whether the player used computer assistance.
Thus all the tech-talk takes a back seat to simple numbers when it comes to getting results. The question remains, are they enough?
The Issue
It took me two days to run the main test with two top programs, Rybka 3 and Houdini 3, run several supporting analyses, and then run my statistical analyzer. Writing the report took another week, however, as I felt responsible also for articulating issues of how to evaluate this kind of evidence, and spelling out scientific particulars for due process. The drift of reactions to others’ early scattershot tests also moved my originally-advised intent of writing my conclusions briefly and simply for chess players to writing for experts in statistical fields—and for a student audience such as in a seminar I am running this coming term.
My report gives examples addressing when and why and how odds of “a million to one” should be treated differently from “a thousand to one.” The latter typifies my results in some cases where there was also physical or observational evidence, but here there is as yet none. Here is a different example to the same effect.
Mark Crowther of London has provided an incredible service called The Week In Chess (TWIC), which collects for free download several thousand games played in tournaments over the preceding week. The current week, TWIC 948, has games by over a thousand players—1,010 to be exact—typically 4–6 per player for a weekend tournament up to 9 for an all-week event such as the Zadar Open itself. If one were to dredge all their games, one would expect to find a statistical deviation that would translate to 1,000–1 odds against some kind of “null hypothesis” about cheating. Clearly Inspector Javert should have left the other characters in Les Misérables alone and taken up statistics. The fear of players being fingered this way is remarked by Dylan McClain in today’s New York Times column:
If every out-of-the-ordinary performance is questioned, bad feelings could permanently mar the way professional players approach chess.
Hence my policy has been that such statistical results have meaning only when there is evidence against the player that is independent of performance or move-match tests with computers by others.
With results citing million-to-one odds, however, the considerations are different—at least for chess. To find such a deviation by natural causes, one would need to dredge 20 years of TWIC—and the indefatigable Crowther has just started his 20th year.
Fail-Safes?
A second factor is that my tests are not invariantly correlated to quality of performance. My co-author Guy Haworth—who gave me heroic multiple detailed feedbacks on my report helping it achieve clarity and fairness—alerted me to discussion of a similar mercurial performance by Scottish master Alan Tate, also in Croatia, in 2010. I ran Tate’s games through a screening test, and found only 51% move-matching, compared to figures near 70% in the present case. Indeed, Tate’s defeated opponents had higher concordance to the computer in those games.
My tests have also rendered negative results; my letter notes that in two major international Open tournaments they were determinative for awarding a delayed prize. Thus they are not always “bad news” even when presuppositions are heightened.
My report describes two main tests, which are partially independent. Presumably their combined confidence would be higher, though I have not yet worked out how to do this numerically. Several alternative specifications for the tests, such as using the player’s rating before rather than after the tournament as the main baseline, excluding one (or two) game(s) where public transmission of moves was switched off amid suspicion of him, and excluding moves after (say) move 70 in very long games when the time available to think might be too short for some cheating mechanisms, exhibit much higher deviations. Although the “Intrinsic Rating” component does not accompany a statement of odds, it indicates that the inherent quality of the moves, as judged by computers, was highly significantly beyond what goes with a 2700-level performance.
Thus I claim specific value for my tests beyond being a metric of performance, which buttresses my point in asking the chess world, what shall we do about all this?
The Letter
In a series of fortunate events after breaking his leg playing soccer, Grandmaster Bartlomiej Macieja of Poland traveled with brace and cane into downtown Warsaw to meet me during MFCS 2011, became co-author on a paper with me and Haworth, became husband and father, was hired as a coach by the University of Texas at Brownsville, and became General Secretary of the ACP—not all in that order. Hence it was logical to address my letter to him as well as ACP President, Grandmaster Emil Sutovsky of Israel. Here are some excerpts:
I pose two questions, of which at least the first should be an immediate concern of ACP in conjunction with FIDE and national organizations. The second is a deeper issue that I believe needs consultation with experts in statistics and computer sciences, and with representatives of bodies in other fields that have established protocols for using evidentiary statistics in fraud detection and arbitration.
- What procedures should be instituted for carrying out statistical tests for cheating with computers at chess and for disseminating their results? Under whose jurisdiction should they be maintained?
- How should the results of such tests be valued? Under what conditions can they be regarded as primary evidence? What standards should there be for informing different stages of both investigative and judicial processes?
…
The point of approaching ACP is to determine how the contexts and rules should be set for chess. The goals, shared by Haworth and others I have discussed this with, include:
- (a) To deter prospective cheaters by reducing expectations of being able to get away with it.
- (b) To test accurately such cases as arise, whether in supporting or primary role, as part of uniform procedures recognized by all as fair.
- (c) To educate the playing public about the incidence of deviations that arise by chance, and their dependence on factors such as the forcing or non-forcing quality of their games.
- (d) To achieve transparency and reduce the frequency of improper accusations.
- (e) Finally, hopefully to avert the need for measures, more extreme than commonly recognized ones, that would tangibly detract from the enjoyment of our game by players and sponsors and fans alike.
More simply, I share the worry of many that a few cases of people “being clever” may ruin much pleasure. This extends to accusations I believe have been ill-informed, such as the one noted in the introduction to my “Fidelity” public site. (The data files behind my results are kept private; whether to open them is another hard question.) I hope that certain little details in my report, such as getting such positive results despite there being ten consecutive non-matches in one game and seven in another, will be noticed and deter others from trying to be “cleverer.”
Open Questions
What cases of statistical evidence in your field may best inform this one?
Update (1/15 9:30pm): Slashdot posted a note on this earlier today, and their comment thread has a wealth more of informative comparisons and reactions.
Trackbacks
- Daily Chess News Links January 14, 2013 | blog.chesscafe.com
- Assorted links
- Cheating at Chess
- Cheating in Chess « Polander
- Weekendowa Lektura | Zaufana Trzecia Strona
- Statistical evidence and cheating at chess | Stats Chat
- Links, Links Everywhere
- Interstellar Quantum Computation | Gödel's Lost Letter and P=NP
- Thirteen Sigma | Gödel's Lost Letter and P=NP
- Littlewood’s Law | Gödel's Lost Letter and P=NP
- Android Giochi Xl | Alleged chess cheat can’t be beat
- The Problem of Catching Chess Cheaters | Gödel's Lost Letter and P=NP
- The Chess Detective | Armchair Warrior
- Daily Chess News Links January 14, 2013 – ChessCafe.com
- Bad Tidings » This is beyond everyone’s attantion span, but you would find every bit of it interesting.
- Depth of Satisficing | Gödel's Lost Letter and P=NP
- This is beyond everyone's attantion span, but you would find every bit of it interesting. – Bad Tidings
- The Doomsday Argument in Chess | Gödel's Lost Letter and P=NP
One answer is sufficiently sobering and morally complex that it is seldom discussed:
• Farah et al. When we enhance cognition with Adderall, do we sacrifice creativity? A preliminary study (2009).
• Lanni et al. Cognition enhancers between treating and doping the mind (2008).
• Bogle and Smith Illicit methylphenidate use: a review of prevalence, availability, pharmacology, and consequences (2009).
Sports like baseball and cycling ignored the reality of doping strategies until these strategies became so prevalent that the moral integrity of the sport was destroyed. Is academia similarly at-risk of progressive moral deterioration?
This serious concern raises tough questions that have few or no easy answers. Questions like “Is drug use fair to fellow students?” and “Should academic interviews include drug-screening?” excite passion (to say the least). As baseball and cycling have learned at high cost, ignoring these issues is an optimal strategy in any given year, but not for any given generation.
Actually history shows the McGuire/Sosa home run derby in 1998 practically saved baseball. MLB were more than happy to look the other way at that time. If there was a pill you could take to make you a Pulitzer prize winning writer, would you take it?
Should my surgeon enhance his cognition when operating on me?
Is your surgeon in a competition with other surgeons to see who can operate on you with the best results?
If so, then to keep the competition fair, either all of the competing surgeons should take performance-enhancing drugs prior to cutting you, or none should.
Otherwise, as long as you’re sure the cognition enhancement will only improve his performance and not include various and unforeseen oversights and mistakes, then sure…why not let him enhance his cognition?
But that latter scenario seems somehow less relevant to the competition of professional athletes and chess players.
“Should my surgeon enhance his cognition when operating on me?”
You’re conflating things that matter (surgery) with things that do not (chess, baseball).
Are you a politician, by any chance?
Your Surgeon should unequivocally take all measures to increase the likelihood of you surviving the operation, such measures defined as reasonable under tort law. However, this is chess, not angioplasty. Entertainment, not Rhino-Oto-Laryngiology.
So, perhaps you could ask instead: Should a comedian take a pill to be funnier?
Audience members are nearly allways offered a drug, and sometimes required to purchase it, at performances by comedians. A two-drink minimum is standard.
So you don’t have wonder any longer, yes, indeed, the superfluous “l” in the preceding post is actually a covert transceiver. Cllever of you to notice.
If someone is really suspected of cheating could they be made to play in a Faraday cage?I
I know from talking with USCF tournament directors that certain “electromagnetic transmission” measures are illegal in US law.
It is illegal in the US to build a device that actively jams other radios (by for example, putting out high power signals on the same frequencies). But a room with metal lined walls that radio waves cannot get through (a Faraday cage) is absolutely legal.
The Faraday Cage rules out electromagnetic transmission absolutely. Proper installation is not trivial, but there are venues, I’m told, equipped to prevent real-time disclosure of insider information or just to kill cell-phone interruptions. Any important embassy has one.
Don’t let up on your analyses even if the problem disappears, though, because you might be working toward some insight into the step-function progress of gifted minds or real-world sightings of the mythical creature “inspiration”.
I don’t care how unlikely a given result is; if there is no physical evidence of cheating the player should not be penalized. Otherwise we deny the possibility that an unheralded player might triumph through sheer determination and inspiration. That would kill the game far more than these supposed cheaters.
For the time being, liberal society is indeed still toeing the “innocent until proved guilty” line. However, there is a frightening strain of thought that advocates automatically predicting crimes, preventive detention, etc., traditional rules of evidence be damned in the name of security. Think the gun control debate in the wake of Sandy Hook is over the top? The future voices of “reason” calling for such preemptive interventions will only grow shriller as “Minority Report” technology improves and thus becomes more seductive.
The overarching question is whether the chess world really wants to lead the charge on introducing such a precedent into liberal society’s principles. As one of the premier outposts of thoughtful society, adopting a style of preventive detention in this game would carry significant moral weight.
In the meanwhile, we need to make it clear to potential cheaters that the traditional rules of law enforcement still apply: you can rob a few banks, but if you make a career of it you’ll end up drawing attention and end up full of bullet holes. Fraud detection constantly improves over time and, with statistical analysis as a tip-off, we’ll be watching you very carefully — uncomfortably so. As they say, you’ll draw heat. And when we do catch you, there will be retroactive hell to pay. Doesn’t sound like a career plan, does it Lance?
Granted, this isn’t as satisfying as catching players who even think about cheating, but Chess is a game. I’d rather see some suspected cheaters get away with a few (or many) ill-gotten victories than help stoke the current feeding frenzy on my personal rights.
Doesn’t it sound like a career plan, though? Lance would never have been anything without his doping. Yes, he is shamed today and has lost much of the glory, fame, and wealth he had at his peak, but I would argue that he is much better off than alternate reality Lance who never doped and still toils in obscurity today. He probably dropped out of cycling and peddles insurance in Plano.
JD and Bertie, well understood, that’s why my post and letter+report are yea-so nuanced. Historically there was similar argument over fingerprinting a hundred-plus years ago, and to judge by Wikipedia here, there are still many current hesitations and caveats. Yet liberal society slowly accepted it as standard procedure, and it does not provoke the level of personal-rights feelings that are associated to guns. One reason of course is that a fingerprint is something “physical”, so we humans can accept it as “hard” evidence, even though the feature-matches are statistical and subject to argument over determination. Unlike some comments cited in my report, I don’t claim parity with this or DNA tests—moreover, my action-point is about sufficiency for formal investigation.
A distinction should also be maintained between “heat” and formal determination of penalty. “Heat” unfortunately has happened more widely than I or my numbers would support, sometimes as a result of others doing move tests in a less-controlled fashion, and it already brings much of the negative consequence decried in other comments. My work and letter aim to lessen the amount of convection from “heat”, while some comparative support for my tests having specific value beyond (just) being indicators of high performance is beginning to be collected here.
I am concerned about unintended consequences.
Let us hypothesize that some sort of statistical test has been approved by FIDE and ACP, and is deployed routinely in all cases of doubt.
Surely, a would-be cheater, knowing that the test exists, would merely try to be less ambitious. Surely the cheater would use a randomizer to decide, before each game, whether to cheat or not. A low enough probability of cheating would produce results that would result in a slow, steady increase in ranking; such a rate of improvement could be tuned not to exceed historical precedents. It probably wouldn’t even occur to officials to try a statistical move-comparison test. Such canny cheaters would evade the net.
I don’t know anything about how these tournaments are conducted. But if I were doing this investigation, I would hire James Randi or someone he recommends, to analyze tournament environments looking for ways that an accomplice could aid a player. The strip-search seems misguided to me: the way to go is to have an accomplice send you hand-signals or something.
This comment closely follows the line of a phone conversation I had with Frederic Friedel of ChessBase way back in January 2007—except that the person I mentioned as an example in that context was Persi Diaconis rather than James Randi. Friedel posted words I gave him on his public feedback line last year here. Yes, it’s kind-of like how using antibiotics induces “smarter” germs (albeit more by natural selection than adaptation). As I wrote Friedel privately eleven days ago, all I do is report on what I see. People have known for years that others will do “unofficial” tests, ones already given credence within the chess world—has that increased canniness? The countervailing factor is that a key-move scheme has a higher cost/benefit ratio and more ways communications can go wrong.
I think one need not randomly cheat, but randomly determine “how much” to cheat. Thus, if a computer says choice #1 for a move is “100” quality and choice #2 is “73”, then pick choice #1. If choice #2 is “99”, then often pick move #2. If for one move choices 1-4 are all “98” or higher, pick choice #4. That way one can play almost as well as the best computer, but not duplicate its top move.
Good news: It is possible to adapt tests of matching the “best” or “top 3” moves of programs to this flexible form of cheating.
Bad news: More generally, whatever rule is chosen, if it is publicized it can be gamed. Think about published experimental results. A lot have t near 2.0 (just above the threshold for statistical “significance” and, thus, publishability). If you and/or the powers that be pick Z > 4 as evidence of cheating, I anticipate a fair # of results with Z = 3.8. If you pick Z > 5, then I anticipate a spike of Z’s near 4.8. Good news: That [hypothesized] spike can demonstrate cheating in aggregate. Bad news: Even if it appears, it will not show who among the high-Z folks misbehaved.
The “AE test” noted in my report tries to evade this argument, since it is directly tied to the benefit of a move. Your “bad news” can follow as a selection effect even without cheating: if 45 research do the same idea, one will get into the top-2.3% significance bracket by chance, while the others won’t publish. Note my reference to Jonah Lehrer’s article here.
A few people have suggested I do more comparisons like the one to Alan Tate in the post, which is the only way to answer Bertie’s comment with evidence. I’ve started here. Another answer is that my report references other tests giving odds about 5-sigma even after adjustment; some exceed 5.5 which is 50-million-to-1 odds, needing one to trove 1,000 years of TWIC to find. But just like “amplifying-down” BPP error probabilities quickly hits a floor at the chance of an unrecoverable hardware error, questions about my setup certainly precede anything longer than 30,000-1 (4-sigma, as in a typical test I noted here), let alone a million to one.
A minor point: the selection effect you mention will make a lot of P = 0.05, .04, .03. 02 and .01 – ideally a uniform distribution. Such selection is surely present in published research. A conscious or unconscious manipulations (“Let’s drop an outlier” or “Let’s collect a bit more data as P = 0.06 so far”) will lead to a concentration at P values quite close to the arbitrary cutoff (e.g., P = 0.05). Thus, a one-sided jump from 0.06 to 0.05 can be selection or manipulation, while a spike at P=0.05 that declines at p=0.04 is almost surely mostly manipulation. I believe the evidence is for a spike, not a one-sided jump, so manipulation (often unconscious or barely conscious) is probably even more important than selection. (See Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22.11 (2011): 1359-1366.)
There are techniques in cheating, easy to apply and safe from proof that can be applied in all fields of interest. This is made possible by the world of abstraction, where our abstract tools are intentionally wrongly used. And what makes up a computer program flow but abstraction.
Pulsed microwave transmissions can be shaped to cause short phrases to impinge on the inner ear alone, silent to anyone else, the transmission is also line of sight and not likely to be overheard by any bystanders. The microwave auditory effect, also known as the microwave hearing effect or the Frey effect, consists of audible clicks (or, with modulation, whole words) induced by pulsed/modulated microwave frequencies. The clicks are generated directly inside the human head without the need of any receiving electronic device. The effect was first reported by persons working in the vicinity of radar transponders during World War II. These induced sounds are not audible to other people nearby. The microwave auditory effect was later discovered to be inducible with shorter-wavelength portions of the electromagnetic spectrum. During the Cold War era, the American neuroscientist Allan H. Frey studied this phenomenon and was the first to publish[1] information on the nature of the microwave auditory effect.
Dr. Don R. Justesen published “Microwaves and Behavior” in The American Psychologist (Volume 30, March 1975, Number 3).
Research by NASA in the 1970s[citation needed] showed that this effect occurs as a result of thermal expansion of parts of the human ear around the cochlea, even at low power density. Later, signal modulation was found to produce sounds or words that appeared to originate intracranially. It was studied for its possible use in communications. Both the United States and USSR studied its use in non-lethal weaponry.[citation needed]
Pulsed microwave radiation can be heard by some workers; the irradiated personnel perceive auditory sensations of clicking or buzzing. The cause is thought to be thermoelastic expansion of portions of auditory apparatus.[2] The auditory system response occurs at least from 200 MHz to at least 3 GHz. In the tests, repetition rate of 50 Hz was used, with pulse width between 10–70 microseconds. The perceived loudness was found to be linked to the peak power density instead of average power density. At 1.245 GHz, the peak power density for perception was below 80 mW/cm2. The generally accepted mechanism is the rapid (but minuscule, in the range of 10−5 °C) heating of the brain by each pulse, and the resulting pressure wave traveling through the skull to the cochlea.[3]
The existence of non-lethal weaponry that exploits the microwave auditory effect appears to have been classified “Secret NOFORN” in the USA from (at the latest) 1998, until the declassification on 6 December 2006 of “Bioeffects of Selected Non-Lethal Weaponry” in response to a FOIA request. Application of the microwave hearing technology could facilitate a private message transmission. Quoting from the above source, “Microwave hearing may be useful to provide a disruptive condition to a person not aware of the technology. Not only might it be disruptive to the sense of hearing, it could be psychologically devastating if one suddenly heard “voices within one’s head”.
I was thinking the same thing. There is however also a possibility of using ultrasonics, such as in the “audio spotlight” product which is directional sound beyond the human hearing range. When the ultrasonic beam meets with a physical substance, or a change in density, such as a human skull, the ultrasonic waves reflect, mix, and degrade into degenerate waveforms within the audio spectrum. If it is aimed at you, you can literally hear the sound emanating from inside your own head. As long as the ultrasonic decibels are low enough nobody else is going to hear it. An ultrasonic transceiver surface the size of a lollipop might be enough for such a low level signal across a crowded room, and nobody else would hear it. Its truly amazing to hear this for yourself.
Both the auditory-microwave and ultrasonic “spotlight” methods are easily detected off-axis. They’ll both splatter energy all across the room, with some lobes being almost as powerful as the main beam. Even using multiple, overlapping fields that all meet additively only in space of the targeted listener’s head will still provide easy detection if you know what to look for.
In this arms-race, the judges have the lead, as the receivers are simple and inexpensive, while the cheaters’ emitters are costly, delicate, and require special placement.
I’d like to suggest that the testing proposed in this post is brilliant, and no different than the testing for drugs in athletics: somewhat delayed, and entirely statistical. Let’s use the proposed process, and in addition, equip the rooms with simple electromagnetic and acoustic monitors which record the goings on. Analysis of the “tapes” in the event of a statistical “anomaly” will quite nicely do for me.
I wonder whether it would be possible to detect cheating based on certain properties of the chosen moves that are independent from a simple scalar rating of the quality of that move.
I once wrote a chess playing program from scratch so I have some understanding of the basic methods by which such programs play. These methods are very different from how humans select moves.
Some ideas that come to my mind:
– most chess playing programs do not perform any pruning of moves. So should no make any mistakes that have visible consequences until the evaluation horizon some N plies in the future.
– the horizon of the chess playing program is determined by the available computation time and the properties of the board position. Plus some of the previous moves’ time when thinking on opponents time. Should be possible to estimate it when analyzing the game.
– So what I’d try to do is : estimate the evaluation horizon of a group of moves by checking when and whether the (cheating) player reacts to predictable future gains/losses. Use that to estimate the computational power of the used program, then check whether the horizon stays consistent with a chess playing program of that strength for the rest of the game.
– as long as there are no forcible material games, the chess playing program can only optimize play for positional advantages. Yet computation of positional value should be much more simplistic and deterministic than what a human would think of a position. Maybe one could even determine parameters of the positional evaluation of the used program based on moves early in the game, then check whether the positional evaluation of later moves is consistent with a chess playing program that uses these parameters.
Regarding your first sentence, some of my private reports have described “tells”, and there is a public instance of that here in the discussion of the move “29..Rf8?!” (back in April 2007 when my methods were much cruder). The other ideas are indeed desirable, and are used to varying degrees by online chess servers where timing and program-behavior particulars are known. See the technical-remarks section of my report to ACP.
I should note also that some of my confreres were unhappy with my red-letter words on that page, but:
1. there was physical evidence,
2. I was supporting already-published reports,
3. I was showing a possible narrative not claiming odds, and
4. I had promised USCF tournament directors, including friends since 1972, after this Dec. 2006 meeting and repeated when I addressed the USCF national meeting in Aug 2007, that I would speak out within bounds of propriety.
As it happened the player in question was caught in Oct. 2009 using apparently the same mechanism in a Sudoku tournament, after an open letter and an investigation. This was even picked up by NPR, with reference also to crossword tournaments (noting comment by “mskmoorthy” below which came as I was typing this). But to repeat, there was observational and physical evidence.
Unlikely events are “normal”, so reducing things to a statement of probability really isn’t very good practice. I generally describe this as “sprinkling Gauss water on the problem.”
I’d suggest looking at probability density functions. This leads to combinatorial complexity and data volume issues in the case of chess. But aberrations in the PDF are generally conspicuous and difficult to obscure or mask.
Thanks (to all new commenters here—first comment is moderated then rest are immediate unless flagged). What ideas for PDF irregularity might best work, given that the sample size of played moves itself is typically only in the 200–300 range?
If this person is cheating, it’s not through magic. It is possible to detect any method of communication by controlling the environment of the subject, watching them carefully, and searching their person. Since he was searched and nothing was found, then the search was not thorough enough, or the mechanism of cheating was not on his person (which goes back to controlling the environment).
I think that statistical analysis is a critical tool in detecting cheating. We can’t very well strip-search every player all the time, or conduct all tournaments in carefully controlled rooms. The costs would be astronomical, and it would certainly ruin the fun. Statistical analysis can tell us which players are likely to have cheated, and they can be targeting for more careful monitoring.
However, under no circumstances should statistical analysis alone be enough to invalidate a win, or ban a player. Unlikely things do happen, even very unlikely ones. Sometimes someone will have a very lucky game. People with previously unseen abilities will pop up from time to time.
Probability is certainly a good indicator of where to look, but it is not primary evidence, and it’s a far cry from the actual proof that (should be) required to invalidate tournament results.
The four minute mile:
I see the OP saying they tested this and that through various chess computers like they are a black box. The only black box here is the human brain. All chess computers are based on algorithms, and therefore one in a billion brains can be expected to be able to form a circuit superior out of chance. Couple that with training against a computer from a very early age, and you would have a formidable opponent and potentially much better odds of a “perfect” chess player.
The only real test is to compare each computer move, and the ultimate reason for the move, that is, look into the algorithm and the steps taken to choose the move. Then ask the human chess player the same question.
It would be relatively easy for a human to internalise the most intense algorithm (a trillion times more efficiently) and run it subconsciously, similar to how you drive a car after 5-10 years of experience.
For example, it has been discovered that we all have a Jennifer Anniston neuron: http://www.newscientist.com/article/dn7567-why-your-brain-has-a-jennifer-aniston-cell.html, if one neuron can store so much, what can a interconnected network of 10 do, 100, 1 million.
For a chess player, leaning from the age 3, with only a computer as opponent, they may be able, to internalise and shortcut all the algorithms that the human programmer has equipped the chess computer system with.
If the system a person is learning from is good at playing chess, the human student could eventually learn to play as well or better than that particular set of algorithms.
They will be unbeatable by a person whom has not had this experience, and their moves will be indistinguishable from computer moves, until that is, they meet someone who has had a similar upbringing and response.
Now that will be a good game of chess, or at least a good stalemate.
Actually, your car analogy is an excellent one: I have been using cars virtually every day of my life, for roughly the last 20 years, and yet I still can’t run a mile in under a minute. The notion that a human can learn to play ‘like a computer’ and specifically at computer speeds by playing against computers just isn’t supported by any of our evidence on either chess, computer AI or human cognition. Could you pluck the name of your best friend out of an unsorted phone book in less than a second? If not, then why should you think you could pluck the best move out of a billion unsorted alternate paths in that sort of time frame?
I see more than the two given alternatives. In a high stakes top end chess game with significant money involved, the possibility that a paid off official, or even his opponent could benefit financially from a carefully planned victory should be considered.
Given the two unlikely options presented, pure greed seems like a reasonable third alternative.
This rambling unfocused article essentially says that the player must have cheated because his success is statistically not possible. No evidence is presented that actual cheating took place.
What is fairness, what is enhancement, what is cheating? As drugs and technology provide enhancement, what is a competition about? Smart enough to program and use a computer for help? Willing to jack one’s body up on speed to win? Using metal legs to substitute for the one’s you weren’t born with? I think there needs to be a step back to re-examine what it is that is fair and equal, a redefinition of terms to fit today’s environment.
Lillie, as we grow to appreciate the molecular processes by which pharmaceutical cognition-enhancers exert long-lasting epigenomic effects … we may need to take *two* steps back … at least in regard to pharmaceutical enhancement. The point is that computer cheaters do not unilaterally alter unborn generations … and moreover computer cheaters do not unlevel the playing-field for men and women … whereas pharmaceutical enhancers may do just that.
The easiest policy is (of course) willful ignorance and institutional disengagement … this policy was disastrous for baseball and cycling in regard to steroids, and therefore academia should not follow this path in regard to cognition-enhancers.
Regarding academia, you have an unusual situation where the individual academic’s advantage is society’s advantage. Enforcing a rule saying that there should be no cognition enhancers risks being a form of cartelisation, where the cartel act against the common good for the sake of internally-calibrated ‘fair play’.
In this case, cheating in chess is actively misleading as an analogy, as the rest of us appreciate chess for being a human activity, unimproved by machine or chemicals much stronger than coffee, and the beneficiaries of chess matches are the community who value raw skill.
As those academics who use cognition enhancers on interview would also use those enhancers at critical times, their enhanced performance is representative, and therefore should not be penalised.
As investigation into performance enhancing drugs is likely to be performed by the academic community themselves, possibly against the interests of society at large, I would suggest that wilful ignorance is, in fact, the best policy.
Am I the only person to consider that maybe the cheater had assistance in the audience? Someone sitting in the front row might have a smartphone remotely connected to a powerful computer, and relay the moves to the chess player via gestures, sitting positions, etc.
Is it possible to detect cheating (I do not know if there is cheating ) say in ACPT (American Crossword Puzzle Tournament) – even when the finalists are solving in front of the audience
Will Shortz himself commented on related aspects here.
Under what circumstances (other than physical evidence) could cheating at chess be obvious? In a tournament setting, for example, could an opponent be set up to reveal cheating by another player?
The simplest way to explain seemingly cheating moves without an actual cheat device would be sandbagging: intentional losses to obtain a lower rating. It may be more telling to analyze his pre-tournament record and look for intentional mistakes.
But why? I can understand sandbagging if you are a low Master who wants to enter and win a non-Master tournament, but if your natural skill level is in the high Grandmaster range, why draw suspicion to yourself by sandbagging, as opposed to just doing your best and being recognized for your brilliance?
A better test would be subjective–ask him, after the games, to explain his thinking. Why did he do x y and z brilliant moves? The less he can explain, the more likely he is cheating. Someone as strong as that ought to have a philosophy, a way of seeing the board and pieces, something that they can expound upon or at least give hints to. Another grandmaster ought to be able to tell if they are spouting BS instead.
Wagering.
I think there is a bigger picture here. Why even play a strategy game that a computer, without any information or connectivity advantage, will win.
Because it’s still fun, has a great history, and has more public participation all over the world than any time previously. Computers are still a step behind the best humans at the Japanese form of chess (Shogi), and human supremacy at Go is apparently not threatened in the near future. My best effort at a more computer-resistant “evolution” of Western chess is here.
Google’s AlphaGo AI Go player is yet another indication that computers will eventually dominate humans in everything.
Spectrum Analyzer. You should be able to use a spectrum analyzer and decent antenna to find any EMI emissions from a person. An electronic device will have EMI coming from it because of the various oscillators.
Almost everything these days has computer chips. What if there is someone with an eye implant, heart pacemaker, artificial limb etc with a computer chip? Are you going to have it destructively tested to see if it can play chess? Or what if one day it becomes normal for people to enhance themselves with computer chips? I think its easier if you just legalise it. If everyone knows what the computer recommends, then you are all equal.
Interfere with any form of radio signal by jamming all communications during a chess match to create a secure room for the chess players to battle it out in. No player should be offended by such methods to ensure the game is free from cheaters.
I know nothing of chess at this level so am just playing devils advocate but if they have found no evidence of cheating then surely the statistics show an incredible improvement. If statistical improvement is used as then surely the grandmasters et,al are going to be accused of defending their positions.
Has the guy been through any kind of drug tests?
Is it possible that this man may be a savant when it comes to chess moves? I’m not defending him, it’s just that savants have been shown to be capable of extraordinary talents in narrow fields, and perhaps this is a case.
Perhaps you’re all looking in the wrong place. Consider the possibility that this man is actually far far better than he let on early in his career. I don’t follow chess tournaments but would this low ranking not allow him to pick easy opponents and shape the tournament to his liking? He would then be in a position to show his true genius at a time and place of his own choosing. Personally I prefer the intrigue of the chess world being “hustled” in this fashion. LOL
How could one take any pride in winning by nefarious means? That strikes me as the lowest of the low positions to occupy. I stumble, I read, I study various positions and their outcomes as explained in such books as the Big Two (IMO), Aron Nimzowtsch’s “My System” and Vladimir Vukovic’s “The Art of Attack in Chess”.
Personally, I could not imagine any possible advantage to be gained by cheating in chess. Perhaps I am naive, and there lurks some potential economic gain by winning this or that tournament, but should you or I do so, then I feel that the weight of being a fraud and a counterfeit would outweigh the emotional gains from having “won” this or that tournament.
Perhaps I live in an aquarium in which such petty desires (i.e. the desire to win, to make a living playing chess, to be cited as an expert player with an innovation on this or that opening or middle or end-game) are not important. That’s how I choose to play this game that I love.
I am 65 years old and thus extremely unlikely to even enter, let alone place, in any tournaments hosted here in Toronto. But that does not mean, nor should be equated with the belief, that I do not love this game as much as you young and foxy Turks. I’m an old man, I accept that, and that means, sadly, that I cannot learn new tricks as quickly as young men and women. But that does NOT mean that I have thrown in the towel. I shall battle on, despite my ageing grey cells, and now and then defeat at least one or two of the best and brightest among the new generation. Or maybe not, and that’s not so bad either. In fact, I would LOVE to be trounced by several up-and-coming new faces on the scene, and thence to take pleasure in saying, “Wow, she was soooo good at age 14, she crushed me like an insect in only 15 moves. Wow!”
Some people regard victory as the only acceptable outcome. I am not one of those people. I have played sessions with superior players in which I have lost 20 in a row. From my perspective, what matters is How many moves did you need to defeat me? If I can increase that number, then I’m perfectly happy to lose 50 games in a row.
Your Checkmate doesn’t impinge upon my self-esteem at all. I’m in this game to learn, and each defeat teaches me a little something: bad move at 15, etc.
I’m still a student in this most wonderful game, and always looking to learn something new.
Arthur
> How could one take any pride in winning by nefarious means?
For the same reason one would take pride in winning any game, if winning by cheating is simply seen as a different game. Successfully cheating in this environment is pretty impressive.
I don’t think that an electronic or high-tech solution is behind this case. It can be as easy as having a friend between the public who signals the moves with any code, like moving from a board of the room to another. In fact, a similar technique was used in an uncovered case of cheating involving french masters like Sebastian Feller.
What if you’ve all been had by a Chess Shark? Ivanov was sandbagging his “pre-tourney” scores…holding back the best if you will until the tournament and then unleashed his true identity?
I kind of hope that is what went down. But masterful cheating at a chess tournament is also interesting. IF the latter is the case – I’d like to know how it was done.
Clever.
You know, you don’t need electronics to cheat. I don’t know what this format of this tournament was like (was anyone allowed to stand in eyesight of the player?). However, using a simple code (say, left hand on a specific part of the body for the letter, right for the number, doing one of a few motions for the piece), or just straight up morse code by tapping one’s hand against the arm, a partner in crime could have the electronics on him, and silently communicate moves to the player. Perhaps video of the event should be taken, and any movements in the audience caught. (Casinos have a lot of expertise in this. Perhaps consult them?)
Computer programs are based on algorithms. If a player uses the same algorithms then their games will be similar to those of the computer. If the person reporting the games chooses to falsify the actions of a player all attempts to verify the activity of the player will be useless.
If you have no proof of the person’s cheating and you say that they cheat then you slander them. All that you can say is that their playing is too similar to that of a computer program for you to believe that it is so by chance or by their independent play.
I believe that the “Greedy” Cheaters are easily caught by your method Dr. Regan. This clown Ivanov claims he beat Houdini 10-0 which is absurd. I am more concerned (as I’m sure you are as well) about the player who leaves during 1 or 2 critical points during a game and cheats then to either save his game or find the winning tactic… Take a 2000 player with houdini getting up to go to the bathroom 3 or 4 times during a 5 or 6 hour game…suddenly this player because of his “helper” becomes much stronger and much more difficult to detect (although perhaps a bit more obvious?).
Well understood. One aspect of the frequent comparison to Lance Armstrong is, why did he get “greedy” for the comeback?
Is it is posible that the players with low average scores are deliberate and such player is able to play higher level game on competitive occasion. Would these players then be called Cheats
The New York Times reported today: Chess: A Master Is Disqualified Over Suspicions of Cheating
http://www.nytimes.com/2013/08/18/crosswords/chess/a-master-is-disqualified-over-suspicions-of-cheating.html
Jens Kotainy, a German international master, was disqualified after it was found that his cellphone gave off vibrations that resembled Morse code. The tournament director,
“Goldschmidt had been suspicious of Kotainy because several experts — including Kenneth W. Regan, a computer science professor in Buffalo who is working on a program to detect cheating — found that Kotainy’s moves had matched the choices of a leading computer program almost exactly. Regan said he had found the same pattern in Kotainy’s games during his previous two tournaments.”