Chad Perrin: SOB

15 October 2006

the importance of false positives

Filed under: Cognition,Geek,Liberty — apotheon @ 02:40

I’m going to discuss the importance of false positives today. I’ve done this before, of course. In this case, however, technical examples will only be used as analogies to the real point of this SOB entry — the Military Commissions Act, which President George Bush will be signing into law on the Seventeenth of this month. Don’t worry, I doubt they’ll disappear me. I’m popular enough that people would notice I’m missing, but not popular enough to be in the top-1,000 list of people to disappear.

Back in April, I wrote about how I addressed the problem of blog spam at the time. We might consider, in this case, comments to be analogous to people and blog spam comments to terrorists, all of which are trying to get into the “country” of SOB:

  • There was an objective measure of a preponderance of evidence in place, in the form of checking the number of links in a comment: if you have too many, you might be a spammer in need of elimination from my weblog. As such, incoming comments with “too many” links are fed into a moderation queue, where they are not publicly displayed unless and until I approve them. This is something like taking people aside at Customs in an international airport to check them out when they are found to have more than a predetermined limit of “links” to suspicious organizations. This can be annoying, but isn’t as bad as random checks (dumping one out of every four incoming comments into moderation, for instance), or using a “no-fly list” made up of common words that happen to appear in some spam messages. It might be a bit like ethnic profiling, checking for Islamic connections rather than actual terrorist activity, though — and it might help cut down on the number of spam posts that slip through while still (eventually) letting all non-terrorists through.
  • Like terrorists faced with such a half-baked system of trying to detect terrorists, the spammers started using posts (suicide bombers) that don’t have enough links to suspicious organizations to set off any “alarms”. They started basically “flying under radar”. They even started using the equivalent of blonde-haired Muslim converts named Bob Smith. As such, I started requiring registration (and being signed in) to post at SOB. This was a bit like a national ID card and keeping a database on Un-American Activities performed by each registrant. People chafed at this (silently, for the most part) and didn’t post comments as much. It was seen as burdensome and, frankly, only those who knew me well enough to trust me had any certainty I wasn’t collecting personal information for sale to other spammers. Nobody likes to be registered, and I started to get something of a “false positive” effect in that the system of protection against spammers was also “protecting” against legitimate comments sometimes. Ultimately, I had to discontinue this policy for reasons of sanity.
  • I made a conscious decision to avoid using validation images to ensure that “real human beings” were posting comments as a means of preventing blog spam. Part of the reason for this is that “real human beings” sometimes post spam comments. Another part is that sometimes “real human beings” have trouble viewing such images (as demonstrated by the regular failures of Blogspot/Blogger antispam measures to actually show the image at all). Yet another is that any method that is sufficiently clear to ensure its readability is potentially susceptible to spambots with OCR capability. The biggest problem is false positives: I can handle a little bit of spam slipping through the cracks (and cleaning it up later) if it’s the only way to ensure that no legitimate comments get blocked or otherwise lost forever. What we have here, in analogy, is the principle of preferring that one hundred murderers go free rather than imprison a single innocent man. Detain a couple minutes for questioning, maybe, as long as it’s not enough to drive him away forever — but not imprison.

More recently — just last month, in fact — I commented specifically on false positives in more depth. I opened with the following:

When you’re talking about lines of communication, such as email, blogging comments, and your telephone, you’re talking about a medium that is open to abuse. To make a means of communication immune to abuse is to make it useless as a means of communication. This doesn’t mean you cannot reduce the incidence of abuse through clever tricks, careful use, and whatever jumping-through of hoops may come to mind. It just means that the vulnerability still exists. The key is not to believe we can eliminate abuse of a means of communication entirely, but to minimize it so that the abuse is manageable.

I later clarified and summarized, saying that to truly eliminate any ability to abuse a line of communication destroys it. This is directly analogous to the notion that completely eliminating the ability of malicious people to act in a free society requires eliminating the “free” in society. This is a concept that seems lost on a lot of politicians — apparently more Republicans than Democrats, and almost no Libertarians appear to suffer this delusion, but it does cut across party lines somewhat.

I further said, in that earlier entry:

The first mandate for solving a communications abuse problem is to avoid interfering with communication itself. In the case of systems like spam filtering, this means that one should avoid false positives first and foremost.

In the case of systems like the United States system of law, where the abuse in question is of society’s protections of individual rights and liberties, that means one should avoid violating those rights and liberties oneself in pursuit of protecting them from abuse.

One might say that, in the so-called War On Terror as in my own personal war on blog spam, false positives are the root of all evil. What value is the security of our nation’s innocent citizens if we imprison, torture, and kill them along with the terrorists from whom we wish to protect them?

Yesterday, I commented on the passage of the Military Commissions Act, paying particular attention to its violations of habeas corpus, the redefinitions of torture (and the bit of media attention that has gotten), and the voting record of the House of Representatives (and, later in comments, the Senate) on this particular issue. Today, thanks to a Bruce Schneier security newsletter email I received, I was made aware of a brief illustration of the more obvious problems with the Military Commissions Act that uses some C source code for an example. Kevin Poulsen provides the following example of the Act as C code in his weblog post Bad Code:

if (person = terrorist) {
} else {

For those of you who are not aware of how C code works, I’ll see if I can explain this one.

  1. The term person and the term terrorist here appear to be variables. Normally, in an if conditional statement like this, it should be comparing the values of these two variables to see if they’re identical. Presumably, the terrorist variable might contain a statement such as “kills civilian noncombatants for political purposes”, while we don’t know what the value of person is in this case — thus the need for the comparison.
  2. The operator between person and terrorist is, in this case, =. Unfortunately, that single equals sign is an assignment operator, not a comparison operator, in C. In other words, this code is assigning the value of terrorist to the variable person, thus making person and terrorist identical in the eyes of this code rather than just checking to see if they’re the same. If you actually wanted to do a comparison, you’d use the == operator instead — a double-equal-sign operator. This is subtly critical to the analogy to the Military Commissions Act, which just assumes that somehow the law enforcement personnel who are “detaining” (imprisoning indefinitely or, in the case of citizens, for up to a year, without having to actually have probable cause) people for “questioning” (interrogation and torture) know what the hell they’re doing without considering whether they might make mistakes in picking out targets. It’s as if Congress believes the Department of Justice is infallible and can never “accidentally” detain a non-terrorist, so it doesn’t matter if people get scooped up off the street and made to vanish without due process, because due process only slows down these morally perfect supermen incapable of error.
  3. It gets worse, though — and slightly more subtle, too. The line that reads punish_severely() refers to a function, presumably defined somewhere else in the program. It’s a collection of code that performs some useful bit of action for you whenever it’s called by the punish_severely() syntax. If you wanted to send it a parameter — some bit of information it needs to be able to change the way it operates when it runs — you’d say something like punish_severely(person), thus ensuring that it specifically operates on whatever the person variable currently contains. If you happened to be correctly checking for equivalency between a given person and the value of a terrorist, via the statement if (person == terrorist), the statement punish_severely(person) would effectively send that terrorist-equivalent person off to Guantanamo Bay or whatever. Because no parameter is passed to the punish_severely() function, however, it just sorta does stuff without any regard for what’s going on in the conditional test. In other words, we don’t really know what it is doing at all. If that doesn’t scare you, I don’t know what will.

So, in short, the question “Is this person a terrorist?” has been replaced with “Given that a person we target must be a terrorist . . .” and the implication that terrorists will be severely punished as been replaced with the implication that someone, somewhere, will probably be punished severely — and it likely involves the person detained for questioning and anyone with the same name, or the same hair color, or . . . whatever. We don’t know the target of punishment, we don’t know the form the punishment will take, we don’t really know who’s being punished, and all we know about the person is that we’ve labelled the person a terrorist without bothering to check whether that person is, in fact, a terrorist.

Welcome to the Military Commissions Act. As Kevin Poulsen seems confident will happen, the Supreme Court might catch this in debugging. That assumes, of course, that the code will ever be sent to the Supreme Court for debugging at all. Something he seems to have overlooked is the fact that the wording of the Military Commissions Act specifically makes it difficult for the matter to ever reach a courtroom at all.


  1. […] SOB: Scion Of Backronymics » the importance of false positives apotheon adds insights to the (person = terrorist) code allegory (tags: liberty terrorism) Share and Enjoy:These icons link to social bookmarking sites where readers can share and discover new web pages. […]

    Pingback by links for 2006-10-16 -- Chip’s Quips — 15 October 2006 @ 07:17

  2. I wish there was somewhere that we could go. I think this country is too far gone to really care. ):

    Comment by Alex — 16 October 2006 @ 03:54

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

All original content Copyright Chad Perrin: Distributed under the terms of the Open Works License