Chad Perrin: SOB

27 April 2008

real closures

Filed under: Geek — apotheon @ 07:46

I haven’t talked about lexical closures in a while. There was a brief time, months back, when it was almost all I talked about here — but then I went for long periods without touching the subject here at SOB. I feel inspired to discuss it again. One of the reasons I feel thusly inspired is having read Closures in Ruby and Python at the weblog “A Ship at Port”. The author, “Matt”, expresses his disappointment in a lack of clear explanation for why he should give a damn about the existence (or nonexistence) of proper lexical closures in any given language. I hope to provide such an explanation suitable to helping any of my readers who have the same questions understand, in basic terms at least, some of the practical value of closures.

What’s a “lexical closure”?

For those who aren’t entirely clear on the concept, a lexical closure (or “closure” for short) is a first-class function that has an enclosing lexical scope “closed” around it.

There are a number of implications of this state of affairs in a lexical closure — many of which are often included in definitions of closures, even though they are really consequences of lexically closing a function rather than defining characteristics. Among them:

  • A first-class function is a function that is treated, in computer science terms, as a “first-class object” — also known as a “value”, “entity”, or “citizen”. They can be treated exactly like entities that can be assigned to variables, operated on, evaluated, and otherwise manipulated exactly like data. “First-class objects” have been occasionally defined as entities that belong to a “first-class data type”, in fact — which means that first-class functions are functions that belong to a first-class data type as well. Generally, first-class functions only exist in languages that either do not differentiate between code and data (e.g. Lisp) or provide some mechanism for turning data into executable code and vice-versa (e.g. Java, though it’s kind of an unnecessarily painful process in that language last I checked, and some point to flaws in the resulting Java object that they say disqualify it as a true first-class function).

  • Lexical scoping, generally speaking, is a form of static scoping (determined at “compile time”) where a given scope has access to whatever’s in its enclosing scope, but the opposite is not true. Thus, if you have a scope containing variable food nested inside an enclosing scope that contains variable bard, the nested scope can also access variable bard, but the enclosing scope cannot access variable food — because in the enclosing scope, food is out of scope.

  • A scope is “closed” when whatever it contains “goes out of scope”. For instance, in a lexically scoped language a function has its own scope — and when that function ends, it “goes out of scope”. This means that any variables that were created within that scope cease to exist, their place in memory freed up and any data they carried “forgotten” by the program. The usually way to keep data from within a function that has ended, closing its scope, is to generate a return value that contains that data — which is, in fact, simply data and no longer part of that (now closed) scope.

  • A common way to create a first-class function is to use a normal, named function, and generate a return value that consists of an unnamed function that can be assigned to a variable or otherwise operated on like any other data. Doing this all by itself does not really create a closure, though, because it doesn’t have an enclosing scope “closed” around it — it just creates a first-class function (a function as data, essentially), indistinguishable in practice from a function created as a lambda without the aid of an enclosing function. The lambda kernel method in Ruby can be used for this purpose, for instance.

  • To close lexical scope around a first-class function in the example of creating it inside another function, you need to actually have some connection between the generated first-class function and the enclosing scope. Thus, if there’s a bard variable in the enclosing function’s scope, and the first-class function makes use of that variable somehow, the enclosing scope is “closed around” the first-class function (creating a closure). Otherwise, the enclosing scope simply closes, and that’s it: it’s gone. When a scope closes on nothing but itself, it evaporates; when it closes on a first-class function, however, it becomes a closure — a closed, but persistent, scope, with persistent state.

  • Protection, in computer science terms, is usually applied to concepts of object oriented programming. Encapsulation is a related term. Encapsulation basically means that a chunk of code is encapsulated as a single, (semi-)atomic entity by a surrounding API, so that the only way to access it is via that API. Object oriented languages achieve encapsulation to varying degrees of success. Protection is to a datum as encapsulation is to a chunk of code — it ensures that a given set of data is protected against access from outside its parent entity (i.e. an object) except via a specific API. As with encapsulation, object oriented languages achieve protection to varying degrees of success. In languages where code and data are not strictly separated from one another conceptually (e.g. Lisp), the distinction can get a little fuzzy.

  • A proper lexical closure effectively provides “perfect” protection. Because the enclosing scope of a closure has gone out of scope, it can no longer be directly accessed (in theory), leaving calls to the closure function’s API as the only means of operating on the data from that scope. If the closure function’s API does not provide any means of accessing some of that data, you simply cannot access it. There are, in practice, exceptions to this (of course) — such as in the case of pointers, references, and other ways to “get around” the closing of scope, but these involve basically reaching directly into the memory used by the program and twiddling bits in some way. That’s a messy, dangerous, and usually discouraged means of working on data. Short of using such means of access to produce programming tricks like (for instance) creating objects, first-class functions, linked lists, and closures using references in Perl, these means of direct access to stored data are generally a sign you’re doing something wrong — like creating memory violation conditions as some kind of dirty hack rather than coding something up properly.

Why should I care about closures?

The canonical, “Hello World” style of closure example is an incrementor — something that, every time it’s called, produces a number greater than the previous number. I’ll show examples of how to generate such incrementor closures that are configurable in two ways, and the closure generator will take two arguments to achieve these configurations. The first is a starting number. The second is an incrementing quantity. Thus, you could generate an incrementor that starts at a number of your choosing and, when called, produces a number incremented by a value you chose as well. My examples should work fine on most Unix/Linux system configurations, depending on your execution path configuration, et cetera.

First, I’ll present an example that shouldn’t really work anywhere — the most basic incrementor closure example, presented in non-executable pseudocode. In this example, there are no arguments sent to the generator, and the resulting closure is non-configurable. It just starts at zero, and prints out a value incremented by one each time it is called.

Pseudocode:
  function generate {
    base = 0
    return lambda { return base += 1 }
  }

  incrementor = generate
  execute incrementor
  execute incrementor
Incrementor closure in Perl (where the my keyword creates a variable with lexical scope):
  #!/usr/bin/perl -l

  sub generate {
    my $base = $_[0];
    my $interval = $_[1];

    sub { $base += $interval }
  }

  $incrementor = generate( 5, 2 );
  print $incrementor->();
  print $incrementor->();
Incrementor closure in Ruby:
  #!/usr/bin/env ruby

  def generate( base, interval )
    lambda { base += interval }
  end

  incrementor = generate( 5, 2 )
  puts incrementor.call
  puts incrementor.call
The output of both of these looks like so:
  7
  9

You may ask why you can’t just create a function to do the above for you without all this mucking about with abstract concepts like first-class functions and closed lexical scope. For instance . . .

Perl example:
  #!/usr/bin/perl -l

  sub generate {
    $base = $_[0];
    $interval = $_[1];
  }

  sub incrementor {
    $base += $interval;
  }

  generate( 5, 2 );
  print incrementor();
  print incrementor();
. . . or in Ruby (where the dollar sign on a variable name gives the variable global scope):
  #!/usr/bin/env ruby

  def generate( base, interval )
    $base = base
    $interval = interval
  end

  def incrementor
    $base += $interval
  end

  generate( 5, 2 )
  puts incrementor()
  puts incrementor()

Both of these will produce exactly the same results, run as-is, as the previous examples. They ignore certain functionality that may become necessary in more complex programs, however. For instance, each of them provides exactly one incrementor function, and every time you need an incrementor that starts at a new number you must reset the values of $base and $interval. If you then want to go back to an earlier set of configurations for an incrementor, because you want to increment two things differently, you must restore your earlier values where you left off with them manually. Check out examples in Perl and Ruby.

With the closure example, however, you can simply create new incrementors for new configuration values, and use the old incrementors again when you need them, without so much duplication of code and data — and without the attendant maintenance nightmare. Check out examples for this state of affairs in Perl and Ruby.

Is every single Ruby block really a closure?

So far, I’ve explained closures and provided examples not only of how to make a closure but of how their properties may be of some use to simplify code and improve maintainability. Last but not least, I’ll tackle another subject that helped inspire this post.

Some time ago, discussion of closures on the ruby-talk mailing list alerted me to a bizarre cultural characteristic of the Ruby community: it is the semi-official position of the most vocal members of the community on the subject of closures that every single “block” in the Ruby language is a closure. Keep in mind, as I discuss this, that in Ruby the word “block” is used with a very particular, specific meaning that differs from the casual manner one might use the term “block” to refer to a chunk of code within its own scope or routine in other languages.

The problem with the notion that a Ruby block is automatically and always a closure is that a block can be created that only exists as long as its parent scope — or even goes out of scope before its enclosing lexical scope does. In fact, a block could conceivably be created whose sole enclosing scope is that of the program itself. Obviously, if the program ends (thus going out of scope), the block does too.

This means that the lexical scope within which the block exists may not be closed around that block. The block is essentially a first-class object, satisfying one requirement, but the lexically closed scope around the block may never happen. This, it seems obvious to me, should disqualify some blocks from being closures — which means the statement that all blocks are closures in Ruby is false.

The rejoinder suggests that the lexical context of the block does not need to go out of scope for the block to be a closure. All that is needed, according to people who hold to this notion, is that the block needs to have access to that lexical context. It doesn’t even need to make use of that context at all — doesn’t have to actually access the enclosing scope, but only be allowed access.

There’s one small problem with that:

That’s a defining characteristic of lexical scope.

In other words, by that definition of a closure, anything that contains a lexical scope and exists within an enclosing lexical scope is automatically a closure. This, of course, would mean that any language that provides lexical scoping allows closures.

I don’t know about you, but I find this notion absurd. It’s poppycock. Closures consist of more than just a scope within a scope. WTF meaningful existence can a closure have if it’s nothing but a scope within a scope? Why would anyone have bothered to refer to it by the name “closure”? Where, for that matter, would the term “closure” have arisen if you never had to close a scope to get a closure?

. . . and why would everyone have been lying about Scheme being the first formal existence of a proper closure for all this time? Lexical scoping certainly existed before Scheme hit the scene.

What else is there?

For a previous discussion of lexical closures, it might be worthwhile to check out my SOB entry of August 2006 titled Linguistic Elegance Test: Closures and Lists. This includes some links to explanations and definitions of closures, too.

Any questions?

(dedicated to Sterling)

edit: It suddenly occurs to me that maybe some of the members of the Ruby community fail to differentiate between closures and callbacks. Most, if not all, Ruby “blocks” are effectively callbacks. They do not fit the definition of a closure, but the difference may be subtle and easily missed, depending on one’s perspective.

Why Web2.0 Works (and what doesn’t work)

Filed under: Cognition,Liberty — Tags: , — apotheon @ 01:22

Yesterday, in Is there one thing I must say to the entire world?, I made a comment about the power of a good idea and the fact that, with such an idea, there’s no need for protectionist business practices of the sort employed all too often in creative industries (music, film, dead tree media, software) these days. The intent was to write something in 150 characters or less that made for a message to the world at large.

Somewhere along the way, I had a brain fart and wrote such a message in 150 words or less. It looked something like this:

If you come up with a good idea, you don’t have to force people to buy in. If it’s really a good idea, you can’t stop people from giving you money. If you try to force people to buy what you’re selling by preventing them from sharing it for free, you’ll alienate people and possibly ensure that what could have changed the world for the better will instead fade into obscurity. If people aren’t beating their way to your door, it’s probably not a good idea — and if you’re losing all your money through distribution and marketing, you’re probably dealing with an industry that has made a habit of alienating customers. So please stop telling me what I can and cannot do with what you’ve given me; trust that with a minimum of business sense you can make a living without wielding the law like a club.

There’s more to it than even that. There’s the fact that allowing people to share what you’ve created is the single most effective form of marketing you can employ if your work is actually worth anything. There’s also the fact that people value “originals” — the original, “official” release of something, a live performance, et cetera — which will always drive people to the creator even if third-party low-cast copies are readily available. Finally, there’s the simple fact that with great talent there’s always a way to build a lucrative business model in a free market, whether you have law enforcement agencies and armies of lawyers running around forcing people to behave a particular way or not.

So . . . what does this have to do with Web 2.0?

Some hints might be found in my older SOB entry, how not to make money on the Web:

Any company still in the business of selling software licenses is going to have a consistently tougher time making money as time progresses. Software does not lend itself to being treated as tangible property: an entirely separate body of law (copyright law, specifically) is required to make it possible to even try, and the process of trying creates new expenses in enforcing this unnatural business model based on artificial scarcity (which any economist will tell you is ultimately an unwinnable game).

Something like Montastic could be monetized by nothing more than a Google AdSense banner. Productized software, meanwhile, requires legions of lawyers, distribution channel partners, and insane wastes of development resources creating enforcement software like WGA and other DRM-leaning “solutions”, all of which ends up treating your customers like your enemies when they should be your friends.

What is Web 2.0?

There are people all over the place begging for an opportunity to tell you what “Web 2.0” means. They’re full of buzzwordy ramblings about leveraging this and monetizing that and the implications of the other thing. The closest I’ve seen to actually approaching an insightful explanation of “Web 2.0”, in the form of what works and what doesn’t, goes something like this:

  1. simple, clean, user-oriented, functionality-focused design

  2. building value through user participation

  3. letting the data come to you

  4. non-obvious means of turning a profit that are tangential to the main product or service

There’s an underlying principle to this, however, that is being overlooked by just about everyone. It’s the key difference between successful “Web 2.0” and unsuccessful “Web 2.0 manque”:

  1. Employ a business model defined by the customers.

That’s about it. There’s no imposition of business model from above in the form of legal limits. There’s no trying to make the customer base conform to a preferred way of making money. There’s no coercive marketing, high-pressure sales, or obfuscation. There’s no priesthood of product or service development dealing with the supplicants of the customer base.

Is this a new business model?

It’s all shiny and new — right? Except . . . it’s not. It’s basically the oldest business model there is. It’s a business model of free exchange and cooperative advancement. It’s the most robust form a business model can take:

  1. Nobody can undercut you by giving the customers more of what they want if you’re not withholding anything.

  2. Getting the customer base involved gives them a stake in your business model’s success, so that even if you’re not always at the top of the market niche you’re never drowning or even treading water.

  3. Even if the entire legal system collapses, that doesn’t directly affect your ability to keep making a profit with your current business model (unless, of course, your business model is tied directly to the current legal system without actually depending on it for the form of the business model — but that’s a weird edge case).

Remember: the most valuable innovations typically aren’t innovations in getting people to buy your shit. Instead, they’re typically innovations in how to achieve what you already know they want. It’s recognizing a need for an improvement in the satisfaction of the customers, and not creating new needs for satisfaction.

. . . and making your profit stream rely on the notion of “intellectual property” is a great way to sabotage your own “Web 2.0” business model. When your business starts hemorrhaging money because you can’t effectively stop people from “pirating” your “intellectual property” (note scare quotes in this paragraph around terms used by champions of coercive business models to confuse readers/listeners), just remember the cardinal rule of a failing get-rich-quick scheme before you come whining to me about “thieves”, et al.:

It’s not my fault your business model sucks.

All original content Copyright Chad Perrin: Distributed under the terms of the Open Works License