Chad Perrin: SOB

23 August 2006

Linguistic Elegance Test: Closures and Lists

Filed under: Cognition,Geek — apotheon @ 04:05

I have talked about lexical closures a fair bit here at SOB. They factor heavily into what I’m about to discuss, so I’ll try defining them for those of you who might be unclear on the concept. This assumes some basic understanding of programming concepts, however: my target with this is programmers, after all.

If you visit Wikipedia at the Closure disambiguation page, you’ll find this definition:

an abstraction binding a function to its scope

From there, clicking on the link for Closure (computer science) takes you to a page whose first line of text defines closures thusly:

In programming languages, a closure is a function that refers to free variables in its lexical context.

It then goes on to say this:

A closure typically comes about when one function is declared entirely within the body of another, and the inner function refers to local variables of the outer function. At run time, when the outer function executes, a closure is formed. It consists of the inner function’s code and references to any variables in the outer function’s scope that the closure needs.

Closures are commonly used in functional programming to defer calculation, to hide state, and as arguments to higher-order functions.

A closure combines the code of a function with a special lexical environment bound to that function (scope). Closure lexical variables differ from global variables in that they do not occupy the global variable namespace. They differ from object oriented member variables in that they are bound to function invocations, not object instances.

The thoroughly excellent book Learning Perl Objects, References, and Modules, published in later editions under the name Intermediate Perl, offers the following explanations:

The kind of subroutine that can access all lexical variables that existed at the time it was declared is called a closure (a term borrowed from the world of mathematics).
Closures are “closed” only on lexical variables, since lexical variables eventually go out of scope. Because a package variables (which is global) never goes out of scope, a closure never closes on a package variable. All subroutines refer to the same single instance of the global variable.

In short, then, a lexical closure is (in my own words):

a block of code that can be passed around as a first-class function that contains data that achieves protection and encapsulation because it is assigned to a lexical variable in the closure’s enclosing scope, and has subsequently gone out of scope, though the closure persists

I could probably make that prettier, with some effort, but there it is.

A lexical variable, in case you’re not aware, is a variable with static local scope. That is to say that its scope is definable at compile time and is local to its nearest enclosing scope. In Perl, a variable is lexical if it is declared with my(). In Ruby and UCBLogo, all new variables are lexically scoped by default. Python does not provide proper lexical scoping, so far as I am aware, as its local variables are not fully accessible within scopes enclosed by the variable’s scope (they can be read, but not written to, effectively changing them from variables to constants for enclosed scopes). This limitation can apparently be short-circuited for single array elements, but this behavior looks more like a bug of the language design than a feature.

Lexical closures (also known simply as “closures” for short) require two things:

  1. proper lexical scoping
  2. “anonymous” blocks of code that are first-class functions

Lexical scoping provides a number of benefits to programming languages that implement it. As time passes, more and more often the languages we encounter implement lexical scoping. Some languages, such as Perl, predate the common recognition of the benefits of lexical scoping, and incorporate it into their design later: this applies to Lisp, as well. Other languages have incorporated lexical scoping from day one, and in the future I expect this number to grow prodigiously, as it is likely that any language designer worth a damn will be using lexical scope for years to come (at least until someone invents or discovers something even better).

Lexical scope contributes to cleaner code, reduces the likelihood of scope-related bugs, and because it is defined at compile time, runtime execution for a program with only lexically scoped variables is faster. More benefits than these apply, of course, but I’m sure you can learn more about the subject on your own.

The power and flexibility of functions that can be passed around as parameters for other functions should be fairly self-evident to even a thoroughly mediocre programmer, whether that programmer can immediately envision instances where it is directly useful or not.

By combining these two characteristics of a language — lexical scope and first-class functions — to create closures, we introduce protection and encapsulation. These are characteristics of data bound to code whose value are quite central to the entire object oriented programming paradigm.

I have recently addressed the matter of elegance in programming here at SOB. Within that discussion, and in a follow-up comment to it, I explained and demonstrated the varied potential for elegance as I defined it within the designs of various programming languages. My initial, very simple, demonstration involved merely providing a series of “hello world” programs in a number of different languages so that the amount of scaffolding cruft necessary to achieve this simplest of operations — outputting a string — could be compared between languages. It was intentionally an extremely simple example, as the point was to demonstrate the lack of elegance necessary to achieve such simple results in some languages. This is, I believe, a very salient and useful means of comparing the elegance of code a given set of languages provide for their users.

It is not, however, the only salient and useful comparison, and is by no means a comprehensive metric for judging the potential elegance within a language. In recognition of this, Colin Alston in his karnaugh.za.net weblog suggested that list processing might provide some useful data in a comparison between languages. He went further than merely suggesting it, however, and went on to provide such a demonstration in his weblog entry Elegance: a function of consistency.

(Note: Much of the following may seem confusing due to the fact that the “consistency” essay has been moved. As part of the move, it seems that all comments on the original weblog post there have been lost, thus eliminating much of the context for the following.)

Colin’s heart seems to rest with Python, one of a series of languages that I tend to hold in high esteem in general. Its closest relatives, it seems to me, are Perl and Ruby: a little further afield are the various Lisps. Python imposes some arbitrary limitations on the programmer, however, such as limiting anonymous first-class functions (called “lambdas” in Python, probably as a direct homage to the use of the term in Lisp, despite their limitations in Python) to a single semantic line of code. Another apparently arbitrary limitation is its restriction of scope (as mentioned above) so that variables that might otherwise be lexically scoped are read-only constants from the perspective of enclosed scopes.

This aside, I was surprised to see the length of the Python example provided as a solution to Colin’s list-processing comparison problem. It was not terribly long, of course — much shorter than C, C++, and Java examples — but several syntactic elements longer than I expected. I rather suspect that, like the Ruby example provided by one of his readers, it can be shortened somewhat. The Python example provided contains nine distinct syntactic elements (where the list used to populate the array is taken as a single syntactic element).

The Ruby example provided by Colin’s reader “jerith”, meanwhile, contains eight distinct syntactic elements, but I have shortened it to a mere five elements. Surprisingly, Perl outperforms even Ruby handily: Colin’s reader Jonathan McKeown provided both a seven-element example (beating the previous Python and Ruby examples) and a four-element example (beating my own golfed Ruby example). One might make a case for each of the Perl examples from Jonathan McKeown containing an extra syntactic element in the form of qw// used to define the list, but this is used as a tool to make the code easier to read and write, and is not a necessity of the code, in addition to which it is merely syntactic sugar for quote punctuation in the list definition — and, even if it is counted as a distinct syntactic element, it only brings the Perl examples into a tie with Ruby.

My own Perl example, provided later in the discussion, further golfed the Perl down to three syntactic elements. That’s one third the number of distinct syntactic elements used in the Python example. Thus, my surprise.

I have here a third suggestion for a point of comparison between languages. In this comparison, I use a lexical closure that operates upon a list and returns the results of that operation. The results are output to STDOUT (generally, the command line). The list itself is defined as arguments to the otherwise noninteractive program, thus allowing the program to behave as a filter utility. Each element of the modified list is printed on a separate line in the program’s output. The input and output of the program should look like this:

# programname arg1 arg2 arg3
arg3
arg2
arg1

The number of arguments the program can take should be arbitrary. As one might guess by the above example, the list processing operation I’ve decided on is a simple reversal of the elements in the list. I intentionally, as with the “hello world” example, deliberately chose a task that should be simple enough for a programmer to understand, at least in general terms, what is going on in examples even from languages with which (s)he is not familiar — though in this case there is some general programming knowledge that is not universal (lexical scoping, first-class functions, and the closure that results from these) in use; hopefully any shortfall in required knowledge has been addressed here.

I hope that quite a few of you will contribute examples of how these requirements for a demonstration of linguistic potential for elegance. In the meantime, I will start things off with two examples of my own.

First, Perl:

#!/usr/bin/perl -l

sub foo { my @list = @_; sub { reverse @list }; }

$bar = foo(@ARGV); print for $bar->();

Unlike Colin’s example, one need not agonize over the number of distinct syntactic elements at all in this case, but we are left with another potential point of contention: the -l argument in the shebang line. My take on it is that this is a directive to the interpreter, effectively redefining a particular core function of the language, and not a syntactic element. That being the case, this Perl example contains sixteen distinct syntactic elements, by my count. The closure generator itself contains nine, the function call that actually creates the closure to be used contains four, and the closure call statement where its results are printed to STDOUT contains three.

The closure generator suffers significantly from Perl’s heritage as a language without lexical scoping in earlier versions, and from the fact that in the addition of lexical scope, it was not made the default scope used for new variables. As a result, the @list array must be explicitly declared and assigned for it to have lexical scope, adding three otherwise unnecessary distinct syntactic elements. Similarly, the closure call and print statement contains a syntactic element that is not necessary in some other languages — the for loop. This is only necessary because of the requirement for printing each list element on a separate line, however. If the input list elements contained newline characters, this would not be necessary, and neither would the -l in the shebang line.

Second, Ruby:

#!/usr/bin/ruby

def foo (list) lambda { list.reverse } end

bar = foo(ARGV) puts bar.call

Because of Ruby’s puts() kernel method, there is no need for an interpreter directive as in the Perl example. This does provide for potentially increased elegance of code, if you accept as a given that adding a newline to output as a matter for distinct syntactic treatment increases gratuitous cruft, rather than simply being a lack of syntactic sugar. As I went easy on Perl in this matter, however, I will do the same for Ruby: let us consider puts() a syntactic point of elegance.

That being the case, this program contains a grand total of thirteen syntactic elements (the language keyword end is not counted, being merely a delimiter, any more than braces are in Perl). The same number of distinct syntactic elements are in use in the call to the closure and output to STDOUT, and in the closure instantiation itself, but Ruby is a clear winner in the closure generator definition. While one might point to a number of superficially apparent reasons for this by examining the list of additional syntactic elements involved in the Perl example, these are all necessary only because of the difference between the two languages in how lexical scope is achieved. In Ruby, it is the default behavior of the language, while in Perl it must be explicitly declared.

26 Comments

  1. Python golf for iterating a list of strings:

    print '\\n'.join(['ab', 'cd', 'ef', 'gh'])

    and a python closure for reversing argv:

    #!/usr/bin/env python
    import sys
    
    def rev(list): return lambda: reversed(list)
    
    iterate = rev(sys.argv[1:])
    for item in iterate(): print item

    If you wanted to also print out the name of the command being run, you could eliminate the [1:] portion.

    Comment by scoth — 24 August 2006 @ 04:10

  2. Dude, your wordpress stopped respecting my code blocks!

    Comment by scoth — 24 August 2006 @ 04:12

    1. I added pre tags to your code blocks and added an extra backslash to the linebreak escape character in one of them. Other than that, I’m not sure what changes might need to be made so they appear properly. For future reference, when posting code blocks here, you should probably use pre tags for anything that uses indentation and add an extra backslash for anything in which you need a backslash that doesn’t get parsed out by WordPress.

    2. I see at least nineteen distinct syntactic elements in that Python closure example (and four in the bare list example). Are these numbers anything like accurate?

    3. Is there any chance I could get you to explain what’s going on in that Python code so that I understand better how it works? For instance, could the variable list be modified within that lambda, or is it read-only in the lambda’s scope? What exactly is the 1: doing? Why is the import sys statement necessary?

    EDIT: I just realized that Ruby can do the simple list output in two syntactic elements. puts ['ab', 'cd', 'ef', 'gh']

    Comment by apotheon — 24 August 2006 @ 04:39

  3. I assume that the lexical closure is a requirement of this test, even though it isn’t strictly required in order to generate the desired results. For instance, you could change your Ruby example to simply puts ARGV.reverse.

    Given closures as a requirement, I don’t have much to add. I would opt for Ruby, and you’ve covered that. I’m not strong enough on Lisp or javascript to think that I could provide the best examples for those languages. And none of the other languages that I use frequently support lexical closures.

    An approximation to the same thing can be achieved rather clumsily in an object-oriented language, though. For instance, in C#:

    
    using System;
    
        class Program
        {
        class foo
        {
            string[] list;
    
            public foo(string[] args)
            {
            list = args;
            }
    
            public string[] fooit()
            {
            string[] reversed = list;
            Array.Reverse(reversed);
            return reversed;
            }
        }
        static void Main(string[] args)
        {
            foo bar = new foo(args);
            foreach (string s in bar.fooit())
            Console.WriteLine(s);
        }
        }
    

    As you can see, the requirement of having a class to enclose scope, as well as the required typing and access control, make C# considerably wordier than Ruby or Perl. It also stinks that Array.Reverse only operates in-place, so you have to make your own copy of the array.

    How’s that for a negative example?

    Comment by SterlingCamden — 24 August 2006 @ 12:54

  4. Actually I just now realized that the above code doesn’t copy the array, so the original array passed to Main gets reversed. To avoid that, you’d need to create a new string array and copy the contents (Array.Copy, for instance).

    Comment by SterlingCamden — 24 August 2006 @ 01:29

  5. Yeah, that’s a pretty negative example, alrighty.

    This disturbs me:

     static void Main(string[] args)
        {
            foo bar = new foo(args);
            foreach (string s in bar.fooit())
            Console.WriteLine(s);
        }

    It looks like you have to basically write all your functions inside a separate scope from your program logic, then call them inside your program logic, or something like that. This seems unnecessarily ponderous. Of course, judging by the weight of the code in the functions (objects, classes, methods), it looks like that may be the only way to keep the program logic readable. It’s a little like anti-footnotes, though, where the “footnotes” come earlier than the main text, and are longer than the main text, and are furthermore completely critical to understanding the main text, the way that code is written.

    It seems suboptimal.

    On the bright side, it’s possible to write Ruby for .NET now.

    Comment by apotheon — 24 August 2006 @ 01:34

  6. So change fooit() above to be:

    public string[] fooit()
            {
            string[] reversed = new string[list.Length];
            Array.Copy(list, reversed, list.Length);
            Array.Reverse(reversed);
            return reversed;
            }
    

    Comment by SterlingCamden — 24 August 2006 @ 01:37

  7. Yeah. Technically, you could write all of your functions as static methods of the Program class, but then you don’t get any of the scoping you were looking for.

    I’m planning to check out Ruby for .NET. That could ease my life.

    Comment by SterlingCamden — 24 August 2006 @ 01:38

  8. There is also IronPython for .NET now too :)

    Comment by Colin Alston — 24 August 2006 @ 10:17

  9. I was bored, so I came up with this example for JavaScript:

    function foo() {
        var myArgs = new Array(foo.arguments.length);
        for (i = 0; i < foo.arguments.length; i++) {
            myArgs[i] = foo.arguments[i];
        }
        function bar() {
            myArgs.reverse();
        }
    
        bar();
        for (i = 0; i < myArgs.length; i++) {
            alert(myArgs[i]);
        }
    }

    It’s rather pointless, as reverse() changes the original array. But what’s annoying is that it doesn’t work on arrays which are implicitly cast — you have to declare an array object and then populate each value explicitly, hence the need for the first loop. (There’s no copy() method in the JS array object.) Otherwise this example could be pretty damn short, excluding the HTML needed to run the example.

    Comment by cluebyfour — 25 August 2006 @ 01:39

  10. I took the liberty of formatting your code, cluebyfour, and eliminating your short follow-up post mentioning the fact that the formatting didn’t work properly. It has become clear that for some reason the process of posting comments here causes use of pre tags to be parsed out for everyone but me, thus requiring me to tag along reformatting everyone’s code. This is something I’ll have to see about fixing, especially since discussing source code is becoming such a hot hobby at SOB.

    Is there a means of reversing the list yielded by an array that would be as succinct as the use of reverse() as you demonstrated without actually changing the contents of the array, or would this require yet more overhead in addition to the original array declaration and assignment loop? I’m actually somewhat surprised that Javascript is so limited that one cannot simply drop a complete list into an array variable in Javascript: whatever its shortcomings, I didn’t think that was one of them. I’ve (obviously) not used enough of it to run into problems such as this, however.

    Comment by apotheon — 25 August 2006 @ 03:17

  11. Here’s another example of using closures (in a Perl/Tk application). The part of the problem we’re solving here is to have nine differently-marked widgets, each of which when clicked changes the text on a bigger label widget. I’ve tackled it by binding a closure to mouse button 3 for each widget:

    for my $text ('1' .. '9') {   # create $widget - how isn't important   $widget->bind('<3>', sub {     $label->configure( -text => $text);   }); }

    For each widget, the second argument to the bind method is a closure capturing the value of $text. (And the enclosing subroutine which creates my $label is also called multiple times – 81 in fact, so it also closes over the $label object).

    This ends up creating 729 widgets, each with a callback implemented as a closure over the appropriate label object and text string. Because there’s more going on in the surrounding code, Perl’s syntax for closures doesn’t look quite as heavy in this example.

    Comment by Jonathan McKeown — 25 August 2006 @ 06:44

  12. Is there a means of reversing the list yielded by an array that would be as succinct as the use of reverse() as you demonstrated without actually changing the contents of the array, or would this require yet more overhead in addition to the original array declaration and assignment loop?

    I could have just written:

    var myArgs = foo.arguments;

    which would have implicitly created an array and assigned the list values of foo, but I would have then been unable to use reverse() for the reason I mentioned before. Thus I would have needed to use a loop to reverse the array elements, so I’d still be looking at the same overhead. (To be honest, I could not get such a method to work last night, which is why I ended up doing what I did.)

    This is the curse of weak data typing, I think (the blessing being ease of use and never having to worry about casting variables to a different type).

    Comment by Brian Martinez — 25 August 2006 @ 08:15

  13. Jonathan McKeown:

    Excellent example! Unfortunately, it’s probably lost on people who aren’t familiar with both closures as a programmatic concept and the way objects are implemented in Perl. The win here, as distinct from previous examples, is two-fold as far as I see:

    1. The closure demonstrated here allows for autogeneration of many encapsulations of persistent, protected state with minimal developer effort.
    2. The syntax for creating a closure with an entire object instance as the closure variable is made simple and elegant by the manner in which objects are instantiated.

    That second point applies equally to closures as to objects in the official Perl object model, of course, since objects are essentially just fat closures.

    Brian Martinez:

    Yeah, I’m a fan of strong dynamic typing with static scope. Weak typing introduces cruft, static typing is a severe pain in the butt sometimes, and dynamic scope is a disaster waiting to happen most of the time (though it’s handy once in a while).

    Comment by apotheon — 25 August 2006 @ 08:33

  14. 2. I see at least nineteen distinct syntactic elements in that Python closure example (and four in the bare list example). Are these numbers anything like accurate?

    Yes that sounds about right. The closure example could be done with for item in rev(sys.argv[1:])(): print item instead of the last two lines, but that doesn’t make it a whole lot better ;-) Don’t take the syntax needed for closures as representative of other features of the language, I just thought you’d be interested to see that yes, in fact, you can do closures in Python.

    3. Is there any chance I could get you to explain what’s going on in that Python code so that I understand better how it works? For instance, could the variable list be modified within that lambda, or is it read-only in the lambda’s scope? What exactly is the 1: doing? Why is the import sys statement necessary?

    It’s not a whole lot different from what you’re doing in the perl or ruby examples so I presume you can figure most of it out. The [1:] is shorthand for the slice function, in this case from element 1 to the end of the array. The slice function makes a shallow copy of the array before performing the slice, therefore the result ends up being a new object (which, if it contained a “reference” to a nested array or hash, the new object would get the reference to the original nested object).

    I have verified that the variable list can be modified in lambda. The lambda single statement limit prevents such operations in my example, but if you use a lambda to call a named or otherwise bound function (perhaps on it’s way out of scope), you could make a lambda of an arbitrary size. In my example, the slice makes a shallow copy of the list, so the object called by reference in the lambda is not the same as the parent class.

    Contrary to what you have heard, python does provide proper lexical scoping. If you’re referencing non-mutable variables from the top level scope, you have to explicitly declare the variable as “global” before you will be able to read or write to them. Otherwise, “if a name is used within a code block, but it is not bound there and is not declared global, the use is treated as a reference to the nearest enclosing function region.” See http://www.python.org/dev/peps/pep-0227/ for more information. It appears that they use closures in some of the examples.

    The “import sys” statement imports the sys(tem) module that has the argv function in it. The system module contains variables and “functions that interact strongly with the interpreter”. I’m not sure if python on all systems would have access to argv. For example there’s python for PalmOS, and I’ve seen at least one non-palm phone that has python, but because utilities aren’t launched from a command line, there wouldn’t be any use for argv.

    The perl analogue to import is use, and they are identical for most intents.

    Comment by scoth — 25 August 2006 @ 01:45

  15. For the most part, this does improve my impression of Python. The fact that one must essentially pass a variable by reference (even if it’s largely transparent) to achieve effects akin to proper lexical scope is a little bothersome, but it’s not a deal-killer. On the other hand, the “one line per lambda” arbitrary limitation is still absurd and kind of a deal-killer for me.

    This, of course, is offset somewhat by the fact that Python shares some features in common with Ruby that make some aspects of code more elegant where Perl lags somewhat. I hear Perl 6 will fix everything, but I heard that about, well, everything — and, apparently, the number 6 in “Perl 6” doesn’t refer to the number of years it will be in development, since it appears it may take up to twice that from the day it was conceived at this rate.

    Considering that argv isn’t available to Python by default, and a library must be loaded to gain access to it, that makes the import sys statement significant in determining the number of discrete syntactic elements necessary to achieve your ends. Bloat += 2, at least. One could make a case for saying that the code in the import statement being added to the entire mix, and that it too must be evaluated, but I think that’s a bit over the top: my guess is that sys isn’t written in Python but, rather, is essentially a part of Python written in C/C++ that has been decoupled. This has its advantages, but it’s still more of a matter of code that needs to be written than of equivalency with a Perl pragma like strict or warnings (which isn’t syntactically significant to the code).

    Thoughts?

    Comment by apotheon — 25 August 2006 @ 02:58

  16. The fact that one must essentially pass a variable by reference (even if it’s largely transparent) to achieve effects akin to proper lexical scope is a little bothersome

    It provides some protection from accidentally accessing the wrong scope when you reference a variable. If you’ve ever gone through the hastle of tracking down a bug due to a scoping issue like that, you might really appreciate it. It’s also less work than turning on strict in perl and declaring “my” for every local variable.

    On the other hand, the “one line per lambda” arbitrary limitation is still absurd and kind of a deal-killer for me.

    Fuctions are accessed by reference, so it’s easy to bind them to a variable name or pass them as arguments to other functions, which should accomplish the same types of things that lambda (as an anonymous function) would. After looking at it a little bit more, I really think that lambdas should be removed just so they don’t cause this type of confusion, on the other hand, as they exist they are syntactic sugar.

    Considering that argv isn’t available to Python by default, and a library must be loaded to gain access to it, that makes the import sys statement significant in determining the number of discrete syntactic elements necessary to achieve your ends. Bloat += 2, at least.

    It is part by default, but isn’t part of the core language. You could also add a configuration file to import sys (probably into the current name space if you wanted). I wouldn’t call this bloat, I would call it modularization. I hear what you are saying (because it needs to be typed in at some point), but it actually keeps bloat out of the core language, which makes it easier to release new versions instead of just promising them for 12 years ;-)

    my guess is that sys isn’t written in Python but, rather, is essentially a part of Python written in C/C++ that has been decoupled.

    It’s almost certainly written in C, most of the included modules are.

    Comment by scoth — 25 August 2006 @ 05:26

  17. By the way, if you’re almost done with “Learning Perl Objects, References, and Modules”, I think I’d like to read it again (or finish reading it). Though this time more for the abstract programming concepts than the actual implementation.

    Your recent language posts have been pretty thought provoking and have led me to explore and understand more about Python than previously.

    Comment by scoth — 25 August 2006 @ 05:43

  18. It provides some protection from accidentally accessing the wrong scope when you reference a variable. If you’ve ever gone through the hastle of tracking down a bug due to a scoping issue like that, you might really appreciate it.

    Are you saying that lexical scope solves these problem? That’s true, and I agree. My complaint was about the manner in which it must be accomplished.

    It’s also less work than turning on strict in perl and declaring “my” for every local variable.

    This is true. There seems to be an almost-equivalent trade-off between Perl and Python with regard to how lexical scope is implemented, at first glance. I think the fact that lexical scope is (so far) at least apparently only available implicitly, and yet not default for Python, is quite dangerous at times. Implicit behavior should only apply to default behavior most of the time. My preference is for how languages like Ruby and UCBLogo handle it: lexical scope is both implicit and universally default.

    By the way, that should say declaring “my” for every lexical variable, as “local” is also the term used for localization of dynamically scoped variables in Perl (something different entirely from lexical scoping).

    Fuctions are accessed by reference, so it’s easy to bind them to a variable name or pass them as arguments to other functions, which should accomplish the same types of things that lambda (as an anonymous function) would.

    So do you mean that there’s some way to accomplish a closure in Python using a function of more than one line of code as a return value?

    It is part by default, but isn’t part of the core language.

    Toe-may-toe, toe-mah-toe.

    it actually keeps bloat out of the core language

    It’s a trade-off. It’s surely a trade-off in Python’s favor for the most part, taken in a vacuum. By slimming Python down a little, you impose some scaffolding overhead when writing code. Python seems to do this with a number of things that aren’t really well-suited to that treatment, though — at least, not in the aggregate. Command line access and regular expressions are prime examples of things that are really best suited to direct inclusion in the language, in my opinion. By separating them out, you get benefits in specific cases and a very minor benefit in most cases, but there are some annoying downsides as well. The completely asinine syntax for regexen in Python springs to mind.

    Comment by apotheon — 25 August 2006 @ 06:00

  19. . . . and yeah, I’ll give your book back next time I see you — maybe Tuesday. If you’re at Static this Sunday, I’ll give it to you then.

    Comment by apotheon — 25 August 2006 @ 06:41

  20. You keep on trying to imply that python doesn’t do dynamic lexical scoping… which it does… by default… since 2002. What makes you think that it doesn’t? Perhaps I can clear up the misunderstanding.

    So do you mean that there’s some way to accomplish a closure in Python using a function of more than one line of code as a return value?

    Yes. The link I included I think even contained an example of a factorial computation using a closure. I am not sure why you have such a hard-on for closures though. They’re a functional programming hack to provide functionality like an object model. It seems that it’s much more elegant to use classes if your language is at all OO friendly. Maybe you can explain why closures are better than classes.

    Here’s a multiline closure for you from the command-line interpreter:

    >>>def closure(list): ...   def subfun(item): ...      list.append(item) ...      if item.find('spam') < 0: ...         list.append('spam') ...      print ' '.join(list[:-1]), 'and', list[-1] ...   return subfun ... >>> dialogue = closure(["We have spam", ]) >>> dialogue('sausage') We have spam sausage and spam >>> dialogue('spam') We have spam sausage spam and spam >>> dialogue('bacon') We have spam sausage spam spam bacon and spam >>> dialogue('tomato') We have spam sausage spam spam bacon spam tomato and spam

    The link you posted to wikipedia’s article on Closure (computer science) also contained an external link link on Python closures.

    P.S. I treated the code block like it was a <font face=”monospace”> tag since <code> is currently busticated. I used a bunch of &nbsp;s and &gt;s. It really made my eyes bleed.

    P.P.S. Python’s syntax for working with regular expressions is much less elegant than perl or ruby. When I was first learning perl, it took me a long time to remember what all of the ~ operators were about, but it was a worthy sacrifice to the gods of regexen.

    Comment by scoth — 25 August 2006 @ 09:05

  21. I’d also like to point out that ARGV is harder to type out because it’s all caps than sys.argv.

    Comment by scoth — 25 August 2006 @ 09:37

  22. dynamic lexical scoping

    I have no idea what that means. It looks self-contradictory to me. I’ll let you know whether or not I think Python does “dynamic lexical scoping” as soon as I figure out what it means.

    The link I included I think even contained an example of a factorial computation using a closure.

    I’m afraid I don’t know Python quite well enough to be positive I know what I’m reading when I look at it, sometimes. I wanted to amake sure I understood what was going on. I also only skimmed the page, trying to hit the high points. I’ll probably find more time to read and ponder it later.

    They’re a functional programming hack to provide functionality like an object model.

    Er, not precisely. Full lexical closure first appeared with Scheme, as far as I’m aware, which appeared about the same time object oriented programming became a practical reality. Functional languages were already approaching that state of affairs for some time before Scheme finally refined the concept to the point currently recognizable as a lexical closure.

    It seems pretty clear that you’re looking at the comparison of closures and objects from the common point of view of greater familiarity with the OOP paradigm than damn near anything else in programming. On the other hand, there are programmer communities where it is pretty much taken for granted that objects are a poor man’s closure. There are debates about which is better — arguments between people who know what the hell they’re talking about far better than either you or I do.

    Of course, unlike the way a lot of Java programmers behave with their noun-obsessed object orientation, I’m not trying to suggest that all of programming should be organized around the use of closures. I just happen to be discussing demonstration of the use of closures for the reasons indicated above.

    Maybe you can explain why closures are better than classes.

    They’re better for some things and worse for others. They can even be combined with classes and more commonly recognizable objects to leverage the capabilities of both when that’s appropriate and useful.

    re: ARGV/@ARGV vs. sys.argv I dunno. It’s kind of a tradeoff in some respects. I like the recognizability of the all-caps version, particularly since it’s a default global variable whose direct use should always be minimized. I also like not having to type all-caps. I think I’d rather that it be replaced with some kind of implicit arg-grabber function that can be used to assign to the array (or whatever) of the programmer’s choice to avoid the whole mess.

    Comment by apotheon — 25 August 2006 @ 11:46

  23. I’ll let you know whether or not I think Python does “dynamic lexical scoping” as soon as I figure out what it means.

    My bad. Python is a statically scoped, a.k.a. lexically scoped language. But, just because I’ve made a mistake doesn’t mean you have to be a smart-ass.

    You still haven’t defined what you mean by proper lexical scoping, or why you believe that Python falls short of this. You alluded to something about passing by reference but I wasn’t clear on what you were referring to.

    I think that part of what may be going on is that you don’t understand python data-types or appreciate the subtleties of mutable vs. non-mutable types. I should also mention that python doesn’t deal with references or pointers explicitly, so when I talk about references, I’m talking about how it deals with data-types behind the scenes in an attempt to explain mutability. Perl’s arrays are handled the same as python’s lists, and perl’s hashes are handled the same as python’s dictionaries, that’s about all that they have in common. Python’s methods for dealing with integers, floats and strings are subtly different from perl’s. Python also offers a few data types that perl does not (arbitrary length long integers, touples, and sets), and perl has a pointer class that python does not.

    There may be some things about the way that Python implements scoping that mean that it can’t be used for generic, multiline closures. Maybe the spam closure I wrote was a special case, but I have some ideas about getting around the apparent limitations and remain unconvinced (or perhaps unaware) that they exist.

    Even if Python’s lexical scoping implementation isn’t completely text book, and by that it sacrifices some of the power of closures, I think that it’s worth it. In my experience, Python’s scope model is less confusing than perl’s and you’re less prone to unintended side-effects from not fully understanding the scope model, or screwing up the syntax. This is also a form of elegance, even if it means that you have to resort to a little bit of crufty OOP scaffolding in order to reproduce the functionality you’d need from a closure.

    Anyway, I’m leading off into an area further and further away from your original subject. Perhaps it would be better to continue this at another time, venue, or article. Your call.

    you’re looking at the comparison of closures and objects from the common point of view of greater familiarity with the OOP paradigm than damn near anything else in programming.

    I guess that’s fair. The language comparison that you’ve been doing with closures and polymorphism is at an academic level. My formal education consisted of OOP with C++. I’ve worked with a few other languages, but my programming theory was mostly limited to C++. This also reminds me that my academic experience with polymorphism was limited to parametric polymorphism because C++ is a statically typed language. I’ve used the other kind of polymorphism since, but didn’t realize that’s what it was called. Some of your recent discussions have drawn more on my university education than anything I’ve needed in the jobs I’ve had.

    I’m afraid I don’t know Python quite well enough to be positive I know what I’m reading when I look at it, sometimes.

    Then why do you keep acting like you can judge it’s relative merits? You end up spreading disinformation and you sound like a language bigot. Though I will give you props, you’re not as flippantly dismissive as you were at one point.

    I don’t really think closures are a hack, by the way. I just wanted to push your buttons and see what you really think. Closures can definitely be simpler and shorter to write than classes, but they only provide encapsulation and persistence, and none of the benefits of the inheritance model, polymorphism, or the modularization of classes. Sometimes you don’t need all that extra functionality though. Closures seem like a neat tool that I haven’t been using.

    Comment by scoth — 26 August 2006 @ 01:38

  24. But, just because I’ve made a mistake doesn’t mean you have to be a smart-ass.

    You’re reading too much into what I said.

    You still haven’t defined what you mean by proper lexical scoping, or why you believe that Python falls short of this.

    I stopped saying that when you pointed out that it does it. I just don’t like the notion of having to pass something by reference to attain proper lexical scoping, which seems to be what you were saying to me.

    You alluded to something about passing by reference but I wasn’t clear on what you were referring to.

    To answer that, I’ll quote something else you said:

    Contrary to what you have heard, python does provide proper lexical scoping. If you’re referencing non-mutable variables from the top level scope, you have to explicitly declare the variable as “global” before you will be able to read or write to them. Otherwise, “if a name is used within a code block, but it is not bound there and is not declared global, the use is treated as a reference to the nearest enclosing function region.”

    (emphasis mine)

    perl has a pointer class that python does not.

    Are you talking about Perl references here? If so, this may be the source of some confusion regarding your use of the term “reference” above.

    In my experience, Python’s scope model is less confusing than perl’s and you’re less prone to unintended side-effects from not fully understanding the scope model, or screwing up the syntax.

    You should probably just explain Python’s scope model to me, then, either here or elsewhere. I thought I had a grasp on it previously, but telling me that it provides lexical scoping by simple mechanisms like def foo(bar) kinda disputes my previous understanding. That looks more like Ruby scoping, which is at odds with what I’d understood of Python’s.

    This is also a form of elegance, even if it means that you have to resort to a little bit of crufty OOP scaffolding in order to reproduce the functionality you’d need from a closure.

    I will not dispute the value of helping protect the programmer from himself. That always comes with a tradeoff, though, because it assumes the language designer knows what the programmer wants — at least, it does if the programmer is actually prohibited from doing certain things, rather than merely encouraged by the language to do things another way (which is also protective, though not as inviolably so). I wouldn’t call something that imposes additional work on a programmer trying to accomplish some task “elegant”, though, even if it encourages greater elegance in certain cases.

    Anyway, I’m leading off into an area further and further away from your original subject. Perhaps it would be better to continue this at another time, venue, or article. Your call.

    Up to you. I don’t mind a little off-topic discussion. I’m also perfectly okay with deferring some or all of this to another time and/or place.

    Then why do you keep acting like you can judge it’s relative merits? You end up spreading disinformation and you sound like a language bigot. Though I will give you props, you’re not as flippantly dismissive as you were at one point.

    It’s a matter, mostly, of not being up-to-date on Python (I keep discovering things I once knew have changed), not being in practice on the Python that I actually do know, and simply not finding Python very readable. My eyes glaze over when I look at it. I’m mostly talking about those three factors when I say that I can’t read Python very well (though, of course, it should also be taken as a given that at this time you certainly have a heck of a lot more practical knowledge of Python than I have). What I do know about Python, I mostly know in an academic sense: I’ve read Python instructional materials, surveys of the language, and so on. I’ve played around a little with translating between it and other, similar languages, with fairly trivial examples.

    Closures can definitely be simpler and shorter to write than classes, but they only provide encapsulation and persistence, and none of the benefits of the inheritance model, polymorphism, or the modularization of classes.

    If by “provide” you mean “implicitly incorporate the syntax for”, you’re right. On the other hand, I’m pretty sure all of these other things are easily created within the code, and without having to write a whole lot of extra code. The only thing lacking is a unified means of doing so, as far as I can tell. Closure syntax is, as far as I can tell, usable as an already complete, alternative object model — at least in Perl. The pervasiveness of the official object model in something like Ruby might interfere with such a thing to the extent that closures could be used to enhance the language’s official object model, but not to effectively replace it.

    Closures seem like a neat tool that I haven’t been using.

    Unlike many other programming concepts (such as the basic concept of programming itself), the simple “hello world” approach to demonstrating how a closure works is also actually useful sometimes. The “hello world” itself, for instance, is entirely useless in the world of “Real Programming”: one simply doesn’t write a program that does nothing more than print out the words “hello world” for any purpose other than demonstration. An accumulator closure, on the other hand, can be quite useful:

    sub foo {
      my $bar = 0;
      sub { ++$bar; }
    }

    It would, of course, be more useful if more functionality were added to it (such as separating its useful return value from its incrementation), but it can be pragmatically useful as is. In fact, I’m planning to use a more complex version of that in a web application that, among other things, generates statistical data from survey results.

    Comment by apotheon — 26 August 2006 @ 03:57

  25. […] of lexical closures, it might be worthwhile to check out my SOB entry of August 2006 titled Linguistic Elegance Test: Closures and Lists. This includes some links to explanations and definitions of closures, […]

    Pingback by Chad Perrin: SOB » real closures — 27 April 2008 @ 07:46

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

All original content Copyright Chad Perrin: Distributed under the terms of the Open Works License