The Resurgence of Dynamic Coding

December 14th, 2009

In the past few years, dynamic languages have enjoyed a surge in popularity. Not just among the coders who would have been using them all along, but even around the coders whose managers keep saying things like “Enterprise” and “Leveraging Synergistic Content Management Social Media Next-Generation Beta Podcasting Outside the Box”. (Side note to such managers: your coders’ eyes glaze over as a defensive mechanism. It’s nothing personal; it’s just like what happens to you when they say things like “Oh of En Log En Virtual Machine Scalable Output Stream Socket Runtime”.)

The crowd that was using Java and before that C++ is now delivering applications in Ruby and Python, and even languages that have been around forever but were never really popular among the mainstream crowd, like Perl, Scheme, and even the quirky but humble Javascript. Anything I could say about this particular phenomenon has likely been covered in any number of blogs; I don’t intend to gush about being allowed to write Ruby at work.

What I do intend to discuss, however, is the resurgence of dynamic code, and of dynamic constructs (such as closures) in high-level languages in recent years. They’ve been right there all along, but where did they go?

Back in the old days, when “portability” was not a concept that had worked its way into the hearts or minds of brave hackers the world over and “upgrade” meant that a DEC engineer came over to your office with a soldering iron, people wrote programs to run on one machine. One of the things this afforded those coders was the ability to write self-modifying code. Rather than telling the machine what to do directly, you could give the machine some rules and tell it how to tell itself what to do.

This sort of construct is wonderful in its power, an application of the principle that whenever you can make the computer do work rather than doing it yourself, you should. After all, the computer (being deterministic) is more reliable than you are, it has a near-infinite capacity for tedium (which you definitely do not), and besides all of that, it’s the whole reason Babbage did all of that tinkering to begin with. If you fail to automate tedium, you are not just wasting your own time, but Charles Babbage’s time as well. Certainly, we can’t have that on our collective conscience, can we?

The use of dynamic coding precedes structured coding, which, in fact, precedes the physical births of the current generation of coders. Not to waste your time reminiscing (or date myself), but when I first started programming, Object Orientation was already a hardened, entrenched concept. Data structures, structured code, all of these things were handed to the twenty- and thirty somethings that comprise us technical grunts. But dynamic coding (as such) was missing from the collection of concepts we were handed. It has always been around in one form or another, since the very beginning, even uses of it that boggle the mind with their cleverness. It was just gone. Here it is now, and everyone is re-opening classes to add new methods, everyone is passing around closures, everyone is writing VM bytecode compilers, and it is definitely here. Where did it go, and why?

As much as I love the C programming language, it has caused a few gaps to appear in our heads, and I’m going to blame it for this particular gap. (Dennis Ritchie: you are still my home-boy and I hope we still cool if you happen to read this.) You can’t exaggerate C’s influence on programming. Nearly every popular language has stolen its syntax, for example, even languages where it made no sense to do so. (I name no names.)

And one thing that C lacked was run-time code generation. Having done a lot of C coding, and having been spoiled by languages in which this is possible in one sense or another (lambdas and evals, code as data, creating functions from scratch, sending them to other functions to execute directly), I’ve often been frustrated at this gap in C’s functionality. Technically, code is possible to represent as data in C, in that you can have a pointer to a blob of machine code in memory and you can manipulate and execute it, but this is pretty rare to see in the wild, due to portability concerns. Through the C compiler, you could tell the computer how to tell itself what to do (i.e., take your C code and turn it into machine code), but that was only one level; you couldn’t easily tell the computer to tell itself more C code.

And in languages that take after C (which wholly comprised mainstream languages when I started coding) tend to make this same omission. C++ lacks it (although I have heard that you can do creative things with vtables). Over the years, dynamic code was relegated to the black magic category, voodoo that decent coders avoided. Even Java, which runs on a VM and thus has no portability issues with code generation, curiously lacks this feature. Java is getting closures soon, and you can simulate them now (although closures are just one piece of the puzzle), but nearly every JVM language includes at least some of these dynamic features, and they’re no longer considered dark arts. Meanwhile, the Forth, Perl and Common Lisp guys are occasionally peeking out the window at the mainstream and noticing that the differences are slowly fading.

Could this gap have been avoided? That’s a slightly harder question to answer. C was, for many years, the Way Coding was Done. Forth had an appropriate mix of low-level access and dynamic constructs, but a portable and fast Forth is a rare beast indeed. In the height of C’s popularity for application programming, a speedy enough Lisp was also rare. Perl has always had eval() (which doesn’t really count as a construct and is a somewhat unstructured string eval, but which is a fairly basic building block if you want the other constructs), but closures are fairly recent in the history of Perl. It’s conceivable that Objective-C could have overtaken C++, but that still has not happened.

Nowadays, however, JIT compilation is enjoying new-found popularity. For VMs, it’s been almost a given, but even JIT machine code compilation is enjoying some popularity. There are a couple of fun semi-toy languages if you’d like to see how it’s done, such as Potion (by the beloved and missed _why) and EaRing (by the talented and difficult-to-ignore Zed Shaw). EaRing, in particular, is of interest for its use of Lightning , which is a library of C macros that will portably generate machine code at run-time (although you’ll need the latest from their git repository if you want it to work on x86-64), thus bringing a sort of higher-level coding facility to C. (If you are interested in seeing a 250-or-so-line demo of Lightning, there is LBF , and if you are reading this a few months in the future, you can check the Pez project , which will be using Lightning for its FFI by then. Full disclosure: I am involved in those two projects.)

So even C gets dynamic coding nowadays, albeit in a slightly different form than real data-as-code languages, but by generating a function that dereferences some pointers you compile into it and by being careful with your static variables and memory allocations, you can get real, live closures in C.

So, wherever it went, dynamic code has made a resurgence and this is for the best. The mainstream is catching up with techniques that would have seemed esoteric or even frightening to many coders ten years ago, which benefits even those for which dynamic coding techniques never died: no one gives you a funny look for saying “lambda” now, and no one gets upset if they see you’ve generated code at run-time. It may have taken us a while to collectively catch up with history, but we’ve made it on this front, and at the pace we’ve kept up in this decade, the 2010s should be just as interesting for our profession.

Tagged with: dynamic code

Related Posts

Author

Pete Elmore

Pete is a Developer on the Pane team.