ack in the days of our fathers, programming meant focusing on learning one language, one platform, and one environment, and mastering it over a span of years. Those years are long behind us, along with half-decade project development times and bell-bottomed pants. It’s time to take a hard look at the “state of the union,” per se, and see how you can think about how to work better, rather than just putting in more time.

Thanks, Ken Getz!

As frequent readers of this magazine will already know, Ken Getz used to prognosticate and discuss all manner of things technical in this space, offering his insights and experience. But Ken has decided to “hang up his spikes,” as they say, and Rod and the crew at CoDe Magazine asked me to help fill in for his absence.

Let’s be honest, nobody could ever replace Ken’s unique perspective, but it’s flattering to be asked to try. In fact, it’s a touch intimidating, because Ken had a perspective that stretched across decades. Fortunately, I have a solid crew of people with me at ThoughtWorks, the consulting company I work for (hence the name of this column), who are willing to step in and help me try to walk in Ken’s footsteps.

Research to Real-World: The Relevance Gap

In the UNIX world, developers have known for years that sometimes the easiest way to attack a problem is to create “a program that writes a program.” Sometimes they use Perl, often combined with code-generation. Sometimes they use regular expressions, either fed to a library that returns hits from a string, or compiled into an actual code library that can be included with your code, (another example of code generation, but in binary form). A third trick was to create an entirely new tool, sometimes known as creating a “little language.” Today, in a grinning endorsement of the truism that “everything old is new again,” this notion of creating the “little language” rises again under the new moniker “domain-specific language” (or DSL), and currently enjoys tremendous media discussion.

The subject of programming languages?or, more specifically, building programming languages?is one that most “line-of-business” developers find hard to relate to. This isn’t surprising given that most programming language research has been restricted to the academic and research communities. These restrictions occurred not because those communities don’t want their research widely read or consumed, but because line-of-business developers didn’t see it as relevant or useful.

Unfortunately, the “relevance gap” meant that a developer out hunting around on the Internet looking for cool new tools or libraries to help build a highly-scalable (meaning highly-concurrent and thread-safe) web application might run across some papers or discussions that reference this concept of “functional languages.” Further research might lead to some papers on why functional languages are great for concurrency, sparking more interest. A bit more reading turns up a reference to a concrete implementation of a functional language called OCaml.

OCaml, for those who’ve not spent a stint in the research or academic community, meshes the functional and object-oriented programming approaches in some interesting ways. For that reason, among others, it enjoys a pretty good following in academic circles, including some interesting forays into the concurrency space. So our intrepid developer begins working with the language, learning the new syntax and playing with the new libraries, and decides that it’s interesting enough to use for building something real. She decides to prototype it for use on her current project.

Like any good horror film, at this point the audience can see it coming, but our developer can’t, for whatever reason.

She runs into problems. Not tiny little problems like finding a version that runs on Windows, but larger problems. Obstacles that cannot be overcome in any reasonable timeframe. Building a UI with OCaml is awkward and difficult at best, because it tries to be platform-agnostic and thus relies on other adaptation layers to reach the underlying widget set. Accessing Windows-only features (such as printing) will be a pain, requiring her to build a “trampoline” layer to access native C-based Windows APIs to do anything. Trying to connect with any modern relational database implementation will require?patience.

The problem isn’t really our developer’s fault, nor is it OCaml’s, to be honest. It’s not that OCaml isn’t interesting?it is?but out of the box (so to speak), OCaml isn’t a language that a line-of-business developer can use in any sort of pragmatic sense. Which means, unfortunately, that not only did our developer waste her time researching the subject and discovering its applicability, but she also learned that academics and researchers are not nearly as interested in creating real-world, production-ready solutions as they are in exploring ideas, at least to the point where they can prove or disprove an idea. How likely is our frustrated developer to trust something coming out of the academic or research community in the future?

Therein lies the heart of the problem: academics and researchers focus on trying to solve problems that line-of-business developers may not face until five or ten years down the road, and that forward-looking focus often comes at the expense of helping to build today’s solutions.

Stay with me, friends, there is light at the end of this rather bleak tunnel.

A Framework for New Languages

In the past couple of years, an interesting trend?starting with the Java Virtual Machine’s original release?has begun to take root in the .NET world as well: Some researchers started building their languages to run on top of the CLR. They do this not just to make their languages and tools available for .NET developers to use (although that’s a nice side benefit), but because when they target the CLR, a large number of problems they don’t want to solve (creating a garbage collector, a module-loading facility, access to the underlying platform, and so on) is already solved for them. They simply build their language to emit IL instead of native code, and suddenly, they can take advantage of everything the CLR offers, from ADO.NET to WPF to Castle to NHibernate to?you name it.

All this leaves academics and researchers in a better place, but so far it still ignores the line-of-business developer. This article started with a developer looking for a way to build high-scale concurrent applications. So far, I’ve established that while academics and researchers improve the development world, delivering their benefits back to the “mainstream” development community is a slow process. It takes a while for new concepts and ideas to filter their way from research to mainstream. After all, it took object-orientation something like 25 years to move from the academic/research world into the mainstream programming world.

Fortunately, the new programming language renaissance is being created on top of the platforms already available in commercial enterprise settings?the JVM and the CLR. So when an academic decides to move the OCaml programming language to the CLR, line-of-business developers can use the work almost immediately.

For example, one Microsoft Research project called “AbsIL” made it easier for developers to take compiled assemblies, untwist them back into a tree structure (called an Abstract Syntax Tree) and modify the code before re-emitting it as compiled assemblies. This is intrinsically a functional type of operation; given the same input, the code will always generate the exact same output. Consequently, as the Microsoft researcher responsible for AbsIL built that library, he also sat down to produce a CLR version of OCaml.

But to make OCaml-on-CLR work correctly, he first had to implement a form of parametric polymorphism for the CLR, which he created as a series of source patches on top of the Microsoft open-source .NET implementation, now known as the Shared Source CLI, or “Rotor.” That set of source patches is still available to this day, named “Gyro.”

Still not convinced that all of this has relevance to the line-of-business developer? You might change your mind when you realize that the series of source patches to “Rotor” became the seed for the generics model that Microsoft folded into the CLR in the .NET 2.0 release, and that the CLR-based version of the OCaml language came to be known as “F#,” which will be included in Visual Studio 2010.

In fact, the next generation of the .NET platform is about to explode with new languages. In addition to F#, Microsoft will release two more languages via the open-source community: ports of Python (IronPython) and Ruby (IronRuby). The rest of the community is also stepping up to the languages plate, with new languages such as Boo, Nemerle, and P# (Prolog#) that add even more variety to the mix.

Get In Front of the Wave

Moreover, this increased focus on creating languages makes it easier for developers to build their own languages?whether a “DSL” or something that makes it easier for other line-of-business developers to focus on their chosen tasks. Using some good examples already available, such as Joel Pobar’s “Good For Nothing” compiler from TechEd a few years ago, a developer can produce a first simple custom language in about a day?even with no prior experience in building languages.

This isn’t to suggest that every single project anyone works on from here on out is going to require a custom development language or a DSL; like all new tools, there is a time and a place for building a custom language or a DSL. But it is fair to say that the next five or ten years is going to see an explosion of new language interest and implementation, and ?tis the wise developer indeed who looks to get in front of the wave to ride it, instead of paddling like mad from the back.