Rethought version 6 due this year

In an interview, the Perl founder says a long-awaited and feature in December 

Despite criticisms such as it having a "cryptic syntax," the Perl language has remained prominent in language popularity assessments, even if popularity has declined and a planned upgrade has been slow to appear. Designed by Larry Wall, the scripting language is suited for tasks ranging from quick prototyping to Web programming and system management tasks, and it's part of the prominent LAMP (Linux Apache MySQL Perl/PHP/Python) open source stack. At the recent FOSDEM conference in Brussels, Wall revealed intentions to have the long-awaited Perl 6 release out in a beta version in September and generally available by December. Wall answered some questions from InfoWorld Editor at Large Paul Krill via email about what's planned for the language and responded to criticisms.

InfoWorld: At what stage of development is Perl 6 and what exactly is expected this Christmas?

Wall: We're mostly done with making it work right and have a good start on making it work fast. As we just announced at FOSDEM 2015 in Brussels, we now think we understand the "works right" part well enough that we'll have a good shot at nailing down a validation test suite for the first official definition of Perl 6 this year as well as providing at least one implementation that passes that validation suite. This is the first version of Perl that will be defined primarily by a test suite, not by a particular implementation. Call the language definition 6.0 for now, though we reserve the right to give it a more "standardly" name.

As for speed, what can I say? By the end of the year, it'll be faster than what we have now, and what we have now is already adequate for many purposes. Perl 6 is designed to be highly optimizable by the proverbial "sufficiently smart optimizer," but we are still in the middle of that work. Performance can still be a bit uneven depending on what you're trying to do and on how much information you give the optimizer under the system of gradual typing.

Perl 6 is also designed to evolve over time, so at some point there will be new definitions of the language and new validation suites to test those new versions of the language. But we wanted to put a stake down saying the basic feature set is now stable. If we choose to change something after that, it will be done either through a sound deprecation policy or by emulation of the old features. People have built a number of successful businesses on Perl 5 over the years, so we understand both the need for stability as well as the danger of being so stable that you can't easily adapt to the future.

InfoWorld: What are the key improvements in version 6? Why should developers be excited about it?

Wall: In a nutshell, it's what people like about Perl minus lots of what people don't like about Perl, with additional cleanups and powerups that Perl 5 could not do with its original design. Perl 6 is definitely still a version of Perl philosophically, insofar as it's an operator-rich, multi-paradigm language that prefers expressiveness and programmer creativity over telling the programmer how they have to solve the problem.

On the other hand, Perl 6 is also a completely different language that has been rethought and rebalanced on every level, with much stronger support for both functional and object-oriented programming as well as reactive and concurrent programming. There is now pervasive concern for composability, evolvability, readability, and maintainability. We try to "hang everything on the right peg" to allow proper separation of concerns. While we still believe in providing multiple ways to accomplish something, we now try to encourage one of those ways as an appropriate starting default. So we think the language scales both down and up to the level of talent of the programmer; the programmer can start out in "baby talk" without getting into too much trouble and then whenever he or she is ready to learn a new concept, the language is ready to support that and to make it easy to refactor the code as the programmer grows in sophistication.

For developers who are already sophisticated, they'll see that most of the problems endemic to the currently available dynamic languages are solved in Perl 6. We understand lexical and dynamic scoping. We detect most typos at compile time. We don't have a global interpreter lock. We encourage Smalltalk-style trait composition over uncontrolled mixins, multiple inheritance, or monkey patching.

Of course, if someone just wants a laundry list of slick new capabilities, we'll trot out terms like a pervasive meta-object protocol, representational polymorphism, generational GC, higher-order programming, metaoperators and "adverbs," junctions, lazy lists, hygienic macros, grammars powerful enough to parse Perl or any DSL you care to name, world-class Unicode support, built-in scheduling to support multicore and many-core scaling, multiple target VMs including JVM, and so on. Autodocs, there's another [feature].

InfoWorld: What has been the holdup with Perl 6? I just ran across an article stating it was due an "imminent release" in 2010.

Wall: Any article (before now) that made a specific date prediction was not written by me or by anyone under my psychic control. We knew starting out that we were optimizing for the long term, not the short term, and that the reason we had this rare privilege was because Perl 5 was so stable, well-loved, and supported. I have always said that Perl 6 will take however long it takes, and I only care whether the process is convergent. We were given this one chance to jump off the worse-is-better treadmill, and we should take that chance despite the hit that we knew we'd take on market share.

So early on, our slogan, or at least one of them, was "Second System Syndrome Done Right." And how do you do that? Well, you just have to take long enough. Companies can't do that because they have a bottom line and a burn rate. But we're an open source community, not needing to make a profit, only to do good in the world. So you know the saying, "Good, fast, cheap: pick two." Well, by definition our community has to do it cheap, so the saying reduces to "Good, Fast: pick one." And we quite intentionally picked good rather than fast.

And frankly, it takes a long time to implement a largish language, especially if you're aiming for more than just a hodgepodge of features. We call what we do the "whirlpool methodology," where the design and the implementations negotiate continually over the tests that define the product, with information flowing in various directions on various levels simultaneously. But the whirlpool sucks everything in eventually and guarantees convergence. This contrasts to the standard waterfall methodology, where you spec everything first and then implement what you spec and then you write tests. That's what people thought we were doing, but we know that nobody is really smart enough to design and implement Perl 6 that way. Instead, we designed a process that allows for experiments and mistakes along the way. Which we made plenty of, much to the merriment of naysayers.

And yet, here we are.

InfoWorld: Will the Perl 5 line still be developed after the release of Perl 6? Are there any backward compatibility issues?

Wall: Perl 6 was intended from the very start to break backward compatibility; that was the whole point. And soon after we decided that, we decided to break everything that needed breaking and embrace the major overhaul, along with all the cultural noise that goes along with taking your time to do something right. [Also], we're not making the mistake of declaring end-of-life for Perl 5. Perl has always supported older releases as long anyone is willing to love and maintain them. Nothing here changes that part of our culture.

We're not into telling people the one true way to do things. People know their own situations much better than we do.

We do think a migration strategy is important, however. We've made it easy to embed Perl 5 and Perl 6 in each other so that you can convert your modules piecemeal as it makes sense. More importantly, you can run your regression tests at various stages of the conversion.

InfoWorld: It has been claimed Perl has been slower than other languages, such as Ruby, Python, and JavaScript. What's the state of Perl's performance now, and what will performance be like with version 6?

Wall: Well, as they say, lies, damn lies, and benchmarks. Way back in the dark ages, Perl usually beat out Ruby, Python, and Java in performance, particularly for the text-processing space it specialized in. We knew other languages would achieve some level of parity at some point, and in fact, part of the reason for redesigning Perl was that the original design was limiting the opportunities for optimization. Now we've given those other languages a nice head start, but I'll just point out that velocity always beats acceleration in the short term but loses in the long term. And we think we have a better gas pedal.

We've done some work with type specialization and JIT and such, but we've still got lots of headroom for better JIT codegen, for a tracing JIT, for runtime-based feedback to AOT optimizers and incremental compilers. Many of the features of Perl 6 are designed with eventual optimization in mind, such as compact native storage, hyperoperators that can map easily to a GPU or vector processor, concurrency primitives that scale well to multicore, and application-wide policies that can close and finalize classes automatically. 

And the user will have more possibilities for making better use of our gradual typing to give hints to the optimizer in places the profiler points out as hot code. We expect big improvements just from making sure tight loops stay in the CPU's cache. We hope to get to that this year.

InfoWorld: Why did you design Perl?

Wall: I had confidence that I could build something that would work better than anything else available at the time. Unix programs were supposed to follow the mantra: "Do one thing and do it well." Most of them failed on both counts. So a lot of early Perl was simply avoiding the arbitrary limits of the typical program of the era. But Perl also violated the reductionist religion of the time by providing a more holistic language solution that could function, not so much as a tool, but rather as a machine shop for making new tools. I knew enough about linguistics to know I could make a contribution there, even if I was pretty stupid about computer science back then.

These days I know a bit more about computer science, and I still think I can make something that will work better than anything else available now. There are still lots of language tools out there that pose arbitrary limits of various sorts, albeit on a more abstract level. We can do better.