Computerworld is undertaking a series of investigations into the most widely-used programming languages. Previously we have spoken to Larry Wall, creator of the Perl programming language, Don Syme, senior researcher at Microsoft Research Cambridge, who developed F#, Simon Peyton-Jones on the development of Haskell, Alfred v. Aho of AWK fame, S. Tucker Taft on the Ada 1995 and 2005 revisions, Microsoft about its server-side script engine ASP, Chet Ramey about his experiences maintaining Bash, Bjarne Stroustrup of C++ fame, and Charles H. Moore about the design and development of Forth.

We’ve also had a chat with the irreverent Don Woods about the development and uses of INTERCAL, as well as Stephen C. Johnson on YACC, Luca Cardelli on Modula-3, Walter Bright on D, Brendan Eich on JavaScript, Anders Hejlsberg on C#, Guido van Rossum on Python and Prof. Roberto Ierusalimschy on Lua. We most recently spoke to Steve Bourne, creator of the Bourne shell.

In this interview Falcon creator Giancarlo Niccolai, took some time to tell Computerworld about the development of Falcon, the power and influence of C++, and how the new multithreading design in Falcon version 0.9 will innovate the scripting language panorama.

If you wish to submit any suggestions for programming languages or language authors you would like to see covered, please email kathryn@computerworld.com.au

Part of my daily job was taking care of the status of servers streaming real time data through financial networks. A scripting facility in the control application would have been a godsend, as the alert conditions were too complex to be determined, constantly shifting and requiring constant supervision. I was not new to the topic, as I previously worked on the XHarbour project (a modern porting of the xbase languages), and I also did some research in the field.

The workload was heavy; even if the logic to be applied on each message was simple, data passing through was in the order of thousands of messages per second, each requiring prompt action, and each composed of about one to four kilobytes of raw data already de-serialised into complex C++ class hierarchies.

In terms of raw calculation power, the existing engines were adequate, but they were greedy. They considered their task as the most important thing to carry on the whole application, and so they didn't care very much about the time needed to setup a script, to launch a callback, to wrap external data or to provide them with data coming from the application at a very high rate.

It was also quite hard to use them in a multithread context. The only vm designed to work in multithreading (that is, to be used concurrently by different threads – we had many connections and data streams to take care of) was Python, but it worked with a very primitive concept of multithreading forcing global locks at every allocation and during various steps of each script. My test showed that this caused rapid slow down, even in parts of the application not directly related with the script (i.e., in the parts preparing the data for the scripts before launching Python vms).

Of all the possible scripting engines, LUA was the most adequate, but using it from concurrent threads posed some problems (at the time, the memory allocator wasn't threadsafe, and needed to be refitted with wide global locks). Also, having to deal with wide integer data (prices on financial markets are often distributed as int64 with decimal divisor) I was worried about the fact that LUA provided only one type of numbers – 58-bit precision floating point. This also caused severe rounding/precision problems in the financial area, and are thus generally avoided.

There was so much work to do on those engines to make them able to meet the requirements for task that the alternatives were either dropping the idea of scripting the control application, and then the servers themselves on a second stage, or writing something new.

I hate to give up, so I started to work at HASTE (Haste Advanced Simple Text Evaluator), a scripting engine meant to be just a scripting engine, and to drive massive throughput of data with the host application.

Was there a particular problem the language aimed to solve?

The main idea behind the early development HASTE was the “integratability” with existing complex multithreaded applications and the interaction with real-time, urgent and massive data flow.

When I had something working, I soon realised that the ability to deal with raw data and the way the vm cooperated with the host application was very precious in areas where other scripting languages didn't shine like binary file parsing, image manipulation, gaming (not game scripting), and wide/international string manipulation, etc.

At the same time, I found the constructs in other scripting languages limiting. I could live with them, as imperative and prototype-based programming are quite powerful, and the "total OOP" approach of Ruby is fascinating, but now that I had HASTE working and doing fine (the HASTE vm was simpler, and slightly faster, than LUA's) I started thinking beyond the pure needs of the engine.

As a professional, that exact task for which I built HASTE was just a small part of my daily activities. Similarly to the way Larry Wall built Perl out of his needs (to parse a massive amount of unstructured log dataset), I started to feel the need for higher logic to carry on my tasks – complex analysis on structured data, pattern finding and decision making.

I used to work with many languages including C, C++, Java, Assembly, Lisp, Prolog, Clipper/Xbase, Delphi, SQL, and of course Python, Lua, Perl and PHP, and I learned through time to employ the best instruments to solve the problem at hand. I felt the need for a tool flexible enough that it could cover my daily needs and drive new ideas.

Pure ideas are useful for the machine. A pure logic language, as Prolog, can explode the rules into raw machine code, being as fast as possible in finding solutions. Pure functional languages, as Erlang, can parallelise massive calculation automatically and compile time. Pure OOP languages, such as Ruby, can treat any entity just the same, reducing the complexity of the code needed to implement them.

But purity is never a good idea for the mind. The mind works towards unification and analogy, and solutions in the real world are usually more effective when a wide set of resolutive techniques can be employed. This is true even in mathematics, where you need to apply different resolutive techniques (and often also a good deal of fantasy and experience) to solve seemingly "mechanical" problems as the reduction of a differential equation. HASTE was terribly simple, a purely procedural language with arrays and dictionaries, but it had an interesting feature – functions were considered normal items themselves. This gave me the idea of working towards a general purpose language (beyond the scripting engine of HASTE) whose “programming paradigm” was “all and none”.

So Falcon was born with the idea of having pure OOP (so that raw, hard C structures could be mapped into it without the need for a dictionary-like structure to be filled), but not being OOP, with the idea of having pure procedural structure (driven by old, dear functions and call-return workflow) but not being procedural, and with the idea of having functional constructs, but without being functional.

It was also developed with the idea to add new ideas into it besides, and throughout, the existing ideas. A set of expanding concepts to solve problems with new conceptual tools, to reduce the strain needed by the professional developer to find the right way to match its idea with the problem and to create a living solution. Not needing anymore to learn how to think in a language to have it to work out the solution, but having a language moving towards the way the programmer's mind solves problems. I needed a tool through which I could shape easily and flexibly solutions to higher logic problems, in different and ever shifting domains (one day parse gigabytes of binary data in search for behaviour patterns, the other day organising classroom “turnations” for courses in my company) and to do that fast.

If there is one thing I’m proud of in Falcon it’s that it wasn't born for the most exotic reasons, but to address the problem of integration and empowerment of massive applications on one side and the necessity do solve complex logic and highly mutable problems on the other. Or in other words, it was born as a necessary tool.

Few languages were born to address real problems, like Perl, Clipper, possibly C++ and C which actually evolved from research/didactic university projects.