Why Do Most New Languages Fail?
New programming languages are invented everyday. Some of these try to solve domain-specific issues, but others hope to become big players among the general-purpose languages used by software developers. Perhaps it’s simply human nature. There are already many general-purpose programming languages in wide use out there in the real world, but some of us are never satisfied. We can’t help but see the flaws in the existing tools, and out of some OCD-like compulsion, we must attempt to create something better. Unfortunately, most of these new programming languages die shortly after birth.
What makes the difference between programming languages that fail and those that succeed? Some will argue that it comes down to new languages having familiar semantics. Others will claim that it’s purely down to luck. I believe the truth is rather simple: most new programming languages are impractical to use, on a variety of levels. Furthermore, for a new language to succeed, it must, as with any product we hope people will use, fill some kind of niche. It must do at least one thing that the others don’t, or do some things better than the competition.
I believe that most programming languages that fail do so for at least one of the following reasons:
- The language is impractical. This means highly unusual syntax, high verbosity, oddball semantics that don’t work well together. It’s the good old “it works well for the toy problem the inventor wanted to solve, but nothing else”.
- The tools are impractical. The stock implementation of the language only runs on one platform. You have to build it from sources, and it has 50 dependencies, but you don’t have 3 hours to spare just to try this out. The tools have weird command-line flags, it’s unclear how to even compile/run a program written in this language.
- The language can’t interface with anything. Hardly any libraries ship with it, it can’t interface with C code easily. All you can easily hope to write in this language are command-line programs that use stdin and stdout, but your application requires access to a MIDI controller and OpenGL, you can’t use this language.
- The language is slow as hell. The stock implementation isn’t a compiler, but a very naive interpreter. The language runs slower than Python, but has none of Python’s qualities. It’s inventors claim it would be easy for someone else (not them) to make a fast compiler for it, but nobody is doing it.
- The stock implementation is incomplete and unmaintained. Perhaps this language was some cool academic idea. The bare minimum needed for one publication was implemented, but once this publication was accepted, the partial implementation was left to rot. This new language never got past the prototype stage, and probably never will.
For a new language to succeed, it has to be practical. It has to be useful. It has to have something to offer over the competition. This is very challenging because the world doesn’t stay still. The landscape has changed since the invention of Python back in 1991. If you wanted to compete against Python in the dynamic language realm, for example, you’d be competing against Python itself, which is now a very mature product shipping with dozens of libraries and volumes of documentation. Since Python came around, the bar has been raised.
To those who hope to eventually create a new programming language, I would give the following advice:
- Study programming languages in general, but also study languages you wish to compete against. If it’s dynamic languages you’re interested in, look at Python, Ruby and JavaScript. They each have glaring design flaws. To compete against them, you should take advantage of this and do better at the things they do poorly. You should also try to avoid introducing glaring design flaws of your own.
- Study compiler design. Speed matters more than you think. You would be smart to design your language in a way you know for a fact a compiler could easily optimize. This will give you a speed advantage over the competition, and probably make your language more robust as a result. Hint: nonsense, complex, unintuitive semantics tend to hinder optimization. Simple, straightforward, consistent semantics make compiler design and optimization simpler.
- Be grounded in reality. Make your implementation pragmatic. If your language is to become popular, it means you won’t be the only user. Design your implementation to be portable, minimize dependencies on external libraries. Release precompiled binaries. Make sure your language can interface with C, because it is the de-facto standard systems language. Everything has C bindings, you should be able to support those.
- Eat your own dog food. You should be trying out your new language on various problems and making sure it works well for common use cases. You should be trying to anticipate the needs of future users.
- Put the work into it. Your compiler isn’t gonna write itself, and nobody likes half-finished, untested products. If you want to create a successful programming language, you might well have to spend a few years polishing your implementation. It might take a large amount of effort for your project to be complete enough for people to notice this little gem of yours.
I would also like to add that it probably helps to prototype things. If you want to create a new programming language, it might be pretty easy for you to write a simple interpreter and prototype a version of your language that uses a LISP-like syntax. Just remember that there’s often a pretty big gap between prototype and final product.
I’m inflicted with a brother-inlaw that is going to write the next great functional programming language. So I get earful how the C++ I use is evil and how functional languages are the greatest. At least he is off of telling me that I should have used Java now that Oracle owns it. He has lead to me to some interesting reads over time like “Purely Functional Data Structures” by Chris Okasaki. At least he does what he preaches. He is off programming Clojure in an Amazonian Cloud, before that was OCAML on Wallstreet.
The one thing that stands out is no language I know does concurrent programing well. I’m a 3D CAD/MCAE software architect. Day in Day out I work with large complex data structures. The size of problems we can solve is pretty much limited by hardware. Seems like a pretty good area for concurrent programming. We have customers that run our software on 32 Thread/16 Core/128GB workstations. The reality when solving performance problems is that concurrent programing is not the first thing we use, but the last if ever.
The software product I currently work on Enventive is a variational geometry sub-system. Basically all the geometry is stored as a large system of equations which is sent to a solver. Solvers are N^3 and take N^2 memory. Ouch. 5000 equations would 400MB of memory and take minutes to solve. We store the equations in a graph apply some N log(N) graph theory magic to it to break it into many small systems of equations which solve in N time if we have been successful. Our first performance fix was to change the order of the algorithms. Eventually as our customers make 100,000 or even 1,000,000 equation parts we will need use multiple cores to speed things up.
Many things come before using concurrent programming. New features, bug fixing, regular performance tuning, then concurrent programing if any time is left. The only thing parallelized so far is the N^3 solver, but because of the graph theory magic it is not needed very often. Because of the little time and low priority, concurrent programing needs to be easy to apply or it won’t be used much.
I don’t have much opinions about concurrency because it’s not really my field, but it seems to me that in the presence of side-effects, (freely) shared memory is a generally bad idea. Two threads probably shouldn’t be able to just write data to the same variable without any kind of safety.
Perhaps the best thing would be to have a language that allows side effects but discourages them. Outline very specific ways in which data structures can be shared. For example, purely functional, read-only data can be shared without restriction. Mutable data has to be shared through special data structures with built in synchronization mechanisms. The language knows exactly which is which.