Skip to content

Building a Small-Scale Prototype

Watch

I’ve blogged before about my plans to create a programming language of my own. I have a long list of features I’d like to experiment with. I want to incorporate features that are not commonly found in other languages. For instance, the language is going to have an extensible grammar. I’d like to have program images that can be suspended and resumed like in Smalltalk. I’m also trying to sketch out a JIT compiler that compiles itself. Unfortunately, so far, I’ve spent more time thinking about the design and implementation of this language than writing any code. I wrote a decent amount of code, then ended up scrapping it and starting over, only to think that I should maybe start over again.

Part of the problem, I think, is that I keep wanting to plan 20 steps ahead. My language, zeta, is going to have a few features which are fairly uncommon/experimental, such as the extensible grammar. I don’t know ahead of time precisely how these features are going to work out as part of the overall design, there are many small micro-decisions to make, things that could be implemented this way or that way, and that’s where I keep getting stuck. I keep worrying about issues such as performance, garbage collection, multithreading, and how different language features will integrate with one another. Inevitably, I end up having to admit that there are simply a lot of important questions I don’t yet have an answer to. Language design is hard, and so is compiler design.

I think that this is a common problem with ambitious engineering projects. When you’re trying to innovate (or just do things differently) on ten different fronts at the same time, there are simply too many unknowns, too many degrees of freedom. This is why innovation is typically more incremental than revolutionary. People build on trusted foundations, and maybe they add one or two cool new features on top. When you try to reinvent everything, there are just too many problems to be solved at once. Solving multiple problems simultaneously is problematic, because it gets in the way of building your product. You want to create something that’s globally optimal, as good as you can make it, but you keep having to make technical decisions which you simply don’t have enough information to make. Each of these decision points stalls you a little bit, because making the optimal technical decision requires taking the time to pause and gather more data. Worse yet, not all of these technical decisions are independent, some are simply incompatible with one another. Choices you make now will push you down a certain path later down the road. Sometimes, you make a certain set of choices, and later on you realize that you went down the wrong path, and you have to backtrack.

I keep wanting to do the ambitious thing, and getting into complex, elaborate designs as a result. I start building, and I often end up backtracking. I’ve decided to take a step back and ask myself some important questions, such as, why “Why do I even want to create a programming language?” and “What am I getting out of this?” One of the things I’ve come to realize is that creating a programming language as a hobby project for the simple joy of doing it, and creating a programming language which I hope will eventually become popular and dethrone python are two very different projects with very different requirements. Creating a programming language for fun is a project with very loose requirements and much freedom to explore, motivated by the joy of programming I discovered when I was a teenager (and hope to keep alive for a long time). On the other hand, creating a language in the hope of whacking Python in the gonads with a 2×4 is a project likely motivated by my inflated ego, and my failure to integrate mindfulness meditation into my daily routine.

Designing and implementing a popular programming language means I have to care about many issues such as:

  • How learnable/palatable the syntax is to the masses
  • How beginner-friendly the language is
  • Integration with C/C++ code
  • Deployment, how easy it is to install my compiler/VM
  • Being able to generate binary executables
  • Multi-platform support, including mobile devices
  • Unicode support and internalization
  • The absolute performance of my language
  • Multithreading and parallel performance
  • Security of my implementation (exploits)
  • Responding to the wants and needs of the community

All of these issues, and more, are things that necessarily strongly influence the design of a language. They are things that limit my ability to freely explore language design. Most importantly, creating a language with popularity in mind means that this project starts to be very much about caring about the wants and needs of the community. Lots of limitations on my ability to pick the language design I want, and having to care (work for) a huge community means that the project starts to look a lot like a job rather than a fun hobby. Hence, if I want to explore language design, maybe I ought to go about this project differently, scale down my ambitions, and go for something less grandiose.

I’ve decided that I’m probably not ready to build a commercial/popular language. Instead, I’m building a small-scale prototype. I’ve asked myself “What’s the most important language feature I want to experiment with?” The answer is the extensible grammar. What I’m doing, then, is building a sort of Minimum Viable Product (MVP). A tiny minimalist language with just the features I need, so that I can get to the interesting part of experimenting with an extensible grammar as soon as possible. Right now, this language is interpreted only, and it has no garbage collector (it never deallocates). Even this, I’m finding out, is challenging. Even this tiny project involves solving many technical problems, but things are already much simpler, and most importantly, much more fun. Once I have the extensible grammar working well, I’ll move on to adding other features, and exploring other equally interesting technical problems.

Exercise, Depression and the Energy Hypothesis

There’s an increasing body of evidence claiming that exercise can help alleviate symptoms of depression. Some studies are claiming that exercise is just as effective as prescription antidepressants, and possibly even more so. The question I ask myself is: why is exercise helpful in cases of depression? It’s universally accepted that exercise is good for you in many respects. So much so, that we rarely stop to ask ourselves why that might be.

I’ll begin by saying that depression is a complex phenomenon. I don’t personally believe that depression is some one disease or disorder with some fixed set of symptoms or some single cause that we can point to. Rather, I believe it’s an umbrella term that we generally use to describe the experience of people who experience persistently low moods and a lack of energy or motivation for what seems like an abnormally long amount of time. In this post, I’m going to focus on one component or symptom of depression which is known as motivational anhedonia, this is the loss of desire or motivation to engage in activities.

It seems that one of the areas where exercise is most helpful is in helping people find more energy and motivation. I’m going to propose a simple hypothesis as to why that might be, which is that your brain has evolved to save and accumulate energy. It wants you to stay alive and hopefully spread your genes, but ideally, it wants you to do this while spending the least amount of energy possible. Your brain would rather have you accumulating energy than spending it. Body fat is an insurance policy: saved up energy to safeguard you against the bitter famine that might be just around the corner. The reason why many people find it so difficult to lose weight is likely that our brains don’t really want to let us do so.

When the winter comes around, and the days get shorter, many people experience Seasonal Affective Disorder (SAD), which can cause lower moods and difficulty getting out of bed in the morning. Here’s the thing though: it’s probably natural for you to feel tired, to move less and to spend more time sleeping during the winter. Most other mammals probably feel just the same way. This is simply your instinct trying to keep you from running around and jumping everywhere, trying to make you save your energy, because in the winter, there’s much less food for you to find. Sleeping more and moving less might not feel very good, but it’s probably the safe thing to do when resources are scarce.

How does exercise fit into the picture? I propose that your motivation to move around and do things reflects your brain’s willingness to let you spend energy. I propose that probably, somewhere in your brain, there’s some mechanism which measures how much energy you need to spend to stay alive, a kind of daily energy budget. This energy budget is likely based on how much energy you’ve needed to stay alive in the past. Exercising involves forcing yourself to spend more energy than usual. When you do this, your brain estimates that your life situation requires you to spend more energy to stay alive, and increases your energy budget.

This hypothesis might help explain why, despite the abundance of food and entertainment found in western societies, depression is on the rise. By many accounts, living conditions in North America are more comfortable than they’ve ever been. Maybe that’s part of the problem though. Maybe the comfort of modern civilization: cars which can get us around without walking, entertainment we can enjoy while sitting on the couch and microwaveable food we don’t even need to cook, have made our lives too easy. These things have made it possible for us to survive without spending energy or moving very much. Maybe, for many people, the way to find more energy is to spend more energy.

Interestingly, there’s also been some recent research suggesting that intermittent fasting, and ghrelin (the hormone which produces the feeling of hunger) might stimulate neurogenesis. As such, it’s possible that intermittent fasting might help combat depression. This also fits with the energy hypothesis, in that when there is a calorie deficit, and energy sufficiency isn’t guaranteed, the brain becomes motivated to have us spend energy so that we might be able to find more food.

I’ll conclude by saying that I do not, in any way, mean to trivialize depression, or the suffering that’s involved. I’ve been depressed, and I know very well that the last thing you feel like doing, when it seems like your world is crumbling down, is to hop on a threadmill and run. I also know that the last thing you should tell a depressed person is to “just snap out of it”. If you have a depressed friend, and you want to help them, then my advice to you is to offer them support, kindness, and a lot of patience. Lastly, exercise and diet can only go so far. Depression is not one disease, and exercise is not some universal cure. Some people cannot get out of depressed states without the help of professionals and prescription antidepressants. These people need help and support just as much.

Musings on AOT, JIT and Language Design

Lately, I’ve been pondering the strengths and weaknesses of static and dynamic languages. In particular, the reaction of the online programming community to Elben Shira’s post The End of Dynamic Languages made me realize that many (most?) programmers out there seem to prefer static languages. This seems to be largely because static languages usually come with better IDE and tool support, as well as static guarantees which programmers value highly (and rightly so).

I’m slowly working on a design for a programming language of my own, and I had a “wait a minute, let’s take a step back” kind of moment. Why? Because the language I was working on was going to be extremely dynamic. It was going to be JIT compiled, dynamically typed, with an extremely powerful macro system based on the ability to hijack and modify the parser while a program is being parsed. I was also considering using a self-modifying Intermediate Representation (IR) as part of the design of the JIT compiler.

Elben’s post made me take a step back because it made me realize that the design of my ultra-dynamic language would make it near-impossible to have much tool support. If you can hijack the parsing of a program while it’s being parsed, and alter the parser using custom code you wrote, then good luck running any kind of static analysis. It’s just not going to work.

I saw that I was a being myopic. I use a bare-bones text editor, no tools, and I’ve only used a debugger a handful of times in the last few years. That’s the way I program, but it’s not the way most people program. I can get away with this because the projects I’ve worked on have all been under 100K lines of code. Designing a language which makes any kind of static analysis, refactoring-assistance or autocompletion near-impossible is a sure way to guarantee the language won’t gain adoption outside of hobbyist circles.

Then, there’s the issue of JIT compilation. I very much like JIT compilers because that’s where my expertise lies, I’ve written my own JIT as part of my PhD thesis. Still, I have to keep an open mind and ask myself what’s best for my language. JIT compilers are fundamentally more powerful in terms of optimization capabilities than AOT (Ahead-Of-Time or static) compilers, because they can adapt the way they optimize code based on the behavior of running programs. They’re great for dynamic languages.

Unfortunately, it seems that the world, at the moment at least, is stepping away from JIT compilation. In order to support mobile platforms or compile to WebAssembly, it seems to me that JIT compilation is impractical. Why? Because it requires generating machine code on the fly. At the moment, it’s unrealistic to just plop a custom binary with self-modifying machine code on the Apple or Android app stores.

I’m thinking I might just design my language so that it compiles to plain old C. This would make it possible to leverage the C compilers that already exist for every platform out there. Rest assured though, this language will be dynamically typed ;)

Have Static Languages Won?

A few days ago, Elben Shira caught the attention of the programming blogosphere with his post entitled The End of Dynamic Languages. The key point from this post is in the following statement:

This is my bet: the age of dynamic languages is over. There will be no new successful ones.

Like him, I’ve noticed that despite the fact that there have been an enormous number of new programming languages coming out recently, the overwhelming majority of them are statically typed. Elben and others make the argument that this is because static languages are better equipped to deal with larger projects, they have better tooling, and programmers prefer them.

I’m fairly invested in dynamic languages. I happen to be completing a PhD thesis on techniques for optimizing dynamic languages for performance (JavaScript in particular). I’ve also been slowly working, in my spare time, on a dynamic language of my own. I like dynamic languages, but I think it’s important, as a scientist and as a human being, not to stupidly fall into the trap of confirmation bias. I’m not going to vehemently defend dynamic languages, or claim that those who don’t appreciate them are unknowing fools. Instead, I’ll be honest and say that Elben’s post, and the significant amount of agreement he found in online communities made me pause and question myself. Are static languages inherently superior? Are they really winning?

That there are less dynamic programming languages coming out is an undeniable fact. I’ve written code in statically typed languages such as C, C++, D and OCaml, and I agree that their type systems help catch certain classes of bugs more easily and rapidly. When writing code in JavaScript, you can run into nasty surprises. Latent, trivial bugs that remain hidden in your code, sometimes for months, until some specific input causes them to manifest themselves.

The problem here though, is that JavaScript is a badly designed programming language. The JS origin story is that Brendan Eich originally designed the language in just two weeks. As such, it has many glaring flaws. If you compare Haskell, a language that came out of type theoretic circles and was carefully crafted by a large research community and extensively studied, to JS, a language designed in two weeks, you find that one looks like a NASA spacecraft and the other looks like a pile of fireworks on the 4th of July in Alabama.

Dynamic languages are at a disadvantage. Most of the mainstream ones out there today were designed by amateurs, people with no formal CS background, or no adequate background in compiler construction. They were designed with no regard for performance, and an impractical mash of features that often poorly work together. Most of the dynamic languages you know are simply poorly crafted. This has resulted in some backlash. I think it’s pretty clear that there’s some amount of prejudice when it comes to dynamic languages.

In universities, computer science professors generally want little to do with dynamic languages. Compiler design courses are focused on statically typed and compiled languages. Type theoretic courses will teach you about Hindley-Milner type inference, but will leave you ill-equipped to understand dynamic typing. Students coming out of your average university compiler and PLT classes have no idea about the challenges involved in creating a JIT compiler, and know little about dynamic typing. It’s no big surprise that these students would go on to create statically typed programming languages.

There might be another factor at play here. Dynamic languages such as PHP, JS, Python and Ruby, in addition to being relatively poorly designed, are the languages that powered the explosion of the web. Nowadays, much (most?) of the programming happening in the world is web development. Much of this work is done by people with no formal CS background. The result is that you have millions of people with less formal education writing code in less well designed languages. JS, Python, PHP and Ruby, and by extension all dynamic languages, are seen by many academics as the programming languages of the riffraff, the unwashed masses, or maybe simply programming languages for relatively ignorant beginners.

Have static languages won? It seems to me that what people really like about static languages is IDE support for things like simple refactorings and autocompletion. Program analysis that can provide some guarantees, find certain classes of bugs without having to run programs with every possible combination of inputs. It’s perfectly legitimate for programmers to want these things. They help alleviate the cognitive burden of working with large (and small) codebases. But, these advantages aren’t inherently advantages of statically typed programming languages. I would argue that Smalltalk had (has) some amazingly powerful tools that go way beyond what the Eclipse IDE could ever give you.

I believe dynamic languages are here to stay. They can be very nimble, in ways that statically typed languages might never be able to match. We’re at a point in time where static typing dominates mainstream thought in the programming world, but that doesn’t mean dynamic languages are dead. So long as dynamic languages do offer advantages, either in terms of expressiveness or ease of learning, they will still be around. You should remember that, in the end, there is no war between static and dynamic languages. There are only tools and tradeoffs.

I will conclude by saying that in my view, programming languages are constantly evolving and influencing each other in what seems like an organic process. Features that are viewed as good ideas tend to spread from one language to many others. Think about closures, for instance. The functional crowd has been in love with closures since the days of LISP, and now, almost all mainstream programming language have closures. Going back to Elben Shira’s post, he states:

We will see a flourishing of languages that feel like you’re writing in a Clojure, but typed. Included will be a suite of powerful tools that we’ve never seen before, tools so convincing that only ascetics will ignore.

I’ve written, back in 2012, about my belief that static and dynamic typing can essentially be combined. You can have statically compiled languages that use type inference to realize what is effectively dynamic typing. That is, the compiler inserts union types where appropriate, and does so automatically. The Crystal programming language is a realization of this idea. This isn’t static languages winning a war over dynamic languages though. It’s the influence of dynamic languages bleeding into static languages. The Crystal language developers are entirely honest about the fact that this language is based on Ruby. Their aim is to build a language which captures much of the flexibility of Ruby’s dynamic typing and also provides you with static guarantees.

Why I Chose Industry Over a Postdoc

About a year ago, I was considering postdoc options. Two university professors had invited me to join their research groups. I wasn’t quite sure what to do. I like research, and I like the work I’ve been doing as part of my PhD. On the other hand, I’ve been growing increasingly frustrated with academia, and more specifically, with the publication game. I’ve had papers rejected several times now. More than once, reviewers who were clearly associated with competing research projects (and did not try to disguise this fact) have shot down my work with unfair, intellectually dishonest and sometimes hostile criticism. In general, I’ve come to feel that, at least in my sub-field, the exploration of new ideas is discouraged, and I’m not getting judged on the quality of my work, I’m getting judged on how well I play the publication game.

The end of my PhD is just a few months away now, and I had to make a choice. I was tempted to continue my research on basic block versioning, but the prospect of working very hard and maybe not being able to publish a single paper made me uncomfortable. Another issue is that the two professors who approached me for postdocs have made it clear that I should apply for a postdoc scholarship. They didn’t have enough money to pay me, and if I didn’t get this scholarship, I couldn’t do a postdoc. They were also putting pressure on me to decide as fast as possible, I got to understand that postdoc positions are limited and it’s a very competitive environment.

Recently, I attended a conference and got to meet a researcher who’s pretty well known in my field. He’s someone I really look up to, someone who’s name I’d seen on several papers that have shaped the development of my own research. We had dinner together a few times during the conference, and discussed various topics. One of the things that really struck me though, is that this guy is in the process of hopping from postdoc to postdoc. He’s struggling to publish his research, getting many of his papers shot down, and having difficulty finding a position as a university professor. He’s a much better academic than I am, and he’s still struggling.

In the meantime, I never really looked for a job, but I’ve been approached by IBM, Facebook, Microsoft, Google, Apple, Twitter, Amazon, Autodesk, AppNexus, Two Sigma, Reservoir Labs, D-Wave, and a few startups. It’s been a stark contrast. On the one hand, academia is offering me a chance to maybe do a postdoc, but only if I’m deemed good enough by the people who judge scholarship applications, and I have to decide now. On the other hand, industry people are bending over backwards to try and get me to come talk to them. I decided to go out and try interviewing for some of these companies, and over a month ago, I made my decision. I signed a generous offer from a company in the bay area.

I have no illusions that industry is some amazing utopia. I’m sure it will take me some time to adapt, and that I’ll miss some of the perks of being an academic. I know I’ll also miss Montreal, the city where I was born and raised. Still, I’ve been in university for over 11 years now, and I really think it’s time for me to try something different. I think that if I continued on the academic path, I’d be headed for stagnation and a burnout. Industry, in contrast, seems full of opportunities to explore. And hey, it won’t hurt that I’ll be making over six times what I get as a PhD student. For the last two years, I’ve been renting a tiny bedroom with a window on a noisy street, and sleeping on an uncomfortable futon that’s hurting my back. One of the first things I’m buying when I make it to California is a queen-sized bed, and the best mattress that money can buy.

The Robotic Revolution

The last three years have seen a slew of impressive deep learning feats. Machines are finally able to effectively extract meaning from the fuzzy thing that is the real world. This is already enabling all kinds of things, from smarter websites to self-driving cars. I’m very enthusiastic about this rapid progress. I think it’s becoming very clear that in the next 20 years, the world will see an increasing robotic presence. I don’t mean to say that androids will instantly become ubiquitous. We’re obviously not there, both in terms of AI capabilities, and in terms of being able to produce lightweight, affordable and nimble robots. What I mean to say is that many of us have already accepted self-driving cars as an inevitability, since there are already working prototypes. The question is then, why stop there?

How much of a stretch is it to go from a self-driving car to a street-sweeping robot? What about an automated garbage-disposal truck with a robot arm that picks up garbage bags without human workers? What about an automated lawnmower you can control with your smartphone? It’s obviously going to take some time for these things to be developed, but I think we can all agree that the automated garbage-disposal truck is not that far-fetched. How much more of a stretch is it to go from such a robot to one that restocks store shelves? Those who have warned us of robots taking human jobs are most likely right.

In the domestic realm, there is already a Berkeley prototype of a robot that can fold laundry. I think one of the biggest hurdles there is going to be the cost. The economic reality is that going from a university prototype to a useful product requires several years and a large monetary investment in R&D. Furthermore, no investors are going to fund the development of such products if the amount of people who can afford to buy them is too small to make a profit. This means that even though we’re very close to being technologically capable of building domestic robots, it’s going to take some time before they reach the market.

Still, I think at this point, it’s only a matter of time. Robots are going to become increasingly present in the world around us, and this will likely generate a feedback cycle. The range of robotic capabilities will expand, the cost of components will go down, and as robots enter the global mindshare, people will become increasingly likely to want to apply robotics to various tasks. We’re about to see a robotic revolution. At this point, it seems inevitable.

There’s Too Many Programming Languages!

It’s an opinion that often comes up in development circles. We’re in the middle of a sort of language boom right now, and every time someone comes along and announces a new language, other people suggest that we should all stop creating new programming languages, because there’s already too many out there. Common complaints include the amount of effort needed to constantly learn new languages, the fragmentation caused by the constant introduction of new languages and the lack of innovation in these.

If you feel that new languages require too much effort to learn, the first thing I have to say is: tough luck. Programming is very much about constantly learning new things and being able to learn on your own. Once you’ve mastered a few programming languages, you should begin to see repeating patterns and find that the skills you’ve already acquired are very transferable. For instance, I’ve never written a single line of Go, but I’ve used C, C++ and D. I’m sure I could get started writing Go code within a few hours, and become reasonably proficient within a week. Go doesn’t scare me. Another important thing to realize is that not knowing a programming language isn’t necessarily a flaw you have to correct. Case in point: there are still many job postings out there for COBOL programmers. It’s perfectly acceptable to specialize in one or a few languages of your choosing.

As for the issue of fragmentation, I think there’s truth to it, but it’s not as bad as people imagine. There are only a few languages out there which are truly mainstream. These have existed for years, and if you know just 3 out of the top 10, you’ll be well equipped to realize personal projects or land yourself a development job somewhere. The new programming languages that come out, for the most part, remain relatively fringe and are used mostly by hobbyists. Most of them will likely die out. I’ve written my PhD project in D, and I’ve found that surprisingly few programmers have ever written code in that language. In fact, none of the programmers I’ve met in person outside of DConf had ever used D.

The biggest kind of fragmentation problem, in my opinion, is self-inflicted. I’ve spoken to people at various companies who told me that their architecture was made of some mishmash of parts written in five or six different languages. That seems like an issue to me, if only because more languages means more dependencies, more breakage, more interoperability problems and more code maintenance issues. That’s not the fault of all these languages for existing though, it’s simply bad planning. The administration there let some capricious programmers get their wish and include new code written in their pet language in the system, with little regard for the added complexity this language soup would introduce.

There’s some argument to be made that many of the new languages coming out lack originality. At the moment, most of them are statically typed and compiled ahead of time and most of them have similar syntax. It’s true that there isn’t a lot of innovation overall, but I don’t think that’s a reason to stop creating new programming languages. The design space is huge, infinite in fact, and we’re only beginning to explore it, in my opinion. Consider that even to this day, all the commonly used languages are based on the editing of text files. Also remember that even the languages which don’t succeed on a large scale, such as LISP, can have a tremendous impact on other languages down the road. Imagine a world where there were only two programming languages in existence: COBOL and Fortran. Is that a world you would like to live in? I would argue that there is a need for new languages to solve new problems.

Follow

Get every new post delivered to your Inbox.

Join 4,346 other followers