The Development of Emblem

(Consider this as a retroactively written blog with no dates. The reason for the lack of dates is that this is an idealized account. Emblem was developed much more opportunistically than is described here, with frequent refactoring and bug fixes.)

Emblem has been developed using incremental development. (If you're not using this, and doing all your design up front, you're doing excremental development.) From a very early stage, the system was usable. Development was started during my last job, and continued through my unemployment. Up to the release, the pace of development was 45 lines of debugged code per day, which is I believe at least four times the industry average. (I've seen figures as low as 3 lines but most sources claim it's around 10 lines.) Beware of anyone who claims to write over 50 lines per day, as that is likely to involve extensive cut-and-paste and would be best done by writing macros or otherwise generating the code on the fly.

The only part which has been formally designed is the 3d graphics. The reason for this is that 3d graphics is heavily mathematical, and involves a lot of vector geometry and trig.

The result of the approach has been code which is highly structured and modular, and which, because it has already been modified so often, is easy to modify whenever this becomes necessary.

The first thing that had to be written was the virtual machine. This is in C, and consists of a loop with a switch statement inside, one case for each VM instruction. Initially, only those instructions needed by the interpreter were written; the rest could be added later.

On top the VM is the interpreter. This was written entirely in C macros (#define statements). A sample (and this, believe it or not, is ANSI C):

DefineF("let")
  Local1 Car
  Local2 /* initialize new env */
  Prog(1)
  Ret

  Params(2)
  Until Local1 Null Do
    Local1 Caar /* var */
    Local1 Cadar Free12 Call("eval") /* val in old env */
    Local2 /* env */
    ACons
    SetLocal2 Pop /* update new env */
    PopLocal1
  Od
  Free11 Cdr Local2 Call("progn") /* use new env */
  Ret
Termin

read was written entirely as read macros, so it's accurate to say that Emblem has no built-in syntax. print in the interpreter is temporary, and is just a call to patom (which is aware of lists).

The next thing to be written was the compiler, which compiles to byte-code. During system build, the compiler is read in (running initially on the interpreter), and then compiles itself. To avoid repeated sytstem builds, I extended the virtual machine so that it was possible to save the world (or image). In order to debug the compiled code, I needed to inspect it, so a debugger (imaginatively called ddt) was added to the VM.

Next, utility code was written to do things like mapping functions and array processing, and to rewrite print .

Then multi-threading was added, and I/O was made non-blocking. After blocking, or executing 10000 VM instructions, the next thread was executed. The first use for this was to implement ledit , which like its ITS namesake allows you to read code directly from an Emacs buffer.

The next thing I did was build a real application, a web server, on top of Emblem. Because of experiences writing CGI scripts in Emblem, I decided to completely rewrite Emblem's I/O, replacing unidirectional files with bidirectional channels. The web werver ran in its own thread.

The next layer to be written was the Emblem Object System. EOS is basically a cut-down CLOS, and supports multiple inheritance, multiple dispatch, before and after methods, and class allocated variables. EOS was a nightmare to write because it contains so much code which writes code before calling it if it finds it isn't there. Multiple inheritance makes things much more complicated because slots can't always be in the same place in instances of subclasses. To get the required performance, I had to modify the virtual machine to be slightly aware of objects and also to modify the compiler.

The object code was needed for Mistletoe, Emblem's GUI. I don't believe in just drawing shapes. For most applications, you want the shapes to be objects, with behaviours. Mistletoe required a major extension of the VM to support Xlib calls, including an event handler to process mouse gestures and keyboard events. The event handler ran in its own thread. Next some widgets were defined, though the emphasis in Mistletoe is on graphics and not on forms interfaces, which might as well be delivered in a browser.

Next, I wrote some hacks, and ported Tetris from Maclisp to Emblem. Then I wrote a wiki to run on Emblem's web server, which I am using to write this. Around this time I also wrote the object browser, so that structured objects in Emblem can be inspected as hypertext using a web browser. Next, on a whim, I decided that 3d graphics would be nice, so I added that. Initially the maths was worked out, then implemented in Emblem, then the time bottlenecks were migrated to the virtual machine.

Then, I prepared Emblem for release, by adding documentation and fixing or documenting known bugs. The release was made late in February.

I should add a bit about testing. This was rather informal, but functions were tested as soon as they were written. I also wrote and ran test code, but many bugs were found in the course of Emblem's day-to-day use rather than running code designed specifically to break things. Because of the system's layered structure, code in lower layers is tested whenever it is called from the layer above.

I sent copies of Emblem to around a dozen people, but haven't heard anything back. I assume they're not using it much. This doesn't bother me and I don't take it personally. There's plenty of good software on my machine that I never use (Haskell, Prolog, Python, etc.) I'd rather continue to improve Emblem, make the occasional release, and gradually build up a user base, than hype it.

Since the release, I have continued to improve it, but the pace of development has slowed. This is largely a morale problem: a realization that I might never get a job developing software again, not through any fault of my own, but because the industry s largely run by people who'd rather fail in the approved way, who prefer to require their staff to follow processes than have the think for themselves, who think that developing software is like building buildings, and who run scared from someone who "might find better ways of doing things".

This page was linked to from

• index
• Emblem

and was last updated on Sep 20 at 01:10.

© Copyright Donald Fisk 2003