Sub Ubi

I’ve discovered that programming while helping students and working with colleagues (both in the room and across the ocean) is actually distracting enough to prevent you from writing good code.

By coming in around 6AM this morning, I got some good quiet time that I was able to use productively, and in a short amount of time wrapped my head around YUV 4:22 (UYVY), and got some drawing routines ported into our Surveyor package. I have some questions for Carl and Jon about this, but for the moment, I’m glad my explorations are (mostly) over.

Last night, I got as far as wondering why I was able to make my images green:

This morning, I quickly isolated the luma component (Y) of the image:

Then, it was on to drawing. I think he sub-sampling for JPEG conversion is confusing me, but we can reliably draw shapes over the image:

Now, when students start doing image processing, they can circle or box in regions they are interested in, and see if their algorithms are returning values that make sense. (They can also print debugging information back to the PC, which is useful as well.) Between these two modes of interaction, I’d say we have a reasonably rich platform/API developing upon which students can develop parallel-safe robotics applications on the SRV-1 using occam-pi.

When my younger brother was… younger, he once stuck a pussy willow bud up his nose. I mean, he didn’t just sorta stick the thing in his nose… he crammed it up deep somewhere in his nasal cavity. I’m surprised he didn’t loose the finger to the finger-biting monster that Shel Silverstein claims lives up there.

Inside everybody’s nose
There lives a sharp-toothed snail.
So if you stick your finger in,
He may bite off your nail.
Stick it farther up inside,
And he may bite your ring off.
Stick it all the way, and he
May bite the whole darn thing off.
(Shel Silverstein, Where the Sidewalk Ends)

Alas, no such biting happened. I do remember him screaming like the end was coming, however, when four adults held him down (one to each arm, one to his legs, and one to his head) while someone gingerly reached up and extracted the pussy willow with a pair of tweezers.

I figure he was around three years old. I would have been seven.

Good times, good times…

Of Pussy willows and Subversion

You see, this comes to mind because my students are using Subversion repositories for all of their coursework. They check things in, I check them out, and comment on their work. I then do a checkin, and they can read my comments right in their code. There’s no complex web-based system, no complex emailing of documents back-and-forth. It works reasonably well, I think.

Now, what should go in a repository? This is a reasonably tricky question.

1. Code. When you’re working on programs, the source code goes in the repository.
2. Documentation. Documentation (preferably written in plain text or LaTeX) should be in the repository. If you’re keen on using closed, proprietary tools, then Word Documents might end up in the repository.
3. Required blobs. A “blob” is a binary document. JPEG images are binary. PDFs are binary in nature. It is a “required” blob if something else needs it to be generated (like a LaTeX document that has an image in it; the PNG or JPEG should be included in the repos).

That’s about it. That still isn’t very specific, so I’ll itemize one or two things that might not go in the repository, and why.

1. PDFs. Generally, PDFs do not belong in the repository. That is, if you found an article that both you and your colleague are collaborating on, the repository is not where the PDF belongs. Why? Because the PDF is never going to change. You should just use a networked drive for this kind of “static” or “unchanging” data.
2. Executables. Programs that you need as part of the process of building or manipulating your software should not, generally speaking, be part of the repository. If you have an EXE that, say, processes one of your files… it should also be in a shared drive. While it might be necessary, it (like the PDFs) will never change. Therefore, you should put it on a shared drive, and let people copy it from there.
3. (By-)Products of a build process. Many programming environments generate many temporary files while building a piece of software. These temporary files that are generated in-between your source and executing on hardware should not be part of the repository. Only the source documents (the C files, and any project configuration files that are part of the IDE) should be in the repos. Temporary log files generated by the compiler should not be in the repos.

In short, only the files that you edit go in the repository. Any files that a program generates should not be in the repository.

Why?

Because your repository grows every time a file changes. If you store big, binary files (like executables produced by a compiler) in your repository, then every time you compile your program, the repos thinks a critical file has changed. In truth, we only want the source code to be tracked… not the output from the compiler. We especially don’t want the log files from the compiler checked in. We simply don’t care how those change from one commit to another, and our colleagues generally don’t care, either.

Now, like all things, there are times when these rules might be violated. For example, I asked my students to put a PDF of their report in the repository. This is because I didn’t want to deal with Microsoft Word documents. I’ll mark things directly into a copy of the PDFs. That is preferable, given the workflow that we’re dealing with, and the fact that some students are using Word, and some are using LaTeX. (I should just require TeX.) So, that “rule” about “no PDFs” was violated for a very specific reason. Likewise, I put PDFs in the class repository so they can easily read and print the documentation on whatever machine they are sitting at… not because it is the “right” thing to do.

So, what goes in your repository? Probably not pussy willow buds.

Some time ago (but only a few posts ago), I ranted about the state of embedded systems development under Linux. If we break the rant down, it basically read like this:

How come I don’t have carrier-grade compilers for every single chip under the sun for free? This is ridiculous!

The truth is, these tools have a cost. GCC is a great resource, but the number of platforms that can be realistically supported through gratis development is actually, quite likely, small. One way that it could be done is if every hardware manufacturer employed one or more GCC developers to maintain back-ends for their CPUs. Perhaps some manufacturers do just that.

However, they’d also have to maintain drivers for the programmers that connect to their chips, and make sure that you could debug the hardware from GDB on Windows, Mac, and Linux—which is a non-trivial amount of work. So, now you’re maintaining a toolchain that goes from the compiler, to the linker, to the driver for the programmer… and there’s probably a standard library to maintain as well.

What I’m starting to see (from being in an engineering school) is that people don’t always have the mentality that they can build things themselves… when it comes to software. Because I wrote a linker for a weird little instruction set, and helped write a bytecode interpreter for that instruction set, II have a different perspective regarding the authoring of software than, say, your typical mechanical engineer. I have no real way to compare, but I suspect I sit down and think about the design and implementation of compilers or distributed systems in much the same way they think about… drivetrains and … moving… things. Or something.

I’ve been wrestling with getting the Transterpreter compiled for a particular MSP430 part. You see, there are a number of parts in the MSP430 family that GCC supports very well—and we have compiled and executed occam-pi programs on those CPUs just fine. But the development board I purchased for $120 has a CPU that is… less-well supported. As in, I need to build a patched version of binutils to get things working. I spent the better part of six hours last weekend debugging what was going wrong and attempting to get things running, and finally realized that my time was worth more than the cost of a tool.

Rowley Associates produce the Crossworks line of compilers for a number of embedded processors, including the MSP430 family of devices. Lets look at their prices:

Right now, I’m too busy to chase after broken builds of binutils. Six hours of my time, valued at $25/hour, is the cost of a compiler that handles the CPU I’m trying to build for, as well as many others. And not only that, but the IDE and tools will work on Windows and Linux… and soon my Mac.

In other words, I get the IDE on Windows, Mac, and Linux for $149. My problems of getting my software to compile on the MSP430FG4619 simply go away. Does this help the next person who comes along and wants to run the Transterpreter on this chip? No. However, they’ll have the same options I currently have:

1. Get the GCC-based tools to work
2. Buy the compiler

If I had infinite time, I would get the GCC toolchain working. But I don’t. I just want the devboard to work, so I can test and demo our VM on a small platform. And because other projects I’m involved in leverage the MSP430, the compiler won’t hurt there, either.

So, as soon as I get another hour or two free, I’m buying the Crossworks tools. I’m no longer a grad student with infinite time—and this is a battle I can afford to win with cash.

I thought it might be worth mentioning (in light of my rants) that it is possible to use the Softbaugh USBP in a Windows XP VM running under VMWare Fusion—that is, on a MacBook.

I was running into problems with VMWare Fusion 1.0beta4. However, I missed the recent upgrade to v1.0 “Gold Master”, or (if you prefer), the final release of version one. In upgrading, Windows XP (running in the VM) correctly found the USBP, and as a result, I can now get on with programming MSP430-based devices on my MacBook. (I think; I have’t actually tried loading code to a device yet, but Windows sees the programming dongle, and I think that’s the important step. I expect things to work at this point.)

The picture above is the Softbaugh PrgUSBP application running in VMWare Fusion’s “Unity” mode. In this mode, the Windows app plays along with OSX as if it were a full-fledged Mac application. This way, I can Apple-TAB back-and-forth between my editor and the programmer, and not have to keep flipping back to the full VM.

As it stands, I think it is sad that embedded programming with chips like the MSP430, ARM9, and other interesting devices is so damn expensive. The programmer was $150, my dev board was $115, VMWare Fusion is $80, and XP… I can’t remember. Granted, many of these things were (in my case) paid by grants, but for a hobbyist, $300+ is rather pricey.

I’d love it if we had a way of programming these chips cheaply ($50 or less), and it worked on every platform. Perhaps, someday, I’ll make that a mission. It would have to be a mission of love, because my guess is that there isn’t enough money in these tools to make it up in volume.

Via BoingBoing, from Wired:

The bill, known as the Protect America Act, removes the prohibition on warrantless spying on Americans abroad and gives the government wide powers to order communication service providers such as cell phone companies and ISPs to make their networks available to government eavesdroppers. (…)

E-mail is plain-text. Currently, I even outsource my email to Google, a company that specializes in search. While I’m not (hugely) paranoid, I don’t believe the government has the right to read my mail. Or listen to my phone conversations. Not without due process, anyway.

I have a growing list of possible projects that students might be interested in working on next year. An easy-to-use, end-to-end secure, open platform for messaging is one of them. PGP with email doesn’t really cut it, sadly; I need a solution that my mother can use without realizing she is communicating with me using military-grade encryption.) I think a good plan with a good group of students should easily get in the door for a Y Combinator summer internship or similar.

I continue to be frustrated by the state of embedded systems programming.

If I want to program an ARM9, I need a special programmer. If I want to program an MSP430, it will require a special programmer. The list goes on and on, for each new chip that I think I’d like to develop code for; the cost is prohibitive, and in each case, it is likely that the tools:

1. Only work under Windows, or
2. Work under Linux if you have a parallel port.

I don’t know about you, but my MacBook doesn’t have a parallel port. And Windows running under VMWare doesn’t always recognize these little programming dongles. And really, I don’t want to use Windows or Linux at all. I want to use my Mac.

I’m thinking that I need a sub-$300 device (sub $100 would be ideal) that can program any small device, whether it is the new Blackfin processor or a crappy little PIC. This implies a software solution, with some mix-n-match cabling coming out of the box.

I’m going to start thinking about this a bit more, because it is a genuine problem in the embedded space. It represents the worst kind of vendor lock-in, and makes my life as an educator and hobbyist a complete nightmare. I need a solution that:

1. Works with any embedded device (via software update, if necessary)
2. Works with any computing platform (Mac, Linux, Windows)
3. Is cheap and portable

In the end, the solution is probably called a “LEGO Mindstorms NXT”, and I should just give up on embedded platforms. But I’m not prepared for that yet.

Along with sketching out an explicit group development process, we discussed several other aspects of our project’s organization. As these were relatively straight-forward things, I’ve grouped them all here.

Wrappers

A ‘wrapper’ is what makes it possible to run the Transterpreter on a new platform. That is, cross-compiling the interpreter for a new architecture is boring and easy. Connecting up the language to the hardware (or underlying operating system) is the interesting bit, and the wrapper is where we do that work.

Up until now, we have typically done wrapper development in branches. However, this is difficult, as it means each wrapper developer must constantly be merging the trunk back into their branch. This is silly, as most of our wrappers we want to remain current, and take advantage of any improvements in the trunk.

So, moving forward, we’re going to try and keep wrapper development in the trunk. Because wrappers are mutually exclusive, we don’t have to worry about one adversely influencing the other, and it will make it easier for us to make sure that changes that help one branch doesn’t hurt another (eg. using buildbot and cross compilers).

This requires us to be a bit more vigilant about…

Tagging

Up until now, we’ve been rather lazy about tagging. As a group, we need to start tagging more often. This might be by deciding on features that should go into milestones, or perhaps through an ad-hoc democratic process, where someone suggests that we tag, and the group agrees (or disagrees) that it is a good time to do so.

This is an area that requires more discussion, but it is clear that the project could do with more versioning and tagging, providing snapshots of known good/stable points.

Libraries

Adam has been doing a lot of work at making libraries for occam-pi cross-runtime buildable. By this, I mean that we have a situation where we have two runtimes (CCSP and the Transterpreter), each which are capable of different things. For example, the interpreter will always be smaller, and therefore more portable to more (embedded) platforms. Unfortunately, if I had developed a C library with occam-pi bindings, it was virtually impossible to use it from both the Transterpreter and the CCSP (KRoC) runtime.

Adam’s work has brought us into a world where we can easily build such libraries for either runtime from the same source with minimal effort. This is excellent, and we’ll be converting all of our libraries to use this new build system over time. Which, brings me to the last bit…

Build systems

Some time ago, I explored the use of Scons as a build system. It represents a spike, and can build the Transterpreter for Linux, Mac (PPC and Intel), the MSP430, and Windows.

However, it’s a bit ugly. It represents my first time working with Scons, and it could be better. Also, we discovered that mingw provides an environment that lets us easily build everything using our Autotools-based environment. So, whereas we thought the Scons build would improve things across the board, it’s not clear that it’s a 100% necessary improvement.

Either way, we’re going to bring Scons into the trunk, and play with it. Because it is completely orthogonal to the Autoconf/Automake build system, it can’t hurt. However, we’re going to be looking at this closely over the next few months, as one of them has to go, and we definitely want our build system to be cleaner/better organized/simplified greatly. If that means using Scons, great. If it means revamping our existing system, great.

Up next…

Those are reasonably straight-forward changes coming down the pipe; as a side effect of these things, we’ll likely do some branch cleanup and repos reorganization. As the time comes for these things, they’ll be discussed on the development list.

The last thing I’ll talk about is a project mission statement and project goals. That’ll come later this week.

So, the first of our diaspora updates involves making our development process more explicit. While our group has, largely, worked this way already, it hasn’t been explicit. This had a lot to do with the fact that the entire team was housed in the same suite of offices at the University of Kent. Now, things have changed.

The first document I want to introduce you to is our development process. We’ve attempted to keep this as short and simple as possible. We want just enough process to keep us moving forward harmoniously, but not so much that developing on the Transterpreter is notfun. Given that we hope that students interested in concurrency, embedded systems, and other cool stuff like that will want to get involved, we also wanted the process to be heavy enough to provide some structure and be pedagogically valuable, but light enough so as not to get in the way of (say) an enjoyable final-year project.

So take a look, and join the development mailing list if you want to discuss it.

Arguably, with respect to England, I’m now the Western Front. That is, last week, I moved from the UK back to my native land of the USA. Kinda sad, really; it’s hard to find cask-conditioned, top-fermented ales in the USA.

On Friday, July 20th, we went ahead and had a group meeting discussing many aspects of the project. Over the next few days, I’ll describe the various parts of that discussion to inform the development list as well as provide a partial record of what we came up with.

To me, the most important part of the discussion was our agreeing some working patterns that we can document and adhere to as a distributed group. As a result, this should make it much easier for new developers to get involved and feel “at home” in the project. This has been missing so far, largely because we were all in adjacent offices for the first few years of the project. That, now, has changed.

I’ll try and do one of these every morning. Admittedly, I’m on vacation right now, but I’m also jetlagged. Or, something… I’m still going to bed around 10PM and waking up around 6AM, which means that I have a few quiet hours in the morning to think and write things like this. We’ll see. Either way, keep an eye on this space, or watch the mailing lists, as I’ll announce the posts in this series to the list, as we (as a group) think they matter.

Somewhere in the past, I took the Joel Test. I like this simple measure of a software project’s health, because it hits on the really important things with respect to quality and maintainability.

The Joel Test is:

1. Do you use source control? Yes
2. Can you make a build in one step? Yes
3. Do you make daily builds? Yes
4. Do you have a bug database? Yes
5. Do you fix bugs before writing new code? No
6. Do you have an up-to-date schedule? Yes
7. Do you have a spec? No
8. Do programmers have quiet working conditions? No
9. Do you use the best tools money can buy? Yes(ish)
10. Do you have testers? No
11. Do new candidates write code during their interview? N/A
12. Do you do hallway usability testing? Yes

This is better than a few years ago. On ever commit, buildbot runs and makes sure we didn’t break anything (on multiple architectures). We don’t fix all the bugs before moving on—but some of those tickets in the database are things like “Write a new linker.” I’ve given us a “no”, but for actual, build-breaking, test-breaking bugs, things get fixed before we move on. We don’t work to a fixed schedule—instead, we tend to work towards features. So, I’ll claim we do have an up-to-date schedule. We don’t have a spec anymore, but we work to the best documentation available whenever possible; this may change in the future. We don’t have the best tools that money can buy, but we do the best with the freely available tools we can cobble together. And, we tend to chat about new features before implementing, which is as close to hallway usability testing as we can get when working on a virtual machine.

So, I’d give us a 7 out of 11; we don’t interview, so I’ll drop it from the list. I think that’s not bad… quite good, in fact, for a small, unfunded open-source project.