Justin Dugger

PuppetConf 2012

2012-09-30T00:00:00Z

Recovered from the post-con crash a while ago, so it's time to write up some thoughts. Last week I attended PuppetConf with my coworkers at the OSL. The OSL attended PuppetConf primarily as a pre-deployment information gathering exercise. We want to avoid common pitfalls, and be able to plan for things coming down the pipeline. Puppet 3.0 was targetted to be released on Friday and clearly that slipped.

The venue itself was nice, but space partitioned poorly. The two main tracks had surplus space, but the three side tracks nearly always had people turned away for space concerns. Supposedly, the recordings will be available shortly, so it may not be the Worst Thing In The World, but only time will tell.

Content wise, one recurring theme is to start small and simple, and not worry about scale or sharing until they become an issue. Designing a deployment for thousands of nodes when you have perhaps a dozen gives new life to the term "architecture astronaut," and there's a certain amount of benefit to procrastinating on system design while the tools and ecosystem mature. Basically, build one to throw away.

Another problem we've been worrying about at the OSL is updating 3rd party config modules in their various forms. The hope is that by explicitly annotating in your system where things came from, you can automate pulling in updates from original sources. Pretty much the universal recommendation here is a condemnation: avoid git submodules. Submodules sounds like the right strategy, but it's for a different use case. In our experience, it dramatically complicates the workflow. At least one person mentioned librarian-puppet, which as far as I can tell is isn't much different than mr with some syntactic sugar for PuppetForge. This is great, because mr was basically the strategy I was recommending prior to PuppetConf.

The Better Living Through Statistics talk was less advanced than I'd hoped. Anyone who's spent maybe 5 minutes tuning nagios check_disks realizes how inadequate it is, and that the basic nagios framework is to blame. What you really want is an alert when the time to disk outage approaches time to free up more disk, and no static threshold can capture that. While Jamie did provide a vision for the future, I was really hoping for some new statistical insight on the problem. It appears it's up to me to create and provide said insight. Perhaps in another post.

R Tyler Croy gave a useful talk on behavior/test driven infrastructure. I'd looked into Cucumber before, but RSpec was only a word to me before this talk. It's certainly something I'll need to take some time to integrate into the workflow and introduce to students. One concern I had (that someone else aired) was that in the demo, the puppet code and the code to test it was basically identical, such that software could easily translate from code to test and back. Croy insisted this was not the case in more complicated Puppet modules, but I'm reserving judgement until I see said modules.

Overall, I'd definately recommend the conference to people preparing to deploy puppet. There's plenty more sessions I didn't cover in here that are worth your time. You'd probably get the most out of it by starting a trial implementation first, instead of procrastinating until Wednesday night to read the basics like I did. Beyond simply watching lectures, it's useful to get away from the office and sit down to learn about this stuff. Plus, it's useful to build your professional network of people you can direct questions to later.

Open Source Bridge Wrapup

2012-07-01T00:00:00Z

Friday marked the end of Open Source Bridge. Just about the best introduction to Portland culture as you can find. Vegan lunches, Voodoo Donut catering, lunch truck friday, and rock and roll pipe organists in the Unitarian's sanctuary.

The keynotes were pretty cool. I'd seen Fenwick's presentation from LCA, and was surprised at how much had changed, hopefully since some of his keystone evidence turned out to be bogus; turns out there's strong evidence that the only "priming" effect was in grad students running the study. I'm still not quite clear on what JScott wants people to run vbox for, but he did have a really good idea about bringing your own recording equipment that I wish I had taken to heart.

Probably the most useful talk I attended was Laura Thompson's presentation on Mozilla's Crash Reporting service, powered by Socorro. A few of the projects the OSL hosts are desktop apps and collecting crash data might be a good engineering tool win for them. A lot of embedded hardware talks that would have been interesting, but not directly relevant to the needs of the OSL. Hopefully they'll be up as recordings soon.

The OSL was also well represented as well in the speaker's ranks: we ran five sessions during the main conference, and two during the Friday unconference. I think next year it would be a good idea to encourage our students to participate as volunteers; getting them facetime with speakers and the community at large can only do us a world of good. I gave a first run of a talk on using GNUCash for personal finance; the turnout was pretty good, given how many people were still at the food carts. I should have recorded it to self-critique and improve.

The "after party" on Thursday was nice. Lance won the 2012 Outsanding Open Source Citizen award, which is great, because he deserves recongition for handling the turmoil at the OSL over the past year. But now I've got to figure out my plan meet or beat that for next year. No small task.

Next up is catching up back at the Lab, and then OSCON!

Solving the Sunday Puzzle with coreutils & grep

2011-08-14T00:00:00Z

Not too long ago, a puzzler admitted to solving the NPR Sunday Puzzle with a computer. Since I hate word puzzles quite a bit, I'd taken similar steps in the past. For example, this recent challenge:

Fill in the 4x4 square crossword with common uncapitalized words, using no letter more than once:

N	A	G	S
E
W
T

My secret weapon? GNU coreutils. Regex are a great tool, but I rarely have to use some of the more obscure features, which hurts on the occasions where they're called for. So the NPR puzzle can be a good way to practice and learn!

Edit: Commenter hggdh points out that the heavy worker here is grep, which is not part of coreutils. If your OS vendor doesn't provide grep, GNU grep sounds like a suitable replacement.

I'm using the American English dictionary provided by Ubuntu /usr/share/dict/words. The format of this file is one word per line. Every form of a word, including contractions and possessives, gets its own line. We use | (pipe) to chain the output of one command as the input of the next. Cat simply outputs a file, and wc -l counts the lines in it.

laptop:~$ cat /usr/share/dict/words | wc -l

98569
I assume no apostrophes are in the puzzle. Grep reads input and outputs only those lines that match a regular expression (regex). Using the -v option to grep changes it to output only lines that don't match our pattern.

laptop:~$ cat /usr/share/dict/words | grep -v "'" | wc -l

74059
That's a lot of words to fuddle around with, so lets winnow this down. Firstly, we only care about 4 letter words. We can use grep to give us only these words, using the regular expression "^....$". Caret (^) represents the start of a line, and $ represents the end of one. Each period is a single free choice character for grep, matching exactly one character in the input.

laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^....$" | wc -l

3174
Having cut the search space by 96 percent, we now turn to the clues for... clues. Fortunately, nags and newts define which letters every word can start with. Grep treats symbols within [] as alternatives, meaning any any one symbol within can match the input. Below alters the regex from 3 to only match words starting with a, g, s, e, w or t.

laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^[agsewt]...$" | wc -l

777
Rules say no two letters repeat in the puzzle, so we'll exclude all words with the letters from nags and newts anywhere other than the first letter. As an alternative to -v, we can use carets inside brackets to indicate "not".

laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^[agsewt][^nagsewt][^nagsewt][^nagsewt]$" | wc -l

131
Next, we can rule out words with repeated letters, like solo and wool. To do this quickly, we'll need to use backreferences. Backreferences can be slow, but since our dataset is so tiny, it will be fine to add it to the end of the pipeline.

cat /usr/share/dict/words | grep -v "'" | grep "^[agsewt][^nagsewt][^nagsewt][^nagsewt]$" | grep -vE "([a-z]).*(\1)" | wc -l

106
Starting to get close! From here on out, this plays a lot like sudoku. Our goal is now to start constructing regex for each word. We replace the leading alternative for a specific letter. To start off, we've only got 7 options for 2 across:

laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^e[^nagsewt][^nagsewt][^nagsewt]$" | grep -vE "([a-z]).*(\1)"

echo

ecru

emir

epic

euro

evil

expo

We now write a different regex without negations to get the same list.

`laptop:~$ cat /usr/share/dict/words | grep "^e[cmpuvx][hipr][cloru]$" | grep -vE "([a-z]).*(\1)" | wc -l`

`7`

Now we build a similar regex for 2 down. Adding in what we know about it's intersection with 2 across (cmpuvx) is the sudoku like step:

`laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^a[cmpuvx][^nagsewt][^nagsewt]$" | grep -vE "([a-z]).*(\1)"`

achy

acid

amid

amok

avid

We rewrite this one as

laptop:~$ cat /usr/share/dict/words | grep -v "'" | grep "^a[cmv][hio][dky]$" | grep -vE "([a-z]).*(\1)" | wc -l

5

Applying the same logic to 3 down yields "^g[ir][lriu][bdlmp]$", and 4 down yields "^s[lu][cilmoru][bdfhkmopr]$".

The last positions in each down regex constructs a new regex for 4 across:

cat /usr/share/dict/words | grep -v "'" | grep "^t[dky][bdlmp][bdfhkmopr]$" | grep -vE "([a-z]).*(\1)"

typo

A unique solution to 4 across!

Revisiting 2 down with this new fact also yields a unique answer. I leave it from here as an exercise to the reader to solve the entire puzzle.

Reducing Interruptions

2011-05-13T00:00:00Z

Been pondering lately how I'd design a desktop environment to improve productivity. Many of the ideas are non-trivial changes, but there's a few papercut sized tweaks I thought I'd document and share. These all revolve around the notion of flow, a state of concentration. The aim is to reduce notification popups and, as a side effect, waste less time futzing with the window manager by batch processing them.

Set Liferea update frequency to once a day. Many feeds specify a refresh of two hours. For news junkies, this might be fine. But none of the feeds I subscribe to are time sensitive. So I set the "Default Feed Refresh Interval" to 1 day. You can override the default for 'important' feeds, if need be.
Set Evolution to check less frequently. This one can be controversial. By default Evolution checks and notifies every 10 minutes. Since the goal is multiple hours of uninterrupted concentration, that's far too many chances of interruption. The challenge is responding to urgent mail without letting the less urgent stuff interrupt. My approach is to turn off general notifications and set alerts for important senders, like the boss and the support ticket system.
Set Gwibber to check less frequently. If someone needs your immediate attention for 140 characters or less, they probably know how to reach you by SMS. You can tweak the notifications to only include mentions and replies, which dramatically cuts down on the traffic. Unfortunately, Gwibber limits the upper bound between refeshes to an arbitrary 240 minutes, so if you wanted a once a day reminder to check everything, you're out of luck there.
Filter noisy RSS feeds. Not all feeds are pure information. For example, the LinkedIn feed throws in tweets from your connections. I'm a fan of feed processing tools, whether it be Yahoo! Pipes or client side XSLT. If you're extra lucky, the feed has a digest; Lifehacker was unique in offering a highlights tag feed, which reduces a weeks worth of posts to a single digest post on Friday. Gawker seems to have killed these tag feeds and posts, after the rousing success of driving users away with their recent redesign. Of course, you can always unsubscribe if you find the signal just isn't worth the noise...
Turn off banshee track notifications. Music is a great way to mask ambient noise in your home or office. Default Banshee however, pops up a notification on every track change. As I'm reading a paper or writing code, this pop up distracts the eye away from the monitor I was looking at. There doesn't seem to be a configuration change one can make, but a clever user on AskUbuntu has found a makeshift solution.

Arduino Update

2010-07-09T00:00:00Z

An alert (and anonymous) reader commented on a previous post about Arduino packaging, letting me know that packaging has been accepted into Debian—both the Java IDE and the underlying compiler toolchain. A big thanks to Phil Hands and Scott Howard for their efforts!

This is a very new package, so patches are still rolling in. The DebianImportFreeze has come and gone, but Scott appears to be on top of things, having just filed a sync request a few hours ago to pull in a few more changes. If he keeps that pace up, he'll make MOTU in no time ;)

Weave synchronization

2009-12-06T00:00:00Z

During the transition from Firefox 2.0 to 3.0, Google Browser Sync was lost to the churn of time and API. This plugin encrypted and synchronized your browser history, tabs and cache among browsers. Since it's disappearance, many people seem to recommend Xmarks. While Xmarks is a fairly robust bookmark sync tool, it neglects other aspects of a profile like cookies or history, making it harder to recommend. Conceivably, Ubuntu One could handle Firefox profiles the way it handles Tomboy and Evolution contacts, but for now, the only plausible comprehensive option is Mozilla's own offering, Weave.

Weave is a browser sync engine. The FAQ gives a brief summary of features: Weave Sync currently supports synchronization of bookmarks, browsing history, open tabs, saved passwords, form entries, and selected preferences. Security settings will be added in a future release. Yes, even tabs. If you leave a tab open on your work box you can copy it to another box (the remote tab remains open). Password recall is advertised as useful for mobile phones; pull complex passwords into your phone's Fennec and skip the tedious caps/numlock/funky character disaster. All of these are configurable, so you can prevent disclosure if you fear distributing passwords to multiple boxes, or fear distributing to Mozilla's hosting service.

I tried giving Weave a spin around a year ago, but it wasn't quite open to public use. There were hints to use WebDAV enabled servers on your own, but the only server code published was more a code testing server (since retired I think), and the plugin outright refused to work without a valid Mozilla labs account, at a time where such accounts were invite only. In fact, I think I couldn't even download it without someone with an existing account giving me the URL; not exactly a participatory open source experience to say the least.

But that was then and this is now; they've made regular progress on Weave and are ramping up for a 1.0 release. Since I've ordered a n900 smartphone, I decided to spend the Thanksgiving holiday setting up a personal Weave server to populate its browser with data quickly. I did more research, consulted the web and documentation, even queried a few IRC channels for user's opinions. It seems they've opened registration to the general public, and there's a new lightweight server published that relies on PHP and SQLite. I've set it up on my personal server to see whether things have improved; when my phone comes in I'll also test integration with Fennec.

It's fairly young, but inherits many design decisions from prior Weave servers, so it should catch up quick and the smaller scope is an advantage. One minor problem is that its only published via tarball and uses (part of) a blog for a website, which does not instill confidence. But with only like five files and no build, it should be trivial to package or even fork and maintain if necessary. And I have it on good authority better publishing is under consideration.

Of course, it would be a bit silly to package a server without the client. If Mozilla announces it formally soon enough, 1.0 plugin and a minimal server could plausibly be in Ubuntu 10.04. So far, the server seems to be working in proper order, but it'll take more and wider testing to have utmost confidence in it.

Server Install Notes

Apache runs as www-data on Ubuntu, so make sure that user has write permissions on the database and read/execute on the PHP code.
Dependencies are fairly obvious: php5-sqlite, php5 and php5-cli (to create your user). It feels a bit goofy to use PHP for CLI scripting, but I guess in some situations it reduces dependencies.
If you self-sign a certificate, you need to store an exception on every browser used. Or replace it with trusted SSL cert.
After a week of initial use, my database currently sits around 5MB; I'm not sure how fast that will grow over time. Probably not much worse than a local FF profile grows.

A note to Mozilla

There's something like three versions of Weave documentation floating around on your wiki, and Google search likes the oldest one best. Probably because even your FAQ links to the oldest server documentation. I expect you don't want this, and should probably take action to change it. Perhaps if documentation had a "current version" alias, people could link to that instead of a specific version destined to go out of date, and warning boxes could appear when viewing old versions. If you need software online help to reference versioned documentation, perhaps you could, you know, use the built- in wiki version control?

How to REALLY make money with Ubuntu

2009-10-28T00:00:00Z

You're all thinking way too hard. If you want to see how people make money on open source, why not check out eBay?

An alternative interpretation

2009-10-24T00:00:00Z

Gerfried Fuchs (rhonda) is a prolific Debian Developer, who maintains a number of Debian packages, some of which I use daily. She also participates in the Debian/Ubuntu Games Team, which has a mission to unite the two projects when it comes to the common purpose of gaming. To that noble end she's considering joining MOTU, but is reluctant to keysign the Code of Conduct, finding an aside within unsavory. She worries:

Anyway, there is this one part with the that CoC that itches me. It's not that one has to sign it with their GnuPG key, but related to it. Making it a requirement to sign it gives the document a much more official character, actually gives it the feeling and impression of a contract and I expect it is meant to carry that feeling. Though, there is this one part in it that I consider off for such a document:

Nobody knows everything, and nobody is expected to be perfect in the Ubuntu community (except of course the SABDFL).

Given that the acronym SABDFL refers to Mark Shuttleworth it means that one has to expect him to be impeccable—which I am sorry but cannot sign.

As best I can tell, her worry is that it declares leadership is infallible, and that any lighthearted humor undermines the serious tone a contract must bear. But jokes can place a lot of meaning into a few words; many legal briefs use them to great effect. My personal interpretation of the aside in question is that we hold SABDFL and leadership to higher standards than the rest of the community, and that we expect the utmost behavior from SABDFL. This interpretation is supported by the Ubuntu Leadership Code of Conduct. I think this particular document does not and has not received the attention the normal CoC has enjoyed; I encourage people to read it now. The aside, I believe, calls to attention the expectations the community has of its leadership.

It also serves as a reminder that SABDFL has considerable power, financial, social and structurally. Without communicating these expectations, there would be to great power without great responsibility. Being able to take SABDFL to task for mistakes the stabilizes the community when an error is made by its public face, and places community checks on his power.

So Gerfried, I'd encourage you to read the Leadership CoC and the governance documents, and then read the CoC again in their light. Perhaps you will then find this aside reasonable, and perhaps you will find more to object to. Either way, you'll discover more about the attitudes of the Ubuntu community you work with, so I think it's worth your time. And if you change your mind, I'd be glad to cheer you on your way to MOTU.

UPDATE: shortly after writing this, the Ubuntu Community Council chose to revise the document. The statement in question is removed, but I still encourage people to understand the community structure of Ubuntu and the expectations we have of leadership.

Fun and Profit with Liferea Conversion Filters

2009-09-13T00:00:00Z

Liferea is an RSS feed reader, with some hidden powers. I use it to automatically download enclosures from the TED feed, and aggregate unread comic strips into a single scroll page in a custom view folder. But today, I'd like to present conversion filters.

Conversion filters are scripts that function according to UNIX pipes semantics to process feeds before display. Liferea performs the task of retrieving the XML, starts the conversion filter, and passes the XML over stdin. The filter prints a converted XML file back to Liferea over stdout, and which happily displays the new RSS feed. For the graphic minded:

A small repository of conversion filters has been started, mainly driven by a different, command line driven RSS reader, snownews. Most published conversion filters are what you'd call a source in dataflow, rather than a filter. Even more are site specific, making this more of a greasemonkey for RSS feeds.

I'll share an example I concocted today. I like the Freakonomics blog, but lately they've added some unrelated content called Quotes Uncovered. I have no idea how identifying the historical source of quotations relates to microeconomics, but I do have an idea of how to use a conversion filter to remove it. But first, we need to talk for a moment about XPath.

XML is tree structured, like a filesystem. I've seen at least two people take this analogy to it's conclusion, and implement an XMLfs via FUSE. XPath also uses this analogy, for a different purpose, describing paths in documents. Paths in UNIX filesystems can be singular, like /etc/apt/sources.list, or plural, like /etc/apt/sources.list.d/ (technically your shell expands this, but play along). Concepts like ../ and are supported by XPath. Instead of directories though, we're going to use XPath to navigate nodes in XML.

Unfortunately, I'm interested in a specific node, based not on it's name but it's contents. So I need some predicate to narrow this. In UNIX shell, find is commonly used for this purpose, but XPath integrates this concept and diverges from UNIX paths. In the case of my Freakonomics cleaner, I want to delete the item node that has Quotes Uncovered in a child title node:

//*/item/title[contains(text(),'Quotes Uncovered')]/..

Liferea doesn't understand XPath though, so we'll need to use the conversion filter to handle. Conversion filters I've seen thus far end up scripting parsing the XML, processing the structure and printing it. Only the middle part is actually going to be unique to conversion filters. Fortunately, there's a program that adheres to UNIX philosophy and handles this: XMLStarlet (in Universe). XMLstarlet reads XML from stdin, and writes XML from stdout, just as our conversion filters must. On a technical level, it converts the command line to an XSLT and applies it to the input. In this case I just need to tell it to delete the section matching that XPath from the feed. That's accomplished with the ed -d option.

So now I just create a new subscription, and put the command in the box:

The result is a one-liner that puts the Quotes Uncovered series to pasture. It's tempting to UNIX pipes to build more complicated filters, but I'm told Liferea isn't currently coded for chained pipes. So if you get more complicated than the basic 3 step Fetch-Filter-Display pipeline, you'll need to write a short shell script for that, or write some really crazy XPath.

Postscript

The Snownews extension repo is interesting, but doesn't have a way to link feeds with scripts. Earlier I compared conversion filters to Greasemonkey. Well Greasemonkey has a partner extension called Greasefire and a website backend userscripts.org for script discovery. It'd be handy to have conversion filter discovery for RSS!

Adeona + UbuntuOne?

2009-08-23T00:00:00Z

Adeona is an OSS anti-theft program. The theory of operation is described as a privacy-preserving device-tracking system in the research publication:

A device tracking system consists of: client hardware or software logic installed on the device; (sometimes) cryptographic key material stored on the device; (sometimes) cryptographic key material maintained separately by the device owner; and a remote storage facility. The client sends location updates over the Internet to the remote storage. Once a device goes missing, the owner or authorized agent searches the remote storage for location updates pertaining to the device’s current whereabouts.

The paper goes on to describe specific goals for Adeona:

network update anonymity
privacy from the thief
efficiency on par with existing tools

I'll argue that most of this is post-hoc requirements from an early decision to rely on OpenDHT (which turns out to be a not so great idea). That said, it's not bad stuff to have; Joey Hess's recent discovery made headlines, and rightly so, when Palm Pre violated pretty much every principle the Adeona wrote about.

However, it's rarely a wise idea for products to rely on academic research projects. As a research project, it might have been acceptable for Adeona to depend on OpenDHT. But this decision should have been revisited when Adeona launched as a community product. And they've paid a price -- OpenDHT announced it would be closing down on July 1st 2009. I'm not clear on whether the OpenDHT infrastructure is too slow and unreliable, or if the software itself is unreliable, but Adeona's own website currently starts with a disclaimer of non-functionality.

So what I'm wondering out loud is, why not integrate Adeona and UbuntuOne? The purpose of using OpenDHT appears to be to find cheap storage for tracking data. Obviously some privacy would be sacrificed. Canonical would basically be able to track some things, like IPs connected to UbuntuOne. While there are already other reliable places for Canonical to snoop IP addresses (access logs from Ubuntu archives), UbuntuOne takes the extra step of authenticating users. I believe though, that the only thing they'd be able to associate is your username with a set of IPs. I guess it comes down to how many ultra-paranoid users object to the privacy concerns of UbuntuOne itself.

Ideally, I think Adeona would be generalized to allow a number of remote storage interfaces. If I understood the system correctly, data is encrypted before being stored. The privacy paranoid would be able to store location data on their own servers, or encrypted email, etc.

So is there something I'm missing? A competing product maybe? Or perhaps the viewpoint of law enforcement; the academic papers seem to neglect recovery rates and concerns from police. Or perhaps something technical. I'm hardly a crypto expert, and currently security researchers seem to be focusing on a proprietary application.

Why isn't Arduino in Debian / Ubuntu?

2009-08-12T00:00:00Z

After my review of the Sparkle Labs Discover Electronics! kit I thought I'd look for related software in Ubuntu. I did find gResistor (Hat tip: Debianized by tuxmaniac), and some drafting tools. But I also found a lot of stuff that wasn't in Debian or Ubuntu.

For example, Arduino makes an interesting platform for hobbyists, students and apparently artists. It's lineage is confusing, inheriting from both Processing and Wiring. I'm not yet clear whether Wiring is a language, hardware or an IDE, but it in turn is based on Processing and adds some microcontroller-y stuff as I understand it. Processing is a language and IDE, and has been used for amazing things. It's all open source, and reportedly works on Ubuntu. People have built amazing things with the Arduino platform as well.

Strangely, hub of activity hasn't resulted in a .deb package, only a two year idle Request For Package for Processing, and nothing on the subject of Arduino's IDE. Having packaging available for this stuff can do a lot to remove trip ups for budding developers and artists. I'd much rather they spend their time fixing their own bugs rather than trying to remove any bugs in integrating the platform with Ubuntu. Does anyone know why or where these efforts have stalled out? Is it legal issues, build problems, or just too convenient already?

Free Software and Computer Scientists

2009-08-10T00:00:00Z

Gabriella Coleman writes on the relationship of academics and Free Software:

One irony (though not entertained in the chapter) has to do with the status of Free Software in the academy: it is pretty weak among CS-ey types and yet Free Software is often identified as a paragon example of the openness and communitarian elements of how academic science is supposed to work. So.. what is exactly going on?

The post seems to confuse the issue of software generated by CS research, and the software needs of education. Simply put: Moodle developers don't earn tenure.

It's not the case, however, that Free Software "is weak among CS-ey types". Looking back on my CS undergraduate studies, my fellow students and I used a number of open source tools. Some are general purpose tools: Apache, OCaml and gcc, CVS, ANTLR, lexx/yacc to name the ones I remember. We even studied source code from a C stdlib. Our 3d graphics course focused on Linux, and OpenGL when it came time for hardware rendering. Some open source software was necessarily open sourced for instruction: NachOS and Minix come to mind. One professor was writing a textbook on formal language theory and the drafts are GFDL.

There are plenty of FOSS research projects. They're just things nobody's heard of or forgot, because they're academic, boring or complicated. If we treat the various Bell Labs as a private academic institution, they've put out stuff like SPIN, graphviz, and so on. University of Edinburg released festival, and Carnegie Mellon developed flite. BSD came from Berkley, an obvious example. The rest of the stuff I know of is source code static analysis tools that go further and take longer than lint, or Eclipse plugins. If anyone knows of Moodle related research for publication in say ACM or IEEE, I'm unaware of it and would enjoy a reference to the paper.

So CS students use and read open source software, and CS researchers create and improve it. The remaining aspect of the question, as I understand it, is why more student projects aren't started to solve the university's problems. It's a bit naive, I think, to imagine a capstone project can replace incumbent software in a semester or two. You could develop a project over years of classes, but there's a problematic theme among educational source code: projects for instructional improvement need to stay in need of improvement. Hence, Minix spawned multiple, unincorporated VM projects, and eventually lead to Linus Torvalds starting his own OS. It's also a bit rude to charge students tuition for the privilege of replacing software you need, and then demand that rights to that labor. Even GRAs are paid tuition or stipend!

A positive example: Cellwriter. UROP paid the student an Undergraduate Research grant, with exceptional results. But in the longer term, who owns the responsibility for maintaining these programs? Undergraduates tend to graduate and land jobs, rarely to continue their work on a capstone course. Leaving the life-span of a project in their hands works fine for the self-selected, but requiring CS students to work on someone else's problem will only work until grades are assigned. I remember my own capstone project, and while instructive, I don't think anyone in the team wants to see that subject or code ever again.

Directing CS departments at open source's deficiencies is a bit like asking EE professors to wire your campus, or ArchiE professors to design your buildings. These people are employed to solve hard and mostly unsolved problems; campus IT is mostly not that. (At this point, I must confess I am a CS academic-in- training who's transitioned to college IT administration, and may have a bias.) It's a tremendous mistake to take these people and build an open source AutoCAD when they should be working on alternatives to the CAD concept! Of course it would be great if innovations from Computer Science departments were Free Software, and many are.

It's therefore incumbent upon IT administration to handle the mundane tasks; we operate in service of the academy, not the other way around. My alma mater actually builds its own online education site, and employs many CS students, but I see little need for CS research on the subject. It's not that IT lacks funding or programmers, but that they have no desire (ability?) to release that code. OSUOSL is the closest positive example I have; they're an organization that promotes the creation of FOSS, by students and staff. But from what I see, their mission is more outreach than... "inreach".

So I guess my point is, it's the duty of IT departments to solve IT department's needs. Hopefully I've convinced you that Computer Science isn't quite so alien to Free Software, and that projects similar to OSUOSL are a better form of FOSS adaptation than reallocating Computer Science faculty and classes.

Streamlined support tricks

2009-06-12T00:00:00Z

Ubuntu is a big project. Nobody really understands everything about Ubuntu; even authoring a Training video on Ubuntu doesn't guarantee sufficient expertise. One great thing about the Internet is that it can bring people with a common interest or problem together. If you have a problem with your car, there are dozens of outlets you can explore to diagnose and repair it. Today I'll explore some of the various ways we can diagnose and repair Ubuntu, and how to moderate the flow from these firehoses.

Official Channels

Obviously Ubuntu was founded by people with a deep understanding of the Internet, software, and collaboration. Over the past ten releases there have been a number of avenues created for people to ask questions and get answers, with varying degrees of noise and signal.

The most high profile of these is Launchpad's bug tracker, Malone (named after Bugsy Malone?). If you have a problem that only changing a program can fix, this is the place. It's modeled on BugZilla and Debian BTS, but brings a lot of new stuff to the table. One such improvement over simple email based tools is the dupe-finder. Given a description, Malone will suggest a set of bugs similar in description, before asking you for a detailed write up of the problem. This is a great way for software to help people with similar problems find each other. But a big project like Ubuntu gets a lot of bugs; setting yourself as a general bug contact is a very not smart idea. If you want to help out with bugs, pick a package or two and set yourself as a bug contact.

A more recent Launchpad addition is the Answers tool. You can ask a question, and people can ask for more information or suggest an answer. Answers manages the state of questions so you can safely ignored questions with accepted solutions. Answers is intended for things that maybe aren't exactly bugs, but simple questions about how to use a program. Like Malone, it also features a dupe checker to help consolidate activity. It's not clear from the website, but you can also set yourself as a answers contact for individual packages. https://answers.launchpad.net/ubuntu/+source/pkg-name/+answer- contact will point you in the right direction, for a given pkg-name. I'm not aware of any RSS feeds for answers, in whole or in part, unfortunately.

We also run two support channels on Freenode, #ubuntu and #ubuntu+1. #ubuntu+1 is for development versions, so it usually focuses around testing packages and undoing damage from package updates gone wrong. #ubuntu itself is the standard support channel, but it can be very daunting. There's currently 1300 people participating in the channel, and it requires a specific etiquette to scale. Because simultaneous conversations are going on, it's best to address questions to the channel, but address all replies to a specific recipient. IRC clients will pick up when their name is used and highlight the reply above the noise of other support conversations. There's also a set of IRC bots to aid in common questions, facts, and channel logging. One neat extra is that if you mention a bug number or URL, the bots can provide the channel with the bug summary. **If you wish to selectively help people via IRC, join

ubuntu, get familiar with your client and set some hilights on packages

you're an expert on**. If you want to get really sophisticated, there are scripts out there for screen+irssi and libnotify that will generate a desktop popup when an IRC hilight comes in.

The mailing lists serve a number of purposes within Ubuntu, but today I'll focus on the support lists. Unmoderated lists can get pretty prolific; ubuntu-users seems to average about a megabyte of mail a month. This translated to 4000 individual emails for the month of May. That's 130 messages a day! Again, a good email client can help trim that down. Gmail will group messages into coherent threads, but we're still looking at perhaps 20 threads a day. Cutting down the volume further may require getting familiar with mail filters that are built into your client. I believe Gmail can store a list to a folder and then bring in an entire thread to the inbox when a message matches a keyword; but I haven't tried this. At any rate, I understand procmail-like tools are key to narrowing large volume lists to something more viable.

Finally, there is the forum. Web forums have a reputation for being very novice friendly, and very time wasting. Forums are also historically polling oriented; if you want to know what's going on currently, you visit the website and refresh the pages. I think UbuntuForums supports subscriptions, but only at the granularity of boards and threads. That's potentially a lot of traffic, so if you want something less demanding, you'll need to subscribe to an UbuntuForum board and filter it yourself in your mail client, similar to the mailing lists. There's no tracking of question status ("problem solved!"); that feature was deemed too hard on the database and disabled.

It seems like web based system have a higher potential to scale; I assume that's mostly due to being backed by a database. But it's also important to be able to customize the system for the support workflow and collaboration, and that's an important distinction between Launchpad and a generic forum. Forums are like the swiss army knife of social websites, you can use them to do lots of things... poorly. I feel custom tailored software will always have an edge over a general forum: wikis for howtos, bugtrackers for bugs, revision control for scripts and programs.

Unofficial resources

So far, we've only covered the things that people use who want to be involved and associated with Ubuntu. The internet is a much wider place than ubuntu.com and subdomains. I'll mention a few places I've found that some people may use instead. Reasons for avoiding official avenues range from ignorance of what's available, to having a established reputations and relations at another place, to better turnaround times and wider points of view.

The first and obvious external support tool is Google websearch. It's a great tool for surveying the hundreds of bugzillas, web forums and blogs on the net. For many people, websearch is the first go-to tool when they encounter a problem; any distribution that stops Google and search engines in general from indexing their bug tracker is doing their users a disservice. Launchpad uses an interesting URL scheme to make their database indexable; I hope to see how they implemented it sometime in July.

The first actual website I'll mention is AskMetafilter. Metafilter is free to read, but requires a 5 dollar one time fee to post. As a result, they have managed to build a community of pleasantly eloquant posters. And they're full on board with web 2.0 features like tagging and RSS feeds. The volume is already quite low as a result of the 5 dollar hurdle, especially since you can only ask one question a week and have to wait a week before asking a question.

But generally, you probably don't want to pay to help other people out. An interesting new group of sites has been built with some very worthwhile design insights to motivate quality responses and questions. There are countless question and answer sites out there; Google Answers (now retired), Yahoo Answers, etc. There's also specialized sites like Experts Exchange that do some unsavory tricks and generally profit from crowdsourced labor that happens spontaneously on the web. StackOverflow is a site designed to shift the rules for programming questions on the web towards open access. But more interesting for Ubuntu is it's companion site, ServerFault, which is geared towards servers and system administration, a topic that will be more relevant to Ubuntu than programming questions typically get. They support RSS feeds, so you can subscribe to all questions tagged Ubuntu. I hope people working on LP answers observe what is and isn't working for ServerFault.

And obviously, if a specific program is having trouble, there's usually an IRC channel, mailing list, bug tracker or forum organized by upstream. If you're going upstream for help, just be careful. It can be frustrating for user after user to come in and complain about a bug in Ubuntu that is fixed in upstream's latest release. If you're looking to help people, getting involved upstream is always a good pick. If you're still reading, hopefully this lengthy post has given you some ideas on how to target specific support request topics, and save yourself some time wading through noisy communication channels. Helping out doesn't have to be an avalanche of data!

mount sshfs at boot

2009-05-25T00:00:00Z

Currently, Upstart in Ubuntu does not generate network events. Instead it calls traditional sysvinit. By default NetworkManager is installed and running; rather than emit network events to Upstart, it contains a run-parts dispatcher (/etc/NetworkManager/dispatcher.d/) which itself simply relies on ifupdown's run-parts dispatcher (/etc/network/*.d/). In particular you care about /etc/network/if-up.d/ and /etc/network/if-down.d/

First set up a unencrypted ssh keypair, so you can ssh and mount the point without a prompt. Write a script, place it in /etc/network/if-up.d/ and make executable. The following was discovered on UbuntuForums and was sufficient for me:

#!/bin/sh
## http://ubuntuforums.org/showthread.php?t=430312
## The script will attempt to mount any fstab entry with an option
## "...,comment=$SELECTED_STRING,..."
## Use this to select specific sshfs mounts rather than all of them.
SELECTED_STRING="sshfs"
# Not for loopback
[ "$IFACE" != "lo" ] || exit 0

## define a number of useful functions
## returns true if input contains nothing but the digits 0-9, false otherwise
## so realy, more like isa_positive_integer
isa_number () {
    ! echo $1 | egrep -q '[^0-9]'
    return $?
}

## returns true if the given uid or username is that of the current user
am_i () {
        [ "$1" = "`id -u`" ] || [ "$1" = "`id -un`" ]
}

## takes a username or uid and finds it in /etc/passwd
## echoes the name and returns true on success
## echoes nothing and returns false on failure
user_from_uid () {
    if isa_number "$1"
    then
                # look for the corresponding name in /etc/passwd
        local IFS=":"
        while read name x uid the_rest
        do
                if [ "$1" = "$uid" ]
                        then
                                echo "$name"
                                return 0
                        fi
        done </etc/passwd
    else
        # look for the username in /etc/passwd
        if grep -q "^${1}:" /etc/passwd
        then
               echo "$1"
                return 0
        fi
    fi
    # if nothing was found, return false
        return 1
}
## Parses a string of comma-separated fstab options and finds out the
## username/uid assigned within them.
## echoes the found username/uid and returns true if found
## echoes "root" and returns false if none found
uid_from_fs_opts () {
        local uid=`echo $1 | egrep -o 'uid=[^,]+'`
        if [ -z "$uid" ]; then
                # no uid was specified, so default is root
                echo "root"
                return 1
        else
                # delete the "uid=" at the beginning
                uid_length=`expr length $uid - 3`
                uid=`expr substr $uid 5 $uid_length`
                echo $uid
                return 0
        fi
}

# unmount all shares first
sh "/etc/network/if-down.d/umountsshfs"

while read fs mp type opts dump pass extra
do
    # check validity of line
    if [ -z "$pass" -o -n "$extra" -o "`expr substr ${fs}x 1 1`" = "#"];
    then
        # line is invalid or a comment, so skip it
        continue
        # check if the line is a selected line
        elif echo $opts | grep -q "comment=$SELECTED_STRING"; then

        # get the uid of the mount
        mp_uid=`uid_from_fs_opts $opts`
         if am_i "$mp_uid"; then
                        # current user owns the mount, so mount it normally
                        { sh -c "mount $mp" &&
                                    echo "$mp mounted as current user (`id-un`)" ||

                                    echo "$mp failed to mount as current user (`id -un`)";
                         } &
                elif am_i root; then
                        # running as root, so sudo mount as user
                        if isa_number "$mp_uid"; then
                                # sudo wants a "#" sign icon front of a numeric uid
                                mp_uid="#$mp_uid"
                        fi
                        { sudo -u "$mp_uid" sh -c "mount $mp" &&
                                echo "$mp mounted as $mp_uid" ||
                                echo "$mp failed to mount as $mp_uid";
                        } &
                else
                        # otherwise, don't try to mount another user's mount point
                        echo "Not attempting to mount $mp as other user $mp_uid"
:
                        echo "Not attempting to mount $mp as other user $mp_uid"
                fi
    fi
    # if not an sshfs line, do nothing
done </etc/fstab
wait

If you have a wifi or otherwise unreliable connection, place the following in /etc/network/if-down.d/:

#!/bin/bash
# Not for loopback!
[ "$IFACE" != "lo" ] || exit 0
# comment this for testing
exec 1>/dev/null # squelch output for non-interactive

# umount all sshfs mounts
mounted=`grep 'fuse.sshfs\|sshfs#' /etc/mtab | awk '{ print $2 }'`
[ -n "$mounted" ] && { for mount in $mounted; do umount -l $mount; done; }

The final step is to make sure that your connection starts at boot, if you want to start networking (and sshfs) before anyone logs in.

Open Source as an Enterprise Strategy

2009-05-21T00:00:00Z

After the dotcom crash, a new argument about open source arose: "What if the company you bought software from goes under? Open source gives you a viable route to find consultants, move development internally, and so on. You're not dependent on any one group's success." You can find this logic behind the Ubuntu Foundation pledge. The argument is slightly compelling, but I can't think of any examples where a company with paying customers simply liquidated.

However, another argument springs to the mind of many after a recent event: "What if the company you bought software from is itself purchased? What protections do you have from aggressive monopolists? With open source, you always have a viable route to avoid particular nasty players." Without a strong competitor available, you have a far worse bargaining position when it comes to things like support and additional features.

That recent event? Blackboard purchased yet another competitor. Blackboard is growing to represent everything IT people hate in the software industry. They applied their trademark to stop what I assume was typosquatting. They filed restraining orders against people who intended to present software vulnerabilities at a conference. Most egregiously, Blackboard was awarded a patent for online teaching websites ("learning systems"). I am not a patent lawyer, but I do know that dozens of places, including my alma mater, had been working on, and may have even deployed, similar systems at the time the patent was filed.

What's most striking about this is that Blackboard is a Linux based system. It's completely a LAMP stack. That heritage might be why Blackboard offered a small compromise days after the SFLC announced the USPTO was re-examining Blackboard's patents. The jist of the pledge is to not pursue patent claims against open source projects, unless those projects integrate with proprietary software. This gift isn't as generous as it seems; the number one question from many colleges looking for software is "Does this integrate with Banner?" Banner is basically the administrative lifeblood of these places. Enrollment, accounting, performance metrics, etc. It is of course, proprietary and the most boring kind of software no open source hacker would write to scratch a personal itch (I'm sorry Mark, but SchoolTool is far from where colleges need to be in the running).

So in the face of the announcement that Blackboard had bought up its largest competitor using their earnings from that quarter (!), a number of people are suggesting that open source is strategic to their enterprise:

"It's going to probably push some people off the fence on the whole open- source question," said Nicole Engelbert, lead analyst for education technology at the consulting firm Datamonitor, who believes that it is possible for Blackboard to improve as a result of its purchase of Angel. "Some people are going to say, 'That's it, I'm going to definitely invest in Moodle or Sakai.'"

Ubuntu ships Moodle (but not Sakai). We also ship restricted drivers. I wonder what Blackboard, Canonical, and the Ubuntu community think of that.

Garmin hosts a Linux User Group meeting

2009-05-10T00:00:00Z

Garmin is a sizable consumer electronics firm situated in the Kansas City metro area, and yesterday they graciously hosted a KULUA meeting in their conference space. Why would they do this? Because they use Linux in their products and are hiring.

Tallgrass Technologies catered lunch, gave a presentation on OpenStorage and held a buisness card raffle for Royals' tickets. A few local Linux aficionado's gave quick presentations on postgresql and Mylyn. It has been suggested that at some point slides and videos of these will be posted. Garmin also gave a talk (slides), but there wasn't any invitation for questions, so those of you hoping to hear about upstream involvement or more transparency will be disappointed. Still, I'm glad they chose to distribute the slides and talk at all. Oh, one neat trick the presenter showed off was ssh'ing into a nuvi device. It's rare to find embedded platforms that leave dropbear installed, so thats pretty sweet. Not sure what ant was running for though; surely they don't build on the device itself.

The campus itself is much, much bigger than when I was last invited to visit. The parking garage appears to have been custom engineered to move Garmin products, as the place is an unnavigable labyrinth. The lobby atmosphere was laden with an awkward senior year interview type tension: five well dressed complete strangers sit in uncomfortable silence, perhaps even secretly sizing one another up, and wait for a company employee to escort you to the interview room user group meeting. I feel like I got the evil eye a bit, dressed in my Ubuntu polo and jeans. Probably because Garmin is hiring, and won't let you forget this fact. In fact, they're one of the few places in town that is hiring, even though they just announced major profit drops in the last week.

Certainly, there was a large turnout, and it highlights demand for user group meetings in Johnson County and Kansas City. If there are any Ubuntistas in the area, you may want to get in contact with me.

Backups galore

2009-05-02T00:00:00Z

I thought I'd do a quick survey and figure out what backup tools are readily available in Ubuntu. I found 17 that might be worth mentioning. Many are front-ends, and front-ends-to-the-front-ends, so to keep them straight I cooked up a quick diagram with GraphViz:

So there's a lot, and I may have missed a few nodes or edges. So far, my favorite is probably mrb:

Package: mrb Description: Manage incremental data snapshots with make

rsync mrb is a single, self-documenting, executable makefile, which aims to make trivial the task of maintaining a set of compact, incremental, rsync mirrors of your important (and sometimes rapidly changing) data. . It relies only on the time-hardened industry tools GNU make and rsync. Snapshots may be taken at any opportune interval. Multiple snapshot targets can be configured in a modular fashion, so fast changing data can be separated from static bulk data, with snapshots of each scheduled or triggered on demand, as may be appropriate for each.

At first you think, "Perfect, how hard can a make frontend to rsync be?" 15 kilobytes of Makefile later, you realize this may not be as brilliant as it sounds on paper. But it is well commented and relatively user friendly, which may actually impede the code's readability (as if anything involving whitespace syntax could be readable). Such gems as:

# If I have to explain this one, then I guess you are just reading this

# 'for the articles' -- but I hope you'll have enjoyed it anyway...

I haven't tried mrb yet, but I might put it on the TODO list. But first, I'll need to give jwz's method a full vetting.

Ubuntu is for ricers?

2009-04-21T00:00:00Z

nixternal takes funroll-loops a little too literally, and discovers that there's more Ubuntu stickers than Gentoo.

Obviously, stickers are a passive tool to communicate. Conversation starters, if you will. And certainly people can tie up identity with a popular brand. I recall seeing a poor grad student who didn't have a MacBook with an Apple logo glued over their Acer logo (poor Acer). She quickly scurried away as the next lab got started, so I didn't get a chance to ask the story behind it. I have a hard time putting myself in the shoes of a Gentoo user and wanting to explain Gentoo to someone who isn't using it. It would just serve to highlight the gulf between nerds and normal people. So I can imagine why there aren't many Gentoo stickers in the wild.

nixternal also asks for laptop stickers, and I just happened to finish photographing some I received in the mail today. Martin Owens chronicles how ZaReason came to offer them. Here's an amateur photo with a yellow lamp bulb and white flash:

Goal Setting

2009-04-20T00:00:00Z

Ubuntu 9.04 will mark the tenth release of Ubuntu. Rather than party and engage in self congratulation, I'd like to engage in a retrospective.

The rocky trail traveled

In November I helped facilitate a collaboration between a Fedora Board member and the organizers of Ubuntu Brainstorm, to test some social network datamining. I'm assuming the conclusion was that the votes were too sparse to draw any strong conclusions about niches interested in specific subjects. Certainly, there's an element of privacy that restricts what can be done here even if it is viable.

I joined MetaFilter and started following the ubuntu and linux tags on AskMeFi. So far, I've racked up four or five "best answers". It's a very similar project to Answers, except there's a five dollar sign up fee that appears to boost the coherence of the questions and replies. And it has RSS feeds, which is nice for turning down the rate of flow out of the firehose.

In January, I tried to get ubuntu CDs in the local college library. They refused a free donation, citing a budget crunch. They also feared a slippery slope where once they started accepting some software, they'd have to start buying other programs for circulation. Amusingly, the library already carries Ubuntu books, and provides the CDs they come with. I believe the reason it was refused was political rather than fiscal: officially accepting software means budgeting for software, which would likely slant the budget in favor of the technology librarians over other subject areas.

Where I specifically fell down over the past six months was in testing and fixing Jaunty. Traditionally, I use my TabletPC to test tools in ubuntu+1 that support the TabletPC hardware. Fingerprint readers, wacom, handwriting, and so on. Unfortunately, I had to loan that device out to a family member who's computer has gone out of commission. Add on top of that a new job with far less spare time, as a system administrator, and triaging and testing fell behind.

Blazing a new way forward

I don't feel like I've accomplished much in the past cycle. To better motivate myself, I'm going to publicly commit to making some changes. For the future, it's time to right-size my goals. I'll do some new stuff, and drop some of old.

For sure:

Promote Ubuntu. I've volunteered to present the Ubuntu netbook remix at an upcoming LUG meeting. Hopefully I'll have enough CDs and stickers to go around.
Evaluate desktop backup tools. We have a lot of backup tools in Ubuntu. Newly minted Ubuntu Member mterry even wrote and packaged one. We always tell people to make backups before upgrading, but there isn't a lot of attention paid to it by the community. I'm taking notes on 8 backup tools thus far. Since I've got a desktop that's been running Ubuntu, and upgraded from warty through the ages to jaunty, keeping around the accumulated changes interests me.
Share-alike. I'll revisit the library's policy on software and try the public libraries instead if they still can't accept free software.

Maybe:

Address bug #290159. There's a patch, I forwarded it upstream for review, and it sorta stalled out. Unfortunately, the upstream author is also the Debian maintainer, so there's no extra opportunity for collaboration and peer review there.
Test and triage wacom. Without 24/7 access to hardware, triaging reports is hard and testing is even harder. If I get it back, I'll be in a better place to dedicate some time to handle it.
Pitch in for the Ubuntu education project. I have experience with popular existing web courseware tools, but not Moodle. Helping out with that may be interesting. I'm not sure where exactly this is being organized though.
Package KeePass 2.0 In a previous post I asked about software for managing team secrets. Keepass 2.0 fills a niche we need at work, so it'd be nice to have it available on Ubuntu workstations. Upstream is dicey though; a single developer who doesn't publish a public source repo, just binaries with corresponding source. Hopefully he'll publish a new version that fixes one or two bugs we've encountered and reported in testing.

Step Down Considerately:

Fingerprinting in Ubuntu. This is a bad idea who's time has come. Unfortunately, there's too much bad to unwind. Thinkfinger in Ubuntu is an SVN snapshot of a dead project, who's packaging I don't fully understand. fprint is potentially the replacement but I haven't had time to read how PAM changed since Hardy. But basically, now that I know that it can work, I'm not sure it should.

I wholeheartedly agree

2009-03-02T00:00:00Z

Colin puts together a coherent rant about bug triage with Ubuntu. I'm glad he was able to meditate on the subject and produce a document with clarity and specific improvements; I wrote a private rant that included the phrase "Bug Assassination Squad: We kill bugs in their infancy". Probably not diplomatic enough to publish.

More diplomatically, one thing I'd like to add to Colin's commentary is about bug duplicates. Some people seem to be in the habit of marking duplicate bugs invalid. Yes, there are a lot of bugs, so going to the effort of finding the dupe and marking it means you'll be slower at closing bugs. But you'll have a better bug database as a result. And it will spare me the effort of searching through all the package's bugs to discover there is no such duplicate. Certainly, if you believe there's a duplicate report, you have a better idea of where the dupe is than the person who submitted the bug, who went through LP's own dupefinder to report the bug in the first place.

While I do feel vindicated that this practice is contrary to guidelines, the fact that it still happens is not encouraging.

Ubuntu for Network Engineers

2009-01-26T00:00:00Z

Keith Tokash posted a review of Ubuntu 8.04 from the perspective of a network engineer who switched from Windows XP to Ubuntu. It's been picked up at Slashdot as well. Rather than post a comment there, I thought I'd share a more in depth response.

A quick synopsis before hitting individual points: much of the review centers around Linux as embodied within Ubuntu, rather than the things Ubuntu brings over the other Linux distributions. Balancing the praise are a few thorns, places where Ubuntu isn't sufficient for his needs. On to a few specific points, then.

He mentions that Evolution and Exchange don't get along well. This is hardly news, and a significant barrier where I work as well. And frankly, it get gets even worse than he reports. Evolution uses the Outlook Web Access (OWA) to connect to Exchange. This works okay in Exchange 2003, but not Exchange 2007. For people working in a large organization, you're unlikely to see a move to a mail server with wider compatibility, but there's hope: OpenChange is working on a library to implement MAPI, so clients can access Exchange on the same terms as Outlook. I can't find it now, but there was some discussion about getting Debian packaging going and getting this done in the Jaunty timeline. It's by no means final, and won't help people preferring LTS, but by the next LTS, I would think Evolution will connect to Exchange 2007. Hopefully Exchange doesn't change much in the meantime!

He also mentions Visio, a diagram program. Visio is certainly a well polished tool, and I've yet to find anything comparable. I hate dia, OpenOffice renders like crap, and Inkscape's diagram tool isn't usable. GraphViz and dotty are on my list to try out, but the lack of a GUI is going to put off a lot of people, and frankly the screenshots give me no faith in their power to make things pleasing. Most diagramming tools in Ubuntu revolve around UML in one way or another.

He also cites the lack of a password manager compatible with PasswordSafe. I've been looking into this myself, and as it happens, he's wrong. Password Gorilla is a compatible program, in package password-gorilla (available since hardy). Unfortunately, if you search for "passwordsafe" in the repos, Gorilla doesn't come up, which explains why he might not have found it. Personally, I like KeePass 2.0 over PasswordSafe. Keepass works great in Jaunty, not so great in Hardy or Intrepid. No package yet, but I may publish one in a repo.

Finally, there's also a few complaints that are far more mundane. For example, turning off 2-page view in Evince (GNOME's PDF reader). Instead of looking in the View menu, he installed Adobe's Linux PDF client! Worth a chuckle, I guess. (If you're reading Keith, it's a checkbox under View->Dual).

All in all it seems Ubuntu is doing a good job of putting together a coherent desktop. Many of the rough edges are being worked on, but Visio remains a sore spot for the foreseeable future.

A cheap media remote

2008-12-19T00:00:00Z

It's been a while since I blogged about something I pay attention to in Ubuntu, get ready! Today I talk about using a Wiimote to watch videos.

If you already own a Wii and a cheap Bluetooth adapter, there's a lot of fun stuff you can do in Linux with the Wiimote. The simplest and most immediately useful is to use it as an alternative input device.

A quick rundown: install package wminput, modprobe uinput, and run sudo wminput -d . wminput will listen on bluetooth for the wiimote, and translate into mouse cursor movement and keystrokes according to it's default config file. Press 1+2 on the WiiMote to put it into discovery mode, and it should connect in a few seconds.

I use the following as my wminput config file:

# remote - settings appropriate for totem/mplayer/mythTV

Wiimote.A = KEY_SPACE
Wiimote.B = KEY_ENTER
Wiimote.Up = KEY_VOLUMEUP
Wiimote.Down = KEY_VOLUMEDOWN
Wiimote.Left = KEY_LEFT
Wiimote.Right = KEY_RIGHT
Wiimote.Minus = KEY_BACK
Wiimote.Plus = KEY_FORWARD
Wiimote.Home = KEY_ESC
Wiimote.1 = KEY_F
Wiimote.2 = KEY_PROG2

Put it in ~.cwiid/wminput/default and you won't have to specify it on the command line! The idea is that D-pad left and right scan through the video, up and down turn the volume down, A triggers pause/play, and 1 toggles fullscreen. I don't know what KEY_PROG2 is good for, but it felt wrong to leave a button unmapped. No tilt sensing or mousing here, since I want to be able to set the wiimote down and enjoy the show.

Some might think that KEY_PLAYPAUSE would be more function specific than KEY_SPACE, but they are wrong; none of the players appear to support that. My hope is that in the future I can find settings more universal; my USB keyboard has buttons that trigger rhythmbox, so maybe that mechanism can be exposed.

There hasn't been a lot of activity upstream, aside from applying a patch from Mario Limonciello (superm1). I'm not sure if it's a sign of maturity or of project failure.

New Ubuntu Store

2008-11-20T00:00:00Z

Seemingly in response to an earlier post of mine, Canonical has recently launched a US based store. Since they've gone to the trouble to risk taking my advice, I suppose it's only fair that I shill a bit for them.

Prices before were close to 40 dollars for a shirt after taxes and shipping. They're now closer to 25, and I think I'll put in an order at that price.

The store's new, so it's not yet perfected -- they ship a fun set of stickers via UPS Ground, a 7 dollar charge for something that could probably be sent for 42 cents. That's okay, system76 still offers stickers for the price of a SASE (?) as a draw (and sales lead) to their store.

One product suggestion: Aluminum case badges.

Ubuntu on ARM

2008-11-13T00:00:00Z

Canonical has just announced that they'll be bringing the ARM arch to Ubuntu. This is quite exciting -- ARM devices have been getting more and more features, and as LinuxDevices reports, there's been unofficial builds by Nokia for a while. Centralizing the builds should help reduce the effort needed to set up Ubuntu on such platforms. For example, currently builds happen after release and are announced when they're complete; we can now expect regular builds throughout development and a release date people can plan around.

There's some interesting hardware coming my way that I'd like to try this on, but I won't share just what it is until I actually have one in my hands, since they've sold out their first small batch and having problems in manufacturing.

In the meantime, I'll be hanging out in #ubuntu-arm observing.

Brainstorm reconsidered?

2008-11-10T00:00:00Z

Jef Spaleta, Fedora contributor and member of the Fedora Board, offers a dissenting view on the value of websites like Dell Ideastorm for community based projects:

It makes some sense when you have dedicated engineering resources to spend and are looking for ideas to spend it on. But if you want to grow new volunteer involvement, I don't think the Idea Storm implementations we are seeing make sense for that. The popularity of an idea simply is not enough. We have to have a mechanism which helps individuals turn personal interest in to personal action...instead of encouraging them to wait for someone else to take their idea and run with it.

Ouch. He's not the first person to recognize there's something wrong with Brainstorm. My personal favorite is #11730: "Make developers pay attention to ideas on this site". The rhetoric of Brainstorm, inherited from Ideastorm, is that users submit ideas and developers implement them. The disconnect between this and reality is causing frustration. The Dell Ideastorm revolves around identifying items for Dell to spend money on to make their product more valuable to customers; Dell managers have a profit motive to pay close attention and allocate resources to make things happen.

In contrast, Canonical has a commitment to making Ubuntu available free of charge, which affects how the profit motive is expressed. In Canonical's focus to make development sustainable, their paying customers are OEMs like Dell, not end users. Under those constraints, they should reasonably focus on the set of items that both users and OEMs want (and might possibly pay for). Automated hardware change recognition and Deviant Art contests don't seem to quite fit that mold.

The rest of Ubuntu contribution is basically the fantastic result of an appropriate social contract. Volunteers seek to improve Ubuntu for their needs and those like them, and contribute that back to everyone. It's a classic stone soup story; the soup tastes great and everyone gets more than they gave. Such people don't need a website to figure out what they want to fix, they need guidance on how to fix it.

Jef's alternative suggestion is to use the votes to find clusters of people interested in a common set of goals, and give them a communication medium to make their goals reality. That sounds nice but I think it's a lot harder than it sounds to find a group of people who can get things done. MOTU has a series of failed failing focused teams: BitTorrent, Java, Games, Audio. Even the ubuntu-laptop team seems to have fallen apart, with no apparent leadership or goals. I don't think it's a matter of subscribing people who vote for a set of features to a mailing list.

From the outside looking in, I'd say that Debian has had more success with their focused team based efforts. Probably because in many cases, it was quite simple to organize a cluster of people interested and already taking personal action. There's probably a critical mass of core competence needed before team approaches can be successful; recruiting is important, but you can't do it without a group who really knows their stuff and can write down the important parts for those that follow.

Within Ubuntu, the OpenWeek is supposed to function like a cross between a conference and a Freshman Activities Fair at the beginning of the semester, and helps address the need for recruiting informed volunteers. However, I think the IRC format, as is, is not appropriate. The first half of most sessions runs like a file dump from a text file, and the latter half runs like a Q&A. The main purpose is to engage the wider community in the development projects within Ubuntu. The Q&A is really useful in getting people to ask questions and maybe even participate in development, but the IRC format hurts the presentation of information in the first half. If we want Ubuntu to further evolve beyond the command line, I think it might be more appropriate to record a 20 minute video presentation or tutorial as a prerequisite to an IRC Q&A session. Perhaps this is something Loco's can get involved in producing with developers?

Another approach may be to make sure high rated items on Brainstorm see treatment at OpenWeek. I imagine fixing suspend and resume would be a massively popular Q&A. Or a session on the highly popular idea, suggested for Jaunty: faster boot times.

OpenMoko or Android G1?

2008-11-08T00:00:00Z

Having killed my cellphone with a good cycle in the washer, I'm in the new phone market I guess. Besides simply replacing the model I damaged ("the cheap option"), I'm also looking at OpenMoko and Android. Trouble is, they're both the same price, and nobody really has published a direct comparison.

I like the idea of running Linux on a phone, and having a wide degree of freedom to customize, but with other fun hardware on the way, I'm not sure how much time I'd really devote to tinkering with a phone I'd rather have working 100 percent of the time. So I'm looking for three kinds of advice:

why I should get an OpenMoko from people who own one
why I should get an HTC G1 from people who own one
a comparison from people who own either but have used both

The last category strikes me as rare, but it can't hurt to ask.

Bootchart refresher course

2008-10-30T00:00:00Z

Boot times are becoming a major hassle again to everyday users. NYTimes is reporting that many laptop vendors are resorting to a technology called SplashTop to boot a small Linux environment from BIOS, rather than incur a lengthy Vista boot penalty, documented in this New York Times video. This can save you a good deal of time if you need to write a quick email and turn the computer back off.

Of course, if you want anything outside the provisions of SplashTop, you'll have to boot into a different OS. SplashTop seems like the wrong solution to the right problem; if the full OS boots too slow, why boot a crippled OS rather than improve the full OS boot time? Thankfully, one of the release goals for Ubuntu 9.04 is faster boot time.

Bootchart is a great tool to record and chart relevant information about what resources are in demand during boot. Originally dreamed up and implemented by Fedora, it's quickly become a tool used by many.

Installation And Use On Ubuntu

The package name is bootchart; you can use your favorite command line or GUI package manager to install the package from main. By default it saves generated charts to /var/log/bootchart/, and will record one for every boot (except when booting in "profile" mode, which monitors disk access during boot to determine which files to read ahead).

The chart itself contains 3 sections:

A description of the system
A pair of graphs describing CPU & iowait and disk activity/throughput during boot
A Gantt Chart showing long lived processes and their status: waiting for disk, running, sleeping or zombie

The most immediately useful purpose of bootchart is to objectively discover the boot time, and where the bottlenecks are. For example, this bootchart shows a length of 33 seconds, which isn't bad, but we can quickly discover a 5 second dead zone, with a suspicious process called "resume". We can also see that disk throughput peaks at 23MB/sec during readahead, but isn't sustained. Daniel Stone documents a similar series of diagnoses from 2004.

I happen to have a very large archive of bootcharts starting from Dapper forward one machine, and from Feisty forward on another. And if you've installed bootchart in the past, you might too. This can be useful to figure out when a drastic jump occured, and lead you to which changes to Ubuntu might have triggered it. You do risk wasting a lot of disk space, of course.

Secret features

Ubuntu comes with scripts to upload charts to a website (disabled by default). From /etc/init.d/stop-bootchart.d:

wget -O /dev/null -q --post-file $CHARTS/$base-$count.png http://bootchart.err.no/upload/$(lsb_release -si)/$(lsb_release -sc)/$(uname -r)/$(uname -m)/$boottype

Make sure you get permission from the owner or change that URL to a server you control before running it in this manner.

Bootchart can also record to SVGZ, a compressed vector image format. Bug reporters feel SVGZ is a lot smaller in practice, and reduces boot time on already long boots (Ironic!). SVGZ also has the advantage that with a bit of thought (ditch the XML comments and use sensible ID attributes instead) you could data mine a collection of charts. For example, you could graph the boot time over the course of six months, or calculate the ratio of time spent running readahead versus total boot time. A lazy survey of my October charts for Intrepid shows boot time has declined personally from 47 seconds to 34.

One slightly dangerous feature is that bootchart doesn't show all running processes:

Some processes were filtered out for clarity -- mostly sleepy kernel processes and the ones that only live for the duration of a single top sample. This skews the chart a bit but is definitely more comprehensible.

The dire future

Bootchart hasn't released since late 2005. A new developer unleashed a fury of 26 commits one day in August, but hasn't touched it since and hasn't announced any release. Some of the patches appear to fix bugs reported in Sourceforge but nobody's touched the bug since being reported, even to close them. There remain a number of open bug reports, but nobody seems to be reviewing reports or patches at this time. This is worrisome because a developer's ability to fix flaws is only as good as the tools available to objectively see them.

Given the goal of faster boot time, it seems unfortunate that a fundamental tool is currently without a functioning exchange for ideas, patches and bug reports. I hope one of the first actions taken on this track is to reform the bootchart project into a vibrant community of participants and observers from a plethora of projects. I'm finding discussions going on about bootchart on the edges of the open source community, but none at the theoretical center of activity. Collaboration is part of the Code of Conduct, and I hope our bright minds gathering at Mountain View in December take it to heart.

I owe someone a thank you

2008-10-17T00:00:00Z

Sometime during the last few weeks my Intel HDA based Toshiba Tecra M7 tablet finally supported headphone jack-sense. No longer will I feel like an amateur plugging in headphones and fumbling about figuring out how to silence the speakers but not the headphones. Yay!

Training materials

2008-10-12T00:00:00Z

I've had the opportunity recently to review some training materials as part of my job. VTC offers a lot of streaming video tutorials on various software. I decided to search for Ubuntu and one series showed up. Charles Griffen authored and published these video tutorials, and they're available from a wide array of sources. Amazon retails a DVD version for $99.95.

To inform the broader Ubuntu community and to learn from the mistakes of others, I've decided to publish a critique of Charles Griffin's Ubuntu Tutorials:

Information

The content itself is basically a brief overview of the default Ubuntu Desktop components, spread out over five and a half hours. The audience appears to be expected to be comfortable with Windows and computers in general as he walks them through burning and verifying a LiveCD. In addition to the basic overview of the GNOME desktop software, he covers briefly how to use the command line, WINE/Crossover/Cedega, and Automatix.

The tutorials are based on Ubuntu 6.10. This is very old and unsupported software, and the videos haven't been updated to reflect any changes in behavior or software. If revisiting the video every six months takes too much time, 6.06 LTS might have been a better choice for the longevity of material.

He spends five minutes on legal restrictions on media formats, but doesn't communicate solutions or even the fundamentals of the problem of patents. For example, DVD playback isn't illegal, if you've negotiated a patent license, or if someone has negotiated one on your behalf, like Dell does for their customers. Canonical even offers such things for sale, if you wish to protect yourself from liability.

The videos completely neglect the Wiki and Launchpad as an avenue for support, instead suggesting the purchase of official support from Canonical or community support from forums. While speaking about the community, he neglects the foundational Code of Conduct that lay out the etiquette expected among developers and the community.

He advocates the use of EasyUbuntu and Automatix in his video. By the time these videos were published, Automatix was known by developers, including Technical Board member Matt Zimmerman, to be fatally flawed. After Matthew Garrett published a document reviewing the flaws, I feel it's irresponsible to continue publishing a recommendation of Automatix.

Narration

Charles's voice is smooth and comes across well over the recording. The pacing is quick enough that the potential boredom of the subject isn't multiplied by numbing slowness. He is well rehearsed and professional. Much of this seems to come from his work on LinuxReality, a podcast oriented towards new Linux users (now defunct).

Most of these tutorials come with subtitles. This is great for accessibility, but there are several errors, some of which are substantial. The file browser is universally misspelled "Nodilus" (it is "Nautilus"). More importantly, the command line tutorial makes a specific point about forward slash versus backslash, while the subtitles get it precisely backwards!

The diction is annoying. He uses the word "one" to refer to you, the viewer, a lot. Your English teacher in high school might have marked you down for informality, but nobody takes their advice seriously, and neither should you. Otherwise, the tutorials remain relatively free of jargon and accessible to the audience. Where important jargon is used, it is explained adequately.

Presentation

The video itself is based entirely off of screen capture streams, even when a diagram or two would be far more illuminating. The VTC website offers Quicktime or Flash playback, which may annoy advocates of open codecs. Admittedly, it is a bit of a challenge today to host video that is universally playable today. (HTML5 offers hope, but that's a subject for another time).

The VM image used for recording is in need of upkeep -- update-manager is prompting for updates in the notification area, and the volume is muted or broken. Busted audio might not be so bad except the tutorial covers some audio applications... without working audio.

The video isn't high quality. The recording was done at maybe 5 or 10 frames per second, and the resolution is too small to adequately display some applications, making the whole experience look cramped. Still, it's high enough quality that most text is readable, a problem I've encountered personally when running Desktop OS's at TV quality resolution. The low resolution does have the advantage that it can play from a DVD to a regular television and still be readable.

Conclusion

These tutorials are outdated, but convey a wealth of information about what to cover and how. Personally, I was a bit disappointed that the tutorials didn't teach me anything new about the Ubuntu Desktop, even about programs I haven't used much. (Does anyone use Evolution seriously?)

I get the impression that making such a set of tutorials takes more effort in planning, recording, and editing than a single person can muster. I may seem harsh in writing this critique, so let me be clear: Griffin makes a valiant solo effort, but the rapid pace of Ubuntu and Linux in general is depreciating the quality of his instruction. The Screencast Team brings a lot of expertise and knowledge to the table, and if they decide to do an "Ubuntu Introduction" project in the future, I hope they'll consider how to improve upon the works of others, and find ways to cope with the high rate of change!

Is Gentoo dying?

2008-09-28T00:00:00Z

A recent cnet article suggests that Ubuntu is eating other distribution's lunch. In particular, one distribution is reported to be falling apart: Gentoo. Gentoo was very popular among my friends at the time I adopted Linux, but from what I've seen, the project fell apart as developers were unable to come to consensus or resolve conflicts.

From what little I know of Gentoo users and the project, it's closer to say that Gentoo is becoming an unofficial set of distributed overlays than a centralized approved project with trusted developers and so on. Gentoo's core appeal isn't under attack by Ubuntu -- building from source and customization for performance are central and remain relatively unique. If Ubuntu's focus on desktop usability or six month release cycle are appealing enough to Gentoo users that they leave the project, then build from source and customization were simply means to an end and Ubuntu has improved the Linux landscape for the better.

The article cites Google Trends as evidence of Ubuntu rocketing off to outer space while Gentoo stagnates. I'm reminded of the current 5-a-day discussion; there's a certain amount of danger to blindly trusting metrics. This Google Trend shows "Ubuntu" approaching "Dell":

(Orange = Dell; Red = Ubuntu; Blue = Gentoo)

It's a large leap to say that Ubuntu is as popular as Dell; certainly Ubuntu is a small fraction of Dell sales. One thing I do know is that Ubuntu is very googleable. All the mailing lists are archived, IRC is publicly logged, the forums even have a special search engine mode for faster indexing and engine retrieval, bug pages aren't blocked in robots.txt, and the wiki is used extensively. This wide array of information is something a tool like Google Search can aid in, and is far simpler than say searching the wiki individually, then Launchpad and so on. Perhaps Ubuntu users are heavier users of Google Search, compensating for what could be a smaller user base. We can't really infer user base size from Trends, just growth patterns.

What does seem clear from the trend is that interest in Ubuntu isn't growing as quickly today. Every new release causes a bump in search volume but there isn't as much sticking around after the release. Maybe Ubuntu works better today, so people aren't Googling their problems as often? I am, shall we say, open to alternative explanations.

"Be collaborative"

2008-09-18T00:00:00Z

Greg K-H gave a talk at the Linux Plumber's Conference ostensibly about the Linux Ecosystem, but appears to have been primarily about Ubuntu's leading commercial sponsor, Canonical. I wasn't in attendence; as a mere amateur, I'm satisfied to read reports and watch videos from conferences like LPC. It's a lot cheaper on the wallet, certainly.

But I do wonder about his accounting. It's simple enough to measure the rate of change of the kernel, the speed of releases and so on, but it's much harder to referee the score. I haven't seen any published scripts to automatically attribute his tally, and I imagine publishing such a thing might be an invasion of privacy. Still there are questions in my mind. If someone chooses to contract a kernel hacker to write a network driver, who does Greg attribute? Does his account of X.org include any work done by Daniel Stone?

Greg is right though, that it takes a lot of work just to tread water with the kernel. The rate of change will bowl you over, which I'm told is one reason Redhat hires a small army of kernel developers to backport patches to RHEL, and I'm sure their customers love them for it.

The Good News

There is, I think, an unprecedented opportunity coming up to collaborate. Fedora, openSuSE and Ubuntu will all be shipping a .27 kernel in a stable release. It occurs to me that this would be a perfect time for one of those extended stable kernel cycles that Greg mentioned in that Google Tech Talk some time ago. I hope this idea is brought up during a conference about solving problems and the "kernel ecosystem".

Beautiful

2008-09-17T00:00:00Z

I won't ruin the RSS with a huge image, but here's a graphical flow of iptables/netfilter. Netfilter is an awesomely complex beast, as the image shows.

Unfortunately, the author has some scathing reviews of Ubuntu 8.04, concluding:

An absolute fail distro. Really, I recommend Windows XP anytime over Ubuntu. (But of course, SUSE/Fedora in the pole position still.)

I appreciate Jan's contribution and dedication to kernel.org and elsewhere; I will leave it to others to debunk or affirm his individual complaints. But framing the complaints of a kernel hacker as applicable to Windows users is disingenuous.

Warrior Rabbit totem

2008-09-08T00:00:00Z

As seen on #ubuntu+1:

#ubuntu+1.log-14:23 < pwnguin> hmm. apparently next week is ubuntu+2 naming time

#ubuntu+1.log:14:24 * pwnguin votes for jackalope

I'm so very sorry.

Chromium Build Instructions?

2008-09-02T00:00:00Z

I have located the Linux build instructions for Chromium, what you might know as Google Chrome.

Interesting notes thus far:

Instructions include a reference to "Ubuntu 8" for retrieving build dependencies ;)
an SVN checkout that retrieves ANOTHER set of source code management tools not in wide use
uses bison for grammars; I've become a big fan of ANTLR for describing a grammar
"BSD licensed," so it should be license compatible with Mozilla (and IE and opera)
checkout of source code takes 16 minutes to grab 666MB and crashes:

Traceback (most recent call last):

File "home/jldugger/Desktop/chromium/depot_tools/release/gclient.py", line 925, in result = Main(sys.argv)

File "home/jldugger/Desktop/chromium/depot_tools/release/gclient.py", line 921, in Main return DispatchCommand(command, options, args)

File "home/jldugger/Desktop/chromium/depot_tools/release/gclient.py", line 885, in DispatchCommand return command_mapcommand

File "home/jldugger/Desktop/chromium/depot_tools/release/gclient.py", line 837, in DoUpdate return update_all(client, options, args)

File "home/jldugger/Desktop/chromium/depot_tools/release/gclient.py", line 713, in UpdateAll deps = get_all_deps(client, entries)

File "home/jldugger/Desktop/chromium/depot_tools/release/gclient.py", line 576, in GetAllDeps solution_deps = get_default_solution_deps(client, solution["name"])

File "home/jldugger/Desktop/chromium/depot_tools/release/gclient.py", line 537, in GetDefaultSolutionDeps deps = scope["deps"] KeyError: 'deps'

Once built, Chromium will do nothing of value to Linux users today.

Currently, I'm downloading their tarball, hoping that it works better than the checkout instructions did.

A recipe for smoother IRC

2008-08-29T00:00:00Z

IRC is a common method of communication within Ubuntu and many other projects. I believe Ubuntu policy is that nothing is official unless it's on a mailing list, but there's still some benefit to having a rendezvous communication like IRC with others. But frequently, I see things like the following:

* novice has joined #ubuntu-foo

< novice> Can anyone help me with bug #53632?

< novice> I applied the patch, but it's not building correctly

< novice> Am I doing something wrong?

_ Five minutes of silence go by _

< novice> Is anybody here?!?!

* novice has left #ubuntu-foo

The first question can really be anything relevant to -foo, but the end result is all too common; the person leaves before anyone glances at the channel to respond. You might imagine it's a simple problem to solve with more peoplepower, but it's more a question of latency than throughput. So having tools to handle that latency is nice.

The ingredients

A remote server running sshd
An account on said server
Screen and irssi installed on said server
ssh client on your local machine

Steps to take

ssh into the remote server: ssh host.com
run screen: screen -S irc
run irc: irssi -C freenode.net
Hit Control-A followed by D to detach from the screen session. Irssi will continue running until you reattach
set up an GNOME launcher with the following command:

gnome-terminal -e "ssh -t host.com screen -d -r"

Click the icon, authenticate with the ssh server and presto, IRC as if you never left

Tips for maximal enjoyment

Public Key encryption can make life a bit simpler
Control-A followed by ? (question mark) will bring up the help dialog for screen
alt-3 switches to window 3 in irssi, and so on. This continues for the qwerty row as well
screen -x will attach multiple connections to the same session instead of disconnecting the others
This page has a more detailed description of the above

Enjoy!

Now you can just ask questions without having to leave, or monitor a channel for relevant questions and messages without missing anything. Traditionally, putting a nick in a message highlights the line on the intended recipient's screen, or does other interesting things. A common technique to establish rendezvous over IRC is a simple ping:

crweb: ping? I have a question about Qt on embedded

If nobody gets back to you immediately, be prepared to wait, especially on relatively small and slow channels. If you need to shut down your local computer, just hit Control+A D to detach. On busy channels like #ubuntu, it's wise to use nick: highlighting to sort out your conversation from the others.

Arora versus Aurora

2008-08-09T00:00:00Z

Why should I have to change my name? He's the one who sucks!

-- Michael Bolton, Office Space

Arora is a next generation browser project publicly released April 11th, 2008, with a focus on bleeding edge UI elements.

Aurora is a next generation browser project publicly announced August 4th, 2008, with a focus on bleeding edge UI elements.

Besides the obvious differences, one has working code, and the other is a Mozilla Labs prototyping project.

Ubuntu Gear

2008-08-08T00:00:00Z

One last post on the subject of money.

I know that the dollar's been on a recent downward trend, but this still feels exorbitant. But the old "strong dollar in name only" policy cuts both ways -- maybe it's time to set up a shop in the land of cheaper exports? Shipping itself adds nearly 50% to the price!

UPDATE: fixed the shop link!

GNOME a11y Contest

2008-08-05T00:00:00Z

Keeping with the theme of money in Linux, today I present you GNOME Outreach Program: Accessibility. The Program has raised $50,000 in pursuit of three goals:

Increase awareness of accessibility and its relation to computer applications
Encourage and inspire people to work on accessibility
Help the free software community improve its accessibility

These come from the rules the front page encourages everyone to read. Divided into Long-Term and Short-Term Tasks, the rules read more like a sweepstakes than an employment agreement or contract. Still, the fundamentals are oddly close to a work for hire: you perform a set of tasks, and are rewarded with money. $6000 for Long-Term tasks, $200 for Short-Term tasks.

Short-Term Tasks

On the subject of short-term tasks, the tasks list page says:

Are you a developer who wants to become more familiar with accessibility? Are you an artist that can draw? Maybe you might also be interested in becoming a module maintainer some day. A great way to get started is by fixing bugs, and we're offering you a way to get paid to do it. :-)

Short term tasks are two week affairs that pay out $200 each (in bundles of five) a construction similar to Google's "Highly Open Participation Contest". Unfortunately, I doubt any of the people completing them expect to claim any prize money. It's a simple matter of math: if any task takes more than 30 hours, flipping burgers pays better (where I live).

Bundling by fives is crazy. It discourages participants simply by the unknown quantity of effort and massive investment before payout. But it gets worse. A sufficiently clever person can nullify all monetary incentive for some tasks and a sufficiently conscientious person will avoid completing tasks they don't intend to claim a prize on. An example: there's only five "Create 10 Accessible Icons" tasks posted. Completing one task effectively reserves the rest for you. Certainly, if you try one, finish it, and decide the other four tasks are too much work to bother with, you've removed the intended monetary incentive, albeit accidentally.

This assumes the flow of tasks is stagnant, though I've seen only evidence supporting that assumption. If you can't find five tasks you think you can claim before anyone else does, you can either wait until new tasks come along or start now and hope they do. Worse still, many of the small task bugs marked as completed were done without knowledge of the Outreach Program it seems. A total of six small tasks are marked completed:

The first two bugs were closed by an Ekiga developer as WONTFIX. Folks, it doesn't get any more classic Bugzilla than that. The middle two were fixed by a developer being paid for their work by Sun. Only the last two were claimed tasks under the Outreach Program. So of the 30 total Short-Term task claims, 2 have been fixed thus far via Outreach.

Long-Term Tasks

Long term tasks are closer to Google Summer of Code in nature: 6 month projects netting $6000. Thus far, there have been two proposals accepted; "MouseTrap: Head Tracking via Lowcost Webcam" and Magnification.

MouseTrap was actually working two months ago. You can see video of it in action:

(Link if you don't see the Flash Player above)

And the magnification task is being worked on by the developer of Compiz's Enhanced Zoom. They're both talented people and I'm sure the money motivates them to continue the work they started outside of GNOME.

Will they meet their goals?

With half a year yet to go, it seems unfair to make any doom and gloom predictions. So instead, let's examine where they stand now in relation to their goals.

Have they helped improve free software accessibility? Yes. Not as much as they hoped, but there's time left to save it. They've fixed a few bugs, enticed a few projects towards GNOME that could have large impact.

Have they encourage and inspire people to work on accessibility? They've certainly encouraged the two Long-Term authors, though maybe not inspired them. Another three people seemed inspired enough to pick up some short term tasks, but none seem on pace to net five Completed Tasks.

Have they increased awareness of accessibility? I'd say not enough. This is an "outreach program", but it almost seems GNOME needs "inreach" to convince their own project members accessibility matters. Normally when I think of outreach I think of an expert organization going to the public to share that expertise, not an organization in search of experts. Closing bugs as WONTFIX that Foundation and Sponsors are trying to pay someone to fix does not look like an organization full of accessibility experts. If GNOME leadership doesn't convince the rank and file that it needs accessibility, outside contribution faces a hurdle that could leave them bitter on the subject. This sort of failure jeopardizes any success found in the previous two questions.

Still, $6,000 is a cheap price for getting some of the Compiz people thinking about how GNOME might ever integrate their work. And a cheap price to get the ball rolling on head tracking within GNOME. If we're lucky, in some distant future we might see eye tracking as well!

Origami Ibex

2008-08-02T00:00:00Z

TED published a talk from Robert Lang titled "Idea+Square=Origami" (18 mins):

He talks about art and it's relationship to math and engineering; its really quite amazing where he goes with it. But even if you hate math or art philosophy, there's something amusing embedded in the talk: an origami Ibex. He has two examples, here and here. He even adds details in the horns.

But just as importantly, Linux source and builds are available for the software he wrote to help him design origami. You find it all from the TreeMaker homepage. If you use it and like it, be sure to thank him! It'd be really neat if someone came up with directions for making an Ibex to keep people busy in the run-up to the Ibex release party :)

George Carlin's rolling in his grave

2008-07-24T00:00:00Z

Wikipedia offers the following definition of etiquette (emphasis mine):

Etiquette, one aspect of decorum, is a code that governs the expectations of social behavior, according to the contemporary conventional norm within a society, social class, or group. Usually unwritten, it may be codified in written form.

The Ubuntu Code of Conduct is our etiquette, codified in written form; it is also universal. It covers:

"behaviour as a member of the Ubuntu Community, in any forum, mailing list, wiki, web site, IRC channel, install-fest, public meeting or private correspondence"

Written in the affirmative, it offers several adjectives relevant to how Ubuntu development should be done. This universality means making rules and interpretations for Planet Ubuntu based on the CoC might reasonably apply elsewhere. Jono Bacon, member of the Ubuntu Community Council and Canonical's appointed Ubuntu Community Manager once stated he felt excited to work at a place where he didn't have to turn off who he was during work hours or separate what he liked from what he did, and we may shortly discover how Jono feels about diminishing that. Despite the clear wording of universality, Fabián Rodríguez has suggested that the Planet needs more formalized rules than the ones inherited from Debian (English only and "don't be annoying") and the CoC. His position appears to be that including objectionable words and phrases, and objectionable ideas are not respectful, and therefore violate the CoC. I respectfully disagree.

Fabián Rodríguez writes:

I don’t expect anyone to change their “WTF” and “STFU” attitude, just leave it outside this project. Setting up a category to carry only Planet Ubuntu posts may help.

And clarifies in a comment:

Although I am brining up the CoC because we have one, I think it is such common sense I am a bit surprised I even got comments on IRC asking what is wrong with WTF’ing here and there or A**holing now and then. Nothing really. But take it elsewhere. And I’ll gladly meet you there, but it won’t be under my @ubuntu.com hat.

It should be obvious that sending harassing, demeaning or confrontational email to Ubuntu or Debian or any other developers isn't suddenly okay if you didn't use @ubuntu.com as the From address. Similarly, it shouldn't matter whether you tag a post with "ubuntu"; if you act or write from a position within the community, and the audience associates you with Ubuntu, you should follow the guidelines as an ambassador of Ubuntu to the larger Linux community, or at least make a note that you are not acting or writing as a member of the Ubuntu community in cases where it might not be clear. At any rate, if you are an Ubuntu Member, then, you should be worried when someone tries to redefine the Conduct you agreed to follow.

The logical conclusion of Fabián's position is that to be respectful nothing that sets off anyone's triggers may pass through Planet Ubuntu's gates. This is a dangerous place to be: one can think of dozens actions that might be considered offensive to some. As Jordan Mantha eloquently put it:

trying to legislate morality is both undesirable and incredibly difficult for the Community Council to do. They are trying to represent a community made up of people from nations and cultures all over the world, and it’s essentially impossible to satisfy both the moral sensibilities and personal liberties of everybody at the same time. I’m also fairly sure it is neither their right nor their charter to tell people what is and is not offensive.

There exist a number of social, religious and political taboos that various cultures may find offensive. It feels weird being an American (land of assimilation) calling a Canadian (home of multiculturalism) on this. If Ubuntu, "Linux for human beings," demands that writing obey one viewpoint, it potentially offends another one as censorship. For example, most of us may see a Tibetan language translation of Ubuntu as progress in bringing Free Software to people who need it, but to a few the act may suggest an anti-China political statement, akin to adding a Confederate flag to the distribution. The Code of Conduct is fortunate to say nothing about such dry powderkegs. As long as we can hold beliefs, disagree and still obey they Code of Conduct's demands for consideration, respect, collaboration, and consultation, there should be room in Ubuntu for all of us.

Chilling effects

Attitude is one thing; I think RTFM or STFU are rarely productive statements. But what's appeared goes far beyond that. Fabián prefers that people who don't agree with his flexible interpretation of "respect" go away. I know at least one guy who does. He still contributes to the development of Ubuntu, but what he has to say is less often heard because despite his reputation, he's quite willing to implicitly comply with the infantilization of the Planet and rarely tags posts our way. It's fortunate that it's easy to include individual RSS feeds in Liferea directly, but if I don't know there's an amount of self censorship going on first, I simply lose that insight, no matter how germane it is to Ubuntu, Free Software or the communities that surround them.

But even if we were to agree that some topics are a bridge too far, specific words and phrases found offensive by some have no clear relation with respect, and already are within contemporary conventional norms when used in moderation. In fact, this entire enlightened discussion would have come across as condescending rather than conversational if the language were to be sanitized; by using that language the author communicates that the audience is a peer, which is central to the point. Self-censored writing feels inauthentic. In that thread, the author comments:

I haven't been particularly active in the Ubuntu community (my first introduction to the open source world), largely because everyone is so damned polite all the time, and as a result the discussions seem fairly dry and limited to technical topics.

This is a disappointing failure to integrate, especially since Rhythmbox needs a lot of lovin', and I'd be happy to see Ubuntu play a foundational role in making that happen.

Doing something constructive about it

One thing that can be done is to offer editorial advice ahead of publication. I've often wished to have a few trustworthy people preview my work on this blog and offer suggestions the way kuro5hin does before going public with a writing. It's a bit sad that the advent of blogging software led to the downfall of community driven writing like k5. Stephan's writing comes across as a bit... "stream of thought," and perhaps a round of editorial review can create something more effective at communicating his ideas and getting people to agree with him. I suspect such offers will be treated as censorship, though a good editor offers only advice, not orders.

Since people are seeking, among other remedies, the removal of Stephan's blog from the Planet, I thought I'd do them a favor. As far as I can tell, the current Planet software doesn't implement filtering (the Venus branch might, but will it support queries?), but the entire Planet software is easy to duplicate, and it's output is easy to manipulate. Here's what the Planet looks like without Stephan Hermann. And for comparison, without Fabián's blog. You can find the construction of this relatively simple construction here. If that doesn't float your boat, I've also constructed a simple dirty words filter. Feel free to customize, the defaults come from the expert on the subject. I've also considered running an alternative, unofficial planet similar to Dave Airlied's, but I'm not sure it's possible without coming across as arrogant or causing hurt feelings.

Finally, the Community Council has this topic on their agenda, and if it doesn't get tabled for lack of time, will be heard at the next meeting. If you can't attend, there are logs available for all such meetings on irclogs.ubuntu.com. In the spirit of being collaborative, it seems relevant to invite Emma Hogbin's opinion, as it seems the language of her lecture that started the mess Fabián and Stephan find themselves in now. As an invited speaker to the now canceled Ubuntu Live! event, decisions rendered would clearly affect her future participation with the Ubuntu community.

If you take one thing away from this essay let it be this: Booting members from the project is in no way, shape, or form "collaborative", and should be taken only when all reasonable measures have failed.

Ubuntu advantages

2008-07-20T00:00:00Z

There are some advantages to running Ubuntu I can think of:

apt-get
six month release cycles
at a very low cost
deep access to the process that creates Ubuntu
a large community with an established etiquette

But not among those is "never search the internet for drivers again". There is a perception that this is special to Ubuntu or Linux. Oh, and also that it's true. Which is why the Ubuntu UK podcast mentions it.

The problem, as I've lead you to believe, is that it's not true. Linux does support a lot of hardware. Probably more than Windows, certainly more than Apple. Greg K-H says people want Linux kernel source / binary compatibility for the driver support. What Linux doesn't do so great job of is supporting new hardware. Lets say you run out and build a new computer which includes a Gigafoo motherboard, with a pq43 chipset, which only hit the market recently and you want to install Ubuntu 8.04. You can't. (Specifics changed because my source is unavailable for details at the moment.)

Unsurprisingly, the hardware is only supported recently, and the 8.04 kernel is based on an older kernel. 8.10 will probably support this out of the box, but not 8.04. Technically, 8.04 LTS might be slightly different. But otherwise, there's a six month gap in Ubuntu's process where you might fall into unsupported territory (see? process does matter!). You certainly can't go to the vendor's website and download some drivers to make it work.

You could build a new kernel with the new drivers you need, but good luck finding where they are and whether they're ready yet. Someone came into #ubuntu-kernel today asking why his upstream kernel didn't build and the short answer is "it doesn't." The long answer is "you need to update your local copy, and if it's still broke get in touch with upstream about it." So even though your hardware might not be supported, you can still download something and make it work. If you build your own kernel, know C well enough to debug FTBFS and don't mind losing support from Ubuntu kernel devs. So in a malevolent sense, the Ubuntu UK guy was right -- not only do you (in posession of newer than six month hardware) need to download and build drivers from the net, most of you straight up can't. Somehow I don't think they quite meant it that way.

This isn't specific to Ubuntu or Linux. Microsoft carries a lot of drivers, but recognizes that they have to support new hardware released after them. Hence in Windows XP you have a floppy disk to provide the SATA drivers that XP didn't have the foresight to include. There appears to be a way to do this but it's not documented, and historical sentiment (fun drinking game: drink every time someone mentions Ubuntu in a Debian political debate) tells me they're not interested in helping out some poor guinea pig install Ubuntu.

Going forward, technologies like DKMS and practices like stable release updates can breath some life into this problem, but it's every bit social as it is technical. I think people have a bit of a right to be irate when they're promised something that isn't true, and I hope Ubuntu Marketing and everyone else understands this and agrees.

Where is Ubuntu Title used?

2008-07-17T00:00:00Z

The following up on the ArtTeam mailing list today:

I always assumed it (the font) to be a convenience in order to display the brand consistently. I've never really thought of using it for anything other than "ubuntu". Are there any examples where it is used for other purposes?

I came up with a few answers, but I'm sure there's more out there. So I'm asking my readers: Where have you spotted the Ubuntu titling font?

I think as an experiment in font usability, I'll switch all the system fonts on my machine :-)

Ubuntu is not perfect

2008-07-16T00:00:00Z

Nick Ali, author of the Ubuntu Weekly Newsletter (among other things), writes that Ubuntu doesn't get a fair shake in a Pingdom report on update site availability. Microsoft had a measured 100% response to pings, Apple a 99.9%. On the subject of Ubuntu it says:

Ubuntu on the other hand came off worse, with only a 98.64% availability for its main repository. That is a total of 1 day, 5 hours and 45 minutes in the three months of this survey.

This brought out an ugly side of Ubuntu Marketing: the self-appointed Defenders of Ubuntu, ready to strike down all perceived slights against it in forums, blogs, and wherever else informed discussion rears its ugly head. Nick Ali (not a Defender, and out of character today) is apparently upset that the headline is misleading (would "Microsoft, Apple trounce Ubuntu" be sufficient?). 'Ed Vim' comments to the Pingdom post itself, saying:

I find this article to be a bit misleading, not only in the title but also the content. The title is just another headline grabber for other sites to use in the typical MS FUD campaign. And let’s face it, a plain vanilla Ubuntu or OS X system with no updates will still be safer and more functional to use than a clean Windows install. This ‘alleged’ update situation is a very minimal issue, As for content, just read the previous comments.

Ubuntu lives in a glass house. You can see nearly everything we do (even the nasty comments on general computing sites). So ask yourself this-- who reports network availability to Ubuntu and why hasn't anyone cited them yet? I have no doubt that Microsoft and Apple have someone looking at this and making sure it's up to par.

People are making out like its unfair that Ubuntu is penalized for a downtime on its archive because of a new release. I agree in one sense, but not completely; the time period over emphasizes the release week and would have been more relevant to Ubuntu users, current and potential, if a six month or year long period were measured instead of the one month coinciding with a new release. But we're only talking about doubling the length of the study. Assuming that the entire outage was contained to release, we'd still be lucky to get two nines.

Still, I don't mind the comparison. Rather, we should feel honored to be chosen as a representative of Linux as a whole, and try to live up to such expectations. One thing not spoken thus far, is that a separate mirror exists for security updates, hopefully so that exactly this problem doesn't spill over into slower security. It would be interesting to see how that archive matches up with the general archive.ubuntu.com availability, but currently both seem to point to the same data center.

And we do have serious downtime problems during this time period that no amount of amortizing can undo this semi-annual anomaly. Mirror admins jokingly compete to see who's got the biggest pipe during release weeks. The main archive is known to be unusable during this time, and the website itself typically reverts to a low overhead, high information view to compensate for the rush. Some clever people decide to upgrade a few days before release to avoid the rush (if only they'd do it before freeze, we'd have more testers!). If we seriously intend to fix Bug Number One we need to hold ourselves to the same level of accountability that we do Microsoft or Apple. The existence of unofficial mirrors isn't an excuse, and in fact may be a security hole. We have talented people and source code, but without the willingness to accept how we look in the mirror as truth, we cannot truly improve.

So the next time you encounter criticism of your favorite software, I ask you to please take a moment to ask yourself two questions: Is it true? and How can I make it better?

The Ubuntu Canvas

2008-07-12T00:00:00Z

Dear Lazyweb,

Can anyone give me statistics on the following Ubuntu userbase properties:

distribution of screen size, resolution, and DPI
distribution of virtual and physical color bit depth
subpixel rendering usage

A reported statistic from the Hardware Database or popcon or something automatically reported would be handy, but inferred statistics like Ubuntu.com or Launchpad visitors and stand-ins like PC users in general are also valuable for validation. I think it's important to know where current users stand in relation to users in general.

This information seems valuable for discussing things like legibility of default fonts in Intrepid and so on. So please share any information!

Cellwriter - My package of the whenever I feel like it

2008-07-09T00:00:00Z

As promised, I've put together a short screencast demoing Michael Levin's handwriting recognition tool CellWriter:

(Link for those who don't see a video above)

The premise is that you write in cells so the tool can do individual letter analysis. You must train it on your handwriting for now -- no default profile is provided. This means substantial work for Latin-based language speakers, and insurmountable challenge for Asiatic languages.

More useful than you'd think

You can use it with a regular mouse, but Cellwriter is intended for use with stylus inputs, like tablets or touchscreens. The software consults with a number recognition engines to determine the input to a given cell: a stroke preprocessor (to accommodate different stroke orders and digitizer aliasing), average distance of the input from sample input, average angles, and frequency weighted word context.

Tablet input might seem like a slow and silly idea, but it's useful in a number of places:

Devices without a real keyboard. Nokia internet tablets come to mind.
Settings where a laptop screen between you and another person forms an unwanted barrier. Classrooms are a good place where an LCD display can be a barrier between you and the lecturer and the chalk board, and a distraction to those around you.
Text input to drawing applications. Sometimes you want to add some nicely rendered text to a drawing, but don't want the hassle of rotating the screen to get access to the keyboard.
Airplanes. Coach is a crowded space, and laptops sometimes fit poorly. Tablet input can reclaim some of the space the airlines keep taking from you in the name of efficiency. I've yet to test this theory, however.

Patches Welcome

CellWriter is a young project, and there are still a number of areas where Cellwriter needs more help:

Default trained database. For some languages the sheer number of characters is overwhelming, and attempting to train for an individual is farcical. Creating and using default data set would improve immediate usability for both Latin and non-Latin languages.
Internationalization. Levin is reportedly working on gettext support, which is an important first step. From there community translation services like Rosetta can aid in bringing the tool to a wider audience.
Librarification. Currently CellWriter is a GTK application; there is an open invitation for volunteers to split the code into a reusable library, as well as a Qt UI for KDE. This could be very useful, if say you wanted GNOME Sudoku to support handwriting input, or a KDE native UI version.
Bug fixing. A number of bugs still exist in CellWriter; if you have experience with GTK and struts, this bug could use your expertise. Additionally, the video above highlights a number of minor UI flaws, some of which I'm told are going to be fixed in the next release. Any bugs, patches or feature requests can be reported at the Launchpad Project page, or on an impromptu forum as bug tracker on Risujin.

Get it today

You can easily install CellWriter in Hardy or Intrepid:

Or you can install it in Debian with

apt-get install cellwriter

Thanks to Michael Levin for his continuting work and to UROP for making it all possible!

Dear UbuntuStudio

2008-07-03T00:00:00Z

I know there's tons of applications of the last post, but I thought there wasn't much in the way of GPL'd implementations. Today, I saw this awesome screenshot, from the audio-visual program CLAM:

The above comes from their Qt based Network Editor application. This is hardly unique among audio applications -- I recall seeing graphEdit years ago on Windows. But I think perhaps there's some great underlying functionality ready to break out for wider uses. Ohloh suggests that it's a mature application with lots of developers, but few comments. CLAM is one of Google's 2008 SoC projects, and some interesting new functionality is on the way thanks to that. One project in particular sounds as useful outside the audio domain: Network Scalability, by Natanael Olaiz (mentor, Pau Arumí). Basically, writing the infrastructure to allow sub-networks, and sub-sub- networks etc. He appears to be making good progress, and I hope he succeeds!

(This is why I always say everything you've ever thought of has already been done on the Internet.)

The popular emergence of apt-git?

2008-06-27T00:00:00Z

It's no secret that Canonical is a large proponent of Bazaar (bzr) and would like to use Ubuntu as a guinea pig for large scale deployments. At UDS Prague, James Westby gave an interview about using "distributed version control systems" (DVCS) for coordinating development. The interviewer is a bit confused about how the Ubuntu flavors interact, so I think an explanation of DVCS and Ubuntu development is in order.

When talking about open source software in general, it's important to keep in mind the concept of versions. Each patch applied to a project can be thought of as a new version of the software. Generally projects release a new version every few months containing a bundle patches. Here's an example based on Xournal:

Xournal turns out to be a pretty good scenario so I'll keep returning to it. Time flows from left to right (but not necessarily to scale). Note the time span between 0.4.2 and 0.4.2.1. There was a pretty critical crashing bug that was caught, patched and released there. However, few new users should use Xournal directly from source rather than go through their distribution. For example, Debian provides packages for Xournal. A picture is enlightening. NOTE: This is not meant to encompass the entirety of Debian's release process:

Several things are going on here. I've organized the picture into three rows: upstream in blue and Debian unstable and Debian testing in red. Debian packagers take upstream releases, add control and build rules, and effectively patch in a debian/ directory. New versions start out in unstable, and after a duration in unstable without serious bugs, that version replaces the old version in testing. The purpose of this is to catch the sort of bugs we saw in Xournal upstream before they cause widespread grief.

Another thing to look at is the June 3rd version in Debian unstable. It has two versions pointing at it. This is because it inherits both the changes from upstream, and the changes that were made in packaging the program for Debian. It is possible that some changes in one source conflict with another. This might happen because Debian patched a bug fix one way, while upstream applied a different fix, for example. In this case the solution is to drop the patch, but sometimes the fix is more complicated. This is what I call the package merge problem in distributions. This is essentially why Debian maintainers hate source code with a debian/ directory provided, since it can cause a conflict with their own packaging efforts.

If the above diagram was confusing, the one below may make you physically dizzy (and it doesn't even contain SRU or Backports!):

That's how Ubuntu works. Blue is still upstream, red is still Debian unstable, and Ubuntu now is gold. Feisty, Gutsy, Hardy and Intrepid are all Ubuntu stable releases; the *buntu flavors all put their packages in these same repositories. They largely all share the same kernel, the same X11 server etc; they just install different packages from those repos by default. This diagram doesn't depict a Hardy version of Xournal because it was a sync from Ubuntu itself--Ubuntu just copied the source from Gutsy into Hardy and rebuilt the whole shebang without really looking at it. Many packages in the "universe" categories do the same thing, except from Debian; there is a script to do this automatically as long as Ubuntu has made no changes to the package in the last release.

This diagram traces the Ubuntu Xournal package to its Debian origins, which are a long ways back. Here, notice that only Feisty derives from Debian. Effectively, Ubuntu has forked this package. This is where distributed version control might make cherry picking patches simpler for Debian, Ubuntu and even Xournal, who's author has reasonably lamented the long lead times. DVCS like bzr or git allow everyone to easily share individual commits within a branch with one another. Debian and Ubuntu source code often seems hidden behind walls, smoke and mirrors; DVCS also provides a commonly understood method of accessing it to upstream, who may want to investigate a heavily reported bug that only Ubuntu users seem to report.

In addition, DVCS also makes it easier for new contributers to participate. Current practice for people without upload access is to grab the source package with apt-get, apply a fix and then generate a "debdiff", attach it to a bug report in LP, and subscribe the appropriate teams. Then the maintainers view the debdiff, apply it, and upload to the build system for deployment. DVCS can make all that happen faster, potentially allowing groups like MOTU to work their magic on more packages and bugs.

So distributed version control is important; Is bzr the right pick?

We might want to know what's popular right now, since DVCS exhibits network effects. Romain Francoise has a very timely graph of Debian package version control system popularity:

SVN is the runaway winner here, but for both Ubuntu and Debian's sake, I don't anticipate this to last. SVN doesn't do enough to help developers with the merge problem. Git is quickly on the move, while bzr is standing still when it comes to adoption. The elephant in the room in this graphic is packages with no version control whatsoever. I imagine that widespread DVCS in Ubuntu is expected to lead to more adoption within Debian. The graph also doesn't distinguish between packages who keep the whole source code in revision control versus those that only keep the debian/ dir.

That report comes on the coattails of discussion at FUDCon on Fedora VCS selection and a rather sad commentary from a Fedora developer (Lennart Poettering):

Yes, with CVS, SVN and GIT I think I have learned enough VC systems for now. My hunger for learning further ones is exactly zero. Let me just code, and don't make it hard for me by asking me to learn your favorite one, please.

Fedora making a unilateral package VCS decision could have consequences downstream. A short conversation can be seen on Planet GNOME about the possibility of moving to Git for code hosting, causing some new edits to DistributedSCM in the meantime. The debate over VCS is not new but Keith offered an insightful gem:

[M]ost of the group will just not bother, and will end up choosing essentially randomly, with a slight bias to whatever is most familiar

This fits well with the Debian VCS-* graphic -- most packages choose nothing, or SVN. Well, if formats matter to enlightened despots, how does bzr stack up against DVCS champ git? To figure this out, I've asked my friend Andy Rueder, who spent a good deal of time digging into git and documenting commands and options:

< aeruder> i've always been a big fan of git due to the simplicity of the repository and the fact that the kernel guys are using it

< aeruder> git is very simple as far as repository format goes... every file has a sha1 from its content, a tree (directory) contains sha1's of files and sha1's of other trees, and a commit simply contains a message, and the sha1 of 0 or more parents and the sha1 of the tree associated with it

What I think this means is that there isn't a whole lot to change about the storage format, and it shows. Daniel Stone, former Ubuntu X maintainer and deadly code ninja, has said of bzr:

bzr fails because I try to clone stuff, and it says, 'oh, you have 0.13, we changed the revision yet again and you need 0.18, but good luck finding anything newer than 0.15'.

This is probably the greatest point I've seen against bzr so far. I haven't personally used either bzr or git seriously, but my impression from those who have is that bzr is still difficult with version compatibility. It seems that, if Bazaar is to be king, "stable" should be shortly added to the front page list of bullet points, so distributions can get tools into the hands of developer-users that aren't hopelessly outdated.

Hello Planet Ubuntu!

2008-06-26T00:00:00Z

Allow me to introduce myself. My name is Justin Dugger and I've been participating in Ubuntu for a while now. First as a tester and now as bug contact and (some) development. Most of what I do is request information from bug reporters and keep friendly relations with upstream. There's a thread on -devel-discuss about making Ubuntu's contributions more visible, and at the moment I think that's exactly backwards. We should be giving out care packages to the heroes upstream that make the bulk of our releases!

Some upstream recent projects I'm currently interested in and would like to thank for their efforts thus far:

Nouveau: For all the (what I shall generously call) activism about closed binary modules I've been reading recently, there's been little fanfare about this project. At the very least, I've seen dramatic improvement over the "nv" driver here, though I imagine other may have different experiences.
CellWriter: There are a few bugs that need to be sorted out, but ultimately, this is a fantastic step forward in Linux handwriting recognition. I should make a screencast demo of this at some point.
Compiz: Colorfilter is a great but obscure plugin that started life as an Google SoC project mentored by Ubuntu and is now in Ubuntu and Compiz upstream. At its core is a set of Cg filters for color transforms with a number of uses. It has filters for high contrast, "inverting" colors, and filters that can remove the colors that the colorblind can't perceive, so you can test your software or presentation.
Xournal: Another tablet oriented program, it lets you keep a high resolution written journal to take notes, make diagrams, whatever. I've found it useful for quick diagramming in technical conversations.
Debian Games Team: This team has been instrumental in keeping Debian and Ubuntu on top of new games and keeping games on top of new build environments. If you're an Ubuntu user looking to contribute upstream, this seems like a good place to start. My particular favorite is Gunroar, and it wouldn't have happened without serious effort from them. Thanks!

If you benefit from any of these (or others), I encourage you to take a moment to offer gratitude. Enthusiasm and praise are the grease on the wheels of Free Software.

Aren't we forgetting someone important here?

2008-06-23T00:00:00Z

Greg K-H leads another charge against the windmills of closed kernel modules. Just who are we supposed to expect to be convinced by a statement that they're very bad?

apt-sync

2008-06-21T00:00:00Z

As a followup to apt-rsync and analysis, I should point out that the people on IRC who suggested it was already done were clearly right. What's not clear, however, is how this might enter Ubuntu or Debian accepted practice. There's been a lot of debate since apt-sync's work over other ways of compressing packages. LZMA, already seen once on this blog, is a frequent contender. Hardy allows LZMA packaging, and I've seen several mailing list threads about it. Unfortunately LZMA and rsync are mortal enemies, shall we say. I'm not sure, but I think that LZMA has won because zsync makes the CD image size problem worse, not better.

An alternative compromise might be to investigate LZMA & squashfs. Currently this represents one of the interesting problems with the Linux kernel. Almost everyone uses squashfs, and while Greg KH can go around declaring that everyone should want their code in the tree and how there's throngs of developers eager to make it happen, the LKML was fairly clear about rejecting inclusion in 2005. There's been some very recent efforts to bring squashfs into mainline, so there is hope. The trouble for squashfs- lzma is that the squashfs maintainer refuses the patches because he's afraid to risk being turned down by LKML again. So squashfs-lzma is distributed as a patch to a patch, and most distros are wary, if they're even thinking about it at all.

But assuming we could use LZMA on squashfs, that would leave Ubuntu more free to ship packages built with an zsyncable package archive, but would still leave a conflict between network bandwidth and disk space.

A small victory

2008-05-28T00:00:00Z

During a search for some open source software, I came across a curiously licensed project. A brief snippet:

You may copy, distribute and use the Software according to the Gnu Public License (GPL) as long as you agree to and comply with any and all conditions in this license.

Limitations

You may with this license only use the Software for non- commercial purposes. [...]

A bit saddened by the missing freedom 0, I wrote the author merely asking for clarification, and in response he's declared intent to relicense under GPLv3. This is only a small victory, as it doesn't currently build for Linux, but it has in the past and I think getting it to build again is a proper reward for this positive step.

Revisiting apt-rsync

2008-05-26T00:00:00Z

I found a fairly thorough review of apt-rsync and where it stands as of 2002. It's six years old but still relevant. An interesting point it makes is that apt-get update could be handled with rsync cheaply.

For now on, I think I'll refer future advocates to that post and ask them to come up with hard numbers for costs incurred. It's important to change the status quo, not for change's sake alone, but for the improvements it brings.

A rebuttal to apt-rsync

2008-05-09T00:00:00Z

Seen on #ubuntu-devel today:

[hi5] Re:rysnc method driver for apt. The context of this can be anything from people over sat connections, 3rd world countries (like iraq where there is no fiber, or copper backbones/lines since the ground is so hard it's almost impossible to run it so everything is expensive sat. based), or saving bandwidth / speeding up updates 1000x for all users / saving money to run repos. Are there no thoughts?

A mess of counter opinion was offered, and some feelings were hurt. I see this pretty often -- someone has an idea to help some people (usually themselves included) and when people shoot the idea down and offer alternatives they get outright angry. Sometimes people invest too much emotion in the solution instead of the problem, I guess. Similar things happened with Automatix, even though Automatix really did stupid things that really did cause some upgrades to break and Ubuntu worked very hard to come up with (I think) suitable alternatives.

[ andrew___] hi5: I think you need to be reading a book on statistics rather than programming, actually. If you can show that debiff will reduce bandwidth by X% and increase CPU usage by Y%, we can have a proper debate about whether it's worth it. Until then, we're all just hand- waving.

With that in mind, I'd like to suggest a quick demonstration of why rsync fails at it's proposed goal of saving bandwidth. Rdiff is basically a binary patching system based on rsync. It should be a good way to measure how much data needs to be transferred. In order to give rsync a fair shot, I'm going to pick two versions of a package and rdiff them. I'll pick them to be close in version, one version bump, if you will. Browsing my apt cache the first satisfactory package to hit my eye was xulrunner (the package causing many people angst with firefox randomly terrible disk performance):

jldugger@jldugger:~rdiff $ ls -lh

-rw-r--r-- 1 jldugger jldugger 7.4M 2008-05-08 22:44 xulrunner-1.9_1.9~b5 +nobinonly-0ubuntu3_i386.deb
-rw-r--r-- 1 jldugger jldugger 7.4M 2008-05-08 22:44 xulrunner-1.9_1.9~b5+nobinonly-0ubuntu4~8.04.0mt1_i386.deb

So it's a fairly big package, plenty of room for rdiffrsync to identify any easy bandwidth savings. Any network overhead should be recoverable on a package this size. First make a signature:

jldugger@jldugger:~rdiff $ rdiff signature xulrunner-1.9_1.9~b5 +nobinonly-0ubuntu3_i386.deb xul.sig

jldugger@jldugger:~rdiff $ ls -lh xul.sig

-rw-r--r-- 1 jldugger jldugger 44K 2008-05-08 22:46 xul.sig

Then make the actual diff:

jldugger@jldugger:~/rdiff $ rdiff delta xul.sig xulrunner-1.9_1.9~b5 +nobinonly-0ubuntu4~8.04.0mt1_i386.deb xul.diff

jldugger@jldugger:~rdiff $ ls -lh xul.diff

-rw-r--r-- 1 jldugger jldugger 7.4M 2008-05-08 22:47 xul.diff

And there you have it: rsync saves no bandwidth at all for .debs. Moreover, these are the two CLOSEST versions. It's not hard to imagine why this is: the .deb file format is an ar archive of two compressed tarballs. The tarballs are going to mostly look like random data, and change size. Just in case minor changes didn't totally screw over rsync by making seemingly random changes all over the tarball, gluing the two tarballs together is going to shift most of the archive around and raise hell as well.

At this point it's appropriate to introduce a lecture (mp3 version) by the author. It's really quite enlightening, but I'll highlight the important bits for the lazy:

the remote update problem is basically: you have two computers connected by a very high latency, very low bandwidth link... a typical Internet link, at least if you're in Australia. So, a piece of wet string, a really pathetic link... and you've got two files. [...] You've got two lumps of data; one sitting on one of the computers and the other sitting on the other computer, and you want to update one of the lumps of data to be the same as the other one.

Basically the rsync program works fine on compressed files, the actual binary works fine, but the rsync algorithm is not very efficient on compressed files. [...] gzip uses dynamic Huffman encoding, which means if you change one byte in the file, everything after that point in the file changes in the compressed data. Problem; that means of course that rsync will be terrible, unless, of course, the change is toward the end of the file. [58m, 31s]

Most importantly, near the end:

yes, handling renames is something I do want to do, partly because of the stuff Steven has been doing with his apt-proxy stuff, where he's actually done an rsync-based apt-proxy system. And look for it; I think it's called apt- proxy, isn't it Steven?

Apt-proxy essentially concludes the same thing, and restates a common hypothesis that rsync is CPU intensive. So unless you can convince gzip to adopt an rsync amenable compression, the apt-rsync is on life support. There are other ways one might go about this, but rsync isn't it today. I leave you with a bit of hope:

There are always more efficient algorithms than rsync. If you have structured data, and you know precisely the sorts of updates, the constraints on the types of updates that can happen to the data, then you can always craft a better algorithm than rsync.

More raw events

2008-05-07T00:00:00Z

If you're interested in Linux and tired of shitty "top 10 commends" on Digg, recycled articles and news filters like Slashdot, here's two interesting things I've found in the past few days:

FOSDEM 2008 videos -- a developer conference to present new software and technologies. That page has the past 4 years worth of videos

Ubuntu Open Week was last week. There was a bit of last minute schedule changes so some of them were short on good questions. Often they seem like someone cutting and pasting a wiki page, which is unfortunate. It might be time to revisit the idea and tweak some things.

I love the internet

2008-01-11T00:00:00Z

They say that open source didn't really take off until the internet was popular, and open source probably milks it for all it's worth. But sometimes the Internet isn't enough, and several conferences have grown from it. Ubuntu their Developer Summits, O'Reilly hosts OSCON, and some enterprising guys in Austrailia host Linuxconf.

Linuxconf 2008 is coming up shortly, and one of the reasons I like it is that unlike the other conferences, it takes the Internet to heart. Nearly all lectures are recorded and made available for poor / lazy folk like me to view at my leisure. It's a great way to kill a day or two. Several look promising:

Dave Jones will be giving a lecture on what he does I guess
Matt Garrett will be giving a talk on suspend to disk. I wish he'd give a lecture on open source bioinformatics tools available some time
* Dave Arlie has a talk scheduled about open source 3d drivers
A pair of debianites are going to talk about why big companies don't like Debian
* A talk on setting up an apt repo
A security lecture on Debian Security
* open source animation tools demo by Bdale Garbee's daughter Elizabeth
MPX - multi pointer X. for those with tablets

There's tons more. If you live in town, I envy you.

Linux Fingerprint Developments

2007-11-15T00:00:00Z

It's been a busy week for Fingerprint Authentication in Linux. Timo announced that pam_thinkfinger is being retired, and a new project will supposedly arise from it. So it's effectively being abandoned. Normally I'd say it's a bad thing, but it's a solid driver at the moment, and most of the problems arise from the PAM module. Requiring uinput has been a source of pain as it's partly considered insecure. The good news is that Thinkfinger is a whirlwind of activity at the moment preparing for a final release, reviewing and integrating patches that have been floating around. Hopefully upstream won't die as much as branch.

Additionally, theres a new Ubuntero pushing on including BioAPI in Ubuntu. He seems a bit of a novice though, so don't hold your breath. I don't quite see the purpose in pushing BioAPI exactly, but if it works, great for everyone.

Finally, fprint announced an initial release this week. They aim to create a generic library with which they can support all kinds of fingerprint hardware. Unsurprisingly, security is "of interest, [but] not a primary objective". That's right. A security tool who's primary objective is not security. Mostly this is an artifact of the contradiction of using fingerprints as security devices -- they're not as secure as a good password, but they are slightly better than auto login as root.

Kernel Ninja

2003-10-31T00:00:00Z

It's been a while, but I've gotten back into a tweaking mood. Since the last posting I've managed to recover to a Debian build, build a working NAT to share the cable modem, and left for school in August. Linux has held up fairly well; the dorms have started to get very nervous about windows install insecurity. A friend of mine accidentally left the network cable plugged in when he decided to remove XP for win2k, and got blaster as a result. They've had him shut off for a week now, partially because the tech they sent out didn't actually remove blaster. Why they don't just block the port in the first place is beyond me. In contrast, linux grabbed an ip lease early on, and hasn't had any problems since (that weren't my own doing).

So earlier this week I decided to enroll in the graphics course next semester, and having an openGL stack the professor likes lent me the motivation to try attacking the kernel again. For a moment I forgot that I ever built my own kernel sucessfully, perhaps because QNX threw me for such loops when I decided to try it out on the second disk (I was working with some developers to hex edit the boot sector-- eventually I decided that I could do without QNX and its delicious GUI). Anyways I ran into the same old problem with dhcp never getting a lease on my own build; compiling the drivers into the kernel resulted in reversing the aliases, it seems. So what was eth0 is eth1, and vice versa. To fix this, its time to build modules. Then its off to accellerated X and finally GL. Then I can go online and brag about how I have the lowest and therefore least valuable glgears score ever!

Sadly, I was not able to use my picross idea as an AI project; instead I'm working with Thaddaeus Frogley's C++ Fight! framework. I swear, I didn't make up that name. He say's his parents made it up, and I believe him, since they are English, after all ;) So for now, the wx picross project is on the back burner. Interestingly enough, Thaddeus has some stuff with wxWindows on his page, mostly how to set it up in VS6. It's not his fault though, he's spent the last few years working on Grand Theft Auto for XBox and other games, so he gets a get out of DLL-Hell free card.

Oh, its Halloween too. As usual for Kansas, the weather took a sharp cold turn just in time to ruin all the children's day of candy feasting. I remember one time my brother went trick or treating in the snow as a ninja. Well, at least, he was a ninja underneath the down overcoat. Stealth frozen ninja sneak attack!

Tweaking

2003-07-24T00:00:00Z

I once said that the end use of Linux is to customize Linux, after observing my roommate and a friend obsess over little things like which screensavers should appear randomly and which shouldn't appear at all. I've tried to avoid such vanity as tweaking options that only appear when I'm not using the computer, but I have been browsing around lately at some of the various performance tools. I've been toying with hdparm, but it seems its allready maxxed out there.

I've also been trying to recompile my kernel, and I've been getting closer. I've now actually gotten a kernel to boot, and most of the services to run. The hurdle right now is dhcp and dhclient. ifconfig reports the devices as up and running, but the card connected to the cable modem never recieves a lease. As you can imagine, debugging and online support are difficult when you can't get online. To complicate matters it seems I've nuked the old debian install; something about device drivers being overwritten I'm guessing.

Sometimes Debian pisses me off. GAIM has been acting up and crashing lately, so I thought I'd see if there's an upgrade. I know that GAIM does update regularly, several of my friends actually run builds from CVS. The latest version of GAIM in the unstable deb tree is about a month and two releases old. People bitch about Debian being slow to package, this is why. So I grab the 20 meg tarball, and follow the instructions the FAQ suggests while downloading. First problem: ./configure fails on a GLIB test. I've run into these sorts of problems before; usually the hard part is figuring out what the package you need is named. The package search utility on the Debian webpage is supposed to be helpful in this reguard but its usually underpowered. None of the glib results I get back are very helpful. I tell it to search descriptions and try again; the very last result is GAIM itself. From this page I discover the true name: libglib2-dev. They've conviently attached the lib prefixes to all packages reguardless of redundancy. Whispering the true name of the devil into apt-get, allows gaim to configure, build and install. But now the computer beeps every 30 minutes or so unexplicably. It takes me about a day to realize that its GAIM beeping about the online status of people in my list. After reading over the ./configure output again, it seems I'm missing the Audio File Library. Again, the beauty of the Debian package search shines through; search queries can only have a single word. I ask on #debian and someone suggests I should install libao. That doesn't work. The name, it so happens, is libaudiofile-dev. Now this is somewhat aggrevating since searching for "audiofile" returns No Results. Armed with this second library, I trudge onward. Everything appears to be in order now, but I'm sure GAIM and Debian haven't let this victory go unnoticed. Eve now, I'm sure the two of them are independently plotting, awaiting the oppertune time to strike...

And finally, Debian added some more themes to the base GNOME distrobition. These actually make the system not look like amateur crap! I think i've settled on a pretty combination of a few themes.

Birth of a Linux user

2003-01-21T00:00:00Z

Back from break! Tried installing Gentoo after win98 fried itself, now I'm playing with Debian. So far its going much smoother, although it feels odd not knowing whats going on underneath the hood. I gotta say the gentoo user docs are very nice. Time to figure out how to customize and configure X. Maybe sometime I'll try out different colorschemes for this website, like [kuro5hin][1].

[1]: http://www.kuro5hin.org/