A case study in the FAIL metric: Soul-Fu

Fri 29 May 2009

Tom Calloway posts a metric for evaluating open source projects, which I shall dub "FAIL, Assessing Inadequacy Lazily" (FAIL). To help calibrate the scales, I'll provide a case study: Secret of Ultimate Legend Fantasy Unleashed (Soul-Fu).

About Soul-Fu

Soul-fu is the brainchild of Aaron Bishop, original author of Egoboo. They are both dungeon crawler games, with an emphasis on action and influence from Nethack. It's original platform was apparently Windows, but the author justifies some code by claiming game consoles were also targeted. Like Egoboo before it, Soul-Fu is largely abandoned by its creator and the source code provided to all takers on a custom license.

Technology wise, the game includes openGL rendering and cel shading. Shadows are cast from characters and enemies, and equipping armor and weapons changes your appearance in game. Animations involve skeletal motion and are generally decent.

In Aaron's absence, a modest community has organized around the official message board, to mod and hack the code. Mod'ing the game involves loading up a text editor in game, modeled after emacs(!).

So how does this game fare on the FAIL scale? Let's find out.

Applied points of fail in bold, editorial comments in italics

Size

The community SVN helpfully checked in all the game data along with the code to a single project, but it manages to still fit within 100MB

  • The source code is more than 100 MB. [ +5 points of FAIL ]

  • If the source code also exceeds 100 MB when it is compressed [ +5 points of FAIL ]

Source Control

The Soul-Fu community runs their own SVN server, after it was discovered that SourceForge didn't approve of the license.

  • There is no publicly available source control (e.g. cvs, svn, bzr, git) [ +10 points of FAIL ]

  • There is publicly available source control, but:

  • There is no web viewer for it [ +5 points of FAIL ]

  • There is no documentation on how to use it for new users [ +5 points of FAIL ]

  • You've written your own source control for this code [ +30 points of FAIL ]

  • You don't actually use the existing source control [ +50 points of FAIL ]

Building From Source

Soul-Fu originally built via VC.NET; someone later wrote a simple Makefile to get the job done on Linux. These Makefiles have been reverted in SVN, so it's hard to judge this section. To promote clue, we'll say anything not in SVN HEAD doesn't count.

  • There is no documentation on how to build from source [ +20 points of FAIL ]

  • If documentation exists on how to build from source, but it doesn't work [ +10 points of FAIL ]

  • Your source is configured with a handwritten shell script [ +10 points of FAIL ]

  • Your source is configured editing flat text config files [ +20 points of FAIL]

  • Your source is configured by editing code header files manually [ +30 points of FAIL ]

  • Your source isn't configurable [ +50 points of FAIL ]

  • Your source builds using something that isn't GNU Make [ +10 points of FAIL ]

  • Your source only builds with third-party proprietary build tools [ +50 points of FAIL ]

  • You've written your own build tool for this code [ +100 points of FAIL ] In game scripting is done with a custom language, PXSS, and it's own compiler. There's also the conversion of game data to a single file, but I might forgive that here.

Bundling

Soul-Fu was originally published requiring a custom libjpeg. This has been fixed in SVN.

  • Your source only comes with other code projects that it depends on [ +20 points of FAIL ]

  • If your source code cannot be built without first building the bundled code bits [ +10 points of FAIL ]

  • If you have modified those other bundled code bits [ +40 points of FAIL ]

Libraries

Soul-Fu builds against a number of freely available libraries such as SDL, ogg, and openGL

  • Your code only builds static libraries [ +20 points of FAIL ]

  • Your code can build shared libraries, but only unversioned ones [ +20 points of FAIL ]

  • Your source does not try to use system libraries if present [ +20 points of FAIL ]

System Install

The most Soul-Fu has for installing is a .nsi for Windows bundles, and even that doesn't appear to be in SVN HEAD. I wrote debian packing for it, but Soul-Fu itself has a snowball's chance in hell of distribution inclusion.

  • Your code tries to install into /opt or /usr/local [ +10 points of FAIL ]

  • Your code has no "make install" [ +20 points of FAIL ]

  • Your code doesn't work outside of the source directory [ +30 points of FAIL ]

Code Oddities

Honestly, the code itself has a host of oddities that might merit it's own categories, but we're trying to use simple and objective metrics as a stand-in for intelligent analysis.

  • Your code uses Windows line breaks ("DOS format" files) [ +5 points of FAIL ]

  • Your code depends on specific compiler feature functionality [ +20 points of FAIL ]

  • Your code depends on specific compiler bugs [ +50 points of FAIL ]

  • Your code depends on Microsoft Visual Anything [ +100 points of FAIL ]

Communication

Soul-Fu's hub of activity is a phpBB forum. Theoretically, the original author has a mailing list, but I don't think it's available for community use.

  • Your project does not announce releases on a mailing list [ +5 points of FAIL ]

  • Your project does not have a mailing list [ +10 points of FAIL ]

  • Your project does not have a bug tracker [ +20 points of FAIL ]

  • Your project does not have a website [ +50 points of FAIL]

  • Your project is sourceforge vaporware [ +100 points of FAIL ]

Releases

Per the original author's guidelines, official community releases must be approved by a vote. No such vote has happened yet. We'll give it a no release mark and move on.

  • Your project does not do sanely versioned releases (Major, Minor) [ +10 points of FAIL ]

  • Your project does not do versioned releases [ +20 points of FAIL ]

  • Your project does not do releases [ +50 points of FAIL ]

  • Your project only does releases as attachments in web forum posts [ +100 points of FAIL ]

  • Your releases are only in .zip format [ +5 points of FAIL ]

  • Your releases are only in OSX .zip format [ +10 points of FAIL ]

  • Your releases are only in .rar format [ +20 points of FAIL ]

  • Your releases are only in .arj format [ +50 points of FAIL ]

  • Your releases are only in an encapsulation format that you invented. [ +100 points of FAIL ]

  • Your release does not unpack into a versioned top-level directory (e.g. glibc-2.4.2/ ) [ +10 points of FAIL ]

  • Your release does not unpack into a top-level directory (e.g. glibc/ ) [ +25 points of FAIL ]

  • Your release unpacks into an absurd number of directories (e.g. home/johndoe/glibc-svn/tarball/glibc/src/) [ +50 points of FAIL ]

History

Soul-Fu was originally a solo project, intended for commercialization. As such you will encounter stupid code and intentional obfuscation of data objects. The metric may need revision to accommodate the level of peer review code got while proprietary. An additional "original authors abandoned the project when open sourcing it" criteria may be in order, for now we'll call it a fork since the original is still available.

  • Your code is a fork of another project [ +10 points of FAIL ]

  • Your primary developers were not involved with the parent project [ +50 points of FAIL ]

  • Until open sourcing it, your code was proprietary for:

  • 1-2 years [ +10 points of FAIL ]

  • 3-5 years [ +20 points of FAIL ]

  • 6-10 years [ +30 points of FAIL ]

  • 10+ years [ +50 points of FAIL ]

Licensing

The code is licensed under what can only be called a custom "Be Nice" license. I actually asked the OSI to rule on it after the author claimed it was okay and the OSI would approve it. The license itself is hidden within an HTML manual, and generally states that it must remain nagware and noncommercial. It also stipulates that official releases must see community approval. On the bizarre licensing the author has not budged.

  • Your code does not have per-file licensing [ +10 points of FAIL ]

  • Your code contains inherent license incompatibilities [ +20 points of FAIL ]

  • Your code does not have any notice of licensing intent [ +30 points of FAIL ]

  • Your code doesn't include a copy of the license text [ +50 points of FAIL ]

  • Your code doesn't have a license [ +100 points of FAIL ]

Documentation

Soul-Fu code does not self document, and what passes for documentation are some notes the author made while writing Soul-Fu.

  • Your code doesn't have a changelog [+10 points of FAIL]

  • Your code doesn't have any documentation [ +20 points of FAIL ]

  • Your website doesn't have any documentation [ +30 points of FAIL ]

FAIL METER

0 points of FAIL: Perfect! All signs point to success!

5-25 points of FAIL: You're probably doing okay, but you could be better.
30-60 points of FAIL: Babies cry when your code is downloaded

65-90 points of FAIL: Kittens die when your code is downloaded
95-130 points of FAIL: HONK HONK. THE FAILBOAT HAS ARRIVED!

135+ points of FAIL: So much fail, your code should have its own reality TV show.

495 points of FAIL: Soul-Fu.

This is why I have Soul-Fu patches and Ubuntu packaging, but am sitting on them. It would be a drop in the bucket of failure.

Comments !