Spark FS

I went to a talk in the early 1980s which touched on slow scan television, it was mentioned how coding could reduce the size of the data transmitted. That seemed a neat idea, it's easy to think up such data compression schemes.

In the mid 1980s I had a small piece of software published in Acorn User, this intercepted the file system vectors on the BBC micro, so that what was written to disc was compressed but appeared to applications as normal data. Coincidentally the compression technique I made up was used in MicroSoft Windows .bmp files some years later (presumably it is just obvious).

It was a small piece of software because as a system extension on the BBC there was little memory for it. Irritatingly I got it down to 257 bytes, and subsequently someone wrote in with a change that would make it fit in a 256 byte page. Just one byte but a big difference.

Moving on, 1987 came with a boom in PC compatibles and software from bulletin boards (BBS). Here downloads came in .arc files - the SEA arc format, and its modification the Phil Katz pkarc. I ported the SEA command line arc tool to RISC OS in 1988 (so it would actually be OS 1.2 Arthur), it went ideally with Beebug's Hearsay communication program. In a rushed Thursday afternoon, I simply added RISC OS file attributes into the archive headers.

On reflection my extension to the .arc file was a bad idea, it made RISC OS arc files unreadable on other systems. I should have followed PK arc and added extra information at the end of the arc file where it could be ignored by software that did not understand it.

When RISC OS did appear it brought with it a desktop and "filer" windows for manipulating disc contents. It was an obvious idea to copy this easy to use filer interface for the contents of archives - I called my program "Spark" for "super archiver". Directories were handled as archives within archives. This is the RISC OS "sparkive", or .spk format.

A natural progression was to support other archive types - by now things like zip files were appearing on bulletin boards, Spark became a multi-format archive program. I did a free read only version, as a form of advertising and because an archive tool was essential for many users - this I called "SparkPlug". In these days the icon for Spark files consisted of a picture of a Noah type ark, which made sense, you put things in an ark. Acorn did not like this icon. Credit to Jason Williams for designing the Spark flash icon.

There was a self extracting version of SparkPlug (how could you get it if you didn't already have a copy) written in BBC Basic. RISC OS has never done self extracting archives well, because there are no file extensions, it always ends up with the naive user being told to set a file type. There was a limited command line version called 'barc' for the BBC Micro again in BBC Basic - at the time Acorn had a Viewdata system called "SID" and they needed to make downloads available to people with BBC Micros as well as Archimedes.

When RISC OS 3 was being planned Acorn asked me if I could write archive software to be bundled with it. I wrote down various ideas including "the holy grail of an archive filing system". Acorn operating systems going back to the BBC Micro have always made implementing file systems easy. This idea of writing software for RISC OS 3 did not come to pass (fortunately for me) and Acorn invented the Squash module and application - sources now available here.

At some point Mark Smith got in touch (via letter) asking about code for handling Spark files, since he was planning on writing an archive filing system for them. I helped him slightly and eventually published his program "ArcFS" as disc #34 in 1991. Great credit goes to Mark Smith for being the first person to actually implement an archive file system whilst others talked about it.

There were some problems with ArcFS and time was going by with no solution. I decided to write my own archive file system, which became SparkFS, appearing towards the middle of 1992.

ArcFS went on to be a big success with its new publisher "The ARM Club", the free read only version was ubiquitous on magazine cover discs, due to its small size, being hand written machine code.

SparkFS was a multi-format archive file system. It was modular so that new formats could be added as RISC OS modules. It was written in C. Unlike Spark it could write to more formats than just Spark files, Zip and Tar.

At the time there was a split between users whose hard discs were not big enough and needed to store their work in compressed format and people downloading things from BBS and the like who needed acess to standard formats. Spark was a relatively large and slow program. Computer Concepts produced "Compression FS", a compressed filing system which was very fast, it worked by compressing single files in place. For the typical user CFS was a lot better option than Spark. I was very lucky that the Internet took off and then most users needed easy access to archives. Whilst disc space became much cheaper and the complexities of compressed file systems were not worthwhile.

CFS meant SparkFS had to support a fast format, "directory archives", where every object in the archive is stored as a separate object on disc - on reflection not really an archive but it made sense at the time.

By the end of 1992 I was happy that SparkFS was complete - supporting all the important file types.

When RISC OS 3 did appear, I was playing with it and noticed that when I accessed the contents of a PC partition it behaved as if it was part of the RISC OS file system. Acorn had implemented "image filing systems" to support their PC emulator - a PC partition was a file on disc, as an image file it behaved just like a folder. It was obvious this was a much better way of accessing archives. Until this time all these archive and compressed file systems had been using a special field in the file path invented for network file systems. As long as you don't have to type a file path this is no problem and once you do you will be sorry.

Without any suggestion from Acorn, SparkFS became an image file system. Image file systems allow a trick, on RISC OS an application is a folder whose name starts with an !. With slight tinkering via file system vectors it is possible to make RISC OS treat an image file as an application. This is what the ImageFSFix module does - a nice idea, but my (poor) implementation upset someone a few years later.

In 1997 Acorn invited me to write a piece of compression software for the "Network Computer" - the idea was to make compressed single files behave like normal uncompressed files. I called this project "ShrinkWrap", because files were wrapped in GZip files and were shrunk. You can download this software here. A return to the Acorn user software I had written a dozen or more years earlier. I added support for Acorn Squash and Computer Concepts CFS files.

See also:
Downloading Software on Viewdata (CET)

A Comment on Zip files
Zip files were designed by Phil Katz to support different operating systems, extra information can be incorporated in such a way that the archive remains useful on all operating systems. The mistake I made with the Spark file format can be avoided. When I added writing Zip files to SparkFS I defined the necessary extensions to the Zip format for RISC OS files. Since then (1992) Zip has been the best format to use on RISC OS, better compression and cross platform support.

But in 2007 the maintainers of the InfoZip program for RISC OS (a command line Zip program) pointed out to me I had made a mistake. When writing the length of the "extra" field for RISC OS Zip files I had counted the first four bytes in the 'tag' and written the length as 24 bytes instead of the correct 20. I fixed this. The result is that SparkFS Zips can have an extra field length of 20 or 24, possibly RISC OS Zips written by other software can have a length greater than these.

You can download below the code which SparkFS uses to write and read the RISC OS header from Zip files.

In February 2023, Risc OS Open announced they would maintain an open source version of SparkFS:

Looking to spark your imagination?

In the future this will be the source of the most modern version of SparkFS.

SparkFS 1.46 full version as supplied on disc

SparkFS main module source code

Historical interest
SparkFS read only version
SparkFS manual as .pdf
SparkFS manual in Ovation Pro format
SparkFS Zip header code
SparkPlug 2.26 Multi format archive unpacker. Note that in common with much else this requires SparkPlug to unpack it. If you don't have a copy, download the self extracting version below.
SparkPlug 2.26 Self extracting version. Set the file type to BASIC and just double-click.
SparkPlug 2.29 Self extracting version for 32 bits, Beagle board etc. Set the file type to BASIC and just double-click.
SparkPlug 2.29 26/32 bit version - use 2.26 above unless you need a 32 bit version.

Click to return to software index
Click to return to wiki index

Page last modified on March 30, 2023, at 09:17 PM
Powered by PmWiki