Filing system bounties

I reckon it’s about time to crystallise all this into some measurable progress. As I have a long train journey coming up in the next 2 weeks it should be possible to organise all these proposals into a plan which can get some milestones attached and hence bounties. The shopping list is large, complicated, and daunting, but none of the individual elements are impossible!

So, I searched the interweb, and this forum, and collected the following list. Last call for any others? Note, I’m not interested in implementation detail at the moment, just the feature point that needs changing.
For example 64 bit file extents could be implemented by throwing FileCore in the bin and using NTFS, or adding a new (F++) format to FileCore. The feature point is “64 bit file extents”, the rest is implementation detail.

From comp.sys.acorn.hardware thread “RISC OS VFS replacement”
2GB file limit.
Duplication over CDFS and SCSI drivers between vendors.
Duplication of xxxFSFiler.
Duplication of HForm variants.
File locks.
Full permissions.
Non blocking IO.
Pool of > 255 handles.

General
Symbolic links.
Journalling of changes.
Document existing implementation.

From http://www.acornusers.org/wishlist/
UID on each file.
Hiding of system directories.
Support for 2048 + 4096 bytes per sector.

ROOL forum and wiki
Recover more gracefully from “Filecore in use”.
Fileswitch call to enumerate open handles.
Partition support.
Rewrite portions in C.
64 bit disc addressing (API defined, but not internally used).
64 bit file pointers (requires extension to FileSwitch API).
More drives per filing system.
Arbitrary meta data.
Bigger RAM disc (or one mapped as required rather than a dynamic area).
Preview of any format changes to allow disc recovery tools to be updated.
Unicode filenames.
Representation of the map of a big disc that doesn’t take up gigabytes of RAM.
Bug #194 (internal pointers use top 2 bits for flags).
Single zone fixed discs not being layed out or recognised properly.

Increase the limited number of drives on a file system

Unicode filenames

I’ve edited my earlier post to reflect those 2 additions, and commenced the analysis. One train journey and several Eurovision rounds of voting later here come the results:

First, I just pasted the list above into Excel. Then I coursely grouped them, for example ‘full permissions’, ‘UID on each file’, ‘Arbitrary meta data’ are all in the general theme of meta data not currently recorded. Then I combined related items within groups, for example ‘full permissions’ and ‘UID on each file’ is basically ‘Unix like file permissions’.

Second, I classified each feature by affected component, for example some affect FileCore, others FileSwitch, others FileCore and FileSwitch, etc… The task “Rewrite portions in C” was eliminated as it is not a feature point, but an implementation decision. Some extra tasks were inferred, for example adding partition support requires going back an updating clients to use this new API. I colour coded each component.

Third, I grouped the features by dependencies. For example, the 2GB file limit is not tied to any other activity, whereas 64 bit file extents are. The colour coding from step 2 showed that some groups overlap several components.

Fourth, with grouped features by dependency I scored each task by ease of implementation and benefit to typical users. I’m naturally lazy so anything which is a big win for not much effort should go near the top of the pile. I sorted this list by the mean product of ease x benefit for each of the groups in step three.

Fifth, I then estimated how long I thought each task would take. While it may take people less (or more!) time than me to do, this is just a scaling factor. Absolute time isn’t important. This revealed that the grouping had made some things massively longer than others which didn’t seem fair so I took out some (independent) stuff and spread it evenly into other groups. Three groups were eliminated as bounties as they were islands of only a couple of days of effort each.

Voila.

In each of the proposed 5 bounties below, I’ve normalised my effort estimate to give prospective developers some idea of the size of the job.

The cunning planning step means that the shortest job with the most immediate benefit is of course first, in bounty 1.

Next, some scaffolding is put up to jiggle around the existing system to make is more sensible, basically sliding a drive manager in inbetween. Below the drive manager are the block data devices, these know how to turn requests for groups of sectors into commands on the wire (eg. ATA in ADFS, SCSI, SD cards, RAM, even ATAPI if you want FileCore CD-ROMs). The drive manager worries about partitions and presents physical drives as logical drives to the layer above. Bounty 2 probably provides relatively little end user benefit but is an important stepping stone.

Then comes the biggie. My time estimate suggests the implementor will need to consider whether to apply these changes to FileCore or to start with a fresh media layout. Because of bounty 2 the choice doesn’t really matter since FileCore will still be around and able to talk to existing media, whether it is joined by a whole new media layout or whether it gains a new sub format (F++ or H?) gives the same user outcome after bounty 3.

Some of the new features in bounty 3 are then made available through API extensions and C library functions. Things like having a ‘hidden’ bit in the filer attributes and updating it to support 64 bit lengths also get enabled here in bounty 4.

Last comes UTF8 in filenames and corresponding updates to built in FileSwitch clients that can support this but don’t currently. DOSFS also no longer is an image filing system by bounty 5.

Roughly speaking it goes

1. Low hanging fruit
2. Architecture
3. The big format war
4. Uncloak new features
5. Compatibility with other systems

while there are so many other ways to sort these pretty coloured marbles, since the feature points have been collecting since 2007 there needs to be some vaguely objective Excel driven stake in the ground rather than 4 more years of talking. I have further sub-details on the tasks within the bounties which can be shared, assuming these get accepted as official!

Bounty 1 [factor 1]

1. Document existing implementation.
2. Revise fsbash or equivalent.
3. Remove 2GB file limit in FileCore.
4. Update clients as required, esp. DOSFS
5. Support for 2048 + 4096 bytes per sector.

Bounty 2 [factor 2.3]

1. Partition support.
2. Update clients as required
3. Duplication over CDFS and SCSI drivers between vendors.

Bounty 3 [factor 3.5]

1. More drives per filing system.
2. 64 bit disc addressing (API defined, but not internally used).
3. Representation of the map of a big disc that doesn’t take up gigabytes of RAM.
4. Preview of any format changes to allow disc recovery tools to be updated.
5. Journalling of changes.
6. Unicode filenames.
7. 64 bit file pointers (requires extension to FileSwitch API).
8. Update clients as required
9. Full permissions, detailing UID like Unix.
10. Arbitrary meta data.
11. Duplication of HForm variants.
12. Update for format changes

Bounty 4 [factor 2.5]

1. Non blocking IO with interlocks on files already in use.
2. 64 bit file pointers (requires extension to FileSwitch API).
3. Long long versions of stdio functions.
4. Pool of > 255 handles.
5. Hiding of system directories.
6. Duplication of xxxFSFiler.
7. Symbolic links.

Bounty 5 [factor 2.3]

1. Unicode filenames.
2. Unicode translation service module.
3. DOSFS becomes a fileswitch client on its own. Does unicode.
4. LanManFS does unicode.

Non bounties

1. Bigger RAM disc (or one mapped as required rather than a dynamic area). RAMFS will keep up with other changes in 1-5, and making it larger by paging or other scheme is entirely independent of other changes.
2. Bug #194 (internal pointers use top 2 bits for flags). A few hours of search and replace.
3. Single zone fixed discs not being layed out or recognised properly. Probably implement this in the next few weeks having analysed it recently.
4. Recover more gracefully from “Filecore in use”. Parked, as this could expand into a reentrant filing system bounty, but I think what was intended was just some way for a developer to determine what caused the problem in the first place.

Some questions and observations:

Bounty 1-3: This means making it 4GB?

Bounty 3-7 and 4-2 (the same, different parts of the job?) if the new api (extensions) are non-blocking, then blocking could be allowed on old programs for ease of compatibility. (And a module created to use the API on old systems.)

Bounty 2-1: shouldn’t a partioning tool be a separate job? (It would be useful before RISC OS supports accessing extra partitions.)

Bounty 1-3: This means making it 4GB?

Correct.

Bounty 3-7 and 4-2 (the same, different parts of the job?)

In 3-7 the on-disc format is changed to allow for 64 bit extents, in 4-2 the APIs are extended to make use of this feature.

Bounty 2-1: shouldn’t a partioning tool be a separate job?

I don’t think so, you need to add support for partitions, so you’ll need to create the tool at the same time. The two are linked, and the implementor of bounty 2 is probably best placed to make a tool having designed the system.

Hi Rob. A good bit of spreadsheet bashing there. Maybe next time we meet we could go through this in more detail but from the looks of things, that’s as good a selection of file system-related bounties as any I could think of.

> Bounty 1-3: This means making it 4GB?
Correct.

I should have asked whether this is the ADFS format restriction, or whether it is a restriction that would stop things like FAT32FS from accessing 2GB – 4GB files. (I haven’t been brave enough to try).

[edit: I just tried, the filesizes on FAT32FS and DVDFS both seem correct, but attempting a copy gives a stupidly large number of bytes to go, plus no readable memory at this address]

However, unless this is a low hanging fruit, is it needed?
If a new api is created to allow the use of big files, does the old one need changing?

Also, if it is a fix for ADFS format drives, what would be the implications of using a drive with 2 – 4 GB files on an old system?

Also it occurs to me that the term ADFS is often used when referring to the format, rather than the filesystem, would it be good for ROOL (or better still ROOL and ROL) to suggest correct terminology?

In 3-7 the on-disc format is changed to allow for 64 bit extents, in 4-2 the APIs are extended to make use of this feature.

Wouldn’t this be better the other way round? Then network filesystems (i.e. with non RISC OS servers) could be updated to use it imediately.

(Is it actually a good idea to update the ADFS format, anyway?)

>Bounty 2-1: shouldn’t a partioning tool be a separate job?
I don’t think so, you need to add support for partitions, so you’ll need to create the tool at the same time. The two are linked, and the implementor of bounty 2 is probably best placed to make a tool having designed the system.

The important thing I see about partitioning is interoperability (and dual-boot) with other systems.

Ideally the system would allow an ADFS format partition at the start of the drive, which could be accessed by current RISC OS systems, and a standard MSDOS style partition table, which would mark the ADFS format partition as used (there are several unused partition types), and work as normal on other systems.

It looks like this should be possible. (After experimenting with a memory stick and gparted and scsiform)

Even before partition support is added to RISC OS, it would have advantages.

1. It should allow dual boot RISC OS / Linux on a single drive.

2. It would allow USB drives to be used part with ADFS for
and part NTFS/EXFS (e.g. a 500GB drive currently would be somewhat wasted in ADFS format)

3. If a USB drive were partitioned ADFS / FAT32 and FAT32FS were modified to be aware of this and not dismount the AFDS partition we would have support for two partitions on the same drive without any changes to RISC OS.

4. Anyone replacing a drive would be able to use the tool, and hopefully not need to reformat the drive when partition support happens.

5. It would allow testing of the partion arrangement for backwards compatibility, before any work is done on implementation in the OS.

(Note – the initial tool wouldn’t need to be RISC OS based, because most of the uses imply access to a second system)

Also, if it is a fix for ADFS format drives, what would be the implications of using a drive with 2 – 4 GB files on an old system?

You can’t put in a disc of a format that the system doesn’t understand. It isn’t going to work.

And why would you, anyway?

You can’t put in a disc of a format that the system doesn’t understand. It isn’t going to work.

I have read comments that the 2GB limit on ADFS format is due to a signed integer being used instead of an unsigned one. The implication being that fixing this would double the filesize. However what was not clear is whether this would work on existing formats or require a new system.

And why would you, anyway?

If you had a drive that was used on a select system, as well.
The situation would be much like the current situation with a FAT32 drive and a different OS.

what was not clear is whether this would work on existing formats or require a new system.

Where do you think the bug is?

It’s in the filing system, ergo a new one is required.

If you had a drive that was used on a select system, as well.

Sorry, Jess, that’s a daft suggestion. If you use a disc on multiple systems, it has to be formatted such that all systems understand it.

It’s in the filing system, ergo a new one is required.

If the the format is intrinsically 32 bit, and is only limited by wasting one bit, it would work. Some comments I have read imply this. (However it may be intrinsically 31 bit)

Sorry, Jess, that’s a daft suggestion. If you use a disc on multiple systems, it has to be formatted such that all systems understand it.

But that is precisely the situation we have with FAT32FS, it works fine on RISC OS, unless you put files over 2GB on it.

it occurs to me that the term ADFS is often used when referring to the format, rather than the filesystem, would it be good for ROOL (or better still ROOL and ROL) to suggest correct terminology

I think the origin of the misnoma is from the BBC Master era, when BBC Model B owners were used to Acorn DFS formatted floppies and then had the new ADFS format to choose from with a heirarchy of directories and 10 letter filenames. In that case ADFS was both the format and the thing that made the drive spin.

In RISC OS the on-disc format is maintained by FileCore and it asks clients to do transfers to the physical drive. So RAMFS, ADFS, SCSIFS and so on are all saved in FileCore format.

Bounty 5 [factor 2.4]
3. DOSFS becomes a fileswitch client on its own. Does unicode.

I don’t see how this fits in with the idea of partition support and a drive manager. DOSFS would have to be a client of the drive manager, would it not? As would whatever handles FileCore formatted drives.

Bear in mind also that DOSFS is useful as an image filing system because you can currently take images of DOS discs and store them as appropriately-typed files on your hard drive. CPMFS and MacFS do this too. If you’ve lifted the file size limits, it seems to me the main reasons for moving away from the image filing system API for handling foreign disc formats have gone.

But they ought to be talking to a drive manager, rather than via FileCore: the whole business of FileCore handling its own format and also driving the hardware drivers like ADFS is messy.

Or do you mean that DOSFS would remain an image filing system and so mainly talk to FileSwitch? In which case, I agree.

Sorry: this is all a bit confusing, as image filing systems which handle foreign disc formats talk to FileSwitch and FileCore at present.

Unicode filenames

Might be worth pointing out that even though the Filer could probably be adapted to cope with Unicode filenames, at the moment you’d be totally stuck at the CLI. We might need an interim step that shows unicode-named files in some ASCII-understandable way until such time as the CLI is updated to cope…

Is it 2012 already? ;-)
Any chance of an update, has this been completed?
Would additional funds make this more attractive to finish?
Is there another plan?

This bounty has been claimed, meaning work is underway towards achieving all the goals listed on that page. We’re considering opening the bounty for “File system improvements (Step 2)” so people can register their interest/support development by donating to that one. The only thing standing between us now and opening the Step 2 bounty is us agreeing internally about what should be in the second one.

The biggest problem we face, to be honest, is that the really hard RISC OS tasks, the ones we envisioned the bounty scheme helping to drive forward, realistically represent many man months of work – which, when done on the odd evening, weekend, means they will take years to complete. Unless people are able to take time off work to dedicate a lot more time to the task. However, with the bounties not really fetching very much (in cash per hour terms) they aren’t really succeeding as a motivator. We’d need to see the bounties getting into the several thousand pounds mark for those harder tasks to stand a chance of actually getting done (or someone very skilful with lots of free time on their hands appearing out of nowhere).

Thanks for the explanation.
I’m in the same situation of not being able to dedicate as much time as I would like to getting up to speed on RISC OS.

Just to update the topic. DiscKnight 1.50 now correctly handles the new FileCore variant, allowing files of up to 4GB, the multi-zone disc map and the special partitioning for the Raspberry Pi.

One thing I’ve realised is that the there doesn’t seem to be any mention of TRIM support in the filesystem bounty plans. Probably a good thing to slot in somewhere, now that SSDs are the norm for modern machines.

Probably a good thing to slot in somewhere,

…and proper dismounting? ;-)