RISC OS Open
A fast and easily customised operating system for ARM devices
ROOL
Home | News | Downloads | Bugs | Bounties | Forum | Documents | Photos | Contact us
Account
Forums → Bounties →

Filing system bounties

Subscribe to Filing system bounties 49 posts, 21 voices

Posts per page:

Pages: 1 2

 
Sep 24, 2011 7:50am
Avatar Sprow (202) 945 posts

I reckon it’s about time to crystallise all this into some measurable progress. As I have a long train journey coming up in the next 2 weeks it should be possible to organise all these proposals into a plan which can get some milestones attached and hence bounties. The shopping list is large, complicated, and daunting, but none of the individual elements are impossible!

So, I searched the interweb, and this forum, and collected the following list. Last call for any others? Note, I’m not interested in implementation detail at the moment, just the feature point that needs changing.
For example 64 bit file extents could be implemented by throwing FileCore in the bin and using NTFS, or adding a new (F++) format to FileCore. The feature point is “64 bit file extents”, the rest is implementation detail.

From comp.sys.acorn.hardware thread “RISC OS VFS replacement”
2GB file limit.
Duplication over CDFS and SCSI drivers between vendors.
Duplication of xxxFSFiler.
Duplication of HForm variants.
File locks.
Full permissions.
Non blocking IO.
Pool of > 255 handles.

General
Symbolic links.
Journalling of changes.
Document existing implementation.

From http://www.acornusers.org/wishlist/
UID on each file.
Hiding of system directories.
Support for 2048 + 4096 bytes per sector.

ROOL forum and wiki
Recover more gracefully from “Filecore in use”.
Fileswitch call to enumerate open handles.
Partition support.
Rewrite portions in C.
64 bit disc addressing (API defined, but not internally used).
64 bit file pointers (requires extension to FileSwitch API).
More drives per filing system.
Arbitrary meta data.
Bigger RAM disc (or one mapped as required rather than a dynamic area).
Preview of any format changes to allow disc recovery tools to be updated.
Unicode filenames.
Representation of the map of a big disc that doesn’t take up gigabytes of RAM.
Bug #194 (internal pointers use top 2 bits for flags).
Single zone fixed discs not being layed out or recognised properly.

 
Sep 24, 2011 9:15am
Avatar Jess Hampshire (158) 855 posts

Increase the limited number of drives on a file system

 
Sep 24, 2011 9:40am
Avatar Jeffrey Lee (213) 5842 posts

Unicode filenames

 
Oct 5, 2011 7:53pm
Avatar Sprow (202) 945 posts

I’ve edited my earlier post to reflect those 2 additions, and commenced the analysis. One train journey and several Eurovision rounds of voting later here come the results:

First, I just pasted the list above into Excel. Then I coursely grouped them, for example ‘full permissions’, ‘UID on each file’, ‘Arbitrary meta data’ are all in the general theme of meta data not currently recorded. Then I combined related items within groups, for example ‘full permissions’ and ‘UID on each file’ is basically ‘Unix like file permissions’.

Second, I classified each feature by affected component, for example some affect FileCore, others FileSwitch, others FileCore and FileSwitch, etc… The task “Rewrite portions in C” was eliminated as it is not a feature point, but an implementation decision. Some extra tasks were inferred, for example adding partition support requires going back an updating clients to use this new API. I colour coded each component.

Third, I grouped the features by dependencies. For example, the 2GB file limit is not tied to any other activity, whereas 64 bit file extents are. The colour coding from step 2 showed that some groups overlap several components.

Fourth, with grouped features by dependency I scored each task by ease of implementation and benefit to typical users. I’m naturally lazy so anything which is a big win for not much effort should go near the top of the pile. I sorted this list by the mean product of ease x benefit for each of the groups in step three.

Fifth, I then estimated how long I thought each task would take. While it may take people less (or more!) time than me to do, this is just a scaling factor. Absolute time isn’t important. This revealed that the grouping had made some things massively longer than others which didn’t seem fair so I took out some (independent) stuff and spread it evenly into other groups. Three groups were eliminated as bounties as they were islands of only a couple of days of effort each.

Voila.

 
Oct 5, 2011 8:03pm
Avatar Sprow (202) 945 posts

In each of the proposed 5 bounties below, I’ve normalised my effort estimate to give prospective developers some idea of the size of the job.

The cunning planning step means that the shortest job with the most immediate benefit is of course first, in bounty 1.

Next, some scaffolding is put up to jiggle around the existing system to make is more sensible, basically sliding a drive manager in inbetween. Below the drive manager are the block data devices, these know how to turn requests for groups of sectors into commands on the wire (eg. ATA in ADFS, SCSI, SD cards, RAM, even ATAPI if you want FileCore CD-ROMs). The drive manager worries about partitions and presents physical drives as logical drives to the layer above. Bounty 2 probably provides relatively little end user benefit but is an important stepping stone.

Then comes the biggie. My time estimate suggests the implementor will need to consider whether to apply these changes to FileCore or to start with a fresh media layout. Because of bounty 2 the choice doesn’t really matter since FileCore will still be around and able to talk to existing media, whether it is joined by a whole new media layout or whether it gains a new sub format (F++ or H?) gives the same user outcome after bounty 3.

Some of the new features in bounty 3 are then made available through API extensions and C library functions. Things like having a ‘hidden’ bit in the filer attributes and updating it to support 64 bit lengths also get enabled here in bounty 4.

Last comes UTF8 in filenames and corresponding updates to built in FileSwitch clients that can support this but don’t currently. DOSFS also no longer is an image filing system by bounty 5.

Roughly speaking it goes

  1. Low hanging fruit
  2. Architecture
  3. The big format war
  4. Uncloak new features
  5. Compatibility with other systems

while there are so many other ways to sort these pretty coloured marbles, since the feature points have been collecting since 2007 there needs to be some vaguely objective Excel driven stake in the ground rather than 4 more years of talking. I have further sub-details on the tasks within the bounties which can be shared, assuming these get accepted as official!

Bounty 1 [factor 1]

  1. Document existing implementation.
  2. Revise fsbash or equivalent.
  3. Remove 2GB file limit in FileCore.
  4. Update clients as required, esp. DOSFS
  5. Support for 2048 + 4096 bytes per sector.

Bounty 2 [factor 2.3]

  1. Partition support.
  2. Update clients as required
  3. Duplication over CDFS and SCSI drivers between vendors.

Bounty 3 [factor 3.5]

  1. More drives per filing system.
  2. 64 bit disc addressing (API defined, but not internally used).
  3. Representation of the map of a big disc that doesn’t take up gigabytes of RAM.
  4. Preview of any format changes to allow disc recovery tools to be updated.
  5. Journalling of changes.
  6. Unicode filenames.
  7. 64 bit file pointers (requires extension to FileSwitch API).
  8. Update clients as required
  9. Full permissions, detailing UID like Unix.
  10. Arbitrary meta data.
  11. Duplication of HForm variants.
  12. Update for format changes

Bounty 4 [factor 2.5]

  1. Non blocking IO with interlocks on files already in use.
  2. 64 bit file pointers (requires extension to FileSwitch API).
  3. Long long versions of stdio functions.
  4. Pool of > 255 handles.
  5. Hiding of system directories.
  6. Duplication of xxxFSFiler.
  7. Symbolic links.

Bounty 5 [factor 2.3]

  1. Unicode filenames.
  2. Unicode translation service module.
  3. DOSFS becomes a fileswitch client on its own. Does unicode.
  4. LanManFS does unicode.

Non bounties

  1. Bigger RAM disc (or one mapped as required rather than a dynamic area). RAMFS will keep up with other changes in 1-5, and making it larger by paging or other scheme is entirely independent of other changes.
  2. Bug #194 (internal pointers use top 2 bits for flags). A few hours of search and replace.
  3. Single zone fixed discs not being layed out or recognised properly. Probably implement this in the next few weeks having analysed it recently.
  4. Recover more gracefully from “Filecore in use”. Parked, as this could expand into a reentrant filing system bounty, but I think what was intended was just some way for a developer to determine what caused the problem in the first place.
 
Oct 6, 2011 8:40am
Avatar Jess Hampshire (158) 855 posts

Some questions and observations:

Bounty 1-3: This means making it 4GB?

Bounty 3-7 and 4-2 (the same, different parts of the job?) if the new api (extensions) are non-blocking, then blocking could be allowed on old programs for ease of compatibility. (And a module created to use the API on old systems.)

Bounty 2-1: shouldn’t a partioning tool be a separate job? (It would be useful before RISC OS supports accessing extra partitions.)

 
Oct 6, 2011 6:19pm
Avatar Sprow (202) 945 posts

Bounty 1-3: This means making it 4GB?

Correct.

Bounty 3-7 and 4-2 (the same, different parts of the job?)

In 3-7 the on-disc format is changed to allow for 64 bit extents, in 4-2 the APIs are extended to make use of this feature.

Bounty 2-1: shouldn’t a partioning tool be a separate job?

I don’t think so, you need to add support for partitions, so you’ll need to create the tool at the same time. The two are linked, and the implementor of bounty 2 is probably best placed to make a tool having designed the system.

 
Oct 6, 2011 11:35pm
Avatar Steve Revill (20) 1337 posts

Hi Rob. A good bit of spreadsheet bashing there. Maybe next time we meet we could go through this in more detail but from the looks of things, that’s as good a selection of file system-related bounties as any I could think of.

 
Oct 7, 2011 8:44am
Avatar Jess Hampshire (158) 855 posts

> Bounty 1-3: This means making it 4GB?
Correct.

I should have asked whether this is the ADFS format restriction, or whether it is a restriction that would stop things like FAT32FS from accessing 2GB – 4GB files. (I haven’t been brave enough to try).

[edit: I just tried, the filesizes on FAT32FS and DVDFS both seem correct, but attempting a copy gives a stupidly large number of bytes to go, plus no readable memory at this address]

However, unless this is a low hanging fruit, is it needed?
If a new api is created to allow the use of big files, does the old one need changing?

Also, if it is a fix for ADFS format drives, what would be the implications of using a drive with 2 – 4 GB files on an old system?

Also it occurs to me that the term ADFS is often used when referring to the format, rather than the filesystem, would it be good for ROOL (or better still ROOL and ROL) to suggest correct terminology?

 
Oct 7, 2011 9:04am
Avatar Jess Hampshire (158) 855 posts

In 3-7 the on-disc format is changed to allow for 64 bit extents, in 4-2 the APIs are extended to make use of this feature.

Wouldn’t this be better the other way round? Then network filesystems (i.e. with non RISC OS servers) could be updated to use it imediately.

(Is it actually a good idea to update the ADFS format, anyway?)

 
Oct 7, 2011 9:32am
Avatar Jess Hampshire (158) 855 posts

>Bounty 2-1: shouldn’t a partioning tool be a separate job?
I don’t think so, you need to add support for partitions, so you’ll need to create the tool at the same time. The two are linked, and the implementor of bounty 2 is probably best placed to make a tool having designed the system.

The important thing I see about partitioning is interoperability (and dual-boot) with other systems.

Ideally the system would allow an ADFS format partition at the start of the drive, which could be accessed by current RISC OS systems, and a standard MSDOS style partition table, which would mark the ADFS format partition as used (there are several unused partition types), and work as normal on other systems.

It looks like this should be possible. (After experimenting with a memory stick and gparted and scsiform)

Even before partition support is added to RISC OS, it would have advantages.

1. It should allow dual boot RISC OS / Linux on a single drive.

2. It would allow USB drives to be used part with ADFS for
and part NTFS/EXFS (e.g. a 500GB drive currently would be somewhat wasted in ADFS format)

3. If a USB drive were partitioned ADFS / FAT32 and FAT32FS were modified to be aware of this and not dismount the AFDS partition we would have support for two partitions on the same drive without any changes to RISC OS.

4. Anyone replacing a drive would be able to use the tool, and hopefully not need to reformat the drive when partition support happens.

5. It would allow testing of the partion arrangement for backwards compatibility, before any work is done on implementation in the OS.

(Note – the initial tool wouldn’t need to be RISC OS based, because most of the uses imply access to a second system)

 
Oct 7, 2011 10:41am
Avatar Dave Higton (281) 668 posts

Also, if it is a fix for ADFS format drives, what would be the implications of using a drive with 2 – 4 GB files on an old system?

You can’t put in a disc of a format that the system doesn’t understand. It isn’t going to work.

And why would you, anyway?

 
Oct 7, 2011 11:07am
Avatar Jess Hampshire (158) 855 posts

You can’t put in a disc of a format that the system doesn’t understand. It isn’t going to work.

I have read comments that the 2GB limit on ADFS format is due to a signed integer being used instead of an unsigned one. The implication being that fixing this would double the filesize. However what was not clear is whether this would work on existing formats or require a new system.

And why would you, anyway?

If you had a drive that was used on a select system, as well.
The situation would be much like the current situation with a FAT32 drive and a different OS.

 
Oct 7, 2011 11:11am
Avatar Dave Higton (281) 668 posts

what was not clear is whether this would work on existing formats or require a new system.

Where do you think the bug is?

It’s in the filing system, ergo a new one is required.

If you had a drive that was used on a select system, as well.

Sorry, Jess, that’s a daft suggestion. If you use a disc on multiple systems, it has to be formatted such that all systems understand it.

 
Oct 7, 2011 11:50am
Avatar Jess Hampshire (158) 855 posts

It’s in the filing system, ergo a new one is required.

If the the format is intrinsically 32 bit, and is only limited by wasting one bit, it would work. Some comments I have read imply this. (However it may be intrinsically 31 bit)

Sorry, Jess, that’s a daft suggestion. If you use a disc on multiple systems, it has to be formatted such that all systems understand it.

But that is precisely the situation we have with FAT32FS, it works fine on RISC OS, unless you put files over 2GB on it.

 
Oct 7, 2011 6:49pm
Avatar Sprow (202) 945 posts

it occurs to me that the term ADFS is often used when referring to the format, rather than the filesystem, would it be good for ROOL (or better still ROOL and ROL) to suggest correct terminology

I think the origin of the misnoma is from the BBC Master era, when BBC Model B owners were used to Acorn DFS formatted floppies and then had the new ADFS format to choose from with a heirarchy of directories and 10 letter filenames. In that case ADFS was both the format and the thing that made the drive spin.

In RISC OS the on-disc format is maintained by FileCore and it asks clients to do transfers to the physical drive. So RAMFS, ADFS, SCSIFS and so on are all saved in FileCore format.

 
Oct 28, 2011 10:08pm
Avatar Matthew Phillips (473) 400 posts

Bounty 5 [factor 2.4]
3. DOSFS becomes a fileswitch client on its own. Does unicode.

I don’t see how this fits in with the idea of partition support and a drive manager. DOSFS would have to be a client of the drive manager, would it not? As would whatever handles FileCore formatted drives.

Bear in mind also that DOSFS is useful as an image filing system because you can currently take images of DOS discs and store them as appropriately-typed files on your hard drive. CPMFS and MacFS do this too. If you’ve lifted the file size limits, it seems to me the main reasons for moving away from the image filing system API for handling foreign disc formats have gone.

But they ought to be talking to a drive manager, rather than via FileCore: the whole business of FileCore handling its own format and also driving the hardware drivers like ADFS is messy.

Or do you mean that DOSFS would remain an image filing system and so mainly talk to FileSwitch? In which case, I agree.

Sorry: this is all a bit confusing, as image filing systems which handle foreign disc formats talk to FileSwitch and FileCore at present.

 
Oct 30, 2011 1:55pm
Avatar W P Blatchley (147) 247 posts

Unicode filenames

Might be worth pointing out that even though the Filer could probably be adapted to cope with Unicode filenames, at the moment you’d be totally stuck at the CLI. We might need an interim step that shows unicode-named files in some ASCII-understandable way until such time as the CLI is updated to cope…

 
Oct 18, 2012 12:31am
Avatar Malcolm Hussain-Gambles (1596) 824 posts

Is it 2012 already? ;-)
Any chance of an update, has this been completed?
Would additional funds make this more attractive to finish?
Is there another plan?

 
Oct 18, 2012 11:18pm
Avatar Steve Revill (20) 1337 posts

This bounty has been claimed, meaning work is underway towards achieving all the goals listed on that page. We’re considering opening the bounty for “File system improvements (Step 2)” so people can register their interest/support development by donating to that one. The only thing standing between us now and opening the Step 2 bounty is us agreeing internally about what should be in the second one.

The biggest problem we face, to be honest, is that the really hard RISC OS tasks, the ones we envisioned the bounty scheme helping to drive forward, realistically represent many man months of work – which, when done on the odd evening, weekend, means they will take years to complete. Unless people are able to take time off work to dedicate a lot more time to the task. However, with the bounties not really fetching very much (in cash per hour terms) they aren’t really succeeding as a motivator. We’d need to see the bounties getting into the several thousand pounds mark for those harder tasks to stand a chance of actually getting done (or someone very skilful with lots of free time on their hands appearing out of nowhere).

 
Oct 19, 2012 6:20pm
Avatar Malcolm Hussain-Gambles (1596) 824 posts

Thanks for the explanation.
I’m in the same situation of not being able to dedicate as much time as I would like to getting up to speed on RISC OS.

 
Aug 13, 2013 6:54am
Avatar David J. Ruck (33) 526 posts

Just to update the topic. DiscKnight 1.50 now correctly handles the new FileCore variant, allowing files of up to 4GB, the multi-zone disc map and the special partitioning for the Raspberry Pi.

 
Feb 14, 2018 12:01pm
Avatar Jeffrey Lee (213) 5842 posts

One thing I’ve realised is that the there doesn’t seem to be any mention of TRIM support in the filesystem bounty plans. Probably a good thing to slot in somewhere, now that SSDs are the norm for modern machines.

 
Feb 14, 2018 2:06pm
Avatar Rick Murray (539) 10531 posts

Probably a good thing to slot in somewhere,

…and proper dismounting? ;-)

Pages: 1 2

Reply

To post replies, please first log in.

Forums → Bounties →

Search forums

Social

Follow us on and

ROOL Store

Buy RISC OS Open merchandise here, including SD cards for Raspberry Pi and more.

Donate! Why?

Help ROOL make things happen – please consider donating!

RISC OS IPR

RISC OS is an Open Source operating system owned by RISC OS Developments Ltd and licensed primarily under the Apache 2.0 license.

Description

Discussion of items in the bounty list.

Voices

  • Sprow (202)
  • Jess Hampshire (158)
  • Jeffrey Lee (213)
  • Steve Revill (20)
  • Dave Higton (281)
  • Matthew Phillips (473)
  • W P Blatchley (147)
  • Malcolm Hussain-Gambles (1596)
  • David J. Ruck (33)
  • Rick Murray (539)

Options

  • Forums
  • Login
Site design © RISC OS Open Limited 2018 except where indicated
The RISC OS Open Beast theme is based on Beast's default layout

Valid XHTML 1.0  |  Valid CSS

Powered by Beast © 2006 Josh Goebel and Rick Olson
This site runs on Rails

Hosted by Arachsys