RISC OS Filing System Update

This page describes plans for changes to update the RISC OS filing system stack.

Goals

• Support large files (64 bit offsets).
• Support foreign format discs (e.g. FAT32 USB drives) larger than 2GB.
• Support a modern disc format to use as a native filesystem, that will handle large discs more efficiently.
• Provide a standard method of partitioning discs.
• Provide support for inserting additional block driver layers to implement things like RAID.
• Implement file and directory cacheing.
• Provide an interface for storing arbitrary metadata with files on filing systems that support it.
• Add support for symlinks.
• Simplify interfaces (for new code) wherever possible.
• Retain backwards compatibility for userspace applications.
• Retain backwards compatibility for old filing systems.
• Provide a unified formatter which can write any supported formats to any hardware.
• Provide a standard way of creating filer iconbar icons.
• Some support for nonblocking/async IO?

Plan

The following is an initial plan on how the changes can be split into smaller steps that can be implemented and tested individually.

Some of the phases could be done in a different order, or in parallel.

Phase 1

• Design new userspace APIs
• Trivially implement them, mapping to the old ones.
• Provide a patch to do the same for older OSes.
• Ideally work with ROL so the same can be added to RO6.

Phase 2

• Design new interface between fileswitch and filesystems.
• Implement fileswitch changes (possibly as a separate module, so it can be in C).
• Non-disc filing systems (e.g. Sunfish) could then be ported to the new interface.

Phase 3

• Design new block driver interface (common to discs and image filing systems).
• Port some filing systems (e.g. Filecore, DOSFS) to the new interface, and test as an imagefs.

Phase 4

• Port or rewrite some hardware drivers to implement the new block interface.
• Can now test with a normal ADFS/Filecore hard disc using the new stack.

Phase 5

• Implement disc formatter.
• Implement disc partitioning.
• Implement file cacheing.

Phase 6

• Port a new filing system (e.g. ext3, ZFS) to RISC OS, using the new interfaces.

Userspace API changes.

General concepts.

Filetypes.

Load and Exec addresses are only supported on old filing systems, new ones can only set the datestamp and the filetype separately. Therefore the new SWIs do not provide any method of reading or writing the load/exec addresses. Existing SWIs may give an error if used on a new filing system and the load/exec addresses written do not represent a filetype and datestamp. All new interfaces use 32bits for the filetype, and it is expected that new filesystems will store the filetype as 32bits where possible.

Symbolic links.

New filing systems may support symbolic links. Symlinks in any part of a pathname other than the leaf will always be dereferenced by fileswitch (and an error will be returned if they are broken links). New calls have a flag to indicate if the symlink should be dereferenced or not when it is the leafname. Old calls that do not have this flag will always dereference (with the exception of OS_File 6, which will always delete the link and not the target). In new calls that return an object type (OS_File 26 and OS_GBPB 13) a value of 4 is used to signify a symlink. This value will only be returned if symlinks are not being dereferenced by the call, or if the leaf is a broken symlink).

Extended attributes.

New filing systems may support extended attributes, to allow arbitrary metadata to be stored with any file or directory. An extended attribute consists of a name (null terminated, can contain any characters except null), and associated data (which can be any length, and may contain binary data). To avoid clashes with attribute names, they are organised into namespaces. A name should consist of its namespace, followed by a colon, followed by the attribute name. Applications can use a unique namespace for any data they store, to avoid overwriting data from another application. Should namespaces be allocated by the RISC OS allocations service? Two namespaces are currently reserved; riscos for any use by the OS, and user for any use the end user wishes. Any attribute name without a namespace (i.e. without any colon in the name) is reserved for the end user. Filing systems may use extended attributes to store information in if needed, and this may be exposed to the user. For example the filetype may be stored as an extended attribute named riscos:filetype.

File Locking.

OS_Find gains an extra flag on file opening to allow files to be opened by more than one process at once. OS_Args gains calls to lock byte ranges of files.

Changes to SWIs

 
Unless otherwise stated, all new calls behave the same as the ones they are replacing. 

OS_Args 
Use of OS_Args 0 with r1 != 0 is deprecated. OS_Args 10 should be used instead. 
Use of OS_Args 1 is deprecated. OS_Args 11 should be used instead. 
Use of OS_Args 2 is deprecated. OS_Args 12 should be used instead. 
Use of OS_Args 3 is deprecated. OS_Args 13 should be used instead. 
Use of OS_Args 4 is deprecated. Filing systems should be able to sort this out by themselves. 
Use of OS_Args 6 is deprecated. Filing systems should be able to sort this out by themselves. 

OS_Args 10 Read 64bit file pointer 
=> r0 = 10 
   r1 = file handle (must not be 0) 
<= r2 = lower 32bits of file ptr 
   r3 = upper 32bits of file ptr 

OS_Args 11 Write 64bit file pointer 
=> r0 = 11 
   r1 = file handle 
   r2 = lower 32bits of file ptr 
   r3 = upper 32bits of file ptr 

OS_Args 12 Read 64bit file extent 
=> r0 = 12 
   r1 = file handle 
<= r2 = lower 32bits of file extent 
   r3 = upper 32bits of file extent 

OS_Args 13 Write 64bit file extent 
=> r0 = 13 
   r1 = file handle 
   r2 = lower 32bits of file extent 
   r3 = upper 32bits of file extent 

OS_Args 14 Lock file 
=> r0 = 14 
   r1 = file handle 
   r2 = flags 
   r3 = start lo 
   r4 = start hi 
   r5 = length lo 
   r6 = length hi 
<= r2 = bit 0 clear if lock or test succeded, 
        bit 0 set if lock failed. 
        Bit 1 set on test if blocking lock was a write lock 
   r3 = start lo (only valid on test lock if lock would not have succeeded). 
   r4 = start hi 
   r5 = length lo (only valid on test lock if lock would not have succeeded). 
   r6 = length hi 

flags are: 
bits 1:0 1 set lock 
         2 clear lock 
         3 test lock 
bit 2 set for a write lock, clear for a read lock 

This call sets, clears or tests for a lock on a range of bytes within an open
file. The locks are only guaranteed to be advisory, although some filesystems
may choose to implement them as mandatory.  The range of bytes locked may be
larger than the current extent of the file, and if so will apply if the file
extent is later increased.  If an unlock request is made to unlock a region
that does not correspond exactly to a region that was previously locked, then
an error may be returned.  Testing a lock is a check to see if a lock call
with the same length, offset and type would succeed. If it wouldn't, then the
start and offset values indicate the range of the lock that is blocking the
request. Note that if a test succeeds it does not guarantee that a following
lock request with the same range will succeed. 

OS_Find 

Bit 8 is added to the reason code in r0, which indicates that this open
should not block further opens from also opening the file (if those further
opens also set this bit). This allows two processes to open the same file
both for writing, or one for writing and the other for reading. File locking
should be used if needed to prevent concurrent access to the same part of the
file.  Fileswitch may choose to return the same filehandle to the different
processes, or it may return a different handle. Some filing systems may not
support this functionality, in which case they will behave as if the bit had
not been set. 

OS_File 

Use of OS_File 0 is deprecated, OS_File 10 should be used instead. 
Use of OS_File 1 is deprecated, OS_File 25 should be used instead. 
Use of OS_File 2 is deprecated, OS_File 25 should be used instead. 
Use of OS_File 3 is deprecated, OS_File 25 should be used instead. 
Use of OS_File 4 is deprecated, OS_File 25 should be used instead. 
Use of OS_File 5 is deprecated, OS_File 26 should be used instead. 
Use of R2-R5 on return from OS_File 6 is deprecated. 
Use of OS_File 7 is deprecated, OS_File 11 should be used instead. 
Use of OS_File 9 is deprecated, OS_File 25 should be used instead. 
OS_File 11 only has a 32bit size, but can still be used. For creating large files, 
create them as a smaller file then use OS_Args 13 or OS_GBPB 14 to increase the size 
when writing the data. 
Use of OS_File 12 is deprecated, as it is unsafe and may overrun its buffer. 
Use of OS_File 13 is deprecated, OS_File 26 should be used instead. 
Use of OS_File 14 is deprecated, as it is unsafe and may overrun its buffer. 
Use of OS_File 15 is deprecated, OS_File 26 should be used instead. 
Use of OS_File 16 is deprecated, as it is unsafe and may overrun its buffer. 
Use of OS_File 17 is deprecated, OS_File 26 should be used instead. 
Use of OS_File 18 is deprecated, OS_File 25 should be used instead. 
Use of OS_File 20 is deprecated, OS_File 26 should be used instead. 
Use of OS_File 21 is deprecated, OS_File 26 should be used instead. 
Use of OS_File 22 is deprecated, OS_File 26 should be used instead. 
Use of OS_File 23 is deprecated, OS_File 26 should be used instead. 
Use of OS_File 255 is deprecated, as it is unsafe and may overrun its buffer. 

OS_File 25 Write 64bit file catalogue information 
=> r0 = 25 
   r1 = filename (no wildcards, no paths etc.) 
   r2 = flags 
   r3 = filetype 
   r4 = timestamp lo 
   r5 = timestamp hi 
   r6 = attributes 

flags are: 
bit 0 don't resolve symlinks 
bit 1 write filetype 
bit 2 write timestamp 
bit 3 write attributes 

OS_File 26 Read 64bit file catalogue information 
=> r0 = 26 
   r1 = filename 
   r2 = flags 
<= r0 = object type 
   r1 = filetype 
   r2 = timestamp lo 
   r3 = timestamp hi 
   r4 = attributes 
   r5 = length lo 
   r6 = length hi 

flags are: 
bit 0 don't resolve symlinks 

OS_File 27 Write symlink 
=> r0 = 27 
   r1 = filename 
   r2 = flags 
   r3 = link value (null terminated) 

flags are: 
bit 0 create new link if leaf doesn't exist 

OS_File 28 Read symlink 
=> r0 = 28 
   r1 = filename 
   r2 = flags 
<= r1 = link value (null terminated) 

flags are: 
bit 0 return raw link contents (else return resolved filename) 

OS_File 29 Write extended attribute 
=> r0 = 29 
   r1 = filename 
   r2 = flags 
   r3 = attr name (null terminated) 
   r4 = data 
   r5 = data length 

flags are: 
bit 0 don't resolve symlinks 

will create if doesn't exist, or overwrite is does. 

OS_File 30 Read extended attribute 
=> r0 = 30 
   r1 = filename 
   r2 = flags 
   r3 = attr name 
   r4 = data buffer 
   r5 = data length 
<= r0 = bit 0 set if found 
   r1 = length of attr (> buffer length if overflowed) 

flags are: 
bit 0 don't resolve symlinks 

OS_File 31 Delete extended attribute 
=> r0 = 31 
   r1 = filename 
   r2 = flags 
   r3 = attr name 
<= r0 = bit 0 set if found 

flags are: 
bit 0 don't resolve symlinks 

OS_File 32 List extended attributes 
=> r0 = 32 
   r1 = filename 
   r2 = flags 
   r3 = data buffer 
   r4 = data length 
<= r0 = Number of attributes found 
   r1 = length of buffer used (> buffer length if overflowed) 

flags are: 
bit 0 don't resolve symlinks 

OS_GBPB 

Use of OS_GBPB 1 is deprecated, use OS_GBPB 14 instead. 
Use of OS_GBPB 3 is deprecated, use OS_GBPB 15 instead. 
Use of OS_GBPB 10 is deprecated, use OS_GBPB 13 instead. 
Use of OS_GBPB 11 is deprecated, use OS_GBPB 13 instead. 
Use of OS_GBPB 12 is deprecated, use OS_GBPB 13 instead. 

OS_GBPB 13 
=> r0 = 13 
   r1 = directory name (not wildcarded) 
   r2 = ptr to buffer 
   r3 = num objs to read 
   r4 = start 
   r5 = buffer length 
   r6 = flags 
C flag corrupted 

buffer 
0  file type 
4  file length lo 
8  file length hi 
12 attributes 
16 object type 
20 time/date 
25 name (0 terminated) 

flags are: 
bit 0 don't resolve symlinks 

Should we just require the user to call GBPB 9 and then OS_File to get info?
With cacheing it wouldn't be such a penalty as now. 

OS_GBPB 14 write bytes to given 64bit pointer 
=> r0 = 14 
   r1 = file handle 
   r2 = pointer to buffer 
   r3 = bytes to write 
   r4 = file position lo 
   r5 = file position hi 
   r6 = flags? 
<= r2 = ptr to end of data 
   r3 = num bytes not transferred 
   r4 = new file position lo 
   r5 = new file position hi 
C flag corrupted 

OS_GBPB 15 read bytes from given 64bit pointer 
=> r0 = 14 
   r1 = file handle 
   r2 = pointer to buffer 
   r3 = bytes to write 
   r4 = file position lo 
   r5 = file position hi 
   r6 = flags 
<= r2 = ptr to end of data 
   r3 = num bytes not transferred 
   r4 = new file position lo 
   r5 = new file position hi 
C flag corrupted 

flags are: 
bit 0 preload 

OS_FSControl 
28 already broken on discs with more than 4GB of files in total 
38 32bit length on input, but unused so not a problem 
37 canonicalise will follow symlinks (new call needed that doesn't?)