Filecore bug
Chris Hall (132) 3567 posts |
If I use the programme below:
then after 3hr30min on my ARMX6 (with FileCore 3.67 (2-Oct-2015)) some 80619 1-byte 18-character filename files have been copied, in a non-optimum order, and I get ‘broken directory’ – the remainder of the disc is good and DiscKnight cannot fix the problem. It needs a reformat to solve it. Bug reported by e-mail. This error is repeatable and I now have two such broken directories on this disc. The error does not necessarily occur at exactly 80619 files copied. I suspect the problem has gone unnoticed because in a filecore to filecore copy operation, the files are presented in alphabetical order, so that no directory entry shuffling is required. The filer uses a non optimum shell sort (when an insertion sort would be much more efficient) after each and every individual file copy operation. The ArmSort module can sort 100000 strings in about 0.4 seconds on an ARMX6 using a flash sort. In a NAS to filecore copy they are produced in a seemingly random, but repeatable order during a copy. Quoting from an earlier post by Jeffrey: ‘marutan.net has copies of 1309,207/FS and 1309,208/FS’ and the latter says the filename limit for 19 character filenames is: For names of up to 19 characters, we can have up to 87380 directory entries Different errors occur if you try to write similar files to a SparkFS archive. RAMfs (which would be faster and not generate a permanent disc fault) only allows 11310 1-byte 18-character filename files (making 11567 files in total on a 125Mbyte RAM disc). Increasing the RAM disc to 128 Mbytes and only 7951 files could be copied, fewer files because the granularity (?) LFAU (?) worsens with increasing disc size. Hence the error cannot be demonstrated using a RAM disc. Writing to an SD card or to a pen drive is too slow for my patience. |
Rick Murray (539) 13870 posts |
Whoa. Is the directory completely scrambled then? I’m surprised DK doesn’t have an option to simply hack the offending directory out of the map and try to recreate the fragments left behind as lost files. Perhaps the lesson to take away from this is that 77 files was a rather (arbitrarily?) low directory limitation, but 80K is… Uh… A startling number of files. And RISC OS managed 15K more files than FAT32 can store in a single directory. ;-) There is a hard fix and an easy “fix”. The easy “fix” is to clamp at 65K entries. Harder is to delve into the innards of FileCore and see what is happening. What I find strange is that it takes 80K files to trash things. Why not 60K or 15K? Looking at the older (E/F format) directory information, it seems a fairly simple structure. Your files have 18 character names. 19 with terminator? Assume rounded to 20 to word align. There follows four words of information. Let’s assume five words in case the disc address is 64 bit. That means we ought to get by with 40 bytes per entry (depending on how long filenames are actually stored, I may be well off the mark!). For 80619 files, 1612400 bytes, or 1.53MiB. There doesn’t seem anything special about that number to me, so my calculations are probably way off. While you say it is not necessarily 80K files, I am guessing it is around that amount. So the first thing I’d ask is “what changes at that point?”. Does the directory entry stray over some sort of structural boundary? tl;dr: I know nothing about FileCore that isn’t written in PRM2 (!) but I find this intriguing… |
Chris Hall (132) 3567 posts |
Whoa. Is the directory completely scrambled then? Who knows? It cannot be opened, giving the error ‘directory broken’. DiscKnight says disc is good. You can rename the directory but it gives an error and leaves the directory name ambiguous, DiscKnight can restore the directory name to the original name again but still broken. The directory buffer is the place where FileCore keeps the directory currently being worked on to see how space is being consumed for the directory entries The disc is 250Gbyte in size so approx 256k to 512k per 1-byte-long file so that 80,000 files would use up 20Gbytes to 40Gbytes. |
Chris Mahoney (1684) 2166 posts |
Just out of interest, what’s your total used space? Could your broken directory be crossing the 128 GB mark? |
Chris Hall (132) 3567 posts |
The first broken directory may have crossed the boundary (but I think it did not) but the second did, leaving 75Gbytes free. |
Chris Hall (132) 3567 posts |
I have repeated the test on an almost empty 250Gbyte disc and after 80619 files have been copied, I get ‘broken directory’ again. QED. |