Source download / UnTarBZ2

Having a a few issues trying to download the RISC OS source from the website (not from CVS)…

The download of the CVS archive kept stalling, took about 20 goes to download “all” 52MB using Netsurf. It looks like this might be Netsurf/RISC-OS having issues on the BB XM as it seems OK in Windows. Also the BB XM network interface is only managing 300Kb/sec in RISC OS where as Windows is doing 1.4MB/sec (full speed for my broadband) downloading the same file.

To unpack the tar archive I downloaded the UnTarBZ2 application zip linked on the same page, but it appears the ZIP isn’t quite right as the file-types don’t map through into RISC OS – the files are !Boot,feb etc. Having re-typed the files manually, the application then just throws Directory Checksum errors and won’t unpack anything. The tar extracts OK in Windows (minus file-types of course) so for now I’ve been reading the code in VS 2010 (!).

Hopefully it’s not me going mad as after a week of VFP and ARM manuals my brain is melting :-)

It’s probably nothing to do with your potential brain melt… it sounds like De-tarballing is yet to be concluded…

Try using the self extracting version (see here – last time I checked, there were two different versions of UnTarBZ2, one which worked OK, one which didn’t. But I think that was before Steve did his most recent update of all the downloads, so things may be different now)

My brain is feeling slightly less melted however…

Tried the self extracting version and it extracted itself ok. But trying to extract the tar file using it fails completely with the same Directory Checksum errors. The few files it creates (guessing where the checksum matches by chance) have garbage names. (I used the GUI but as it’s only issuing the command I can’t imagine that’s at fault)

I’ve tried the very same tar file on Windows and it’s fine so it kind of must be the app that’s at fault.

Does someone have a version of !UnTarBZ2 that works on 5.17?

(email alan.peters at tbasoftware.co.uk)

The last set I tried, I unpacked the first level on a windows machine, then dropped the Tar onto !SparkFS on the Beagle.
This kept all the filetypes except TEXT (Don’t know if this is the way the new system is supposed to work?).

Sounds like some things for us to investigate there – 1. the zip version not decompressing correctly on RISC OS and 2. the checksum errors reported (presumably by bunzip2?).

Using bunzip2 on Windows to remove the compressed wrapper gave me a tar file that decompressed ok using SparkFS into a TBAFS folder. Many thanks to Tank for that one.

That’s the good news…

Inside the cvsroot directory within the tar archive the file types are wrong, similar to before with the !UnTarBZ2 ZIP archive. The names end with ,filetype,v – for example in the BASIC source directory the first file is !Mk,fd7,v

In other parts of the tar archive filetypes are OK.

Not sure if this is a bug or just a limitation of something that I don’t fully understand yet – but it does make this version of the source rather unusable without writing a quick hack to fix the filetypes :-)

Hang on, if this is the CVS repository, there are two things to note:

1. all of the repository files should have ‘,v’ on the end because that’s how CVS works
2. it is expected to be decompressed on UNIX or Linux as that is the preferred platform for hosting a CVS server ^*

Because of that second point, I create the tar files on our hosted server (which runs a custom UNIX) meaning that the files all have their RISC OS filetypes as a comma extension – this is exactly how the RISC OS CVS repository (i.e. everything inside cvsroot) has always been.

^* At least, that’s the one we have reasonable confidence in working, the RISC OS port of CVS isn’t well tested by ROOL.

Sorry a small part of my post was lost due to A formatting error (which I just corrected. “The names end with ,filetype,v”. I was particularly highlighting the file-type not having been applied, but can appreciate that CVS has to do whatever it does.

Maybe I’m not understanding the purpose of that tar archive of all the code, but I would have thought the file-types and names needed to unpack correctly to make the source build on RISC OS? Also with some parts unpacking with filetypes and others without it seems a bit confusing.

Anyway I can code a quick hack to sort it for now and will take a look at using CVS instead at some point.

Many thanks for all the help!

Although I haven’t downloaded any of the CVS archives myself, I’d assume that they’re a full copy of the CVS database. So rather than using the contents directly, you’re meant to use it to initialise your own local copy of the CVS repository, complete with the full history of each file, etc.

If you merely want to download an archive of easy-to-compile source code then instead of the “CVS” archives you should use the ones that are named by the product that they build – “Disc”, “OMAP”, “Tungsten”, etc. These archives contain the equivalent of what you’d get if you used the CVS “checkout” script to fetch the source from the CVS server. I.e. they only contain the sources required for building that particular product, and more importantly, they contain the right branch/version of each product, something that you might have trouble tracking down if you were to download any alternate archives. Plus of course there aren’t any silly “,v” suffixes on the files, and (if you use the right version of UnTarBZ2!) the files will have the correct filetypes applied when the archive gets extracted.

Of course, there are a few problems with the setup as it is now:

1. The downloads page doesn’t make any clear distinction between the two different types of archive
2. The text at the top of this page links to here for the Disc/IOMD/etc. source downloads, but those files are now on the risc-os-tarballs page
3. Plus the “download our old batches of source code as a single merged-together archive” text on the risc-os-tarballs page is probably obsolete since (AFAICT) you’ve removed the old batch 1-5 downloads
4. If you want to download the source for something that isn’t in one of the Disc/IOMD/etc. products, your only choice is to mess around with the CVS archive or use CVS directly (which can be done from RISC OS, although it’s a bit tricky)
5. Plus the aforementioned problem of the two UntarBZ2 copies being out-of-sync with each other

I don’t think the UnTarBZ2 archives are out-of-sync, I think I just forgot that I needed to create the zip version on RISC OS or RISC OS users will have problems with the filenames.

Yes, the build documentation is out-of-date.

Please note, on the downloads pages, there is an “Info” icon to the right of each download. I suggest you read the page(s) associated with the download(s) in question to understand what it is. In summary, the “CVS” archives are of a CVS repository – this is not source code that you build; it is exactly what it says it is: a CVS repository.

The archives which are called (in their description) “Source code” are exactly that – source code.

And yes, the old Batch releases of source code are very obsolete and have been removed from the downloads pages for that reason. The documentation needs updating. It seems that there’s as much enthusiasm in the community to keep these documentation pages up-to-date as there is within ROOL – close to zero. :)

Thanks, that’s much clearer. I can see I needed the OMAP version. I wrongly had assumed as I wanted to look at BASIC that I could just download any of the archives. Perhaps the full CVS archives could be at the end of the page.

Still can’t get the !UnTarBZ2 app to work so I’m having to work around it on Windows as above. Perhaps a RISC OS compatible ZIP file might be a bit less hassle?

Anyway, Thanks again and sorry to take up so much time with my lack of understanding :-)

Aaaah! The difference that has always puzzled me about the downloads is now much clearer, thanks Jeffrey and Steve. Although the CVS entry is described as ‘The complete cvs source repository as a compressed archive.’ which is confusing for us cvs novices.

Mind you it is also confusing in that the Source Code page seems to have duplicate entries for each item, which look exactly the same except for the date, and the older ones all have a size of 33 bytes so are presumably useless! Is this deliberate, or accidental?

Mind you it is also confusing in that the Source Code page seems to have duplicate entries for each item, which look exactly the same except for the date, and the older ones all have a size of 33 bytes so are presumably useless! Is this deliberate, or accidental?

Those will be the MD5 checksums of the files. Although I’m not sure why the checksum isn’t just listed beneath each download. Underneath each MD5 download it looks like the page is showing the MD5 of the MD5?

Shouldn’t this be automated, with Ticket #62 being fixed? (Although there’s no reference to ‘MD5’ in the listed Changeset #140.)

Although I’m not sure why the checksum isn’t just listed beneath each download. Underneath each MD5 download it looks like the page is showing the MD5 of the MD5?

Hi Jeffrey – I’m afraid I don’t follow what you’re saying there… I’m sure we could improve the layout of downloads pages but we didn’t want the MD5 checksum to take up much room seeing as so few people a) know what they are and b) bother with them.

Guys, I’ve uploaded a fixed version of the untarbz2.zip archive which should extract under RISC OS correctly – it’s linked to from the body text of this page. This is identical to the self-extracting version of the same application, which can be downloaded here.

Note: this version hasn’t itself had any work performed on it so don’t be surprised if you still see those checksum errors when you use it to extract some of the larger tarballs from our site – that still needs investigating…

The documentation needs updating. It seems that there’s as much enthusiasm in the community to keep these documentation pages up-to-date as there is within ROOL – close to zero. :)

Without wishing to nag, on a couple of occasions I’ve found some broken documentation and started to fix it. And then been denied because I was using Netsurf, and the wiki needs Javascript.

Hi Jeffrey – I’m afraid I don’t follow what you’re saying there… I’m sure we could improve the layout of downloads pages but we didn’t want the MD5 checksum to take up much room seeing as so few people a) know what they are and b) bother with them.

I think things have changed a bit since the last time I looked, but here’s what I currently see on the source code download page:

The first download link is the CVS source archive. Then the second download link is the MD5 of the archive. Except you can’t tell which is which unless you hover your mouse over the download link to see what extension it is (or look at the size column). Plus, underneath each download link, along with the date & time info, it looks like the MD5 is being displayed anyway (previously it was only the MD5 download links which had the MD5 shown under the date, but now it seems to show the MD5 under the date for every entry). Except to complicate things further, if you actually download one of the MD5 files and look inside it you’ll see that the contents of the file doesn’t match the MD5 shown on the web page.

Although looking at the dates, I’ve now noticed that the MD5 downloads are about three months out of date compared to the archives. So maybe it was just a glitch which placed the .md5 files on the server and all you need to do is delete them?

Although looking at the dates, I’ve now noticed that the MD5 downloads are about three months out of date compared to the archives. So maybe it was just a glitch which placed the .md5 files on the server and all you need to do is delete them?

You’re absolutely right – those downloads shouldn’t be there at all. I think what happened was that in our migration from the server “Alpha” at our hosting provider, Arachsys, to the server “Beta”, we lost all the stuff which automatically invoked our script(s) for (re)generating the MD5 checksums. From then on, the process has had to be kicked-off manually when we remember to do so. At some point, this process has been performed incorrectly and resulted in all the .md5 file gumpf.

Fixed.

Without wishing to nag, on a couple of occasions I’ve found some broken documentation and started to fix it. And then been denied because I was using Netsurf, and the wiki needs Javascript.

You’re welcome to nag! Drop an email to webmaster@riscosopen.org so Andrew is aware of your issue. Maybe it’d be possible to have a non-JS version of the wiki (although I suspect it’s unlikely).

It’s already been listed as an issue in the bug tracker, by Andrew, so I think he’s aware :) Ticket #278.

I’ve finally managed to get OMAP RISC OS to build after a bit more head scratching. The prepare script makes like much easier – sadly I’d first figured it all out myself from the various documentation pages and forum threads!

There are a few folders in the base of the OMAP source download other than the OMAPDev folder that seem a bit unnecessary. Are they useful for anything or could they be removed?

It might then be an idea to put a ReadMe file in the root along with the OMAPDev folder that explains that you need to run prepare (then EraseCVS – maybe prepare could do this?) to get started.

Many thanks!

I’ve finally managed to get OMAP RISC OS to build after a bit more head scratching.

Bummer. I’ve just re-written the How to build RISC OS wiki page which would probably have helped you.

There are a few folders in the base of the OMAP source download other than the OMAPDev folder that seem a bit unnecessary. Are they useful for anything or could they be removed?

Which folders were you thinking of?

It might then be an idea to put a ReadMe file in the root along with the OMAPDev folder

Yep – that’s a good idea. I’d probably keep the ReadMe very short and just point to the wiki page(s).

EraseCVS – maybe prepare could do this?

I did consider that but I’ve since forgotten why I chose not to. I think I wanted to see if builds work fine without that step – if they do, it makes things easier for getting changes back into CVS.

Which folders were you thinking of?

When unpacked the root directory contains:
Autobuild
Castle
mixed
OMAP3Dev
RiscOS
Utilities

However everything I needed to build was in OMAP3Dev and in my build directory copy I deleted all the other directories.

One other note – I placed the entire tree in a TBAFS archive and it really helps speed things up due our caching logic. It’s also easy to back up the image file rather than all those thousands of files.