CLib vs. oversized client stubs
Jeffrey Lee (213) 6048 posts |
If new entries are added to an existing CLib library chunk, and a program which requires those new entries is run against an older version of CLib, then CLib will fill the extra entries with branches to zero. Wouldn’t it be more sensible to raise a “Shared C Library out of date” error, instead of planting a ticking time bomb in the client app? Or at least, it should make it so that the extra entries call through to a routine which raises an error in a cleaner way than just branching to zero? Also, the code is broken in 32bit builds of CLib (lowering R3 means the branches to zero get overwritten with routine addresses) |
Jon Abbott (1421) 2640 posts |
Funnily enough, I was looking at the branch to zero’s in CLib last week, scratching my head thinking “Who on earth thought this was a good idea?” I agree, CLib should return an informative error and not rely on Branch through Zero which is pretty meaningless these days with high vectors. |
David J. Ruck (33) 1585 posts |
I assume it was done so programs could detect zero entries and perhaps take mitigating action, such as using an alternative internal routine, but has this ever been done in practice? Without mitigation it should be prevented by an RMEnsure in !Run of an appropriate newer SCL, but if that is omitted for whatever reason, filling the entries with routine which returns and error as JL suggests would be sensible. |
Stuart Swales (8827) 1323 posts |
I was pretty sure the Stubs init code did the appropriate SCL RMEnsure? Yeah, raise error. |
Jeffrey Lee (213) 6048 posts |
APCS-32 builds of the stubs have a few RMEnsures they go through, to make sure CLib & its dependencies are relatively up-to-date. APCS-R builds (as used by PlingSystem) don’t. So that seems like another problem. Also any error returned by EnsureCLib gets ignored, so even on APCS-32 the RMEnsures are pretty useless! |
Jeffrey Lee (213) 6048 posts |
I’ve been looking at this today. https://gitlab.riscosopen.org/jlee/RISC_OSLib/-/commits/StubChecks Fixing the CLib init SWI to return an error if the client stubs are too big was pretty easy, and it was also pretty easy to add extra code to the stubs to allow them to detect if an old version of CLib has placed a branch to zero or junk in the stubs. But then that led to the discovery that things which were being built by the HostTools makefile were actually using the CLib stubs that are intended for use on the target machine, instead of using the stubs from the host’s DDE install (so there’s now this set of changes which are aiming to fix that) The final piece of the puzzle, which I’ve not fully dealt with yet, is to try update the stubs so that the library chunks can be dynamic in size, so that unused entries at the end of each library chunk won’t be included in the final program. Otherwise the new sanity checking in CLib and the stubs means that every time we add new entries to an existing chunk (which has happened a few times over the years), anyone who releases software using those new stubs is likely going to require the user to install a new version of CLib as well, even if the new functions aren’t being used. Not the end of the world, but a departure from previous behaviour, and not always the most friendly thing to do. I know how to reorganise things (split some of the library chunks over multiple source files and set up dependency chains between them), but because the way the library chunks are being built is already a bit messy (15+ source files) I think I’ll first have to try and tidy that up (set up some makefile logic to control all the different build configs) otherwise it’ll just turn into a giant unmaintainable tangle of source files. Not insurmountable (it looks like amu can support the same makefile magic that I used before with GNU make), just an extra hour or two of work to dig through all the files and work out the necessary configuration. |
Charles Ferguson (8243) 427 posts |
The simple answer is never extend chunks, and instead create a new chunk for each set of new symbols that you need. That way the chunks won’t be recognised and will error, rather than trying to extend the chunks in a way that is going to cause a problem. The chunk mechanism is already there and protected against problems, so use it. It also results in smaller linkage because you only ever link with the library sections that you need when the content is pulled in from those chunks. This means that if you’re not using the functions, you don’t have to worry about the version of the CLib that you need – you only ever need the version that defined the selection of chunks that you use, and you don’t force the user to continually upgrade. |
Jeffrey Lee (213) 6048 posts |
Yes, but sadly that isn’t what’s actually happened. But in better news I’ve now got some rough changes to implement the dynamically sized library chunks (splitting them for each time the chunks have been extended). I just need to tidy it up a bit more and finish testing the different build targets. |
Charles Ferguson (8243) 427 posts |
(sigh) well that was pretty careless. At least there’s an easy way to stop proliferating the problem – remove the entries from the stubs that exist and republish, and duplicate the extended entries into a new chunk. That way it’s only people who were using old stubs (and their users) that will run the risk of there being a problem, and all future users with the new stubs that don’t use such dangerous operations will not suffer the problem. |
Jon Abbott (1421) 2640 posts |
I agree with this approach as future-proof. |
Jeffrey Lee (213) 6048 posts |
Yes, after doing some more testing I’m inclined to agree. The code I added to the stubs to try and detect when CLib doesn’t fill in all the entries wasn’t working against old CLibs, because the behaviour it was relying on (CLib filling the excess entries with branches to zero) was only added in CLib 5.37. CLibs older than that appear to just fill the excess entries with branch instructions to bad code. So:
|
André Timmermans (100) 648 posts |
StubG was designed to work with various CLib versions from RISC OS 3.1 to same 32-bit ones. |
Charles Ferguson (8243) 427 posts |
As I recall – and I could be misremembering as it was a while back, StubsG detects which of the styles of C libraries is in use through the SWI that works – if the 32bit one works, then it knows that the CLib is 32bit capable and lets things work as before. However, if it is 26bit, the 26bit SWI is called and the returned stubs are patched so that the PSR is restored properly in LR before the call. There is some fun for things like qsort which supply a function to call which will (under a 26bit CLib) be expected to be called with the PSR in the PC, and to return with the PSR preserved, so there is some special dynamic handling performed for these callback functions. This sort of operation won’t usually be a problem if you’re in USR mode, but if you’re in SVC mode it would be utterly fatal to not preserve these operations. Since the SharedCLibrary had already had these changes made itself internally – as it had to support 26bit and 32bit calls in the same sort of way – these changes were understood and relatively simple to do. I’m sure there were a bunch of subtleties, but without looking at the code I don’t remember. For the extended functions which are present in the new chunks, they’re completely ignored and will never be used by the application at all. Instead those functions are provided by StubsG itself. StubsG is a library (ALF, I believe) so it will only include the code that is needed – if you never call the C99 style functions, you’ll never need them linked with your code and so your overhead will be low. However, when you do include those functions, they’ll just pull in the symbols and code from the files in the library. In the early versions of StubsG, I didn’t even bother with things like `printf` which had been extended – as that introduces quite a bit of extra code. However later versions included these formatting functions as well. There is a StubsGS which omits the printf functions, so that you have a smaller binary – the number of times you actually need the 64bit or other formatting operations is pretty low, so this library is much smaller. (slight digression, as Jeffrey mentioned the difficulties in the source for the libraries making things difficult, in his earlier posting) StubsG is actually very similar to Stubs in how it’s built as a component, but pulls in more libraries. And here’s one reason why this was all very easy1 to create… In RISC OS 4, as delivered, SharedCLibrary lives inside RISC_OSLib. Not RISC_OSLib the library, but that was the directory that the source lived in. I suspect this will be very familiar to anyone working with the RISC OS 5 code base. This was because RISC_OSLib was coupled tightly with the SharedCLibrary so that it could be built into the ROM. Why is it built into the ROM? So that !Edit, !Draw, !Paint and (I think) Filer_Action can share its code – so they can be smaller. The fact that SCL builds in RISC_OSLib should not be a reason why they live together. RISC_OSLib is a higher level component than SCL, so why should we have everything in that component. Similarly, the component also built ANSILib and the Profiler library. There are two variants of the SCL (ignoring the APCS configuration), in that the ROM version includes the RISC_OSLib libraries and RAM version does not (plus the ROM version exports its symbols when rom_link’d). Additionally, the SCL had to keep its static data consistent – if it did not then parts of the I/O system which directly referenced static memory would fail (and there were a couple of other bits of the RTSK that needed parts of the system to be the same). And of course any time you changed RISC_OSLib you ran the risk of affecting the ROM build, and the fact that the RISC_OSLib version number was now tightly tied to the SCL version number. This is, of course, nuts. I was going to make a mistake, even though I ‘knew what I was doing’, and updates were being completely hobbled by this structure and the complexity. So I split it up into is constituant parts.
Because each of these components is separate and distinct, they can have their own version numbers, and the SCL doesn’t race – and it certainly doesn’t race based on the RISC_OSLib version. How do the other ROM apps work which used to link against RISC_OSLib? !Edit and Filer_Action are linked against RISC_OSLib just like any other application, and as Filer_Action doesn’t pull in much of RISC_OSLib the duplication is tiny. !Paint and !Draw aren’t in ROM any more, because they’re not vital to the booting of the system, so why waste the space. All of this made the maintenance of the SCL much easier. Care is still needed, but you’re only concerned with a single component at any given time, and you can test them independant of the rest. Things like the static memory areas are still important to get right, but with fewer other concerns you can work with just the bits that you know are important to this component. Reworking the _kernel_system code so that it was actually safe to use in a Taskwindow, or press Escape during a shift, was significantly easier with the code split into separate components. I realise that was something of a digression from the issue of the chunks, but this was one of the main ways that the C library became much more maintainable. 1 Really it wasn’t that easy – working with the CLibrary is a pain, and as the rest of the section discusses, I had to make it simpler because the existing design was completely unmaintainable and would almost certainly result in incompatibilities and difficulties in testing in the future. |
Jeffrey Lee (213) 6048 posts |
MR now up, although it’ll probably be blocked for a while until the HostApp changes go through. https://gitlab.riscosopen.org/RiscOS/Sources/Lib/RISC_OSLib/-/merge_requests/27 |
Charles Ferguson (8243) 427 posts |
I see no documentation in that PR, which continues the lack for those releases. When creating the RISC OS Select documentation for the updated SCL I had to invent documentation because none had been distributed as part of the Pace releases. Surely you cannot still be missing this from the RISC OS 5 SCL? The documentation I’m talking about is:
I do not see any equivalent in the repository, or in that PR. |
Jeffrey Lee (213) 6048 posts |
Correct, we don’t seem to have any documentation for what the different chunks contain. |