More bounties underway!

A couple of weeks ago, we marked the ARMv7 code generation from the C compiler bounty as underway. This is great news for developers helping to ensure RISC OS software remains future-proof.

Today we also marked the Further clipboard support as claimed.

Thank you again to everyone who has ever donated to ROOL – you are directly supporting important developments to ensure RISC OS continues to improve and remains a viable Operating System for years to come!

Oh look, the compiler uses “an intermediate representation”. ;-)

It compiles C code and inline assembler to an intermediate instruction set (known as “jopcodes”), and the ARM backend can generate ARMv5 (and a few ARMv6 instructions) from these jopcodes.

Are there any plans to add C11 extensions? Whoever does the USB Bounty is going to find it a bit messy without them as the NetBSD code has a number of C11isms the last time I looked.

Having just been working on some clipboard stuff as part of one of our projects, I have to say that the further clipboard support is great news.

However, what would be ideal would be a re-implementation of RISC OS Ltd’s Clipboard Holder system, to avoid introducing more different variations of “how to do clipboard”. What I discovered was that every application seems to handle things subtly differently, because it is down to the app in question. !Edit does a nice job, incidentally, so I’m definitely not complaining at ROOL on this one!

What ClipboardHolder did, though, was to standardise things. Because it acted as a middle man (without the application needing to be aware of it – that’s very important), it effectively allowed the clipboard responses to be standardised. In other words, if your program works, it works with everything.

Conversely, I saw (well, we saw, as there’s two of us doing clipboard things on two apps) unpredictable behaviours with some apps on RO5 which worked fine on Adjust with ClipboardHolder. I won’t name names, as the fault may be with the source apps rather than destination apps, but nevertheless when some work beautifully (eg. !Edit) and others are unpredictable or don’t, it’s not ideal.

One final thing is that clipboard needs to be simple to implement. The existing documentation is long and scary. The simplest explanation is probably the (rather old) one at http://www.wss.co.uk/pinknoise/Docs/Arc/Apps/Clipboard.html

However, what would be ideal would be a re-implementation of RISC OS Ltd’s Clipboard Holder system, to avoid introducing more different variations of “how to do clipboard”.

RISC OS 5 has had its own Clipboard module for over a decade, but it seems that I’m the only one that’s ever used it.

Like ClipboardHolder it makes using the global clipboard much easier. In its simplest form you just hand over the clipboard data and let the Clipboard module deal with any requests for that data.

This is particularly useful for Toolbox applications where the Data Transfer Protocol (DTP) is hidden away in the SaveAs object. Without the Clipboard module the DTP would still have to implemented just to support the global clipboard.

Here’s how to put the contents of the BASIC variable Clip$ on the global clipboard:

SYS "Clipboard_Put",0,&FFF,Clip$,LEN(Clip$),"ClipTest"

For ClipboardHolder this would be:

SYS "XClipboardHolder_Copy",0,&FFF,Clip$,LEN(Clip$)

Applications that want to work with both clipboard modules could use something like this:

SYS "XClipboard_Put",0,&FFF,Clip$,LEN(Clip$),"ClipTest",0 TO ;Flags%
IF ((Flags% AND 1) = 1) THEN
  SYS "XClipboardHolder_Copy",0,&FFF,Clip$,LEN(Clip$)
ENDIF

What ClipboardHolder seems to be missing is support for applications that can supply the clipboard data in different formats. For this the Clipboard module has its proxy mode where it will handle all the messaging with the application requesting the clipboard data. Before sending the paste message the Clipboard module will ask the application holding the clipboard for the data in the format requested by the pasting application.

Another important function of both clipboard modules is to provide support for CnP in the Wimp. As the Wimp isn’t a task (so no task handle) it can’t send messages itself and needs a module that provides SWIs. The Wimp communicates with the clipboard module using those SWIs and the module will do the necessary messaging to obtain the clipboard data.

The way this is implemented in the RO5 Clipboard module is more complicated than ClipboardHolder but it avoids the need to forcibly claim the global clipboard. Something ClipboardHolder is forced to do due to a flaw in the original clipboard protocol.

Due to this claiming of the clipboard trying to run applications (eg ClipMan) that did the same on versions of Select with ClipboardHolder was, erm, interesting. Basically the machine froze as they kept claiming the global clipboard off of each other. I believe UniClip also ran into this as it contains code to detect the presence of ClipboardHolder.

If anyone wants to play with the Clipboard module grab a copy of MiniTime which contains a copy of the module as it seems that there is no download for it on the ROOL site.

The simplest explanation

Wow. Now I know why I never implemented clipboard support.

There ought to be a SWI to paste a type-tagged object to “the clipboard” and a Wimp message (with SWI) to notify of change, and a SWI to retrieve a pointer to the previously pasted object.

Anything else is just complication, and what’s all that guff about clipboard ownership? It should be an opaque black box that owns itself and the current object is the last thing pasted.
The model as given in the document suggests that an application must not only be active and resident to handle clipboard data, but must also find a way to handle differences between what was actually copied at a point in time, and what may exist now after subsequent user edits. That’s unbelievably ass-backward.

So anything that improves upon that situation would be a good thing.

BTW, you should name names. Not to shame, but to try to narrow down where exactly the problem lies.

Anything else is just complication, and what’s all that guff about clipboard ownership?

Just think about it, it makes sense: the owner just supplies the list of formats for the clipboard data. If someone wants specific data from the clipboard, the owner supplies exactly this data and nothing else. Saves a lot of memory for certain data.

Think about an application providing access to bitmap graphics. It can supply the data as PDF, Sprite, PostScript, BMP, GIF, PNG and TIFF. Do you really want to have all that data instantly lingering in your computer’s memory?

And that’s exactly the problem the ROL approach to clipboard handling had. The ClipboardHolder implementation was completely alien to the way the clipboard protocol was intended. I am sure you can find the newsgroup discussions about this in the early 2000s.

Thanks Fred. I was basically trying to avoid the situation where programmers need to add new code, or make changes. It does sound like the OS5 module is helpful, although I fear obfuscated behind the mammoth documention file that exists on the ROOL site (it is very, very daunting to those of us with coding-inferiority-complex!).

Anything that can standardise things would be good, though, but as you say, having to write one thing for RISC OS 5, and one for Adjust, is, erm, unfortunate.

Steffen – I agree with what you say, and it is brilliant, right up until the moment you realise that no two programs implement the protocol in exactly the same manner.

By way of example, some programs insist on ram transfers, and don’t degrade gracefully to <wimp$scrap> datasave/dataload methodology. Others are the exact opposite. It means that a correctly written program ends up with a large amount of duplicate handlers for different situations, which is a mess.

Fred – what’s the distribution policy (licence) on Clipboard module? I actually looked for it in ROM a few days ago (I remembered reading about it a long time ago, but didn’t know what happened to it). One of the programs we’re working on will be RISC OS 5 only, so there’s no real need for it to implement all the crap – Clipboard module would be ideal. But, obviously, it’d need to be distributable/downloadable, otherwise we’d have a problem dependency.

By way of example, some programs insist on ram transfers, and don’t degrade gracefully to <wimp$scrap> datasave/dataload methodology. Others are the exact opposite. It means that a correctly written program ends up with a large amount of duplicate handlers for different situations, which is a mess.

It is not really different to drag&drop handling between applications, is it?

In true tradition of RISC OS “not invented here” philosophy, nobody uses that simple, generally available library that would handle all that complexity transparently. Or more precisely, because of the “not invented here” tradition, such a library does not even exist.

Or more precisely, because of the “not invented here” tradition, such a library does not even exist.

Not strictly true, since if the toolbox is used, then all the load/save protocols, including drag-n-drop to other applications, is handled by that.

what’s the distribution policy (licence) on Clipboard module?

Feeling puzzled here because the Clipboard module referenced by Fred is in the CVS under the apache section of the tree and the licence file starts with the text:

Clipholder (which people seem not to have liked from day 1 back in the 2000’s) is obviously a different kettle of fish.

One flaw with the RISC OS 5 clipboard protocol (which the Clipboard module doesn’t solve – not sure about ClipboardHolder) is that programs aren’t notified when the contents of the clipboard changes (and/or there’s no lightweight way for a program to poll the clipboard to ask if the data’s changed). They’ll be notified when the owner of the clipboard changes (via Message_ClaimEntity 4), but the spec says not to send that message if you’re already the owner, so if it’s the same owner but different content then you’re out of luck.

This flaw causes problems for the VNC server – the VNC protocol requires that whenever data is copied to the clipboard on the server, it’s immediately sent over the network to the client. Since RISC OS can’t notify the server when the clipboard content changes, this means that (unless the server owns the clipboard) it has to regularly try and read the full clipboard contents to see if it’s changed.

Fred wrote:

RISC OS 5 has had its own Clipboard module for over a decade

it seems that there is no download for it on the ROOL site.

Hmmm…. I wonder why there’s not very much support for it then?

Steffen wrote:

Or more precisely, because of the “not invented here” tradition, such a library does not even exist.

I’m sure people would welcome you publishing one.

Think about an application providing access to bitmap graphics. It can supply the data as PDF, Sprite, PostScript, BMP, GIF, PNG and TIFF. Do you really want to have all that data instantly lingering in your computer’s memory?

What I’d specify is much much simpler. Images are Sprites (bitmap) or DrawFile (vector). Both are widely supported and system native.
If an application would prefer to use something better (JPEG, ArtWorks, etc) then it can push it to the clipboard as a filetype tagged block of data. The recipient can import it if it understands (like ArtWorks → Ovation), or complain (in which case you can drag-save exporting the data as a known format).

It shouldn’t be up to the clipboard to need to store or cache all sorts of possible formats “just in case”.

was completely alien to the way the clipboard protocol was intended.

Probably not surprising, it’s a really weird implementation that basically drops a lot of the behaviour of the clipboard into the lap of the programmer (well, no surprise there in RISC OS land).

What I’d expect:

SWI Clipboard_Clear
SWI Clipboard_Write <pointer to data>, <data size>, <filetype of data>
SWI Clipboard_Read (returns <pointer>, <size>, <filetype>)
SWI Clipboard_Enquire (see below)

Wimp_Message Clipboard_Changed (info broadcast)

If the Read SWI flags “data read” or anything like that, then an Enquire SWI can be provided to get the data length and filetype without side effects.
It would be expected that the data would be stored (by the Clipboard itself) in a Dynamic Area.
The Read/Enquire SWI would return a filetype of 0 if there’s nothing on the clipboard (pointer/size considered irrelevant if the type is zero).

That’s about as complicated as it needs to be in order to be a functional clipboard.

what’s the distribution policy (licence) on Clipboard module?

Feeling puzzled here because the Clipboard module referenced by Fred is in the CVS under the apache section of the tree and the licence file starts with the text:

Correct, the module was started by Ben (Avison) but its clipboard support was only partially implemented, something I finished in 2007-ish. The other thing that the module was supposed to help with is DnD but that still needs to be done, and from scratch.

It’s probably best however that the Clipboard module is made part of the standard ROM of disc image. Given that the module is required for CnP support in the Wimp (now under way) I’m leaning towards the former.

They’ll be notified when the owner of the clipboard changes (via Message_ClaimEntity 4), but the spec says not to send that message if you’re already the owner, so if it’s the same owner but different content then you’re out of luck.

Yes, that’s the flaw in the original protocol that I was alluding to. It probably made sense at the time to cut down on the amount of messages being sent round given that machines were much less powerful but in hindsight it was a poor decision.

the server [..] has to regularly try and read the full clipboard contents to see if it’s changed.

Resulting in more messages being send round than the original ‘optimisation’ saved. :-(

The other way to find out if the clipboard contents has changed, is to forcibly claim the global clipboard so that other applications need to claim it before they can change the contents.

The other way to find out if the clipboard contents has changed, is to forcibly claim the global clipboard so that other applications need to claim it before they can change the contents.

How frequently would it need to forcibly claim the global clipboard? If several running applications took the same approach, wouldn’t we end up with a similar slow-down where the clipboard is constantly changing hands (so we never know if the contents is changing or other applications are just trying to monitor it too)?

Still, I suppose it’s not something most normal applications need to know about, so I can see it could work for just something like VNC.

How frequently would it need to forcibly claim the global clipboard?

Basically every time another applications claims the global clipboard. Less if you’re only interested in a particular filetype, ClipboardHolder only forcibly claims if the filetype is text, IIRC.

If several running applications took the same approach

That might be possible if each application was does so for a different filetype but if not then there’s a problem. See my account above of running ClipboardHolder and ClipMan at the same time.

The problem of applications not being notified of clipboard contents changing is one I would love to see fixed. As often though the large body of dead, as in not maintained code, makes this difficult. Without that, all that’s needed is for the global clipboard to always be claimed when the contents changes, even when it remains in the same application.

Or more precisely, because of the “not invented here” tradition, such a library does not even exist.

I’m sure people would welcome you publishing one.

My track record suggests otherwise. And I am fairly sure nobody would change their favourite programming language to use my library.

It shouldn’t be up to the clipboard to need to store or cache all sorts of possible formats “just in case”.

The clipboard protocol was designed to allow to speficy the list of filetypes you accept (in decreasing order of preference). This does not mean the the owner of the clipboard (application or module) does store the clipboard internally in various format “just in case” only that he will respond only if the its stored clipboard data matches one of the filetypes from the list or if he is able to translate the clipboard data to one of these filetypes.

The most obvious example of translation is bitmap/vectored images but it applies also to text files (e.g. HTML → text). Basically if an application can export its files into other format, it can do the same for the cipboard.

A pity we don’t have Select’s image converter modules, it would make it easy to let the Clipboard module perform this translation for bitmap/vertored images.

A pity we don’t have Select’s image converter modules, it would make it easy to let the Clipboard module perform this translation for bitmap/vertored images.

Translations always seem to lose detail, so I’m not so sure allowing the clipboard to do any translation is a good idea.

If things are going to be updated can I mention a problem I’ve seen. Say if you load a raw html file into Edit but then want to paste a section of it into DataPower, DataPower won’t accept it as it isn’t filetyped text.

I’m not sure of a way round that problem without changing the receiving applications.

I often load into Edit many different files not filetyped as text: Data, HTML, Marcel… that I want to cut and paste elsewhere.

If things are going to be updated can I mention a problem I’ve seen. Say if you load a raw html file into Edit but then want to paste a section of it into DataPower, DataPower won’t accept it as it isn’t filetyped text.

Are you sure? It works fine on my Pi here. What version of the OS and/or DataPower are you using? I’m using RO5.24 and DataPower Home v1.61

I can’t actually remember which application/data file combination I’ve come across it with. The example was to explain the priciple. I’ll make a note next time I have the problem.