C library for easier HTTP(S)

The other day I mentioned that I was working on a library that made it easier to use the URL, AcornHTTP and AcornSSL modules. I said at the time that it wasn’t quite ready but that I’d share something in “a few days”. It’s been more than a few days so I figured that I should share what I have.

Download HTTPLib – Click the HTTPLib tab. This should be considered “alpha” quality. Source is linked from the same page.

Quick overview
Call httplib_get and pass in a URL, a function to call once the transfer is complete, and optionally a function for reporting download progress (or NULL). For example:

httplib_get("https://server/path/file.txt", downloadcomplete, progress);

void downloadcomplete(unsigned int status, char *content)
{
    if (status == 200) // OK
    {
        printf(content);
    }
}

void progress(unsigned int bytes_transferred, unsigned int total_size)
{
  printf("%d of %d bytes downloaded", bytes_transferred, total_size);
}

status will contain the HTTP status code if there is one, or 0 if it didn’t get that far (e.g. if you click “Reject” on an unverified certificate).

You will also need to add some code to your polling loop using httplib_isbusy and httplib_dowork so that everything can multitask. You will need to allow null events if HTTPLib is doing something:

while (!quit)
{
    if (httplib_isbusy())
    {
        event_set_mask(0);
        event_poll(&event_code, &poll_block, 0);
        httplib_dowork();
    }
}

Remember to set the mask back to something appropriate afterwards :)

Payload body
Some HTTP methods such as POST and PUT require a payload body. An extra char * parameter is available:

httplib_put("https://server/path/resource", payload, putcomplete);

Downloading files
You can use httplib_download to save a file instead of creating a char *. To do this:

httplib_download("https://server/path/file.zip", "local-filename", downloadcomplete);

The downloaded file will have its file type set based on its MIME type.

As of HTTPLib 0.06 you can cancel an in-progress download by calling httplib_stop.

Setting user agent
You can set a custom user agent via httplib_useragent. You must call httplib_useragent_free once you have finished your HTTP session.

Other functions
httplib_head does a HEAD request. Use it in exactly the same way as httplib_get.

Limitations

• Only one transfer can be active at a time

Future plans

• Fix the limitations!
• Add support for caching (with an option to bypass)

Licence
HTTPLib is available under a slightly modified BSD licence. As of 0.02, a copy of the licence is available in the Zip (and no longer cluttering up this post!).

I’ll post any updates in this thread. Hopefully someone finds this useful and easier than massaging the SWIs directly.

And almost forgot to say: I think you’ll need AcornHTTP 0.98 or newer. Earlier versions will probably not work. You’ll also need the URL module loaded, and you will of course need AcornSSL if you’re using HTTPS.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS [blah blah]

Why does nobody seem to understand THAT A BIG WALL OF TEXT IN CAPITAL LETTERS IS HARDER TO READ? Imagine reading the licence aloud AND SHOUTING the large chunk at the end…

Imagine reading the licence aloud AND SHOUTING the large chunk at the end…

Imagine a user that emails the third line techs direct with BOLD UPPERCASE AND RED.
Imagine the same user having their emails drop into the junk mail folder(s) :)

Well, for 0.02 I’ll put the licence inside the Zip so you don’t need to be blasted with it (in this case I forgot to include it and I’d already copied the Zip onto my server). With that said, I think the licence file in my other apps is probably also full of caps because I just copied a template. I’ll make a note to change that…

Well, for 0.02 I’ll put the licence inside the Zip so you don’t need to be blasted with it

I’m pretty sure Rick’s comment was a criticism of the person/people that wrote the original.
My comment was about one of a small set of users within the work domain and the effect her abuse had, which is probably not dissimilar to other wall of caps instances (ignore it)

With ref to the code – might be nice if it could be put into a module – so it could be used from BASIC etc.

Thanks for the work.

I’m pretty sure Rick’s comment was a criticism of the person/people that wrote the original.

I know, but still :)

As for work, we have someone who writes every single email subject line in all caps (but thankfully not the rest of the message!)

With ref to the code – might be nice if it could be put into a module – so it could be used from BASIC etc.

While that would be possible, I’m not sure how much benefit it would be. The URL module already exists, and HTTPLib is basically a simplified C interface into that. If there are specific features you’d find useful in BASIC then it might be more logical to add them to URL rather than have a module “wrapping” another module (which, in turn, wraps the AcornHTTP module!)

If you disagree then please give some examples of what you’d like to accomplish so I can understand better.

My impression was the whole purpose of C was to get everyone playing an endless game of Pass The Parcel…

With ref to the code – might be nice if it could be put into a module – so it could be used from BASIC etc.

Sensible future RISC OS project number 87536: teach BASIC to load and call a user-mode library (ELF, ALF) so that we can stop putting standard application code into the RMA and excecute it with OS privileges.

I am sure Steve will be around in a minute and tell us that Basalt can do this already :-)

I am sure Steve will be around in a minute and tell us that Basalt can do this already

Lol!

I am sure Steve will be around in a minute and tell us that Basalt can do this already

Not yet. ;-)

I did play around with calling SharedCLib routines within Basalt, to enhance printing with printf for instance, but that was a long while ago.

I understand the point about privileged modes, but I still like the parsimony of shared routines called as SWIs.

I understand the point about privileged modes,

We all do, but this is one of the things we’re stuck with – unless somebody fancies having a crack at running user (and most system) modules in SYS mode . . . and dealing with the fallout of pretty much everything that assumes “modules run in SVC mode”.

but I still like the parsimony of shared routines called as SWIs.

Indeed. While SWIs have efficiency issues (recalling the SWI number from something that should be in the code cache, not the data cache etc), there is a benefit to a set of routines that can be called via a standardised interface from just about any language worth knowing the name of¹.
One could argue that the Desktop is exactly that – a set of routines with a SWI interface.
And Sockets.
And FileSwitch/FileCore.
And………

¹ Obviously I am referring to the subset of languages available on RISC OS, not languages in general everywhere (and especially not the vagaries of human languages!)…

HTTPLib 0.02 is now available, and the first post has been updated accordingly. But in summary:

• Handles failure better (better ≠ well!)
• Added feature to download files instead of save to char *
• Removed the user agent parameter from the various functions; a future version will offer a “global” way of setting this.

HTTPLib 0.03 is now available. Changes in this version:

• Fixed a malloc bug, which means that httplib_useragent (which was present in 0.02 but not documented) now works and is supported
• Downloaded files will now be typed according to their MIME type, instead of being hardcoded to Zip.

HTTPLib 0.04 is now available. This one fixes an embarrassing bug which was intermittently corrupting downloaded files (because some numpty – me – was calling the wrong SWI…) and adds an httplib_head function for doing a HEAD request. There is still an issue with some servers not “liking” HTTPLib, which I’ll hopefully get to the bottom of in 0.05.

Version 0.05 is now available (see first post). It supports the rest of the HTTP methods, and is also “better” because it doesn’t have the same code repeated all through it anymore :)

The issue I mentioned in the previous post appears to be a problem with AcornHTTP rather than HTTPLib. I haven’t had a chance to investigate it properly yet.

The URL module already exists,

where? How do we use it?

It’s included in the nightly HD4 image and the docs are here.

Thank you.

HTTPLib has been updated to version 0.06 (link in first post).

• Implemented “progress” function so you can see how far through a download you are.
• Added httplib_stop function to abort an in-progress download.
• No longer uses a fixed 1024-byte block size. It can now download in blocks of up to 1 MB, resulting in a significant performance boost on fast connections.

0.07 released. This version has some minor reliability improvements and is now known to work when called from a module.

The link from first message return 404 error.
Can anyone to send a correct link or upload the sources on Google drive or something?

This link seems to be working.

Apologies! I moved the site yesterday and completely forgot that I had direct links to some of the Zips. Links in first post fixed, although I’ve broken nemo’s link in the process :)

Does anybody have a C or C++ example app that uses HTTPLib? I can’t get it to work although I tried to implement it as shown in the initial post. But I am new to C so the problem is very likely me ;-)