TCP/IP Stage 2 bounty

@Alan, I think there are enough changes for take up the mantle of maintaining it :)

I’m glad to see RO 5 has started returning error numbers the RISC OS way (relative to the SWI base instead of zero based UNIX errno’s). There really should be code to set the error_base% variable depending on the earliest Internet module version that supports this.

Yes, sockets are hard.

Made so by the lack of simple examples.

But, with encouragement, perserverence, and a bit of luck, something clicked into place and I now have a non blocking PROCedure which fetches a web page from a local server. (with adequate progress and error detection)

The PROCedure is currently about 40 lines long (with another 40 for debugging) and is called from the poll loop of an existing program. The bit I was missing was how to tell when the connection was made (or crash out when the connection was going to fail). The comment in the !Antispam source code about removing the loop repeating calls to Socket_Connect prompted me to do just that and repeat the Socket_Connect SWI until I get the response 56 (or an error response 22, for example, – connection rejected [wrong port used!])

Thank you all for your patience and help. If anyone is interested in this simple BASIC code example, I can post it here when I have tidied it up.

There really should be code to set the error_base% variable depending on the earliest Internet module version that supports this.

Test it’s behaviour, not it’s version number. There are at least 4 independent implementations of the RISC OS socket SWI interface.

But in this case you can just check the returned error number and adjust accordingly.

5 implementations. There is a bespoke implementation that we used over here in the US, that I do not think ever made it back to the UK. It is likely the least complete, though it supported the software we used that needed it in the 1990s. Ironically I think it is cleanly enough written that it would take very little effort if any to port to 32bit PC/R15 modes, despite being in assembly lang.

But in this case you can just check the returned error number and adjust accordingly.

For that you have to assume that no future internet module will ever return a standard RISC OS error number less than 256, unlikely but possible.

Instead of testing the version number it might be better to call one of the socket SWIs to provoke the return of known socket error, and see if it is ERRNO or module base+ERRNO.

@Alan, I think there are enough changes for take up the mantle of maintaining it :)

Sorry Dave, I was being lazy just sending the whole thing. The important change is the one in PROCsocketlib, with comment just after FD_SETSIZE% – 128 (96 would be better here).

It refers to changes in the PROCFD… and FNFD… code, where the masks are handled. I don’t have an original version of the library to see what changes I had to make there.

All the other stuff doesn’t affect the operation directly, just made some things compatible with the rest of my code. For example using FNisvar in the error handler – cleaning up after an error could cause an error if the variable hadn’t yet been created. ON ERROR OFF prevents that causing a loop, but I wanted tosuppress the message as well.

The HOSTS bit may be left over from when I was struggling to use name resolution – I’d be surprised if it is in use now, since several of the addresses have changed, and I haven’t kept that file updated.

Actually this thread has reminded me of another variation – VRPC returns error numbers with the base added. My other computers at the time I did the networking did not. This meant I had to make adjustments to error detection.

(My main networking code was done about ten years ago, giving me higher level functions to work with. I’ve had to revisit a few times, most recently about 3 months ago when I found another subtle bug caused by one of the edge cases, when handling the ring buffers. Since triggering it required conditions that depended on the exact size of the data being transferred, and so varied with, for example, the length of someone’s name, I was lucky when I founds a repeatable way to trigger the problem, and eventually identify the cause. It occurred when the ring buffer wrapped for the third time, and the message-end marker was exactly at the end of the buffer, and another message-end marker was left behind at the start of the buffer.)

Hi Folks,

I am once again looking for help. I,m not sure I understand how to set the descriptor set values for Socket_Select.

For Example, to create a set with just Socket 16, is it as simple as
DIM set% 4
set%!0=2^16
set%!1=0
set%!2=0
set%!3=0

EDIT: It appears that the correct code is
DIM set% 4
set%!0=1<<16
set%!1=0
set%!2=0
set%!3=0

Sockets generally are an uphill struggle

I would love to recommend the use of the URL fetcher. It’s a hell of a lot simpler, except…

• Due to an inadequately thought out API, following redirections is a pain in the ass. You’ll get the error 301 or 302 back, but there’s no mechanism (as yet) to extract the Location string and pass it to the client. So you need to repeat the action requesting to have the headers returned, and parse it yourself.
  Unnecessarily complicated, it really ought to return the error response and then have the location available as the data payload.

• Is ZLib still broken? Are we still handing out broken builds of AcornHTTP? :-(

Otherwise, using the URL system is about as simple as creating a session handle, telling it what URL you want to fetch, and constantly polling ReadData until something happens (note the status flags, as this is also where the connection takes place). You pass it a full URI, it handles http/https for itself

I use this in Manga now (broken module notwithstanding) as it’s a hell of a lot simpler and less bother than dealing with sockets directly, and switching to/from TLS is as simple as changing the prefix from “http://” to “https://”.

I can understand sockets for dealing with specific protocols, or rolling your own, but for fetching web pages? Nah, it’s just so much bother. Toss in encryption and… oh bother bother bother.

This is very minimal, but it works:

REM >URLFetch
REM by Rick, 2019/12/26

ON ERROR PRINT REPORT$+" at "+STR$(ERL) : END

buffersize% = 256
DIM buffer% buffersize%

REM The following line written like this "because Textile"
PROCfetch("ht"+"tps://www.riscosopen.org/forum/posts")

END

:

DEFPROCfetch(what$)

  LOCAL session%, status%, reply%, bytesread%

  SYS "URL_Register" TO , session%
  SYS "URL_GetURL", 0, session%, 1, what$

  REPEAT
    SYS "URL_ReadData", 0, session%, buffer%, buffersize% TO ,,,, bytesread%
    SYS "URL_Status", 0, session% TO status%,, reply%

    IF ( (reply% <> 0) AND (reply% <> 200) ) THEN
      PRINT "Unexpected server response "+STR$(reply%)
      SYS "URL_Deregister",, session%
      ENDPROC
    ENDIF

    IF ( status% AND 16 ) THEN
      IF ( bytesread% > 0 ) THEN
        PROCdumpbuffer(bytesread%)
        bytesread% = 0
      ENDIF
    ELSE
      REM Waste some time while the brew matures.
      PRINT ".";
      WAIT : WAIT : WAIT : WAIT : WAIT : WAIT
    ENDIF
  UNTIL status% = 32

  IF ( bytesread% > 0 ) THEN
    PROCdumpbuffer(bytesread%)
    bytesread% = 0
  ENDIF

  SYS "URL_Deregister",, session%
ENDPROC

:

DEFPROCdumpbuffer(size%)
  FOR l% = 0 TO (size% - 1)
    b% = buffer%?l%
    IF ( (b% < 32) OR (b% = 127) ) THEN
      IF ( (b% <> 10) AND (b% <> 13) ) THEN b% = 46
    ENDIF
    IF (b% <> 13) THEN PRINT CHR$(b%);
    IF (b% = 10) THEN PRINT CHR$(13);
  NEXT
ENDPROC

Select is called with 5 values IN REGISTERS as below.

The first of these is the number of sockets we are prepared to work with on this call. I always use FD_SETSIZE% which is set to 128 in my version of the library. Since the system limits the number of sockets to 96, that would be a better value.

The next three registers hold addresses of memory locations where bitmaps for the sockets are held. In my case, each memory area is 4 words, i.e. 128 bits in size. So if I wanted to select socket 3 for reading, the first word in readfds% would hold 1<<3

The final parameter is the address of a two-word (64 bit) area holding the waiting time in seconds in the first word, and in microseconds in the second word.

DEF FNselect(ndfs%, readfds%, writefds%, exceptfds%, timeval%)
  LOCAL R%,V%
  errno%=0
  SYS "XSocket_Select",ndfs%,readfds%,writefds%,exceptfds%,timeval% TO R%;V%
  IF(V%ANDVflag%)=Vflag% THEN errno%=!R%:R%=SOCKET_ERROR%
    IF errno%>error_base% THEN errno%-=error_base%
=R%

I call it as here:

REM enable broadcast port input
PROCFD_SET(broadcast_s%,readmask%)
REM enable listen port input
IF listen_s%>-1 THEN PROCFD_SET(listen_s%,readmask%)
REM enable input from server
IF server_s%>-1  THEN PROCFD_SET(server_s%,readmask%)
REM request immediate return (100uSec)
poll_delay%!_tv_sec%  = 0
poll_delay%!_tv_usec% = 100
events%=FNselect(FD_SETSIZE%, readmask%, writemask%, NULL%, poll_delay%)

In this case, I am looking for input. The writemask is set to 0, the exception handler mask address is 0.

In the null poll return code I then check whether there is anything received. If using Socketwatch then this can be done using the poll return code 13, as well or instead.

Sockets are simple, if you can multithread (a solution that works in RISC OS is to copy the threading cheat used in Contiki OS).

That said most RISC OS programmers do not take the effort to do any form of multithreading, because it is not natively supported by the OS. It needs better taught that Multithreading on any OS is fairly simmple regardless of what the OS and its libraries support.

So look at some of the examples from multithreaded OSes, then think about how that looks with a simple threading technique like that of Contiki OS (protothreads, cooperative multithreading).

Rick, I agree that working at the socket level is unnecessary when things like the URL fetcher exist. Why write your own HTTP client?!

Has anyone raised a bug for the 301/302 issue? Sounds like something entirely solvable, assuming the sources for the fetcher is in gitlab!

Select is called with 5 values IN REGISTERS as below.

Thanks for the explanation. It all helps. Thie mist is starting to lift.

Of curiosity (as I am a strong believer in not reinventing the wheel), does anyone have a URL to get URL Fetcher ?

Potential nevermind. I have looked for it in google. As it is something I had never used I am surprised to find it is part of the OS?

Wow, I thought I at least knew of every significant part of RISC OS, and as I spend a bit of time everyday crawling through the source of the OS nowdays, this is a big WOW how did I miss that, moment.

AND as an FYI for any interested, Rick does a great job of describing “URL Fetcher”s basic usage here:
https://heyrick.eu/blog/index.php?diary=20191226&keitai=0

This provides for a simpler native API for fetching data than I am aware of as a native API on any other OS. Most other OSes you need to add some extra libraries to get this level of simple functionality for fetching data from a given URL.

This has my attention. I am still trying to figure what protocals it supports, so looking for the documentation. Obviously it supports HTTP and HTTPS, though what about FTP, Gopher, etc?

Found the documentation, and it does support FTP, NNTP, Gopher, as well as HTTP and HTTPS:
https://www.riscosopen.org/wiki/documentation/show/URL%20Fetcher%20API%20Specification

Though it looks like we are missing the support modules for the other protocals. So this is a bit of a let down. Though for HTTP and HTTPS we have it made in the sun.

Though it does give the concept for a secondary module, allong the same lines though for upload instead of download. Perhaps call it “URL Poster” or similar. Then even the use of HTTP ‘POST’ commands could be with us.

It can already do POST (as well as PUT, DELETE, etc).

Kool. I had not gotten that far in reading it when posting above, and was a bit misslead on it being a ‘fetcher’. Now the only question is if someone wants to reimplement the other protocal support modules? Or if they are available somewhere already?

https://gitlab.riscosopen.org/RiscOS/Sources/Networking/Fetchers, although NNTP doesn’t seem to be there. I’m not sure whether it ever existed, or whether it was only documented as “this is the sort of thing that can be done”.

I’ve played with the Gopher fetcher in the past and can confirm that it works.

I am once again looking for help. I,m not sure I understand how to set the descriptor set values for Socket_Select.

Don’t bother with it, just read/write/accept non blocking. If you want to know the details of select read the linux man page

As it is something I had never used I am surprised to find it is part of the OS?

It was made widely available way back in 1997 in “Universal Boot” to accompany the launch of Browse and Java. Along with an extended set of Toolbox modules and the Nested WIMP.

Don’t bother with it, ..

I have the code working now. Using the Socket_Select SWI gives my TCP/IP client the ability to wait a long time in a Wimp_PollIdle loop whilst the server collects the data needed in the response.

If you want to know the details of select read the linux man page

Thank you for that.

It was made widely available way back in 1997 in “Universal Boot” to accompany the launch of Browse and Java. Along with an extended set of Toolbox modules and the Nested WIMP.

That would explain why I did not really notice it. It came about with updates that I avoided, and then shortly thereafter the split reality made me ignore the official RISC OS channels until the release of the RPi (and after RO Ltd gave up there side of the game).

Now I should have done more research on what was in RO5 when I upgraded from the RiscPC 600 (original ARM610 CPU, no x86) and A7000 to an RPi 1B in 2012, though I did not.

I am out of touch on some things. I just last month found out that DOSBox runs on OSes other than RISC OS, and was originally developed for other OSes at that.