TCP/IP Stage 2 bounty

Yes, sockets are hard.

Made so by the lack of simple examples.

But, with encouragement, perserverence, and a bit of luck, something clicked into place and I now have a non blocking PROCedure which fetches a web page from a local server. (with adequate progress and error detection)

The PROCedure is currently about 40 lines long (with another 40 for debugging) and is called from the poll loop of an existing program. The bit I was missing was how to tell when the connection was made (or crash out when the connection was going to fail). The comment in the !Antispam source code about removing the loop repeating calls to Socket_Connect prompted me to do just that and repeat the Socket_Connect SWI until I get the response 56 (or an error response 22, for example, – connection rejected [wrong port used!])

Thank you all for your patience and help. If anyone is interested in this simple BASIC code example, I can post it here when I have tidied it up.

There really should be code to set the error_base% variable depending on the earliest Internet module version that supports this.

Test it’s behaviour, not it’s version number. There are at least 4 independent implementations of the RISC OS socket SWI interface.

But in this case you can just check the returned error number and adjust accordingly.

But in this case you can just check the returned error number and adjust accordingly.

For that you have to assume that no future internet module will ever return a standard RISC OS error number less than 256, unlikely but possible.

Instead of testing the version number it might be better to call one of the socket SWIs to provoke the return of known socket error, and see if it is ERRNO or module base+ERRNO.

@Alan, I think there are enough changes for take up the mantle of maintaining it :)

Sorry Dave, I was being lazy just sending the whole thing. The important change is the one in PROCsocketlib, with comment just after FD_SETSIZE% – 128 (96 would be better here).

It refers to changes in the PROCFD… and FNFD… code, where the masks are handled. I don’t have an original version of the library to see what changes I had to make there.

All the other stuff doesn’t affect the operation directly, just made some things compatible with the rest of my code. For example using FNisvar in the error handler – cleaning up after an error could cause an error if the variable hadn’t yet been created. ON ERROR OFF prevents that causing a loop, but I wanted tosuppress the message as well.

The HOSTS bit may be left over from when I was struggling to use name resolution – I’d be surprised if it is in use now, since several of the addresses have changed, and I haven’t kept that file updated.

Actually this thread has reminded me of another variation – VRPC returns error numbers with the base added. My other computers at the time I did the networking did not. This meant I had to make adjustments to error detection.

(My main networking code was done about ten years ago, giving me higher level functions to work with. I’ve had to revisit a few times, most recently about 3 months ago when I found another subtle bug caused by one of the edge cases, when handling the ring buffers. Since triggering it required conditions that depended on the exact size of the data being transferred, and so varied with, for example, the length of someone’s name, I was lucky when I founds a repeatable way to trigger the problem, and eventually identify the cause. It occurred when the ring buffer wrapped for the third time, and the message-end marker was exactly at the end of the buffer, and another message-end marker was left behind at the start of the buffer.)

Hi Folks,

I am once again looking for help. I,m not sure I understand how to set the descriptor set values for Socket_Select.

For Example, to create a set with just Socket 16, is it as simple as
DIM set% 4
set%!0=2^16
set%!1=0
set%!2=0
set%!3=0

EDIT: It appears that the correct code is
DIM set% 4
set%!0=1<<16
set%!1=0
set%!2=0
set%!3=0

Sockets generally are an uphill struggle

I would love to recommend the use of the URL fetcher. It’s a hell of a lot simpler, except…

• Due to an inadequately thought out API, following redirections is a pain in the ass. You’ll get the error 301 or 302 back, but there’s no mechanism (as yet) to extract the Location string and pass it to the client. So you need to repeat the action requesting to have the headers returned, and parse it yourself.
  Unnecessarily complicated, it really ought to return the error response and then have the location available as the data payload.

• Is ZLib still broken? Are we still handing out broken builds of AcornHTTP? :-(

Otherwise, using the URL system is about as simple as creating a session handle, telling it what URL you want to fetch, and constantly polling ReadData until something happens (note the status flags, as this is also where the connection takes place). You pass it a full URI, it handles http/https for itself

I use this in Manga now (broken module notwithstanding) as it’s a hell of a lot simpler and less bother than dealing with sockets directly, and switching to/from TLS is as simple as changing the prefix from “http://” to “https://”.

I can understand sockets for dealing with specific protocols, or rolling your own, but for fetching web pages? Nah, it’s just so much bother. Toss in encryption and… oh bother bother bother.

This is very minimal, but it works:

REM >URLFetch
REM by Rick, 2019/12/26

ON ERROR PRINT REPORT$+" at "+STR$(ERL) : END

buffersize% = 256
DIM buffer% buffersize%

REM The following line written like this "because Textile"
PROCfetch("ht"+"tps://www.riscosopen.org/forum/posts")

END

:

DEFPROCfetch(what$)

  LOCAL session%, status%, reply%, bytesread%

  SYS "URL_Register" TO , session%
  SYS "URL_GetURL", 0, session%, 1, what$

  REPEAT
    SYS "URL_ReadData", 0, session%, buffer%, buffersize% TO ,,,, bytesread%
    SYS "URL_Status", 0, session% TO status%,, reply%

    IF ( (reply% <> 0) AND (reply% <> 200) ) THEN
      PRINT "Unexpected server response "+STR$(reply%)
      SYS "URL_Deregister",, session%
      ENDPROC
    ENDIF

    IF ( status% AND 16 ) THEN
      IF ( bytesread% > 0 ) THEN
        PROCdumpbuffer(bytesread%)
        bytesread% = 0
      ENDIF
    ELSE
      REM Waste some time while the brew matures.
      PRINT ".";
      WAIT : WAIT : WAIT : WAIT : WAIT : WAIT
    ENDIF
  UNTIL status% = 32

  IF ( bytesread% > 0 ) THEN
    PROCdumpbuffer(bytesread%)
    bytesread% = 0
  ENDIF

  SYS "URL_Deregister",, session%
ENDPROC

:

DEFPROCdumpbuffer(size%)
  FOR l% = 0 TO (size% - 1)
    b% = buffer%?l%
    IF ( (b% < 32) OR (b% = 127) ) THEN
      IF ( (b% <> 10) AND (b% <> 13) ) THEN b% = 46
    ENDIF
    IF (b% <> 13) THEN PRINT CHR$(b%);
    IF (b% = 10) THEN PRINT CHR$(13);
  NEXT
ENDPROC

Select is called with 5 values IN REGISTERS as below.

The first of these is the number of sockets we are prepared to work with on this call. I always use FD_SETSIZE% which is set to 128 in my version of the library. Since the system limits the number of sockets to 96, that would be a better value.

The next three registers hold addresses of memory locations where bitmaps for the sockets are held. In my case, each memory area is 4 words, i.e. 128 bits in size. So if I wanted to select socket 3 for reading, the first word in readfds% would hold 1<<3

The final parameter is the address of a two-word (64 bit) area holding the waiting time in seconds in the first word, and in microseconds in the second word.

DEF FNselect(ndfs%, readfds%, writefds%, exceptfds%, timeval%)
  LOCAL R%,V%
  errno%=0
  SYS "XSocket_Select",ndfs%,readfds%,writefds%,exceptfds%,timeval% TO R%;V%
  IF(V%ANDVflag%)=Vflag% THEN errno%=!R%:R%=SOCKET_ERROR%
    IF errno%>error_base% THEN errno%-=error_base%
=R%

I call it as here:

REM enable broadcast port input
PROCFD_SET(broadcast_s%,readmask%)
REM enable listen port input
IF listen_s%>-1 THEN PROCFD_SET(listen_s%,readmask%)
REM enable input from server
IF server_s%>-1  THEN PROCFD_SET(server_s%,readmask%)
REM request immediate return (100uSec)
poll_delay%!_tv_sec%  = 0
poll_delay%!_tv_usec% = 100
events%=FNselect(FD_SETSIZE%, readmask%, writemask%, NULL%, poll_delay%)

In this case, I am looking for input. The writemask is set to 0, the exception handler mask address is 0.

In the null poll return code I then check whether there is anything received. If using Socketwatch then this can be done using the poll return code 13, as well or instead.

Rick, I agree that working at the socket level is unnecessary when things like the URL fetcher exist. Why write your own HTTP client?!

Has anyone raised a bug for the 301/302 issue? Sounds like something entirely solvable, assuming the sources for the fetcher is in gitlab!

Select is called with 5 values IN REGISTERS as below.

Thanks for the explanation. It all helps. Thie mist is starting to lift.

It can already do POST (as well as PUT, DELETE, etc).

https://gitlab.riscosopen.org/RiscOS/Sources/Networking/Fetchers, although NNTP doesn’t seem to be there. I’m not sure whether it ever existed, or whether it was only documented as “this is the sort of thing that can be done”.

I’ve played with the Gopher fetcher in the past and can confirm that it works.

I am once again looking for help. I,m not sure I understand how to set the descriptor set values for Socket_Select.

Don’t bother with it, just read/write/accept non blocking. If you want to know the details of select read the linux man page

As it is something I had never used I am surprised to find it is part of the OS?

It was made widely available way back in 1997 in “Universal Boot” to accompany the launch of Browse and Java. Along with an extended set of Toolbox modules and the Nested WIMP.

Don’t bother with it, ..

I have the code working now. Using the Socket_Select SWI gives my TCP/IP client the ability to wait a long time in a Wimp_PollIdle loop whilst the server collects the data needed in the response.

If you want to know the details of select read the linux man page

Thank you for that.