USB MIDI?

So mom has been in hospital and I may have some form of bronchitis. As you can imagine, this week sucked. So getting this done took a lot longer than expected when my mind wasn’t on much of anything at all for the last… <counts on fingers, runs out of fingers> days.

Anyway. This iteration is here now.

Actually, I lie, it is here:

http://www.heyrick.co.uk/random/usbmidi_002_alpha7.zip (23KiB)

Here’s what’s worth knowing (copy-pasted from the Versions file):

• Various internal code tweaks – mostly to remove “magic values”.

• Module should no longer give odd aborts when quit or reloaded; this was due to the “get some data” ticker not being cleared.

• The format of the RX buffer has changed. It is now 2048 bytes in length (enough to hold 256 commands), and every MIDI packet takes eight bytes, as follows: [snip description] Actually asides from making the buffer larger, and supplying valid data for the timestamp, the format has been like that for a while now. You’re supposed to use RxByte or RxCommand, not poke around the buffer directly, so this is mostly academic…

• MIDI_Init now supports the following options:

        R0 =     0 - Reset internal state of device
             bit 1 - Clear RX buffer
             bit 2 - Clear TX buffer
             bit 4 - Clear current error ('B' and 'D' yes, 'X' and '/' no)
             bit 30- Place Real Time messages into RX buffer
             bit 31- Don't do special actions on Real Time messages
      (other bits not supported or not relevant)

• There is another discrepancy with the Acorn MIDI module. If you are receiving MIDI clock ticks, what happens if you receive these on two ports at once?
  Because the API was written a long time ago, the generic selection will apply to the module as a whole, however the realtime/clock options are specific to each port so can be controlled as follows:

        MIDI_IgnoreTiming
          R0 = 0 (receive), 1 (ignore)
          R1 = "RICK" (&4B434952)
          R2 = Which MIDI port to apply this to
          [will currently fault ports > 0]
          
        MIDI_Init
          R0 = Bits 30 (RealTime->buffer) or 31 (RealTime ignore) set
          R1 = "RICK" (&4B434952)
          R2 = Which MIDI port to apply this to
          [will currently fault ports > 0]
          [when used in this way, all other bits of R0 are ignored]
          
        MIDI_FastClock
          Clock type applies to the module as a whole. It is not
          possible to have one port timing by MIDI clock and another
          by ms/cs ticker.

• Just to clarify – note that MIDI_TxByte will:
  a. Collect bytes until a valid command is present, USB data works by sending entire commands, not individual bytes.
  & b. Does NOT support the use of running status.
  & c. Does NOT (yet) cope with SysEx packets. There is minimal support in the TxByte routine, but currently no code to assemble the actual packets for USB transmission.

• The MIDI module now supports timing. Timing works in one of two ways:
  1. If you do NOT enable the Fast Timing, then the MIDI time stamp is an incrementing MIDI beat counter.
  2. If you enable Fast Timing, then the MIDI time stamp is the centisecond counter when the data was received (, multiplied by ten). [the old MIDI module used the millisecond clock to timestamp data the instant it was received, but we poll on a centisecond-based ticker so there is not much point in making the time stamp more accurate than this]

• NOTE – if you select slow timing (MIDI beats) and you then tell the MIDI module to ignore such timing (MIDI_IgnoreTiming) then the resultant data will have NO timestamp.

• NOTE – the timing value is that in force at the time the MIDI command was received. If you change the timing type, this will apply to NEW data, not retrospective to old data still in the buffer…

• Something that was never made clear is what happens if you have, say, a three-byte MIDI command in the buffer and you RxByte and then RxCommand ? In order to rationalise this, the behaviour is as follows: If a command has been partially byte read when an RxCommand is received, the MIDI module return the SAME command as the one we were reading from. In this way, RxByte can be used to “peek” at the data.

• Accordingly, RxByte now works.

• The MIDI beat counter increments for every Timing Clock message received (unless disabled with MIDI_IgnoreTiming) and the Song Position Pointer increments every six beats. This happens in the background regardless of whether or not Fast Clock is in use.
  The MIDI module DOES NOT generate timing messages, it uses those generated by the connected equipment.
  This behaviour is disabled (entirely) if MIDI_IgnoreTiming is used.
  The following SWIs are concerned with this behaviour:
  • MIDI_TxStart – resets the MIDI beat counter to zero.
  • MIDI_TxContinue – no effect.
  • MIDI_TxStop – no effect.
  • MIDI_TxSongPositionPointer – also updates internal counter.

• MIDI_TxProgramChange now has an addition to make it simpler to set Yamaha XGLite voices.
  R0 on entry is the voice/instrument/patch number.
  R1 on entry is the bank select. It comprises (MSB << 8) + LSB.
  R2 on entry should be &4C475859 for R1 to be used.
  Therefore, to select “Velocity Strings” (XGLite voice #299, MSB 0, LSB 45, program 49) you would enter:
  SYS "MIDI_TxProgramChange", 49, 45, &4C475859

• You may have noticed the (x10) in some of the above descriptions regarding time stamps. For compatibility with the Acorn module, the default behaviour is to provide (and expect) millisecond values. However if you set bit zero to MIDI_Options, then the module will revert to using centisecond times (roughly equivalent to SYS “OS_ReadMonotonicTime”). You could also use BASIC’s TIME if you set it like this at the start of your program:
  SYS "OS_ReadMonotonicTime" TO t% : TIME = t%
  but note that TIME can be reset, the monotonic ticker cannot.
  The correct way to set MIDI options without affecting any of the other options values is:

        SYS "MIDI_Options", -1, -1 TO options%, delay%
        options% = options% OR 1
        SYS "MIDI_Options", options%, delay%

This change is NOT persistive. If the clock type is altered (MIDI_FastClock with R0 >= 0), then you would need to do this again to select the Ticker mode of operation instead of the default fake millisecond clock.

      In other words:
        SYS "MIDI_FastClock", 0  (or do nothing, it's the default)
          will use the MIDI beat clock timing
        SYS "MIDI_FastClock", 1  (or any positive number)
          will use the fake millisecond timing derived from ticker
        SYS "MIDI_Options" with R0 bit 1 set
          will use the centisecond ticker
          [you don't technically need to call MIDI_FastClock first in
           order to choose the ticker clock; just call MIDI_Options]

Edit: Just in case anybody asks, &4C475859 is “YXGL” (for Yamaha XGLite). The procedure should work with other extensions to General MIDI that uses bank selection to permit a greater range of available voices…

Rick, I did get hold of a half decent usb keyboard (M-Audio Keystation) -good pdf for it too. I jotted down some notes to get the miditestY program playing ball.

DEFPROCparse_three(a%, b%, c%)
REM rick typo &D0 insteadof &E0
 IF ((a% AND &F0) = &E0) THEN
DEFPROCparse_control(a%, b%, c%)
REM rick typo AND 32 not needed?
IF (b% < 64) THEN
CASE b% OF
WHEN  0: PRINT "Bank Select [MSB]";
....
ENDCASE
REM rick typo b% instead of c%
REM  IF (b% = 0) THEN PRINT " MSB";
REM  IF (b% = 32) THEN PRINT " LSB";
REM  IF ((b%  THEN PRINT " (coarse)";
REM IF ((b% > 32) AND (b% < 46)) THEN PRINT " (fine)";
ENDIF

Not what I had planned to do today, but life’s like that…

It is a small change. The naff-device option (R1 to MIDI_Options SWI) is now either zero or non-zero, and the delay that is used is fixed at 320 microseconds per byte transmitted, plus an extra 320µs to allow for delays in the adaptor. This value, 320µs, is how long it takes for a byte to be sent via serial MIDI.
In this way, transmitting via USB should no longer flood the adaptor and cause data loss.

http://www.heyrick.co.uk/random/usbmidi_003_alpha8.zip (23KiB)

I decided upon this course of action after looking at a bit of Linux code that appears to imply a bunch of hardwired delays for some USB devices; and it seems that there is simply no obvious way for a USB MIDI device to signal “hold off a mo, I’m busy”. This, I guess, is the forward march of technology. One step forward, one step back…

The method used relies upon HAL calls, so if you managed to get the MIDI module working on something that wasn’t running RISC OS 5… then this delay won’t be available to you. It will throw an error when you try to set it via MIDI_Options SWI in this case.

The “delay factor” reported in the device information is no longer meaningful. What you see is how many ticks per second your hardware timer runs at. For RPCEmu (and I presume RiscPC real), this is 2000000 (2MHz). For the Pi, it is 1000000 (1MHz). Other devices? Something along these lines… The delay itself is fixed at 320µs per byte (plus one) and it is calculated on the fly as necessary so nothing to report. ;-)

Oh, and miditestY has not (yet!) been fixed as per Ronald’s description above…

it seems that there is simply no obvious way for a USB MIDI device to signal “hold off a mo, I’m busy”.

There isn’t.

There isn’t with serial MIDI either.

There isn’t with serial MIDI either.

With a UART (or hardware serial chip) you have the ability to know when a byte has been sent so you can then send the next one; plus there is no issue of overrunning capacity as the serial hardware is running at the serial speed.

With USB, we have a buffer of indeterminate size (testing implies about 20 bytes) and a link that is running many times faster than the serial speed and no apparent way of detecting how much space is available in the buffer or anything like that.
It seems, therefore, that the only method left is for the sender to clock out each packet with a delay for how long this packet should take to put on the wire.

Isn’t flow control performed by the MIDI device? I would have thought IN endpoint data flow would be limited by serial speed and OUT endpoint data flow would be NAK’d by the MIDI device until room was available in its buffer. I suppose you are saying that the MIDI device just loses OUT data if it is overrun – odd.

My MIDI module takes data sent to it from Maestro and formats it in MIDI packets then sends it to the file handle for DevicesFS (etc) to process.
A while back in this thread, David posted some mp3s that sound quite different to the same thing playing on my Yamaha (with direct MIDI input); the only explanation we could come to was that for complicated music with 6 or more notes playing at once, if they were all being sent at the same time, the adaptor was simply processing the 4 or 5 that would fit in the buffer and throwing away the rest. This correlates with what I observed tying the serial output to my ’scope and then throwing data at the device. It was just discarding data.
This is actually well known, cheap interfaces are horribly badly implemented, however:

1. it probably isn’t helped by no mention in the spec of a control endpoint for device management (ie can we send data now?)
2. Saying “your interface is poo” isn’t a good response to a person with one of these (even if it is true) when they can reply “well it works on windows!”, doesn’t do us any favours…

An endpoint for flow control doesn’t help unless the device has a big buffer. Thats why USB serial devices do their own flowcontrol so I don’t think that is the problem. Limiting the output to 20bytes per millisec because the serial buffer is 20 bytes doesn’t sound a good solution either as you need to be able to send 32bytes per millisec for it to work properly. If you wait to see if the usb buffer is empty before sending more then it’s too late you will have missed the next millisec slot.

You would expect the USB MIDI device to have at least a 32byte USB buffer – it probably double buffers with 2 32byte buffers to get transfers every millisec. At least that’s how microchip cpus with USB work.

You say notes were missed. Were they missed or could they have been delayed a millisec?

Limiting the output to 20bytes per millisec because the serial buffer is 20 bytes doesn’t sound a good solution either as you need to be able to send 32bytes per millisec for it to work properly.

I’m not. I’m asking the machine to delay 320uS for each byte transferred plus one to account for time taken within the adaptor.
Thus, it won’t be able to saturate the serial side, but it is better than losing notes.

In technical terms, this would be 0.00128 seconds per three byte “NoteOn”, which might seem slow, but remember it would physically require 0.00096 seconds to put on the wire as serial MIDI isn’t very fast, plus however much time the controller chip (some sort of MCU?) needs to actually convert USB packets to raw MIDI.

This isn’t too say it is definitive. I have not yet tested this at all as my keyboard accepts USB directly and hasn’t (as far as I’m aware) lost data in normal use. I’ll need to hook up the old Roland keyboard to the serial adaptor and see how it behaves. But… where did I put it? Hmmm! And where is the power lead adaptor I made? Hmmm!
;-)

You would expect the USB MIDI device to have at least a 32byte USB buffer – it probably double buffers with 2 32byte buffers to get transfers every millisec. At least that’s how microchip cpus with USB work.

That is what is claimed in USBInfo (http://heyrick.co.uk/random/pics/midi_dongle.jpeg), but practical experience suggests either the buffer is smaller or the assembly buffer (where outgoing serial data is assembled) is the problem. I suspect the latter due to data just being lost.

You say notes were missed. Were they missed or could they have been delayed a millisec?

David has removed his broken version of FollowYou, so I’ll need to see if I still have a copy. If basically played with the bass line mostly missing. You can, alternatively, see the data here (http://www.heyrick.co.uk/blog/index.php?diary=20141122) as spikes on the scope. Throwing data at the device would output five commands, sometimes six. Adding a delay meant…well, it is all explained in the blog post. ;-)

I rather suspect this is but a workaround for some extremely shoddy hardware, so we perhaps shouldn’t be surprised at needing to do weird things for it. <sigh&gt

As I see it the device has to have the capability to store 1 packet (32 bytes) otherwise it can’t work and I can’t see how the serial end can be overrun either. I’d much rather believe the problem is with RISC OS USB.

I don’t see that delaying the byte input does anything. Data is only sent over USB every 1ms (USB1 device) it is not sent immediately. So if you start with an empty buffer and put 1 byte in it a transfer of 1 byte is queued with the USB controller the transfer of this data happens in the background. In the mean time you buffer more bytes – you may for example buffer 32 bytes in the millisec. When the first one byte transfer finishes a new transfer is queued made up of all the bytes now in the buffer which in my example is 32. While that is being processed by the controller you are filling the buffer again and so on.

I don’t know how MIDI data is formatted but if it doesn’t contain any timing information I would be inclined to buffer data separately in 32byte chunks and have a multiple of 32byte USB buffer ie only insert 32byte chunks into the USB buffer.

32 bytes would be 8 MIDI messages. That’s too many.

Delaying is intended to slow down how fast data is sent out. It works out just a little over 1ms per NoteOn command so rather than dumping a bunch on to the device at once, the delay will spread it out over several transmissions, just a little slower than real serial MIDI speed.

I should stress – I do not observe these issues with my Yamaha keyboard, so I’m inclined to believe the device is the problem and not RISC OS (and a quick look at Google will come up with enough material to support that!).

For reference: every serial MIDI command is converted into a four byte USB MIDI command. There is no “running status” (a shortcut where multiple NoteOn commands could be one NoteOn followed only by a sequence of note values) and System messages cannot interrupt normal data transfer. Everything is converted to a four byte value where the first byte tells the message type and which MIDI port this is, the second byte is the MIDI command byte and the third and fourth bytes are the parameters (or zero if not used). Basically it is zero padded MIDI data with a prefix byte.

Looks like it might be even more fun trying to get sensible data going the other way… https://audiodestrukt.wordpress.com/2012/11/18/inexpensive-usb-midi-interfaces/

A few years ago I wrote the embedded code for a commercial 4 channel USB-serial MIDI converter. I believe there are now several devices from that manufacturer that use or used my code. I recall the USB bulk endpoint as being 64 bytes (16 MIDI commands other than SysEx), and each buffer was several hundred bytes long.

MIDI should be capable of handling a chord on several instruments simultaneously.

Don’t get confused by the rate at which serial MIDI operates. It’s irrelevant. Buffering should take care of everything.

If the converter you’re using drops MIDI commands, throw it away. Don’t compromise your code by downgrading it to handle a defective-by-design device.

I recall the USB bulk endpoint as being 64 bytes (16 MIDI commands other than SysEx), and each buffer was several hundred bytes long.

That’s because you “did it properly”. ;-)
I’m sure these devices retail for more than a fiver.

MIDI should be capable of handling a chord on several instruments simultaneously.

Indeed. The “about five” notes that I’m seeing is a chord plus two notes of melody on a single instrument. Which is about sufficient to replay your daughter’s piano recital…if she’s six and not a prodigy…

Don’t get confused by the rate at which serial MIDI operates. It’s irrelevant. Buffering should take care of everything.

…provided the buffering is worth a damn, that is. If not, this is when the rate that serial MIDI runs becomes important.

If the converter you’re using drops MIDI commands, throw it away.

I would love to be able to say “your converter is fetid poo, hit it with a large hammer and then chuck it in the bin”; but unfortunately we run into the “works with Windows” problem. I don’t have a USB analyser so I don’t know if Windows throttles output to these devices or what, but if something appears to work on Windows but fails on RISC OS, the average person isn’t going to think that their hardware is crap and couldn’t sensibly buffer, they will instead think RISC OS is crap.
Quoting this post, David Feugey said: Nota: everything sounds OK on Windows, with the same setup.
Need I say more? ;-)

It seems that there are a number of non-compliant devices that appear unable to correctly buffer MIDI data. So much so that FreeBSD has taken the step of implementing a USB “quirk” for the device where it will only send one command per millisecond slice, which is effectively what my module will do for such a device (as the calculated delay works out to be slightly over a millisecond for a NoteOn). http://fxr.watson.org/fxr/source/dev/usb/quirk/usb_quirk.c?v=FREEBSD9#L452

Don’t compromise your code by downgrading it to handle a defective-by-design device.

This is not the default behaviour. To my mind, anything that doesn’t behave like my Yamaha keyboard (an unexciting PSR e-333 with USB input) is broken. As it uses USB MIDI directly, it does it correctly. SysEx and all.

However it is necessary to have a compatibility option that can be enabled for these crappy Chinese pieces of rubbish because telling a user “your hardware sucks in fundamental ways” is never going to be the popular option.

Perhaps the only reason this isn’t more of an issue is because our only MIDI software at the moment is Maestro and, well, who can forget this utter train wreck (and that’s not even MIDI!)?

Did anything happen with the MelIDI open sourcing mentioned earlier in the thread?

I ask because I saw a question about MIDI sequencing software on another RISC OS forum.

That would be great.
http://www.melidi.co.uk/

And perhaps some 32bit port of Anthem too
http://www.arsvcs.demon.co.uk/rci/sound/anthem/anthem.htm

Even if the best is probably MidiWorks

I have a part developed multi-track clips on up to 16 track recorder and player written in Joe Taylor’s AppBasic
which App has to be on icon bar for it to run.
Stopped writing 5 years ago as no time. I assume it still works.
Is for my own needs to record keyboard improvisations using just
a Yamaha DGX Keyboard and do basic editing
I have Anthem which was stripping sustain pedal against my will in files
so it puts sustain back in.

It used to reliably play and record 3 or 4 tracks but rest not so reliable.
The user Interface is unconventional and a bit intricate as last thing I meant to sort out.
Developed it with parallel port midi into a Risc PC and A7000+.
I do not know if will work on R pi.

If someone would like to test and see if fills a gap, it is available.

Stephen Foord

Hi Stephen. I’ll test it with pleasure (expect a little delay)
temp1267@riscos.fr

Hi, shoot me a copy….and if it doesn’t work with USB MIDI, I can figure out why. ;-)

heyrick1973 at yahoo dot co dot uk

Update of the MIDI module. Entirely recoded the USB device detection.

http://heyrick.co.uk/random/usbmidi_004_alpha9.zip

Code review (half-assed excuse to not go out and mow something) got some bugs fixed. And I even had time to mow for an hour before sundown while listening to Chihiro Onitsuka…

https://www.heyrick.co.uk/random/usbmidi_005_alpha10.zip

Here’s the blurb:

0.05  2017/04/18

      Now outputs the correct event numbers for receiving data (0) and
      error occurred (1).
      
      Now correctly cancels any outstanding callback on module exit
      (though the chance of this situation happening is pretty slim).
      
      Removed everything to do with delaying data for crappy hardware
      interfaces. The requested delay is kicked over to the HAL to do
      with the high resolution timer(s).
      
      Fixed a typo that stopped MIDI_TxByte SWI and *MIDIUSBSend from
      working (was discarding events due to not recognising the MIDI
      command because the command part was being extracted twice).
      
      Numerous tweaks and tidyups to the code.

Just to clarify the third one down – the original code tried to time and calibrate a busy-wait loop of stacking and unstacking a bunch of registers a few tens of thousands of times to cause a delay, but while it worked, the delay was very variable so it sort-of-worked at best. Then, rummaging around the HAL documentation, I discover a call that can delay for much shorter periods than possible with the 100Hz clock. The sort of periods I am looking for. So I just use the HAL method and took out the now-unused code. Yeah, it sucks to delay the entire machine for microseconds at a time while sending data…but that’s the price you pay if you choose to purchase a USB to MIDI interface that costs less than a Happy Meal…