You can be exempted on the basis of being covered by the limited capability for work exceptional circumstances rules, usually called Regulation 29(2)b
As far as I understand it, Regulation 29 relates to limited capability for work, i.e. for entry to the WRAG. I believe it's Regulation 35 that deals with the limited capability for work-related activity exceptional circumstances, leading to the support group if you satisfy it.
I would say appeal. I realise it's probably stressful, but it seems to be the only way to get them to properly look at your case.