As others have said, if it's clearly stated on invoice and customer actually received a copy of invoice (this will help if they call their credit card company to dispute, and a lawyer has limited room to play except for a simple bully tactic via a letter - after that it will get expensive for the customer to pursue for a few hundred dollars - risk/reward) then you have a contractual right to stand your ground on the policy - however, there is the goodwill aspect especially if you can move the product quickly.
That's your call and how you manage your business' brand/reputation.
Maybe an alternative to the re-stocking fee language is a non-refundable deposit portion of the special order sale, in the future. This will require 2 transactions so you only refund the refundable portion versus trying to split an single transaction which is more likely to result in a dispute.
I assume you pay for the product in full when you order or do you pay when it is eventually shipped to you? That is - are you holding the float in the meantime or are you funding the money between the time of the refund and when the backorder is filled and you can sell it from inventory?