I think you'll need to get the whole armrest unit out, and the procedure you need to follow has been very kindly detailed on buellwinkle's website here: http://www.buellwinkle.com/r170/
When you get there, click on the DISASSEMBLY icon, and thereafter on DASHBOARD AND CONSOLE, and in that set of instructions go right on to page 4 and start by removing the radio at step 16 (wiggle something like a pen knife blade in the slits at each side of the bottom of the radio fascia and jiggle the fascia out a bit at each side in turn), skip step 20 for your purposes, and carry on to step 26, which is as far as you need go.
Meantime, after removing the screws in step 24, you have to pull the brake handle off- it's very stiff unless you squirt wd40 or similar down between the rubber and the steel shaft. Then uncrew the 2 screws fixing the rear wall (inside the storage box of the arm rest unit) to the bulkhead behind the seats, pull the whole armrest unit backwards while pushing it down at its front, to free it from the gear shift section seen in step 24.
It is getting the brake handle off, and access to the 2 screws in photo 24 that are the most tiresome parts
Take it easy and give yourself an hour or 2, including a break for a beer after the brake handle has succumbed!