Using SPI Ports (SCK, MISO, MOSI) as General IO?

Hi all!
 

TL;DR: I'm out of general IO pins, I need just one more and was hoping MISO/MOSI could be the answer. Am I using them wrong?

I have a DFRobot Firebeetle ESP32-e board that I'm currently using with Arduino IDE. 

Currently I have a matrix of switches setup but ran out of standard IO ports to use (all ‘D’ labelled pins as well as A4). I notice in the documentation that SCK, MOSI and MISO are also listed as IO ports, but when I try to use these like the others for IO I get some weird behaviour. 

This consists of the first switch in the matrix (hooked up to MISO and D2) working perfectly. All other switches in the row hooked up to MISO seem to swap rapidly between itself, and the switch directly infront. All other rows behave as expected (even when hooked up to MOSI).

I have a second board and a second switch Matrix so I tried this again and could replicate the issue. I then tried swapping from MISO to a D pin and the row worked fine there without issues. 

The issue at this point is mostly that I'm out of available pins, don't have the real estate for a breakout board, and was hoping to sort of push through with just one more pin. Is there something I need to include in my Arduino Sketch to allow me to use MISO/MOSI pins as general IO? Are there any other pins I could consider using?