My guess is your program (either yours or some demo program) just has a fixed table with what combinations of the sensor values give which colour. If you have a setup with a particular ambient light, it kind of needs calibration. You should tell your system what colour each paper is. And your system would set up a lookup table according to that. After that, the system could recognise any colour.

On forum.arduino.cc, someone suggested to add pullup or pulldown resistors to the pins. Which pins would benefit from them?