If that rubber is starting to degrade to the point where its wearing off from cleaning it, or some mild cleaner or Armor-All is going to wreck the surrounds - you may be ready to replace the surrounds anyways
As for the reactiveness of the coil/suspension mechanism of that type of speaker - the spider takes on most of the work, NOT the surround. The surround is just there to seal it up and let the speaker "float" in the middle.
You can order parts and DIY the surround or bring it/ship it to a place to have someone else do it if you are not comfortable with that kind of thing. I used to replace surrounds amongst other repairs of speakers, amps and other audio/electronics equipment.
TLDR; just clean it, if it gets wrecked it can be fixed.use oil or armor-all, Windex, diluted vinegar...whatever you do DO NOT use acetone - it WILL dissolve the rubber.