I'd suggest a fabric shaver (aka: clothes defuzzer). It's designed to remove lint and fuzz from clothes without damaging the fabric. It'll likely remove the slightly larger fibers as well.
Doing a quick online search, there seems to be a wide variety of them as well as a wide variety of retailers having them available.
Random image from the internet for example:
There's also a different type of lint remover that isn't electric or adhesive, which might work.
Alternate lint remover
As I was looking for the defuzzer, I stumbled across another type of lint remover.
I'm not affiliate with this product, retailer, or site in any way, but it is a more specific suggestion, so I'm including the link along with the pic.