How do you remove fuzz from spandex?

So I got a bunch of foamy stuff from this matt on my black spandex shorts. And I can't get it off D: Help! Btw already tried lint lifter, didn't work. Thanks!