If it's spring tails, which it sounds like to me, then just leave them. As Tindo said they eat decaying matter.
I once got rid of them on an orchid dousing the media in soapy water, then washing through with plain water... but the are in all my plants (orchids and non-orchids) and eventually they spread back to the one I had cleaned up.
They seem to get more numerous in ones that need repotting, but I have them in everything and they do no harm.