Don't forget to take your plants out of the pot first
One method I have heard of working is to put the empty pots in the dishwasher, and run through the longest cycle and use your normal dishwasher detergent. The only problem is, this may take several washes.
or
Clean off any loose residue or old soil, using a wire or plastric brush. (a wire brush can mark the pots though)
Mix a bleach solution using one part bleach to 10 parts water. Fill a bucket with this solution.
Sterilize your pot by placing it in the bleach solution to soak, then soak the pot in clean water for two to three hours. use a wire brush to get rid of any stuborn stains and rinse the pot again and soak in water for another hour to get rid of the bleach.
Let the pot completely dry and inspect it for remaining mineral deposits. If there are any deposits, you can make up a solution of one part vinegar to 10 parts water and soak the pot overnight in a bucket.