It is not acceptable to use the word "retarded" in a derogatory manner. It does not mean "foolish" or "stupid". It is hurtful, rude, and just plain assholish to use this word. It is even more assholish to try to defend it. So just don't. It won't fly.
It doesn't matter if you "don't mean it that way", any more than it matters that your best friend is gay when you call someone a fag. It doesn't matter if you have nothing against the disabled, any more than it matters that you're really not a racist when you joke about how cheap Jews are. It doesn't matter. What you're saying is hurtful. And never acceptable.
And you know what else? If you hear someone say it, or read someone who uses it? CALL THEM ON IT, every bit as much as if they said 'he's such a nigger' or 'she's such a cunt' or 'what a fag!'. Those words hurt. And so does "retard".
So just stop.