Yes, it can mean, "I don't want to do that." It can also mean, "I don't want to do anything with you, but I don't want to be so rude as to say so." I remember my feelings being hurt, when I first figured that out.