UPDATE: The American Heritage Dictionary agrees with you. They list /yôr/ as an acceptable variant pronunciation for your but not for you're. However, since they don't say what dialects this pronunciation is found in, I don't know whether the speculation in my original post below is correct.

ORIGINAL POST: In the U.S., there are a number of accents which don't have the phoneme /ʊɹ/ in moor. (My mother, from rural Illinois, had one of these accents, and I used this accent's pronunciation for many of these words when I was young.) In these accents, the phoneme is replaced by either /ɝ/ as in purr or /ɔɹ/ as in pore; the rule is that generally, words that are pronounced with /jʊɹ/ (pure, cure) the phoneme is replaced by /jɝ/, and words that are pronounced with no /j/ (poor, moor, sure), the phoneme is replaced by /ɔɹ/.¹ It may be that in some of these accents, you're and your get disambiguated by having you're rhyme with purr, and your rhyme with pore. I do know that pronouncing you're with /ɔɹ/ sounds wrong to me, whereas pronouncing your either way sounds fine.

I'm just guessing, but it's possible that one of these accents influenced your pronunciation of your and you're.

¹ This rule only works for the final syllable of words. Otherwise, the sound generally changes to /ɝ/.