We have this phenomenon where I live where people won't leave your house. Just as they're getting up to go they'll start telling another story. They'll put their shoes on and have another tale to tell. Then with one hand on the door they'll be reminded of something else.
Please get out of my house. I love you, but go.
Image Transcription:
Everett and Mrs True are entertaining two guests. One of the guests stands to leave: "Well, True, I guess we'd better be going home."
Everett rises out of his chair, knocking it over: "I'm glad to hear you say that! I'm tired and sleepy and I have to get up early in the morning! For heaven's sake, don't sit down. Good night, GOOD NIGHT!!!!"
I wouldn't worry too much about getting three down votes. Honestly, those are rookie numbers, and I'm sure as a fellow queer person you'll get many more for just existing. Which is incredibly sad, but that's life for you.
Some instances actually specifically disallow downvotes for similar reasons. My home instance doesn't allow them because brigading has been a problem in the past.
At the end of the day, you can't ever know exactly why somebody's downvoted. The question you have to ask then, is "does it matter?"
I'm not saying that to diminish your feelings. It hurts when people are hurtful. But downvoting someone for being different is the refuge of hateful people who are otherwise impotent at effecting change in their own life. It's sad, miserable, and frankly pathetic.
Consider it another way: if transphobes are seeking to make you feel bad by giving you a thumbs down, then not giving a fuck becomes an act of defiance and a source of strength. Haters gonna hate.
As to your specific question wondering if maybe you should ignore people asking about your pronouns: that's up to you. You don't owe anyone an explanation of your identity - particularly not if someone is asking in bad faith.
If someone asks you and you want to tell them "no," again that's up to you. But I can practically guarantee you're going to get downvoted when you could have ignored them.
For what it's worth, if anybody can't figure out your pronouns from the context of how you talk, that's kind of on them.