This is always an odd one because it depends on how people ask the question, it’s can be asked in a genuinely inquisitive and friendly way or it can be asked in an aggressive way. I was born in the UK but if someone asks me where I’m from my default answer is usually “my parents are from India” because you can usually tell that’s what they really want to know so why not just give them the answer, it’s not something to be ashamed of.
Funnily enough, many, many years ago, on a night out with the lads, I was approached by an attractive woman. I initially thought she was chatting me up. Nope. She asked if was I Irish? 'Well, my parents are from Ireland', I responded. With that, she was off back to her mates. 'Yep. Irish'. It seems that there had been some kind of debate/bet about her being able to recognise Irish people. So, milky white people get it occasionally too.
But I am with you. Surely, even if a bit presumptive, awkward, and indicative of some unworldliness, most people asking you this question are probably just trying to be nice. But in the case of accents, the question is more understandable. I got it rather a lot when I lived in the US. And I was once complimented by a Mexican US immigrant on the standard of my English, presumably, not realising that we speak English in the United Kingdom (perhaps I should have said that 'I am from England'). I was somewhat lost for words.