Is the expression 'Do you speak English?' confusing in Korea?

When I try to ask if someone can speak English in Korea, I use the expression '영어 하세요?' However, it seems that the other person gets confused every time I use this expression. For example, when I used this expression with a staff member at a café in Gangnam, the person often responded with '영어?' After that, I often had to say '한국말 조금 해요' for them to understand. I wonder if other foreigners have had similar experiences. I would like to seek advice on why '영어 하세요?' is such a difficult expression to understand, or if there are more natural expressions. What aspects can be improved when using this expression? If you have any experiences or advice, please share.