Its a bug in QuickSight. The most recent instance similar to yours had an input that the issue started occuring when the dataset had joins between tables. Your case seems similar.
The solution is to create a calculated field to multiple the latitude/longitude by 1000 and use the same in the map