I have used uMap before and would recommended.

Some features:

• not sure about the max amount of points you can place on the map but it's quite a mature software so I'd expect it to be capable of 20k points
• "Add POIs: markers, lines, polygons", i.e. markers would be you points
• "Embed and share your map"
• "Choose the license for your data"
• "Batch import geostructured data (geojson, gpx, kml, osm...)"
• "Manage map options: display a minimap, locate user on load…"
• "And it's open source!", i.e. you don't have to worry about costs but can decide later if you're happy with the solution to donate to the project

About your remarks:

• "separate datasets for each city" -> I think you could solve it with layers, e.g. one layer in the map for each city, if I understand your request correctly.