When I used Merlin in Europe in spring, where I know bird sounds well, it overlooked about 50% of birds, and made a mistake 3-5% of times. In practice, in about 10 minutes it claimed one bird which was not calling. Merlin records audio, so I can be sure this was a mistake, not a bird it overlooked.
When I used Merlin in Mexico, every interesting sound I checked first with the real recording of the species. If it was identical and distinctive, I could start looking for the bird visually. Still, it sometimes made completely false claims (I remember some Greenlet was claimed out of the blue - no similar bird singing), and many interesting sounds it did not name.
Unfortunately, Merlin does not learn - if a bird or a sound is not in the algorithm, then Merlin will always overlook that species, until a new version of the algorithm comes up. Which is a bummer for birders, and makes some interesting errors if ornithologists use data gathered directly or indirectly from Merlin to e.g. count birds.