How reliable is E-bird? (1 Viewer)

I should say straight away that I’m a great fan of the E-bird concept. I’ve only relatively recently discovered it but have found it so useful (and reliable) in Spain that I’ve added links to the relevant sites to my birding guide to Cadiz province (although, shamefacedly, I have to admit that I still need to transfer data from the pile of notebooks sitting reproachfully next to my desk).

However, on a recent trip to Queensland, I discovered that the resource could be very misleading and inaccurate. I was staying at Cassowary House with my old school friend Phil Gregory. During my stay I regularly went for an early AM walk along the Black Mountain Road and on one of my saunters met a birder whose perception of what could be seen there seemed curiously inflated. Even with my very meagre knowledge of Australian birds I quickly realised that several of his target species didn’t occur there. Back at Cassowary House I mentioned this to Phil so we checked the entries for the area. That for the Black Mountain Road got it about right with 91 species (all but a handful causing no raised eyebrows for my host). However, that for Cassowary House listed 213 species. This caused Phil some astonishment since in 20 odd years of living there he’s only had about 120 species! As the author of two field guides (inc. one on north Queensland) and a new Helm monograph, Phil’s no slouch as a birder and, although he readily conceded he might have missed a handful of species, dipping 90 odd birds was out of the question. On closer inspection it was plain that some folks were posting records from other sites. Red-necked Stints, Sharp-tailed Sandpiper and other waders were surely seen at Cairns Esplanade rather than over dense rainforest. This much was obvious. However, other reports may simply have been a matter of error and much less obvious to the untutored eye (species that appear to exist nearby for example but are strictly limited to west of the Great Dividing Range such as Atherton Scrubwren). One observer listed at least three species Phil had never seen there, all on the same day. Phil immediately contacted E-bird to express his concerns but several weeks down the line the faulty list is still there and Phil’s had no response.

Now I realise that E-bird depends heavily on unpaid moderators and such like and that the occasional error will inevitably creep in but the level of misinformation involved here seems unacceptably excessive. Although I was able to twig something was wrong not everyone has a local guru to hand. It’s hard for users without the guidance I enjoyed to sort the wheat from the chaff. If some sites/areas lack moderation, as seems to be the case here (and unlike the situation I’ve found in Spain), then perhaps some sort of ‘caveat emptor’ warning that the list is unmoderated should be added. Hopefully, this level of misinformation is exceptional but have others had similar problems or is it very unusual?

Reliability / ablity of the observer is the greatest problem and it's long been recognised as a limitation I think. I've certainly been present when some crap is being uploaded and one bloke even started entering stuff I'd told him I'd seen.

I've also seen people who are virtually addicted and getting their sightings entered after each day is like a high, they cannot communicate in any form until they finish inputting their sightings. I think there are some people whojust want to see their name published?

I think it is very accurate, as long as you are aware of the outliers.
If you look at the illustrated checklist for a site, you can see the calendar view of sightings, plus the "date last seen".
If either of these are historic, or the graph just has a single slither of green, then chances are, it was a rarity, or an error. If you're seeing a species with >50% for a given month, then you have a good chance of seeing it there.
If the graph is solid Green January to December, then you are going to be sick of the sight of them by the end of your stay!

I think the opposite is true also.. it still needs people to look beyond what they are expecting to see. eBird definitely isn't the complete list of everything that is possible at a site. Especially in underwatched areas.

There are two issues at play. The first is simple identification error, which is very frequent. The second is the use of hotspots, which are the easy option when inputting records. This is usually why you get records of waders in rainforest, or shy forest species in the middle of cities.

Sadly, some birders seem to use ebird as their personal listing software, and just dump records, often with an x rather than an actual count, in random locations in a country, often the capital city. Personally I would like to see ebird close the accounts of birders who do this.

EBird is in my opinion actually the least useful of the three alternative projects: observado/waarneming and ornitho. Did you check these, John?

Currently, eBird is to be taken with a grain of salt. If a species is reported regularly from a site, this can be expected. Otherwise, just ignore it unless you know the observer.

I use it preparing lists for locations to visit. For example, if you visit an urban park on another continent, you know that you can expect only maybe few percent of birds regularly found in that country. Use ebird to see what expect and what you can miss. Then it is also good for finding recent records of common species. For example, when I visited USA in spring, I used it to track first records of arriving migrants in the general area. I also have several programming scripts and subroutines to extract from ebird lists a list of species unseen and seen by me.

What annoys me is that pseudo-scientific publications use records from eBird and suchlike without a credible way to estimate and compensate for fake sightings. Such papers are just a say-so. And there is a number of issues which are quite hard to address statistically, for example self-accentuating propagation of bias (people go to places which ebird suggests and put species which ebird suggests, spinning fake into overdrive).

Other weakness of eBird are too big hotspots - sometimes covering a 30 mile stretch of a road, or a whole national park from desert to forest to waterbodies.

Another weakness is false multiplication of records. Often there is a hotspot with 4 or 6 observations, which were obviously one observation by a visiting group.

Ebird and similar projects might improve by introducing a ranking system for observers, plus a function to select only records from observers with high rankings. The rank would be calculated using rejected sightings and outlier sightings. This would mimic what serious organizations do: they select experienced birders plus are cautious of those who claim too many out of the way records.

Hi Jurek,

jurek said:

EBird is in my opinion actually the least useful of the three alternative projects: observado/waarneming and ornitho.

Click to expand...

Curating as it's done on ornitho is a bit of a double-edged sword. There is a tendency to discount records that are out of the ordinary, even if the sightings might have been genuine.

Not to say that personally, I didn't find it helpful to have my mistakes pointed out, but I a while ago I read an article that stated that curation in such recording systems introduced as many biases as it removed, and I find that plausible.

With regard to some of the effects you pointed out, if you talk to the German ornitho.de people, they state that they have statistical methods to avoid such biases, but in the talks they give at fairs they also show that there are biases they can't compensate for, which sends a self-contradicting mesage.

Additionally, I specifically asked one of the team about precise reporting procedures, and he just said, "Oh, don't worry, just start recording as you see fit" ... that's probably how most people to it, but it doesn't really instill confidence in the scientific consistency of the platform.

I'm also turned off by ornitho.de because there are no interfaces to actually do "citizen science", and from what you just described about ebird, there seem to be some useful interfaces on that platform, even if your examples only show the individual observer's application.

There's also naturgucker.de, by the way. It's a general biology observation platform, so it doesn't offer the kind of specialized fields ornitho has, but it still seems to have a lot of fans.

Regards,

Henning

DMW said:

There are two issues at play. The first is simple identification error, which is very frequent. The second is the use of hotspots, which are the easy option when inputting records. This is usually why you get records of waders in rainforest, or shy forest species in the middle of cities.

Sadly, some birders seem to use ebird as their personal listing software, and just dump records, often with an x rather than an actual count, in random locations in a country, often the capital city. Personally I would like to see ebird close the accounts of birders who do this.

Click to expand...

I suspect also, the bulk import tool introduces errors. I was surprised to see my summary page, a list of 90 species for Jordan. A country I have never visited!
Looking at the records, they were from the Negev desert Israel side (which I have visited!). I have no idea how these have jumped the border, but they were 100 records from 20 years ago. I don't think anyone is going to lose sleep or be sent on a wasted journey resulting from them. Even I can't be arsed changing them.

In total, there are 737 million bird sightings on the system, wow!

There are problems with it, but most are caused by people inputting records sloppily. Maybe being lazy and using broad 'hotspots' rather than pinpointing locations accurately. Or worse, listing everything from a day's birding as one site even though they may have travelled a fair distance.

The Jordan/Israel blip that Peter mentions is a fault of eBird though. It seems to struggle to recognise borders - a site in Leicestershire that I enter comes out as in Notts, and according to eBird records I entered rightly as southern Azerbaijan are shown as in Iran !

Steve

As DMW says - the larger problem in this case is not poor identification/stringing/list inflating or anything like that. It is using a hotspot to log birds from a larger area. This particularly happens on birding tours where someone stays at a lodge and then visits various sites surrounding the lodge but all the sightings end up being recorded for the lodge.

I think the Hotspot model is both one of eBirds strengths but also weaknesses. Ideally there would be a real-time location capture for each bird entered, assuming the lists are not entered after the fact. Hotspots would be defined as areas not points, so that all lists within an area would automatically be part of that Hotspot. And Hotspots would be hierarchical. So you could have a Hotspot for each trail or valley or area in a National Park, say, but also a Hotspot for the entire park that covers all the finer resolution Hotspots. I've heard rumors that area-bounded and hierarchical Hotspots are in the works, but I've heard these rumors for a long time so who knows.

All this being said, I still think eBird is the easiest to use and most powerful of the tools out there, and is the one I use by far the most. I also disagree with Jurek's suggestion that using eBird data in a publication is pseudo-science. Certainly eBird is fraught with problems but it's also a massively useful tool and there is a ton of signal in the noise. Whether you publish a credible paper or something sketchy has more to do with how well the authors understand their source data and how rigorous they are than with what their data sources are. Certainly eBird has lousy records all over the place, but it's not that hard to sift the signal out of the noise. If you want to look at, say, spring arrival dates for migrant species across large areas over many years, movement of irruptive species, or as applies to more of us here, plan a birding trip, eBird is an amazing data set. The signal greatly outweighs the noise. If you want to understand the distribution of neotropical Swifts or SE Asian Swiftlets well, or try to study at sea occurance of Prions or Diving-Petrels, you ought to know that ID errors are rampant, many IDs are based upon field guide distribution maps, "common knowledge," etc, and museums are still sorting out the proper IDs of specimens to this day.

I read these points both with interest and a level of understanding having used it an others.

As I understand it warnemming places a grid reference on every observation and you input species & locations separately. (This is of course not the location of the bird but the observer if the app is used).

EBird links a checklist to the location. It is unfortunate that the tracker, which is available to the observer, is not available to other users of the website - this would help understanding where people have actually been. The data obviously exists!

There are ways of defeating any input system, most of them are more cock-up than conspiracy. The biggest issue is the level of moderation. As I understand it there are two ways that moderators can act. Individual species records can be flagged and followed up, or whole checklists 'flagged'. Flagging them stops them going into the core data set but does not change the original record. The checklist can only be changed by the observer, the moderator cannot do this.

The flagging system is not infallible. You can see a bit of how the system works with the formation of local checklists. If there are few records then the base record square used is 100x100 miles (160x160km)! If there are more records, or it is attached to a hotspot then it quickly comes down to a square of a few miles square. You can see that if the record is only flagged as strange to 10,000 square miles it can include a whole host of strange records and not be flagged.

Hotspot rules are laid out in the eBird notes - it is clear that many people don't follow them generally in posting to them. I suspect that many of the issues or Cassowary house are due to people who stay there putting all their records from a days birding to the single location regardless of where they actually have been or seen.

I too have had the issues of boundaries - I think this is also created by the square system that eBird seems to use. I managed to resolve mine issues by emailing them and they sorted it very quickly.

As a final point, all systems have to trust the inputter. This means that there is likely to a few mistakes. I would be surprised if the error level is particularly high.

I have used eBird to enter my sightings since 2008 on a global basis. I have found eBird data is very reliable in the U.S. where there is an abundance of reviewers with local knowledge of what is likely in specific areas and times; they also have knowledge of local birders. The reliability of data elsewhere is variable but is likely improving over time as more knowledgeable reviewers join eBird and continue to review historical data. I not infrequently get queries from eBird reviewers who have just started reviewing data that I entered several years ago in various parts of the world. If you find some suspect data, I suggest you email them to report it.

jurek said:

Other weakness of eBird are too big hotspots - sometimes covering a 30 mile stretch of a road, or a whole national park from desert to forest to waterbodies.

Another weakness is false multiplication of records. Often there is a hotspot with 4 or 6 observations, which were obviously one observation by a visiting group.

Click to expand...

Ebird typically addresses the big hotspot problem by including an admonition to use more precise locations if possible as part of the hotspot name. But having big hotspots makes it convenient to enter data for some locations where it is difficult to know more precise locations and is clearly preferable to forcing the observer to guess at a more precise location that may be inaccurate.

As for the "false multiplication of records," there is nothing "false" about the situation you reference. All the records are accurate. eBird takes into account in bar charts, etc. the obvious possibilty that multiple observers may be reporting the same bird; that happens all the time with rarities.

John

There are a wealth of ways to interrogate and understand the data which makes it a very useful resource for me. I have no knowledge or familiarity with Australian birds and clearly the database is not used that much for Australia as a Hotspot like Cassowary House has less than 100 checklists for the last two years.

Hotspot for Cassowary House - https://ebird.org/hotspot/L920581?yr=all&m=&rank=mrec

Illustrated Checklist for Cassowary House (including frequency) - https://ebird.org/hotspot/L920581/media?yr=all&m=

Checklist list - https://ebird.org/hotspot/L920581/activity?yr=all&m=

Offending accumulated checklist including sightings from Hastie Swamp, Lake Barrine, etc - https://ebird.org/checklist/S58597084

So in answer to your question here - in my view, it is very accurate. Indeed even the offending accumulated checklist is accurate as listing the waders from Hastie Swamp.

An ebird reviewer could invalidate the entire checklist to remove it from the public tools and records and remedy the issue that you highlight but even without doing that, it took little time to form a view on the location looking at the date of records and the frequency tools.

I have little doubt that as the number of records increase and indeed reviewers find more time to weed out anomalies then accuracy will improve but in the meantime, it is already and astonishingly powerful and accurate tool.

All the best

The more you zoom out, the more accurate it becomes!
If you zoom in too much, you might end up preserving a car park instead of the marshes.

I've only used ebird in the states, and have found it quite useful in getting a sense of patterns and locations for birds (plus the alert function is super helpful for year-listing or discovering new sites). Obviously the strength of ebird is going to vary somewhat...some areas are just underbirded, or if birded regularly not by folks who use ebird, moderation and quality of moderation can vary, and so forth.

FYI, there are ways for scientists to correct for biases in the data if they pull it from ebird.

One other consideration: with some rare birds, people want to put in their records, but do not want to disclose the actual location as a means to protect the bird. In fact, I was asked once to change a location that I had put in eBird for that very reason.

I'm not saying this was the reason for the Cassowary House example, but it is part of the story with eBird.

Logically, treat eBird as a human being who sometimes identifies birds wrongly. Would you trust a human birder who makes mistakes as often as you see them in the ebird database? For me, ebird has trustfulness on the level 'would use this guy for planning a private trip but choose 3-4 such records, so one should be right' but less than the level 'would trust a serious science report coming from this guy'.

pbjosh said:

Certainly eBird has lousy records all over the place, but it's not that hard to sift the signal out of the noise.

Click to expand...

A common mistake in papers is implicit approach 'we use wrong data from ebird because we have no better data'. Another is that a report proofs the data on common and easy species, where the trend is not skewed by an occasional error, but includes also rarer and difficult species, too, where errors are significant.

Weak points of ebird compared to waarneming/observado (in my opinion, the best of the three systems) are:
- Not including other wildlife. Privately it means, that one should keep two record systems if one would want to use ebird. Scientifically, needlessly wasting wealth of data of mammals, butterflies, dragonflies etc., which most birders also notice in the field.
- More limited ways of searching records. In observado search has far more options, including e.g. which birds I could see within X km from my locality?
- Slow to load and difficult to use on a mobile phone without an app (never used an app). Especially its huuuge pictures and small text fields on the main page look terrible on a mobile.
- Focusing on lists not observations. Only a minority of birders make complete lists. Especially few write down very common species, like Mallard or Herring Gull in Europe.
- Focusing on hotspots not squares. Only a minority of birders have so poor orientation that they cannot find themselves more closely than a 30 mile stretch of a road. Practically, it is a waste: how to search a bird inside a huge hotspot? This will also produce many funny biases and errors in analysis.

jurek said:

Logically, treat eBird as a human being who sometimes identifies birds wrongly. Would you trust a human birder who makes mistakes as often as you see them in the ebird database?

Click to expand...

The mistakes referenced in this thread are not ID errors, but location errors. My experience in the US is that surprisingly few ID errors make their way into the eBird database (at least signficant ones). Filters and flagging are effective control measures. Unusual sightings are not confirmed by reviewers without appropriate documentation--just as in pre-eBird days.

Jim M. said:

The mistakes referenced in this thread are not ID errors, but location errors. My experience in the US is that surprisingly few ID errors make their way into the eBird database (at least signficant ones).

Click to expand...

Well, how would you treat a person who has problems with keeping records and sometimes says he has seen shorebirds in a rainforest? More seriously, big mistakes are also easily picked, but what about the many more smaller mistakes which are not obvious?

To be positive, anybody could set a private border what one can believe and what not. Privately for me, it is: at least 4 records made by at least 3 people over more than one day, and not one group which can be mistaken together. Quite high, but I was burned several times.

More interesting is: when ebird (or observado, ornitho) records become sound scientifically. Here is no fixed number. It is quite a new problem because ebird records are different from standard bird surveys. Many statistical assumptions are violated, for example independence of observations (because people are influenced by what others write on ebird). Medical statistics can help here, and there will be a penalty for lower quality of records. I toyed with writing about it once, about avoiding biases of public wildlife databases, but never had time for it.

jurek said:

Well, how would you treat a person who has problems with keeping records and sometimes says he has seen shorebirds in a rainforest? More seriously, big mistakes are also easily picked, but what about the many more smaller mistakes which are not obvious?

Click to expand...

In bird recording generally (eg in County bird reports here in the UK) there are some errors - misidentification all the time.

However, as pictures go, a number of small errors don't matter so much. Lots more birds will go unnoticed and unrecorded. If some numbers are inflated or birds didn't really occur there, countless thousands others weren't recorded which restores some kind of balance.

It is also perhaps the case that birds can move, or die, or hide. So any list is only a snapshot, or indication.

You'd hope that the tools and technology will develop and smaller and smaller errors and inconsistencies will be picked up on as time moves on ...