A good link to check out for rare birds in Delaware is here at the Delaware Valley Rare Bird Alert.
It covers Delaware, New Jersey, and Eastern Pennsylvania.
Information from the above mentioned site shows that on May 9 there were twenty-one Caspian Terns at Gambacorta Marsh.
They were also present the same day at Prime Hook NWR.
I cannot tell you anything about these places, I have not been to either one.
I was lucky enough to see a Caspian Tern at both Speedwell Forge Lake and Middle Creek WMA in Lancaster County here in Pennsylvania back on April 18.
The Tern at Speedwell was plunging in the water after fish, quite successfully too. After catching five fish in as many attempts it moved off to the north.
It was really spectacular to watch up close at it was fishing.
Good luck with your quest.