Hi Sean
Glad that you discovered the valleys behind Hveragerði. I haven't been walking there for years but I remember when I first went there almost 20 years ago how astonished I was when I walked up to a waterfall in the river and suddenly felt this big waft of warm air in my face - I hadn't realised that the river was warm until then.
The obvious suggestion for a walk near Reykjavík is to walk up Esja, the big mountain to the north of the city. The advantage of this is that it is close to the city (20 minutes by car), there is an obvious path and there will be other people there (as a safety precaution). You don't have to go to the top (in fact reaching the top can be tricky in winter) but you can reach a good vantage point after an hour or so uphill, which takes you to about 600 metres asl. I found this info in English n it
http://stuckiniceland.com/south/perfect_hiking_reykjavik/
You can branch off lower down and walk through the forest show in the photo if you don't want to go too high.
Another good one, which takes you into the heart of an old volcano (which should excite any 9-year old and 52-year old) is the walk to Búrfellsgjá. In fact the same people who did the above link also did one for this walk (great minds think alike)
http://stuckiniceland.com/south/great-hiking-trail-near-reykjavik-if-you-can-find-it/ . This is certainly less strenuous than Esja and in many ways more interesting (although lacking the high mountain scenery). The directions on that link should get you there but in winter don't be tempted to drive much beyond where you park your car - the road is not ploughed. This site is also only 20 mins a most from the city.
If there's still plenty of snow around you might even enjoy just walking through the tree plantation at Öskjuhlíð in the city (this is hill where "the Pearl" or "Perlan" is located). Nothing spectacular but the conifers are pretty in the snow and there are plenty of trails.
Birding wise, you might see Ptarmigan on either of the first two I mentioned, Raven and Snow Bunting, but little else. Birding in Iceland in winter is mainly a coastal pursuit. The walk along the north shore of Reykjavík is nice in good weather, best for birding west of the harbour right out to the lighthouse at Seltjarnarnes (quite a long walk but doable) and Gyr Falcon occurs here. You wouldn't want to do it on a day like today though.
I know you don't need to be told this but I'd best say it anyway: Check the weather (
http://en.vedur.is/) and road conditions (
http://www.vegagerdin.is/english). After a very mild autumn it's very wintry at the moment and in conditions like we've had for the last few days, both of the above routes would have been out of the question for anyone.
There are many more possibilities but these should give you something to start with.
EDIT: I've just checked the links in the Búrfellsgjá article and they don't work. I'll try to post a map at some stage.