A couple of notes here. First off, the thistle seed, more appropriately called nyjer seed, will NOT sprout. Nyjer seed is indeed a non-native, and therefor, the laws prohibit any non-native seed from being imported without sterilization. Odds are, if you're getting weeds growing, it is something else, feeding off the excess nutrients from the decaying seed hulls.
Now, on to the goldfinches. I've read in the past about the feeding habits of most backyard birds, as it concerns bird feeders. The research done shows that birds typically have 3 preferences or so, give or take, when it comes to where they feed. They have a mental ranking system of those sources, and make those rankings based on favorite foods, water, and of course safety. If a bird finds a favorite type of food, near water, and with lots of good cover to feel safe, they will return over and over, unless or course any of the things they like disappear. If that food source is empty, they move on to location #2, then #3, and so on. Should the #1 source not be restocked in a week or 2, they will stop coming. Now, on the other hand, should that food source ALWAYS remain stocked, and water available, they may not even stray at all. In the case of your neighbors, that is most likely their "spot", and unless something drives the birds away, they will not stray from ole reliable.
Now, the good thing in all of this is new birds. For example, last winter, I had 1 measley pair of cardinals that would feed in my yard. Throughout this year, new babies were born, established new territories and such, and behold! I now have about 12 cardinals showing up every morning and evening. Same with my chickadees, my flock has increased from a whoppin 3 last winter, to about 15. Blue Jays as well, and unfortunately, house sparrows too. So, in all this jibber-jabber, what in the world did I say? Well, for short, you're doing everything right, but you're trying to take birds away from a feeding station that they love, and probably don't want to stray from. But, give it time, forget your neighbor's birds, and wait for you very own birds to make your yard their "spot".