Extract everything that looks like links from a large amount of data in python

I have about 5GB of html data that I want to process to find links to a set of websites and do some additional filtering. Right now I am using a simple regex for each site and looping through them looking for matches. In my case, the links may be outside of the "a" tags and not quite well formed in many ways (eg "\ n" in the middle of the link), so I try to grab as many "links" as I can and check them later with other scripts ( so BeatifulSoup \ lxml \ etc). The problem is my script is pretty slow, so I am thinking of any ways to speed it up. I am writing a test suite to test different approaches, but I hope to get some advice :)

Right now I'm thinking about getting all links without filtering first (perhaps using a C module or a standalone app that doesn't use regex, but a simple search to start and end each link) and then using regexp to match to me. necessary.