Should I implement mixed use of BeautifulSoup and REGEX or rely solely on BS
I have some data that I need to extract from a collection of html files. I'm not sure if the data is in a div element, table element or merged element (where the div tag is a table element. I've seen all three cases. My files are big - up to 2mb and I have tens of thousands of them. So far I've looked at the td elements in tables and looked at the lone divs. I think the longest time is the souped file , over 30 seconds. I played around with creating a regex to find the data I needed and then looking for the next table tag close, tr, td or div to determine what type of structure my text is contained in find the corresponding open tag, cut off that section and then wrap it all in open and closed HTML tags
stuff
<div>
stuff
mytext
stuff
</div>
so I create a line that looks like this:
s='<div>stuffmyTextstuff</div>'
Then I terminate the line
def stringWrapper(s):
newString='<HTML>'+s+'</HTML>'
return newString
And then use BeautifulSoup
littleSoup=BeautifulSoup(newString)
Then I can access the power of BeautifulSoup to do what I want with newString.
This works much faster than the alternative, which first checks all cell contents of all tables until I find my text, and if I can't find it, test the entire contents of the div.
Am I missing something?
a source to share
BeautifulSoup uses regex internally (which is what separates it from other XML parsers), so you are most likely just repeating what it does. If you want a faster option, use try / catch to try to parse lxml or etree first, then try BeautifulSoup and / or tidylib to parse the parsing HTML if the parser doesn't work.
It sounds like what you are doing, you really want to use XPath or XSLT to find and retrieve your data, lxml can do both.
Finally, given the size of your files, you should probably parse path or file usage so that the source can be read incrementally rather than being stored in memory for parsing.
a source to share