Is it possible to segment a document in BeautifulSoup before converting it to text based on my parsing of the document?
I have some html files that I want to convert to text. I have been playing around with BeautifulSoup and have made some progress in understanding how to use instructions and can send html and return text.
However, my files have a lot of text formatted using table structures. For example, I might have a paragraph of text that is in a td tag in a set of table tags
<table>
<td> here is some really useful information and there might be other markup tags but
this information is really textual in my eyes-I want to preserve it
</td>
</table>
And then there are "classic tables" that have data in the body of the table.
I want to be able to apply an algorithm to a table and set some rules that determine if the table breaks out before converting the document to text.
I figured out how to get the characteristics of my tables - for example, to get the number of columns in each table:
numbCols=[]
for table in soup.findAll('table'):
rows=[]
for row in table.findAll('tr'):
columns=0
for column in row.findAll('td'):
columns+=1
rows.append(columns)
numbCols.append(rows)
so I can work with numbCols and use len of every item in the list and the value in every item in the list to analyze the characteristics of my tables and determine which ones I want to keep or discard.
I don't see an elegant way to use this information in BeautifulSoup to get text. I guess what I am trying to understand, suppose I parse numbCols and decide that out of ten tables in a particular document, I want to exclude tables 2, 4, 6 and 9. So the html document part includes everything but those , tables. How can I segment my soup this way?
The solution I came up with is first identifying the position of each of the open and closing table tags with finditer and getting the spans, and then zips the spans with numbCols. Then I can use this list to trim and concatenate pieces of my string. Once this is done, I can use BeautifulSoup to convert the html to text.
I'm sure I can do it all in BeautifulSoup. Any suggestions or links to existing examples would be great. I must mention that my source files can be large and I have thousands to process.
I didn't have an answer, but I'm getting closer
a source to share
Man I love this stuff Assuming in the naive case that I want to drop all tables with any rows with a column length greater than 3 My answer
for table in soup.findAll('table'):
rows=[]
for row in table.findAll('tr'):
columns=0
for column in row.findAll('td'):
columns+=1
rows.append(columns)
if max(rows)>3:
table.delete()
You can do whatever processing you want at any level in this loop, you only need to identify the test and get the correct instance to test.
a source to share