Ruby HTML scraper written in Hpricot having problems with escaped HTML
I'm trying to clean up this page: http://www.udel.edu/dining/menus/russell.html . I wrote a scraper in Ruby using the Hpricot library.
problem: the HTML page is escaped and I need to display it without saving
example: "M&M" should be "M&M"
example: "Entrée" should be "Vegetarian Entrée"
I tried using Ruby's CGI library (not too successful) and the HTMLEntities gem I found through this post.
HTMLEntities works during testing:
require 'rubygems'
require 'htmlentities'
require 'cgi'
h = HTMLEntities.new
puts "h.decode('Entrée') = #{h.decode("Entrée")}"
blank = " "
puts "h.decode blank = #{h.decode blank}"
puts "CGI.unescapeHTML blank = |#{CGI.unescapeHTML blank}|"
puts "h.decode '<th width=86 height=59 scope=row>Vegetarian Entrée</th> ' = |#{h.decode '<th width=86 height=59 scope=row>Vegetarian Entrée</th> '}|"
correctly gives
h.decode('Entrée') = Entrée
h.decode blank =
CGI.unescapeHTML blank = | |
h.decode '<th width=86 height=59 scope=row>Vegetarian Entrée</th> ' = |<th width=86 height=59 scope=row>Vegetarian Entrée</th> |
However, when I use it on an open uri file, it doesn't work as expected:
require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'htmlentities'
require 'cgi'
f = open("http://www.udel.edu/dining/menus/russell.html")
htmlentity = HTMLEntities.new
while line = f.gets
puts htmlentity.decode line
end
Wrongly gives things like:
<th width="60" height="59" scope="row">Vegetarian Entrée</th>
and
<th scope="row">Â </th> // note: was originally ' ' to indicate a blank
but handles M&M correctly, getting:
<td valign="middle" class="menulineA">M&M Brownies</td>
Am I handling escaped HTML incorrectly? I don't understand why it works in some cases and not others.
I am running ruby 1.8.7 (2009-06-12 patchlevel 174) [i486-linux]
Any help / suggestion is greatly appreciated. Thanks.
a source to share
HTMLEntities seems to work, but you have an encoding issue. The terminal you are printing is probably set up for latin encoding and barfs on utf-8 outputs your script outputs.
What environment are you using ruby in?
The reason "&" correctly displays that it is an ascii character and thus will display the same in most encodings. The problem is that it doesn't have to happen once in the XML document and it might get in trouble later when you feed the decoded file to hpricot. I believe the correct way would be to parse hpricot and then pipe what you retrieve from the document to HTMLEntity.
a source to share