Converting html objects to their values in python
I am using this regex for some input,
[^a-zA-Z0-9@#]
However, this results in a lot of html special characters being removed inside the input, like
#227;, #1606;, #1588; (i had to remove the & prefix so that it wouldn't
show up as the actual value..)
Is there a way that I can convert them to my values so that it satisfies the regexp expression? I also don't know why the text decided to be so big.
a source to share
Given that your text has numeric but unnamed entities, you can first convert your byte string, which includes the defs xml entity (ampersand, hash, digits, semicolon), to unicode:
import re
xed_re = re.compile(r'&#(\d+);')
def usub(m): return unichr(int(m.group(1)))
s = 'ã, ن, ش'
u = xed_re.sub(usub, s)
if your terminal emulator can display arbitrary unicode characters print u
will show
ã, ن, ش
In any case, you can, if you like, use your original RE, and you won't accidentally "catch" the entities, but just the ascii letters, numbers and a couple of punctuation characters that you specified. (I'm not sure what you really want - why not accented letters but just ascii for example?), But ifwhatever you want it will work).
If you have named entities in addition to the numeric encoded ones, you can also apply the standard library module htmlentitydefs
recommended in the other answer (it only deals with named entities that map to Latin-1 codepoints, however).
a source to share
You can customize the following script:
import htmlentitydefs
import re
def substitute_entity (match):
name = match.group (1)
if name in htmlentitydefs.name2codepoint:
return unichr (htmlentitydefs.name2codepoint[name])
elif name.startswith ('#'):
try:
return unichr (int (name[1:]))
except:
pass
return '?'
print re.sub ('&(#?\\w+);', substitute_entity, 'x « y &wat; z {')
The following answer is issued here:
x « y ? z {
EDIT: I figured out the question as "how to get rid of HTML entities before further processing", hope I didn't waste time answering the wrong question;)
a source to share