What am I using to match MS Word characters in regEx
I need to find and remove all non-standard ascii characters that are in a string (usually delivered there by MS Word). I'm not really sure if these characters are ... both the fancy apostrophe and the double quotes and all that. Is it unicode? I know how to do this with ham-handed [az, etc. Etc.], but I was hoping there was a more elegant way to simply exclude anything that is not on the keyboard.
a source to share
Probably the best way to deal with this is to work with character sets, yes, but for what it's worth, I've had some success with this quick and dirty approach, character class
[\x80-\x9F]
this works because the problem with the "Word characters" for me are the ones that are illegal in Unicode and I have no way of misinforming user input.
a source to share
Microsoft applications are known for using fancy characters like curly quotes, em dashes, etc., which require special handling without adding any real meaning. In some cases, all you have to do is make sure that you are using one of your extended character sets to read the text (for example, windows-1252 instead of ISO-8859-1). But there are a few tools out there that replace these quirky characters with their peer but universally supported equivalents. Google for "Demonizer" or "AsciiDammit".
a source to share
I usually use a JEdit macro, which replaces the most common of these with the more ascii accessible version, that is:
- hyphens and dashes before the minus sign
- suppress points (one char) for multiple points;
- dot list item for asterisk;
- and etc.
It adapts easily to Word / Openoffice / independently, and of course, modified to suit your needs. I wrote an article on this topic: http://www.megadix.it/node/138
Greetings
a source to share
My solution to this problem is to write a Perl script that gives me all characters that are outside the ASCII range (0 - 127):
#!/usr/bin/perl
use strict;
use warnings;
my %seen;
while (<>) {
for my $character (grep { ord($_) > 127 } split //) {
$seen{$character}++;
}
}
print "saw $_ $seen{$_} times, its ord is ", ord($_), "\n" for keys %seen;
Then I create a mapping of those symbols to what I want them to be and replace them in the file:
#!/usr/bin/perl
use strict;
use warnings;
my %map = (
chr(128) => "foo",
#etc.
);
while (<>) {
s/([\x{80}-\x{FF}])/$map{$1}/;
print;
}
a source to share
What I would do is use AutoHotKey , or python SendKeys or some kind of visual baseline that would send me all possible keys (also with offset applied and unapplied) into the Word document.
In SendKeys this will be a script of the form
chars = ''.join([chr(i) for i in range(ord('a'),ord('z'))])
nums = ''.join([chr(i) for i in range(ord('0'),ord('9'))])
specials = ['-','=','\','/',','.',',','`']
all = chars+nums+specials
SendKeys.SendKeys("""
{LWIN}
{PAUSE .25}
r
winword.exe{ENTER}
{PAUSE 1}
%(all)s
+(%(all)s)
"testQuotationAndDashAutoreplace"{SPACE}-{SPACE}a{SPACE}{BS 3}{LEFT}{BS}
{Alt}{PAUSE .25}{SHIFT}
changeLanguage
%(all)s
+%(all)s
"""%{'all':all})
Then I would save the document as text and use it as a database for all preemptive keys in the keyboard layout (you might want to replace the default input language more than once to get absolutely all of the displayed characters).
If char is in the text document of the result - it is displayed, otherwise not. No regex needed. You can of course add a range of characters within a script or program.
a source to share