Parsing in Python: what's the most efficient way to suppress / normalize strings?
I am parsing a source file and I want to "suppress" lines. This means converting each string like "bla bla bla + / *" to something like "string", which is deterministic and does not contain characters that might confuse my parser because I am not interested in the value of the string. One of the problems here is string formatting like "% s", see my note on this below.
Take, for example, the following pseudocode, which might be the contents of a file that I am parsing. Suppose strings start with "and the escape character is" - "":
print(i)
print("hello**")
print("hel"+"lo**")
print("h e l l o "+
"hello\n")
print("hell""o")
print(str(123)+"h e l l o")
print(uppercase("h e l l o")+"g o o d b y e")
Should be converted to the following result:
print(i)
print("string")
print("string"+"string")
print("string"
"string")
print("string")
print(str(123)+"string")
print(uppercase("string")+"string")
I am currently considering it as a special case in code (ie it detects the beginning of a line and "manually" runs to the end with a few extra parts in the way). If there is a Python library function that I can use, or a good regex that can make my code more efficient, that would be great.
A few notes:
- I would like the "start of line" character to be a variable, eg. 'vs'.
- I'm not familiar with Python code at this point, but I'm planning and the problem is clearly getting more complicated because lines can start in multiple ways and must end in a way that matches the beginning. I'm not trying to deal with this right now, but if there is any well-established practice I would like to know about it.
- What worries me the most about this "suppression" is the case of string formatting with the likes of "% s" which are meaningful tokens. I am not currently doing this and have not thought it through completely, but if any of you have any suggestions on how to handle this, that would be great. Please note that I am not interested in the specific type or formatting of in-string tokens, it is enough for me to know that there are some tokens in the string. Note that this might be important here: my tokenizer is not nested because my purpose is pretty simple (I don't compile anything ...).
- I am not entirely sure if it is possible to escape the line start character. What can you say about the general ways of implementing this language in most programming languages? Is the assumption of double occurrence (like ") or any set of two characters (like" \ ") enough to avoid? I need to consider other cases (think languages like Java, C / C ++ , PHP, C #)?
a source to share
Option 1: To clean up your Python source, try the built-in tokenize module . It can correctly find strings and other tokens in any Python source file.
Option 3. Use pygments with HTML output and replace anything with blue (etc.) with "string"
. pygments supports several dozen languages.
Option 2: For most languages, you can create a custom regex replacement. For example, the following sanitizes Python source code (but it doesn't work if the source file contains """
or '''
):
import re
sanitized = re.sub(r'(#.*)|\'(?:[^\'\\]+|\\.)*\'|"(?:[^"\\]+|\\.)*"',
lambda match: match.group(1) or '"string"', source_code)
A regular expression is operating correctly, even if the string contains backslash ( \"
, \\
, \n
, \\
, \\"
, \\\"
etc all work fine).
When you create your regex, be sure to match comments (so your regex replacement will not touch lines within comments) and regex literals (for example, in Perl, Ruby, and JavaScript), and pay attention to backslash and newline matches ( for example, in Perl and Ruby, a string can contain a newline).
a source to share
Nowhere do you mention that you are taking an approach using a lexer and a parser . If you don't actually do that, take a look at, for example, the tokenize module (which you probably want) or the third party PLY module (Python Lex-Yacc). Your problem requires a systematic approach, and these tools (and others) provide it.
(Note that after you've provided the code, you can apply another specialized tokenizer to the contents of the strings to detect special formatting directives, such as %s
. In this case, the regex can do the job.)
a source to share