Find dependencies in python / script source

Question

I have a bunch of simple Python scripts with simple expressions [1]:

C = A+B D = C * 4 

I need to execute them, but most importantly, I need to know which objects I rely on; in the previous case, objects A and B are external dependencies. For instance. given that I have the previous code in the var source, I want to be able to:

 deps = { "A" : 1 , "B": 2} exec source in deps 

therefore it is strictly necessary to know how to build dict prints.

I was hiding in the Python ast module, but I had no idea.

---

[1] simple mathematical aggregations, within for loops, no more.

Answer 1

You can tokenize the Python source code using the tokenize module from the standard library. This will allow you to find all the variable names used in the script.

Now suppose we define "independence" as any variable name that comes immediately before the = sign. Then, depending on how simple your script code is (see Cautions below), you can define variable names that are not dependencies as follows:

 import tokenize import io import token import collections import keyword kwset = set(keyword.kwlist) class Token(collections.namedtuple('Token', 'num val start end line')): @property def name(self): return token.tok_name[self.num] source = ''' C = A+B D = C * 4 ''' lastname = None names = set() not_dep = set() for tok in tokenize.generate_tokens(io.BytesIO(source).readline): tok = Token(*tok) print(tok.name, tok.val) if tok.name == 'NAME': names.add(tok.val) lastname = tok.val if tok.name == 'OP' and tok.val == '=': not_dep.add(lastname) print(names) # set(['A', 'C', 'B', 'D']) print(not_dep) # set(['C', 'D']) deps = dict.fromkeys(names - not_dep - kwset, 1) print(deps) # {'A': 1, 'B': 1} 

---

Warnings

• If your scripts contain statements other than simple assignments, then names may be populated with unwanted variable names. For instance,

   import numpy 

  will add both 'import' and 'numpy' to the names set.

• If your script contains an assignment that uses left unpacking side tuples, e.g.

   E, F = 1, 2 

  then the naive code mentioned above will only recognize that F not a dependency.