Dynamic importing stuff in Python
In my artificial intellingence project's prototype Okai (which is completely written in python), I did some dynamic importing which surprisingly was unknown to many. I think you might be interested in knowing about it as well.
Before I explain dynamic importing, lemme explain traditional import too for the sake of readers of all levels. To follow along, you must be atleast familiar with basics of Python. If you don't understand anything, read anyway... you will learn something new. Let's start with the most basic example and increment on it. If you feel bored, skip to next section.
Very familiar import. Here, we are simply importing
os is now available in the scope as a "module" object (Python treats literally everything as an object) identified as
import module as something
import random as r
Again, we are simply importing a module. But now, the module will be available as given identifier (here it is
r) in the scope.
import module's module
Note that you can't import methods and properties of a class without importing class.
. is the scope resolution operator in python. It simply allows you to go a level deeper in the namespace and find the next identifier ie.
request in above example. but
urllib whole is imported and request is not defined! Why is that?
Something seems to be wrong here. But it is how it should be. To keep things interesting, let's continue our journey and pin it for later.
import module's module as something
import urllib.request as request
We have fixed the buggy import we did before. Now the
urllib is not available in the scope, request is the scope and refers to
from module import module
from urllib import request
It works exactly like the above fix but more self explanatory and simpler. Remember, we still have a question pinned. We need to find the answer.
from module import all
from urllib import *
This one will import modules that have their identifiers listed in
**all** list or simply, exist within the namespace of urllib's
**init**. Since we used
urllib does not exist in the scope. But all the modules from within
urllib 's namespace now exist with their default identifiers! So many identifiers within same scope can be dangerous. Consider the following example:
walk = 'Some string' from os import *
I just committed a crime!
walk has been overridden by
os.walk ! To avoid such incidents, avoid importing
*. As a thumb rule, do imports at the beginning of your code.
from module's module import module
from urllib.request import Request
It works as expected. Nothing new. You can keep going deeper in the modules as long as you want. Anything that comes inbetween
import does not exist in global scope.
import self's module
Let's say that you wanted a module named as
sys in your package, and you made one. But now, how do you import it when there is already a standard module with similar name? You use relative importing.
from . import sys
This will import
sys from within current package.
from .sys import mod
This will import
mod from within
sys module that resides in current package.
from .. import sys
sys from the parent package. But we still have that question pinned. Let's find the answer to that question. We need to find how import works behind the scene first.
How import works
We can always look in the documentation and find how import works. If not willing to do so, you can always dump global identifiers available with
globals(). Other than
__builtins__ list usually contains most of the built in classes and functions (as strings). If you
dir(__builtins__), you will find
__import__. This is invoked by import keyword. if you look in the help,
it explains what
__import__ accepts as argument:
Help on built-in function __import__ in module builtins: __import__(...) __import__(name, globals=None, locals=None, fromlist=(), level=0) -> module Import a module. Because this function is meant for use by the Python interpreter and not for general use it is better to use importlib.import_module() to programmatically import a module. The globals argument is only used to determine the context; they are not modified. The locals argument is unused. The fromlist should be a list of names to emulate ``from name import ...'', or an empty list to emulate ``import name''. When importing a module from a package, note that __import__('A.B', ...) returns package A when fromlist is empty, but its submodule B when fromlist is not empty. Level is used to determine whether to perform absolute or relative imports. 0 is absolute while a positive number is the number of parent directories to search relative to the current module.
It also answered our question! When importing modules with scope resolution and
fromList is empty, it returns the topmost object.
Note that is also indicates that you can use
importlib.import_module() to programmatically import modules. Or should I say dynamically. As
**import** provides functionality to import the modules, you can utilize it for dynamic importing. But that would be a bad practice as the documentation depreciates its direct use. So let's see what
importlib can do for us.
Hey import library, import module named something...
Let's checkout how
import importlib help(importlib.import_module)
As you can see, in the following,
import_module accepts name of the module and package.
Help on function import_module in module importlib: import_module(name, package=None) Import a module. The 'package' argument is required when performing a relative import. It specifies the package to use as the anchor point from which to resolve the relative import to an absolute import.
importlib, you will see that it also contains default
**import** as well. Obviously, it is just a wrapper over
But it allows us to import modules by their name. Following is the code snippet of okai. Highlighted line is responsible for dynamic importing.
import importlib from mind.thought import thought class action(thought): def do(action, act_on): if(type(act_on) != thought): act_on = thought(act_on) todo = action.split('.')[-1] if( not callable(action) ): try: action = importlib.import_module('actions.'+ action) except ImportError: print("\n[ok]: I don't know how to - " + action + '.\n') # return thought('raise', 'EXIT_FAILURE') return thought('') try: toAct = getattr(action, todo)() return toAct.do(act_on) except Exception as e: print("[ok]: I can - " + action.__name__ + " but cannot - " + todo) print("[...] Because " + e.args) print() print(e) else: return action(act_on) return thought('') # return thought('raise', 'EXIT_FAILURE')
That's that. You now know how import works and how to do dynamic importing. I hope you learned something new.
This article first appeared on my old blog, then dev.to. Since both are now dead, reposting here.