One of the great mantras in Python is avoid repetition. And yet, when working on a Django app or with a new third party library, I often find myself stuck in the same pitiful cycle:
Start the Shell; encounter a bug or unexpected behavior; close the Shell; make some changes to my code; restart the Shell. And so on….
This sucks. Not only does it end up wasting 3-4 seconds for each Shell restart on my Macbook Air (those seconds really add up over time), it also inevitably leads to soul crushing frustration.
As an alternative to incessantly restarting the Shell, you could use the builtin Python reload() function, which reimports a given module within the program. Unfortunately, issues still crop up:
- reload() must be passed a module as an argument, meaning you have to import the entire module at some point in your program.
- Typing reload(module_name) still takes too much time if done ad naseum
- Django models.py modules can’t be reloaded normally due to the AppCache singleton.
- People forget about the builtin in the heat of the moment, it just happens.
My solution (inspired by the builtin, multi-threaded Django server) was to add an auto-reloading thread to Shell_Plus.
Shell_Plus, an the extremely helpful script found in the Django Extensions library, is a Django Management Command which spins up an embedded Shell. Within the embedded Shell, the script sets a number of objects in the global scope on startup via (i.e. your Django models and settings variables).
The major change here is that, before entering the mainloop of the IPython Shell, a Watchdog observer thread (another great library) is kicked off, which listens for file system events. When a relevant event occurs, the thread automatically reloads the module into the global scope of the embedded Shell via a global dictionary. It’s fast and completely transparent to the Shell user.
This rather large gist (https://gist.github.com/4499367) contains the code for the reloader thread, along with a heap of documentation.
At this point, I should mention that I decided to add some black magic to make the reloader as powerful as possible.
When a class definition is changed in a file and reloaded in the shell, all of the instances of the old version of that class inside the Shell’s global scope are dynamically assigned the reloaded class.
old_instance.__class__ = RefashionedKls
The implications of that last point are a little crazy. While inside a pdb debugger, you can add, delete, or modify class methods and immediately see the result inside the Shell session. I’ve used it to great effect while debugging and generally experimenting with code but the danger is obvious so beware. Again, check out the gist to see more details.
You can find my Django Extensions fork at https://github.com/Bpless/django-extensions.
Even if you are not developing a Django app specifically, you can build off this concept of auto-reloading embedded Shells for any Python project.
One alternative is restarting the kernel from the menu (shell_plus –notebook). At least that’s how I tend to do it.
Viewing on a nexus 4… Only thing I can see without scrolling is your mug. Why?
Haha, I don’t know. I just changed the wordpress theme. I honestly haven’t looked at it on mobile yet.
…I really need to just design the blog myself.
This is a really cool idea. I’ve been looking for a way to do this for the past while. Unfortunately it’s not working yet on my local dev machine in a virtualenv.
I did a little bit of debugging and I think the issue is that the ReloaderEventHandler thinks that the project_root is the virtualenv root (~/.virtualenvs/web) instead of the directory where I store the actual code (~/workspace/web). A lot of virtualenv users store their virtualenvs in a single place and their code in other places on their filesystem.
Not sure if you designed with this pattern in mind. If you didn’t, I’ll gladly do a fork and add some code to support this.
Either way, thanks a lot for writing this. It’s going to save a lot of time
.
Thanks Omar, it’s awesome to hear that you’d like to use this. I decided to default to the virtualenv intentionally. My goal was to make it easy to work with python packages installed in your virtualenv. This is the relevant code:
autoreload_path = os.environ.get('VIRTUAL_ENV', getattr(settings, 'PROJECT_ROOT', False))It looks first for the VIRTUAL_ENV environment variable and then falls back to the Django settings variable called PROJECT_ROOT.
In hindsight, I should have defaulted to PROJECT_ROOT, instead of the VIRTUALENV. Definitely more sensible.
autoreload_path = getattr(settings, 'PROJECT_ROOT', os.environ.get('VIRTUAL_ENV')What do you think?
Hey Ben,
I also experienced frustration with this for a long time and I came up with a solution that I’m personally very happy with now. Hopefully you’re editing in vim because that’s what I used. Ok, here it is:
1) open your .vimrc
2) Paste this line in: map !! :w:exe “:!python ” . getreg(“%”) . “”
3) Save and restart that mutha fucka
Now when you want to run the python script you’re working on just type ‘!!” and vim will open a temporary shell (full-screen). When the script finishes or you Ctrl+C you’re right back to where you were. I should also mention that this FIRST saves the file and then runs your script. Finally, make sure the script you’re running has a __main__(). Haha. Oh yeah, and if you’re wondering why I chose “!!”–it just looks badass.
Let me know what you think.