RSS Feed
Download our iPhone app
Browse DevX
Sign up for e-mail newsletters from DevX

By submitting your information, you agree that devx.com may send you DevX offers via email, phone and text message, as well as email offers about other products and services that DevX believes may be of interest to you. DevX will process your information in accordance with the Quinstreet Privacy Policy.


Developer's Guide to Python 3.0: Python 2.6 and Migrating From 2 to 3 : Page 3

Python 3.0 has been released. Are you ready to migrate your code? Find out what you need to know to make the switch.




Building the Right Environment to Support AI, Machine Learning and Deep Learning

The Migration Process

Assuming you've digested all the information, heard all the warnings and recommendations, and you have made up your mind to port your Python 2 project to Python 3. How would you go about it?

Before getting started, you should know that there's a tool that can help, called 2to3. The 2to3 tool is a Python program that automatically converts Python 2.6 code to Python 3.0 code. It has a modular architecture and includes plug-ins that can fix specific issues. While it's very useful, 2to3 can't (or at least doesn't) do everything for you—there are several issues that you need to fix manually.

Here are the step-by-step instructions:

  1. Make sure you have good test coverage and that all the tests pass (under Python 2.x).
  2. Port your code to Python 2.6.
  3. Make sure all the tests pass using that version.
  4. Run your code under Python 2.6 with the -3 flag.
  5. Fix any warnings.
  6. Run the tests again (still under Python 2.6).
  7. Run the 2to3 tool.
  8. Run your tests under Python 3.
  9. Fix any problems.
  10. Repeat steps 8 and 9 until all tests pass.

Here's an example that uses this process to migrate a simple program that populates a dictionary with some values, prints them and writes the sorted values to a file. This program has some elements that are not compatible with Python 3. Here's the code:

import os import sys def main(): print '=====================================' print 'Python 2.x to Python 3 Migration test' print '=====================================' print 'Python version is:', sys.version d = dict(a=1, b=2, c=3) assert dict.has_key('a') print 'The items of d:' for k, v in d.iteritems(): print '%s: %s' % (k, v) v = d.values() print 'First value is', v[0] if os.path.isfile('values.txt'): os.remove('values.txt') f = file('values.txt', 'w') print>> f, sorted(v) f.close() def test(): main() s = open('values.txt', 'rb').read() print s assert s == '[1, 2, 3]\n' if __name__=='__main__': test()

The preceding program is self-testing. Running the script executes the test() function, which calls the main() function and verifies that the correct results are in a file. Test coverage is important because Python is a dynamic language, and the tools are not perfect. Unless your tests cover your entire codebase, you can't be sure whether the migration was successful—it might fail at runtime when the interpreter gets to the untested areas of the code.

Step 1: Ensure All Tests Pass Under Python 2.x

Here's the test output under Python 2.5

===================================== Python 2.x to Python 3 Migration test ===================================== Python version is: 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] The items of d: a: 1 c: 3 b: 2 First value is 1 [1, 2, 3]

That's the expected output. In addition, the assert statement in the test() function didn't complain, so all the tests pass (just one test in this case).

Step 2: Port Your Code to Python 2.6

In this case, the code itself didn't require changes, so I simply ran the program under Python 2.6.

Step 3: Make Sure All Tests Pass

Here's the output:

===================================== Python 2.x to Python 3 Migration test ===================================== Python version is: 2.6.2 (r262:71600, Apr 16 2009, 09:17:39) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] The items of d: a: 1 c: 3 b: 2 First value is 1 [1, 2, 3]

So far, it works perfectly. Note that the program always prints the Python version it is running under. This can be useful during migration when you run the same code under several interpreters.

Step 4: Run Code in Python 2.6 Using the -3 Flag

The -3 flag warns about Python 3 compatibility issues that 2to3 can't solve. These issues include: dict.has_key(), apply(), callable(), coerce(), execfile(), reduce() and reload(). I'm not sure why 2to3 can't handle most of these issues automatically, but that's how it is. When running the program with the -3 flag you get a nice deprecation warning that even tells you what the fix is:

article_5.py:11: DeprecationWarning: dict.has_key() not supported in 3.x; use the in operator assert d.has_key('a')

Step 5: Fix the Warnings

In this case, I replaced the code d.has_key('a') with 'a' in d.

Step 6: Repeat Tests (Still Under Python 2.6)

Everything works fine even with the -3 flag.

Step 7: Run the 2to3 Tool

Here is the result of running 2to3 on our program (use the -w switch to overwrite the original file):

> 2to3 -w article_5.py RefactoringTool: Skipping implicit fixer: buffer RefactoringTool: Skipping implicit fixer: idioms RefactoringTool: Skipping implicit fixer: set_literal RefactoringTool: Skipping implicit fixer: ws_comma --- article_5.py (original) +++ article_5.py (refactored) @@ -2,33 +2,33 @@ import sys def main(): - print '=====================================' - print 'Python 2.x to Python 3 Migration test' - print '=====================================' - print 'Python version is:', sys.version + print('=====================================') + print('Python 2.x to Python 3 Migration test') + print('=====================================') + print('Python version is:', sys.version) d = dict(a=1, b=2, c=3) assert 'a' in d - print 'The items of d:' - for k, v in d.iteritems(): - print '%s: %s' % (k, v) + print('The items of d:') + for k, v in d.items(): + print('%s: %s' % (k, v)) - v = d.values() - print 'First value is', v[0] + v = list(d.values()) + print('First value is', v[0]) if os.path.isfile('values.txt'): os.remove('values.txt') # The 'file' function is NOT handled by 2to3 f = file('values.txt', 'w') - print>> f, sorted(v) + print(sorted(v), file=f) f.close() def test(): main() s = open('values.txt', 'rb').read() - print s + print(s) assert s == '[1, 2, 3]\n' if __name__=='__main__': RefactoringTool: Files that were modified: RefactoringTool: article_5.py

All the print statements were fixed (including the print>>). The dict's iteritems() method was fixed to simply items() (which returns an iterator in Python 3). It wrapped the d.values() call, which returns a list in Python 2.x, in a list. That's pretty good. The original file was saved with a .bak extension and replaced with the refactored code.

You can run 2to3 on entire directories also and it has many command line options.

Usage: 2to3 [options] file|dir ...

Options: -h, --help show this help message and exit -d, --doctests_only Fix up doctests only -f FIX, --fix=FIX Each FIX specifies a transformation; default: all -x NOFIX, --nofix=NOFIX Prevent a fixer from being run. -l, --list-fixes List available transformations (fixes/fix_*.py) -p, --print-function Modify the grammar so that print() is a function -v, --verbose More verbose logging -w, --write Write back modified files -n, --nobackups Don't write backups for modified files.

Step 8: Run Code in Python 3

Make sure you have Python 3 installed, and then run the tests again using Python 3. Running the sample program gives the following output:

===================================== Python 2.x to Python 3 Migration test ===================================== Python version is: 3.0.1 (r301:69597, Feb 14 2009, 19:03:52) [GCC 4.0.1 (Apple Inc. build 5490)] The items of d: a: 1 c: 3 b: 2 First value is 1 Traceback (most recent call last): File "article_5.py", line 35, in <module> test() File "article_5.py", line 29, in test main() File "article_5.py", line 24, in main f = file('values.txt', 'w') NameError: global name 'file' is not defined

What happened? The program is using the file function that was a built-in function in Python 2.x, but doesn't exist in Python 3. The 2to3 tool doesn't handle it and the -3 warning flag is unaware of the problem. This is the reason it is so important to have good test coverage.

Step 9: Fix Problems

The fix in this case is very simple; just replace the file function with the open function:

f = open('values.txt', 'w')

This time, when you run the tests, you'll see that the fix worked, but now the assert fails:

b'[1, 2, 3]\n' Traceback (most recent call last): File "article_5.py", line 35, in <module> test() File "article_5.py", line 32, in test assert s == '[1, 2, 3]\n' AssertionError

Step 10: Repeat Steps 8 and 9 Until All Tests Pass

What's the problem now? In Python 3, when you open a file in binary mode and read from it you get back a bytes object, not a string. You can fix the problem by opening the file for reading as a text file (use 'r' instead of 'rb'):

def test(): main() s = open('values.txt', 'r').read() print(s) assert s == '[1, 2, 3]\n'

That last change solved all the problems. Now, all the tests pass and the migration is complete.

Comment and Contribute






(Maximum characters: 1200). You have 1200 characters left.



Thanks for your registration, follow us on our social networks to keep up-to-date