7

I want to run a Pythonic project using Python compilation (.pyc or __pycache__). In order to do that in Python2, I haven't any problem.


Here is a simplified example in a Python2 project:

  • Project tree:

    test2
    ├── main.py
    └── subfolder
        ├── __init__.py
        └── sub.py
    
  • Compile:

    python -m compileall test2
    
  • Project tree after the compile:

    test2
    ├── main.py
    ├── main.pyc
    └── subfolder
        ├── __init__.py
        ├── __init__.pyc
        ├── sub.py
        └── sub.pyc
    
  • As you can see, several .pyc manually generated. Now I can run this project using main.pyc as fine, which has a relation with the sub.py:

    python main.pyc
    

    Out:

    Hi
    Bye
    
  • main.py content:

    from subfolder import sub
    
    print('Bye')
    
  • sub.py content:

    print('Hi')
    

Now I want to retry this behavior in a Python3 project.

Here is a simplified asyncio (available in Python3) project:

  • Project tree:

    test3
    ├── main.py
    └── subfolder
        ├── __init__.py
        └── sub.py
    
  • Compile:

    python3 -m compileall test3
    
  • Project tree after the compile:

    test3
    ├── main.py
    ├── __pycache__
    │   └── main.cpython-36.pyc
    └── subfolder
        ├── __init__.py
        ├── __pycache__
        │   ├── __init__.cpython-36.pyc
        │   └── sub.cpython-36.pyc
        └── sub.py
    
  • As you can see, __pycache__ folders manually generated. But I cannot run this project using main.cpython-36.pyc which has a relation with subfolder:

    cd test3/__pycache__
    python3 main.cpython-36.pyc
    

    Out (I expected that produced the Hi Bye message):

    Traceback (most recent call last):
      File "test3/main.py", line 2, in <module>
    ModuleNotFoundError: No module named 'subfolder'
    
  • main.py content:

    import asyncio
    from subfolder import sub
    
    async def myCoroutine():
        print("Bye")
    
    def main():
        loop = asyncio.get_event_loop()
        loop.run_until_complete(myCoroutine())
        loop.close()
    
    main()
    
  • sub.py content:

    print('Hi')
    

Question:

How can I run this project (above Python3 project) using __pycache__ folder?

Or

How can I run a Python3 project with the relation between subfolders using python compilation?


[NOTE]:

  • I cannot use the python compileall (Python2 compile) in the above Python3 project due to the asyncio method.

  • My Python(s) version is Python2.7 and Python3.6

1 Answer 1

12

You can enforce the same layout of pyc-files in the folders as in Python2 by using:

python3 -m compileall -b test3

The option -b triggers the output of pyc-files to their legacy-locations (i.e. the same as in Python2).

After that you can once again use the compiled cache via:

python3 main.pyc

The way the loading of modules works since PEP-3147, it is impossible to use pyc-files from __pycache__ folder in the way you intend: If there is no *.py-file, the content of the __pycache__ is never looked-up. Here is the most important part of the workflow:

   import foo
     |
     |
     -- >  [foo.py exists?]  --- NO ----> [foo.pyc exists?]  -- NO --> [ImportError] 
            |                                     |
            |                                    YES
           YES                                    |--> [load foo.pyc]
            |
            |-> [look up in __pycache__]

                               

That means, files from __pycache__ are only looked up, when a corresponding *.py-file can be found.


Obviously, building python scripts with a Python version 3.X in this way and trying to run the resulting pyc-files with another Python version 3.Y will not work: Different Python versions need different pyc-files, this is the whole point behind PEP-3147.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.