0

I want to embed a Python interpreter in my C++ project using the pybind11 embedded interpreter.

During runtime, Python modules from different locations can be loaded and deleted.

To achieve this, I used importlib to refresh my Python cache. However, the cache remains unchanged, and if I try to load a module with the same name but from a different location, the old module is still loaded.

Thanks for your help.

i tried this small example

#include <pybind11/pybind11.h>
#include <pybind11/embed.h>
namespace py = pybind11;

void addModulePath(const std::string &module) {
    auto sys = py::module::import("sys");
    auto importlib = py::module::import("importlib");
    auto path = sys.attr("path");
    py::print(path);

    path.attr("append")(module);
    importlib.attr("invalidate_caches")();
    py::print(path);
}

void deleteModulePath(std::string& module) {
    auto sys = py::module::import("sys");
    auto importlib = py::module::import("importlib");

    auto path = sys.attr("path");
    py::print(path);
    path.attr("remove")(module);
    importlib.attr("invalidate_caches")();
    py::print(path);

}

PYBIND11_EMBEDDED_MODULE(changeModule, m) {
    m.def("add_module", &addModulePath, "add module path");
    m.def("delete_module", &deleteModulePath, "delete module path");
}

int main() {

    constexpr auto test1Path = "/tmp/test1";
    constexpr auto test2Path = "/tmp/test2";

    py::scoped_interpreter guard{};
    py::module_ module = py::module_::import("changeModule");

    module.attr("add_module")(test1Path);

    auto test = py::module_::import("test");

    test.attr("test")();
    module.attr("delete_module")(test1Path);

    module.attr("add_module")(test2Path);

    test = py::module_::import("test");
    test.attr("test")();
    return 0;

}

the folders test1 and test2 does both contain a file called "test.py" which prints its folder name.

I don't want to restart my interpreter.

7
  • What version of Python? Looks a bit like bugs.python.org/issue45703 Commented Aug 13, 2024 at 14:54
  • What are you trying to accomplish by loading modules with the same name, but different code? In anything more than a toy problem this could lead to bugs where module foo holds a reference to version 1 of test and module bar holds a reference to version 2 of test. Just because you've deleted a module from sys.modules does not mean it is no longer accessible in the interpreter. Have you tried just adding /tmp to sys.path and importing test1.test and test2.test? Commented Aug 13, 2024 at 14:58
  • Oh, invalidate_caches() isn't enough to load new source code of a module that has already been imported. If a module exists in sys.modules, then that module is always used in preference to loading the module from disk. You need to either delete the module from sys.modules or use importlib.reload(mymod). reload() preserves the globals of the module. So names that were present in the old source, but not the new source will be preserved. Commented Aug 13, 2024 at 15:07
  • 1
    @DanMašek I use Python 3.12.4 Commented Aug 13, 2024 at 15:13
  • 2
    Generally you can't reload extensions at all. And lots of things (e.g. Numpy) will just crash if you try Commented Aug 13, 2024 at 15:37

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.