[FIXED] Module not found if graalpy is packaged in Spring Boot Jar

Issue

I'm building a Spring Boot (3 RC1) application with some Python code (GraalVM 22.3). Running the application in dev mode works. After building Jar with Maven I get an error:

Caused by: org.graalvm.polyglot.PolyglotException: ModuleNotFoundError: No module named 'pystac'
    at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:399) ~[org.graalvm.sdk:na]
    at ch.so.agi.sodata.stac.ConfigService.readXml(ConfigService.java:116) ~[classes!/:0.0.1-SNAPSHOT]
    at ch.so.agi.sodata.stac.SodataStacApplication.lambda$init$0(SodataStacApplication.java:60) ~[classes!/:0.0.1-SNAPSHOT]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:767) ~[spring-boot-3.0.0-RC1.jar!/:3.0.0-RC1]
    ... 13 common frames omitted

The python.Executable shows to the graalpy executable packaged in the Jar: file:/Users/stefan/sources/datenbezug/sodata-stac/target/sodata-stac-0.0.1-SNAPSHOT.jar!/BOOT-INF/classes!/venv/bin/graalpy

Grepping the Jar shows the pystac module in the Jar file, e.g. BOOT-INF/classes/venv/lib/python3.8/site-packages/pystac/item.py

Creating the context with:

var VENV_EXECUTABLE = ConfigService.class.getClassLoader()
                .getResource(Paths.get("venv", "bin", "graalpy").toString())
                .getPath();

var context = Context.newBuilder("python")
                .allowAllAccess(true)
                .option("python.Executable", VENV_EXECUTABLE)
                .option("python.ForceImportSite", "true")
                .build()

Is it possible to put the whole python stuff including third party libs into the FatJar? Or did I just miss some packaging magic?

Solution

The issue is that the default Truffle filesystem, AFAIK, only supports the actual filesystem of your OS, i.e., it does not "see" resources in the jar file. This is why it works in dev mode when the resources are just files on the filesystem.

There are two options how to deal with this:

• do not put the venv into resources, but deploy it separately, such that it ends up somewhere on the filesystem
• implement custom Truffle filesystem, but I am not aware of anyone who tried this with virtual environments yet, so there may be some rough edges. Most notably GraalVM LLVM Toolchain that is used to build Python native extensions for execution with GraalVM LLVM engine produces system specific binaries, so your jar will not be as portable as one would expect from a jar. If you only use pure Python packages, it may be fine.

Answered By - Steves
Answer Checked By - Senaida (JavaFixing Volunteer)