Help installing esa_snappy

[jeffdiamonddw]

I have been trying to install esa_snappy for a number of days with no success so far. I need help please. details are below.
I have also tried python3.7 and various other things such as using pip to install jpy .

If anyone can help me or provide a script that works I would be very very grateful!

Thankyou!

Jeff Diamond

**************************I began with this ubuntu on an aws ec2 instance:
ubuntu@ip-172-31-15-21:~/snap/bin$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy

**************************Then ran the following script:
#install python tools
sudo apt update
sudo apt-get install -y -o DPkg::Options::=--force-confdef cmake python3-dev python3-pip

#install java
cd ~
wget https://download.oracle.com/java/21/latest/jdk-21_linux-x64_bin.deb -O jdk-21_linux-x64_bin.deb
sudo apt-get install -y ./jdk-21_linux-x64_bin.deb
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk-21-oracle-x64/bin/java" 2
sudo update-alternatives --set java /usr/lib/jvm/jdk-21-oracle-x64/bin/java
export JDK_HOME=$(java -XshowSettings:properties -version 2>&1 | grep "java.home" | tr -s ' ' | cut -d' ' -f4)
echo export "JDK_HOME=$JDK_HOME" >> ~/.bashrc
echo export "JAVA_HOME=$JDK_HOME" >> ~/.bashrc
source ~/.bashrc

#install snap
cd
wget https://download.esa.int/step/snap/9.0/installers/esa-snap_sentinel_unix_9_0_0.sh -O /home/ubuntu/esa-snap_sentinel_unix_9_0_0.sh
chmod a+x /home/ubuntu/esa-snap_sentinel_unix_9_0_0.sh
/home/ubuntu/esa-snap_sentinel_unix_9_0_0.sh
echo 'export SNAP_HOME=/home/ubuntu/snap' >> ~/.bashrc
source ~/.bashrc

#install jpy
sudo apt install maven
git clone https://github.com/jpy-consortium/jpy.git
cd jpy
python3 setup.py bdist_wheel
find /home/ubuntu/jpy/dist/ -name *.whl | xargs -I{} cp {} /home/ubuntu/.local/lib/python3.7/site-packages/snappy
export LD_LIBRARY_PATH=/home/ubuntu/snap/jre/jre/lib/amd64/server/

#configure snappy
cd /home/ubuntu/snap/bin
./snappy-conf /usr/bin/python3 /home/ubuntu/.local/lib/python3.10/site-packages/

************************ ***********************I get this error
ubuntu@ip-172-31-15-21:~/snap/bin$ ./snappy-conf /usr/bin/python3 /home/ubuntu/.local/lib/python3.10/site-packages/
/home/ubuntu/snap/bin/../platform/lib/nbexec: WARNING: environment variable DISPLAY is not set
Configuring SNAP-Python interface...
java.io.IOException: Python configuration failed.
Command [/usr/bin/python3 ./snappyutil.py --snap_home /home/ubuntu/snap --java_module /home/ubuntu/snap/snap/modules/org-esa-snap-snap-python.jar --force --log_file ./snappyutil.log --jvm_max_mem 10G --java_home /home/ubuntu/snap/jre/jre --req_arch amd64]
failed with return code 10.
Please check the log file '/home/ubuntu/.local/lib/python3.10/site-packages/snappy/snappyutil.log'.
at org.esa.snap.python.PyBridge.configureJpy(PyBridge.java:232)
at org.esa.snap.python.PyBridge.installPythonModule(PyBridge.java:149)
at org.esa.snap.rcp.cli.SnapArgsProcessor.processPython(SnapArgsProcessor.java:103)
at org.esa.snap.rcp.cli.SnapArgsProcessor.process(SnapArgsProcessor.java:49)
at org.netbeans.modules.sendopts.DefaultProcessor.process(DefaultProcessor.java:202)
at org.netbeans.spi.sendopts.Option$1.process(Option.java:387)
at org.netbeans.api.sendopts.CommandLine.process(CommandLine.java:317)
at org.netbeans.modules.sendopts.HandlerImpl.execute(HandlerImpl.java:62)
at org.netbeans.modules.sendopts.Handler.cli(Handler.java:69)
at org.netbeans.CLIHandler.notifyHandlers(CLIHandler.java:234)
at org.netbeans.core.startup.CLICoreBridge.cli(CLICoreBridge.java:82)
at org.netbeans.CLIHandler.notifyHandlers(CLIHandler.java:234)
at org.netbeans.CLIHandler$1.exec(CLIHandler.java:268)
at org.netbeans.CLIHandler.finishInitialization(CLIHandler.java:447)
at org.netbeans.MainImpl.finishInitialization(MainImpl.java:256)
at org.netbeans.Main.finishInitialization(Main.java:92)
at org.netbeans.core.startup.Main.start(Main.java:316)
at org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:123)
at java.lang.Thread.run(Thread.java:748)
Python configuration error: Python configuration failed.
Command [/usr/bin/python3 ./snappyutil.py --snap_home /home/ubuntu/snap --java_module /home/ubuntu/snap/snap/modules/org-esa-snap-snap-python.jar --force --log_file ./snappyutil.log --jvm_max_mem 10G --java_home /home/ubuntu/snap/jre/jre --req_arch amd64]
failed with return code 10.
Please check the log file '/home/ubuntu/.local/lib/python3.10/site-packages/snappy/snappyutil.log'.

******************************************* and this is the resulting log file
ubuntu@ip-172-31-15-21:~/snap/bin$ cat /home/ubuntu/.local/lib/python3.10/site-packages/snappy/snappyutil.log
INFO: Installing from Java module '/home/ubuntu/snap/snap/modules/org-esa-snap-snap-python.jar'
WARNING: Architecture requirement possibly not met: Python is x86_64 but JVM requires amd64
INFO: Installing jpy...
ERROR: The module 'jpy' is required to run snappy, but no binary 'jpy' wheel matching the pattern
'jpy-{version}-cp310-{abi_tag}-linux_x86_64.whl' could be found.
You can try to build a 'jpy' wheel yourself, then copy it into
"/home/ubuntu/.local/lib/python3.10/site-packages/snappy", and then run the configuration again.
Unzip the jpy sources in /home/ubuntu/.local/lib/python3.10/site-packages/snappy/jpy-.zip, then
$ cd jpy-
$ python setup.py bdist_wheel
$ cp dist/*.whl "/home/ubuntu/.local/lib/python3.10/site-packages/snappy"
Or get the source code from https://github.com/bcdev/jpy and follow the build instructions:
$ git clone https://github.com/bcdev/jpy.git
$ cd jpy
ERROR: Configuration failed with exit code 10

[SabineEmbacher]

Dear Jeff,

have you tried to find help for this problem in the forum?
E.g. using the expanded search?
https://forum.step.esa.int/search?expanded=true&q=snappy

Best regards
Sabine

[dolaf]

Hi Jeff,

thanks a lot for your detailed bug report.

From the messages in your 'Install jpy' and 'Configure snappy' steps, it seems that Python versions 3.7 and 3.10 are mixed up, which might be a source of the problem:

python3 setup.py bdist_wheel
find /home/ubuntu/jpy/dist/ -name *.whl | xargs -I{} cp {} /home/ubuntu/.local/lib/python3.7/site-packages/snappy
export LD_LIBRARY_PATH=/home/ubuntu/snap/jre/jre/lib/amd64/server/

#configure snappy
cd /home/ubuntu/snap/bin
./snappy-conf /usr/bin/python3 /home/ubuntu/.local/lib/python3.10/site-packages/

However, in the past we received various feedback from users who had similar problems during the installation, and we agree that this procedure is not very user-friendly. Therefore, the snappy interface has been significantly revised for the new SNAP 10 which will be released very soon. In especially, Python versions up to 3.10 will be supported, and the required jpy wheels will come with the snappy module, thus a manual jpy installation will no longer be needed.

The latest documentation on the current installation procedure (for SNAP <= 9) can be found here.

Also note that the current SNAP-Python interface is called 'snappy', whereas the new, revised module will be called 'esa_snappy' from SNAP 10 onwards (documentation in preparation here).

We hope this will help at least for the future.

Cheers,
Olaf

[jeffdiamonddw]

Thanks everyone. I found a solution by Ferni at https://forum.step.esa.int/t/problems-with-snappy-and-python-3-8/36833

This is the resulting script run on ubuntu22:

#some apt-get installations
sudo apt update
sudo apt-get install -y -o DPkg::Options::=--force-confdef cmake python3-dev python3-pip

#install python3.8
sudo apt update
sudo apt install -y software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install -y python3.8
sudo apt install -y python3.8-distutils
sudo apt install -y python3.8-dev

#more apt installations
sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev git

#install java
cd ~
wget https://download.oracle.com/java/21/latest/jdk-21_linux-x64_bin.deb -O jdk-21_linux-x64_bin.deb
sudo apt-get install ./jdk-21_linux-x64_bin.deb
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk-21-oracle-x64/bin/java" 2
sudo update-alternatives --set java /usr/lib/jvm/jdk-21-oracle-x64/bin/java
export JDK_HOME=$(java -XshowSettings:properties -version 2>&1 | grep "java.home" | tr -s ' ' | cut -d' ' -f4)
echo export "JDK_HOME=$JDK_HOME" >> ~/.bashrc
echo export "JAVA_HOME=$JDK_HOME" >> ~/.bashrc
source ~/.bashrc

#install esa snap toolbox
cd
wget https://download.esa.int/step/snap/9.0/installers/esa-snap_sentinel_unix_9_0_0.sh -O /home/ubuntu/esa-snap_sentinel_unix_9_0_0.sh
chmod a+x /home/ubuntu/esa-snap_sentinel_unix_9_0_0.sh
/home/ubuntu/esa-snap_sentinel_unix_9_0_0.sh
echo 'export SNAP_HOME=/home/ubuntu/snap' >> ~/.bashrc
source ~/.bashrc

#download wheel for jpy
wget https://anaconda.org/Terradue/jpy/0.9.0/download/linux-64/jpy-0.9.0-py38_3.tar.bz2
tar -xvf py-0.9.0-py38_3.tar.bz2

#run snappy config
cd ~/snap/bin
./snappy-conf /usr/bin/python3.8

#copy wheel file to snappy installation and run snappy-conf again
cp ~/jpy_wheel/jpy-0.9.0-cp38-cp38-linux_x86_64.whl ~/.snap/snap-python/snappy/
cd ~/snap/bin
./snappy-conf /usr/bin/python3.8 /home/ubuntu/.local/lib/python3.8/site-packages/

#put some environment variables in .bashrc
export PYTHONPATH="${PYTHONPATH}:/home/ubuntu/.snap/snap-python/snappy"
echo export PYTHONPATH=${PYTHONPATH}:/home/ubuntu/.snap/snap-python/snappy >> /home/ubuntu/.bashrc
export SNAP_HOME=/home/ubuntu/snap >> /home/ubuntu/.bashrc
echo export SNAP_HOME=/home/ubuntu/snap >> /home/ubuntu/.bashrc
source /home/ubuntu/.bashrc

[dolaf]

Hi Jeff, thanks for the info. Good to hear that the installation sequence now works for you!

[h-a-graham]

Just wanted to add my solution here - I was also having trouble with the python module installation on linux, the following bash script solved it for me:

#!/bin/bash
set -e

# This script builds the ESA SNAP Python API and runs the tests

# Set the paths to:
# ESA SNAP Python API
ESA_SNAPPY="$HOME/.snap/snap-python/esa_snappy"
# ESA SNAP binaries
ESA_SNAP_BIN="$HOME/esa-snap/bin"
# ESA SNAP virtual environment
VENV_LOC="$HOME/virtualenvs/snap"

# Create the virtual environment
if [ ! -d "$VENV_LOC" ]; then
  python -m venv "$VENV_LOC"
fi

# Activate the virtual environment
source "$VENV_LOC/bin/activate"

# Copy the jpy wheel to the binary folder
cp "$ESA_SNAPPY/lib/jpy-0.13.0-cp310-cp310-linux_x86_64.whl" \
  "$ESA_SNAP_BIN/jpy-0.13.0-cp310-cp310-linux_x86_64.whl"

# Copy the python module to the virtual environment
cp -r "$ESA_SNAPPY" "$VENV_LOC"/lib/python*/site-packages

# Run snappy-conf to set up the python module
"$ESA_SNAP_BIN/snappy-conf" "$HOME/virtualenvs/snap/bin/python" "$VENV_LOC"/lib/python*/site-packages

# Install missing dependencies and pytest
pip install numpy
pip install pytest

# Run tests
cd "$VENV_LOC"/lib/python*/site-packages/esa_snappy/tests
pytest

[dolaf]

Hi, thanks for posting this. However, note that with SNAP 10 (released on April 18th) it should no longer be necessary to manually copy jpy wheels or the esa_snappy module. After you selected the Python of your choice (base or a virtual environment), you can directly run snappy-conf, as you did in your script.

The latest how-to is here.
We try to frequently improve the documentation, following recent user feedback.

[h-a-graham]

Thanks for the link to those instructions! Just to confirm though that I did attempt that approach first and I believe that I'm running the latest snap version:

My OS is Ubuntu 22.04.4 LTS

I'm playing around with making an R wrapper for GPT so I'll have a go at a clean install soon and try linking up python again and report back...

[h-a-graham]

quick update - if I find more time test I will try update further but I suspect the issue may have resulted from using the installer GUI to attempt setting up the python environment in the first instance. When I install snap in headless mode with -q, the snappy-conf util works exactly as expected...