proxmox/docs/archive/00-meta-pruned/JNA_WHY_NOT_WORKING_REVIEW.md

Why JNA Is Not Working Properly (VMID 2101 / Besu)

Purpose: Review root causes for JNA failures in Besu on VMID 2101 (Core RPC) and how to verify/fix them.

---

1. What JNA does in Besu

• JNA (Java Native Access) is used by Besu (and its dependency OSHI) to call native libraries (e.g. for hardware/OS info, Udev).
• At runtime JNA extracts a small native library (e.g. libjnidispatch.so) into the JVM temporary directory (java.io.tmpdir) and loads it.
• If that directory is read-only, missing, or not writable by the process user, JNA fails to initialize and you see:
  • NoClassDefFoundError: Could not initialize class com.sun.jna.Native
  • or "Read-only file system" when writing the native lib.

So JNA “not working properly” in our context means: the JVM’s temporary directory used by JNA is not writable or not correct.

---

2. Root causes (why JNA can still fail)

#	Cause	What happens	How to check
1	Default tmpdir is read-only	Kernel remounted root read-only (ext4 errors). Default java.io.tmpdir is often /tmp, which is then read-only. JNA cannot extract the native lib.	pct exec 2101 -- mount | grep 'on / ' → if ro, then root is read-only. Run make-rpc-vmids-writable first.
2	-Djava.io.tmpdir not applied	BESU_OPTS must be passed to the JVM; Besu does pass it. If the unit file was edited by hand or is an old version, the -Djava.io.tmpdir=/data/besu/tmp might be missing or wrong.	`pct exec 2101 -- grep -E 'BESU_OPTS
3	/data/besu/tmp not writable by besu user	JNA runs as the besu user. If /data/besu/tmp is root-only or missing, JNA still fails.	pct exec 2101 -- ls -la /data/besu/ and pct exec 2101 -- su -s /bin/bash besu -c 'touch /data/besu/tmp/.w && rm /data/besu/tmp/.w'
4	noexec on /data or /data/besu	If the mount has noexec, the extracted .so cannot be executed and JNA fails.	pct exec 2101 -- mount | grep /data (if /data is a separate mount)
5	Install never completed	Fix script stalled at "Installing packages..." (apt). So /opt/besu is missing or incomplete, and besu-rpc.service may be missing or not have the tmpdir option. Besu never starts, so “JNA not working” is really “Besu not installed”.	`pct exec 2101 -- test -f /opt/besu/bin/besu && echo ok
6	Conflicting or missing JNA on classpath	Besu ships a bundled JNA. If the system or another lib provides a different JNA version, you can get class loading errors. Less common with the tarball install.	Only if you see classpath/version errors in logs; then reinstall Besu from clean tarball.
7	Java version	Besu 23.10.x supports Java 17. Wrong or very old Java can cause odd native loading behavior.	pct exec 2101 -- java -version (expect openjdk 17).

---

3. What we already do (and one gap)

• Make root writable: make-rpc-vmids-writable-via-ssh.sh runs e2fsck on 2101 so root (and /tmp) can be mounted rw again. That removes cause 1 for the filesystem.
• Set java.io.tmpdir: We set -Djava.io.tmpdir=/data/besu/tmp in:
  • install-besu-in-ct-standalone.sh (unit template: Environment="BESU_OPTS=-Xmx2g -Xms1g -Djava.io.tmpdir=${BESU_DATA}/tmp").
  • fix-rpc-2101-jna-reinstall.sh (post-install: mkdir -p /data/besu/tmp, chown besu:besu, and grep -q java.io.tmpdir ... || sed -i ... to add it if missing).
• Besu passes BESU_OPTS to the JVM, so -D options in BESU_OPTS are applied (see Besu docs).

Gap: The fix script often does not complete (apt stalls at "Installing packages..."). So in practice:

• Either Besu is not (re)installed (no /opt/besu/bin/besu), so the unit file we patch is never created or is old.
• Or the unit exists but post-install step never runs, so -Djava.io.tmpdir=/data/besu/tmp is never added and the service still uses default /tmp. If /tmp is read-only again later, JNA fails.

So “JNA not working properly” is often a mix of: install not finishing and/or tmpdir never applied because the fix script never reached the step that sets it.

---

4. Verification checklist (on the host / in the CT)

Run from a host that can SSH to r630-01 (192.168.11.11), or from inside CT 2101:

# 1. Root and /tmp writable?
pct exec 2101 -- sh -c 'mount | grep "on / "; touch /tmp/.w && rm /tmp/.w && echo "tmp ok"'

# 2. Besu installed and unit has tmpdir?
pct exec 2101 -- test -f /opt/besu/bin/besu && echo "besu binary ok" || echo "MISSING"
pct exec 2101 -- grep -E 'BESU_OPTS|java.io.tmpdir' /etc/systemd/system/besu-rpc.service

# 3. /data/besu/tmp exists and writable by besu?
pct exec 2101 -- ls -la /data/besu/
pct exec 2101 -- su -s /bin/bash besu -c 'touch /data/besu/tmp/.w && rm /data/besu/tmp/.w && echo "besu can write tmp"'

# 4. Service status and recent log (JNA error?)
pct exec 2101 -- systemctl status besu-rpc.service --no-pager
pct exec 2101 -- journalctl -u besu-rpc.service -n 40 --no-pager

If (2) shows no java.io.tmpdir in the unit, add it and reload:

pct exec 2101 -- sed -i 's|BESU_OPTS=-Xmx2g -Xms1g|BESU_OPTS=-Xmx2g -Xms1g -Djava.io.tmpdir=/data/besu/tmp|' /etc/systemd/system/besu-rpc.service
pct exec 2101 -- systemctl daemon-reload
pct exec 2101 -- systemctl restart besu-rpc.service

---

5. Recommended order of operations

1. Ensure 2101 is writable
   Run ./scripts/maintenance/make-rpc-vmids-writable-via-ssh.sh (once; we fixed the lvchange -an issue).

2. Complete the 2101 fix so install + tmpdir both run

   • Run ./scripts/maintenance/fix-rpc-2101-jna-reinstall.sh and let it run to the end (or increase STEP2_TIMEOUT in the full maintenance run).
   • If it stalls on apt again, log into the CT and run apt-get update && apt-get install -y openjdk-17-jdk wget by hand, then re-run the fix script so it does the Besu install and the post-install step that sets /data/besu/tmp and -Djava.io.tmpdir in the unit.

3. Verify
   Use the checklist in §4. Confirm besu-rpc.service contains -Djava.io.tmpdir=/data/besu/tmp, that /data/besu/tmp exists and is writable by besu, and that journalctl -u besu-rpc no longer shows JNA/NoClassDefFoundError.

---

6. References

• 502_DEEP_DIVE: 502_DEEP_DIVE_ROOT_CAUSES_AND_FIXES.md — Read-only CT, 2101 JNA fix.
• RPC troubleshooting: docs/09-troubleshooting/RPC_NODES_BLOCK_PRODUCTION_FIX.md §9 (JNA / NoClassDefFoundError).
• 2101 changes: VMID_2101_CHANGES_AND_FAILURES.md.
• Besu JVM options: https://besu.hyperledger.org/public-networks/how-to/configure-java/pass-jvm-options