Recovering a Suspended Hermes Telegram Gateway

Symptoms

• Hermes stopped responding in Telegram.
• hermes gateway status was misleading at first because systemd was repeatedly attempting to start a replacement process.
• Telegram configuration and network connectivity were not the root cause.

Evidence

The process list showed two important details:

ps -eo pid,ppid,stat,etime,%cpu,%mem,cmd --sort=-%cpu \
  | grep -Ei '[h]ermes|[t]elegram|[b]ot'

The manually launched gateway had process state T:

PID 4540 ... STAT Tl ... python -m hermes_cli.main gateway run

T means stopped by job control, commonly after Ctrl+Z.

The service journal showed why recovery did not happen automatically:

journalctl --user -u hermes-gateway.service -n 100 \
  --no-pager -o short-iso

Gateway already running (PID 4540).
hermes-gateway.service: Failed with result 'exit-code'.

The suspended process was doing no work but still owned Hermes’s PID/lock.

Root cause

A gateway started manually from a terminal was suspended instead of terminated. The enabled systemd user service then entered a restart loop because Hermes believed the suspended gateway was still running.

Recovery

First identify the current stale PID. Do not reuse the example PID blindly.

ps -eo pid,ppid,sid,pgid,tpgid,tty,stat,cmd \
  | grep -E '[h]ermes.*gateway'

Stop the managed service while removing the conflict:

systemctl --user stop hermes-gateway.service

Ask only the confirmed stale process to terminate. A stopped process may need CONT to handle its pending TERM signal:

kill -TERM STALE_PID
kill -CONT STALE_PID
kill -0 STALE_PID 2>/dev/null \
  && echo "still present" \
  || echo "stopped"

If the exact stale process remains after several seconds:

kill -KILL STALE_PID

Start Hermes through its supported service:

hermes gateway start
hermes gateway status

Verification

Inspect the sanitized state:

jq '{pid,gateway_state,active_agents,platforms,updated_at}' \
  ~/.hermes/gateway_state.json

Expected result:

gateway_state: running
active_agents: 0
telegram.state: connected
telegram.error_code: null

Confirm that Telegram polling and the heartbeat are current:

grep -E 'polling confirmed healthy|Connected to Telegram|telegram connected' \
  ~/.hermes/logs/gateway.log | tail

stat -c '%y %n' ~/.hermes/state/gateway.heartbeat

Prevention

Use the managed service commands:

hermes gateway status
hermes gateway restart
hermes gateway stop
hermes gateway start

Do not use Ctrl+Z to stop a foreground gateway. It suspends the process and leaves its PID lock in place. Use Ctrl+C for a foreground process, then verify that it exited.

This system already had the user service enabled and systemd linger enabled, so the managed gateway survives SSH logout.