Restoring basicagent.service with Deterministic Operations

basicagent.service should never be a static artifact. It is the runtime spine behind the portal, and if it drifts, both delivery and customer conversations stall.

This article is the practical recovery and hardening pass for the common failures that keep the service from staying green.

1) What is the real failure mode?

Most service incidents are one of these:

1. basicagent.service is configured but not restarted after a content change.
2. Dependencies are healthy in isolation, but the portal cannot resolve routes or publish new content.
3. Background jobs are running while the main process is in RESTARTING churn because of a deploy-time config mismatch.
4. File changes are not synchronized to /opt/basicagent before restart.
5. DNS, port, or reverse-proxy assumptions changed (especially around service aliases and ports).

The important point is that these are not random runtime bugs. They are usually a broken change path.

2) First principles for a service recovery

When an outage appears, do not guess. Capture five artifacts in order:

systemctl status basicagent.service
systemctl show basicagent.service -p ExecStart,MainPID,ActiveState,SubState
journalctl -u basicagent.service -n 160 --no-pager
python3 tools/work_order_engine.py --help >/tmp/basicagent_work_order_help.txt
curl -sS 'http://127.0.0.1:3000/health' | head

If curl fails but logs show startup, route ownership is usually wrong:

• service points at non-existent binary path,
• wrong WorkingDirectory,
• stale environment, or
• missing --release binary refresh.

3) Canonical service contract

basicagent.service should be treated as a contract, not a hand-edited note.

Recommended unit file shape

[Unit]
Description=BasicAgent Rust Portal
After=network.target

[Service]
Type=simple
User=basicagent
WorkingDirectory=/opt/basicagent
EnvironmentFile=/root/agentpipes1/.env
EnvironmentFile=/opt/basicagent/.env
ExecStart=/opt/basicagent/scopelock_portal
Restart=always
RestartSec=3
TimeoutStartSec=30
TimeoutStopSec=30
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

You should pin a versioned binary target and avoid implicit cargo run from runtime.

cp -a /opt/basicagent/scopelock_portal /opt/basicagent/scopelock_portal.bak.$(date -u +%Y%m%dT%H%M%SZ)
install -m 755 /root/dec15/december\ 15th/portal/target/release/scopelock_portal /opt/basicagent/scopelock_portal
systemctl daemon-reload
systemctl restart basicagent.service

4) Why this service sometimes “does not publish”

In this stack, publication is a three-step behavior:

1. content markdown exists on disk
2. the portal process reads it from configured content_dir
3. route layer exposes it, homepage indexes it, and service restart reflects the new catalog

If one step fails, the page can silently stay off the front page.

common breakpoints

• content file missing expected frontmatter (title, description, slug)
• invalid date metadata and rank ordering fallback behavior
• service still serving older artifact path after deploy
• front page filtering excludes path for publish logic

5) Front-page surfacing logic you should test after every publish

Use this quick check directly after edits:

curl -sS https://basicagent.org/ | rg -n "Latest published articles|Latest published technical"
curl -sS https://basicagent.org/api/home-feed | sed -n '1,140p'
curl -sS https://basicagent.org/admin/content-audit | head -n 80

Then validate:

• basicagent-service appears in article list
• publication-ready pages indexed increases
• blocked or date-gated pages does not increase

6) Code-level hardening in the current stack

The safest changes are deterministic:

• track source file modified date for publication ranking if published/updated is missing;
• treat future published dates as hard blocks and render reason in admin audit;
• avoid manual content copies and keep one canonical source directory;
• restart service only after deployment write completes;
• keep publish artifacts in a single step with checks before restart.

That ordering matters more than any code tweak.

7) Recommended daily command loop

Run this from monitoring once per deploy day:

set -euo pipefail
python3 tools/bgmon.py status basicagent_service_watch_01 || true
systemctl is-active basicagent.service
journalctl -u basicagent.service --since "1 hour ago" --no-pager | tail -n 80
curl -sS http://127.0.0.1:3000/health
python3 tools/data_status.py --json | sed -n '1,120p'

If the site is healthy but article freshness is stale, do this:

cd '/root/dec15/december 15th/portal'
git status --short
python3 -m pip show --version >/tmp/pip-version.txt 2>/dev/null || true
ls -l content/pages | wc -l

8) Contact capture and service state

The service must not treat /contact as a dead-end form. Every lead capture should carry intent, retention stage, and source tags through to POST /contact, then to your admin/content pipeline.

This is where reliability connects to revenue:

• stable service uptime
• stable contact flow
• deterministic follow-up queues

That combination is the single highest-leverage technical posture.

9) What we changed in the portal runtime

This deployment now:

• records source file date for publish ranking when explicit dates are absent;
• keeps basicagent.service checks in an auditable checklist;
• ensures missing content can still surface by freshness once published files exist in source;
• includes this guide for operator continuity.

The stack is now less brittle because publishing does not depend on one metadata field.

If you want a stronger pass, next step is a fully scripted deploy job that gates on admin/content-audit output and blocks restart on unexpected blocked items.

Build narrative

Follow a coherent path from thesis to lab notes to proof-of-work instead of isolated pages.