Thanks for the suggestion

What would you suggest instead?

Install fresh tomato to this and you’ll get a much better AP with very good firewall and QOS and traffic inspection. Also good SNMP for monitoring

https://www.ebay.com/itm/256808892926

https://www.freshtomato.org/

Cloudflare tunnels are cheap(free if it’s just a couple), simple, and really great.

https://caddy.community/t/caddy-cloudflare-tunnel/15929

You’re using something in front of caddy right?

Atleast refuse basic headers and close connections

add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";

server {
    listen      80 default_server;
    listen      [::]:80 default_server;
    listen      443 default_server;
    listen      [::]:443 default_server;
    ssl_certificate certs/server.cert;
    ssl_certificate_key certs/server.key;
    server_name _;
    return      444; #CONNECTION CLOSED WITHOUT RESPONSE
}

Ports, any NAT, internal IPs. The first part of an organized attack is getting environment enumeration down. If a bad actor can map your network they can more efficiently direct their attack.

It’s a custom nginx proxy to the kube api. Too long to get into it. I was hired to move this giant cluster that started as a lab and make it production ready.

Thanks for the feedback

Coredns and an nginx reverse proxy are handling DNS, failover, and some other redirect. However it’s not ideal as it’s a custom implementation a previous engineer setup.

We’re thinking of moving to it from a custom coredns and flannel inplementation in a k3s 33 node cluster.

Tangentially what’s your opinion on Traefik?

https://traefik.io/

Well yes and no. The rate at which you get your data back is where the gotcha lies anything up to 8TB is free if you send them $280 and they’ll refund the money once they get the drive back. Anything over 8TB is where it gets pricey.

Bsckblaze doesn’t care, they know they’ll get their money when @Unforeseen@sh.itjust.works tries to get data back from backup.

Are you backing up files from the FS or sre you backing up the snapshots? I had a corrupted journal from a power outage that borked my install. Could not get to the snapshots on boot. Booted into a live disk and recovered the snapshot that way. Would’ve taken hours to restore from a standard backup, however it was minutes restoring the snapshot.

If you’re not backing up BTRFS snapshots and just backing up files you’re better off just using ext4.

https://github.com/digint/btrbk

Do you rely on snapshotting and journaling? If so backup your snapshots.

You don’t have a lot of confidence in five year olds

Explain like I’m five.

Considering it’s tailscale, one may want to have the service fail though as tailscale is sometimes not used for convenience but security concerns instead.

Delay the start of your containers with the tailscale dependency. Are you using required or depends_on in your docker-compose.yml

https://hatchjs.com/docker-compose-conditionally-start-service/

If you’re using kubernetes you can make the requirements at the pod level

https://stackoverflow.com/questions/69423932/container-initialization-order-in-pod-on-k8s

Edit: If using docker-compose.yml you can set a condition on a healthcheck

You can also specify a condition that must be met before the service is started. For example, the following configuration will start the web service only if the db service is healthy:

version: ‘3.7’
services:
web:
image: nginx
depends_on:
– db
condition: service_healthy

The service_healthy condition checks the health of the db service using the docker-compose ps command. If the db service is healthy, the web service will be started.

In my humble opinion the point of self hosting an LLM is so that the data doesn’t leave your LAN.

It’s as complex as you make it, is linux native, is scriptable, doesn’t use YAML, is native to the OS, and is free as in beer. Just like SNMP. however they’ll also get logs at a central server they can drill into if needed.

Which I believe fulfills the requirements of OPs post.

Sidenote, self-hosting is absolutely overkill just as a theory and process. I often read responses to suggestions as this or that is overkill, or complicated, or non-trivial effort.

The self hosting community is a broad spectrum of users , from those with home labs on an old dying laptop to those with a full rack setup. People have different needs and interests. Some are learning infra and devops for work or to get into a new job. Some are privacy minded. Some are trying to get the most bang for their buck. Some just want to pay for a cloud hosted solution. Some just want an automated home. Some run a home business.

Edit: to the point of your valid and helpful SNMP post, most syslog servers also will ingest and report on SNMP traffic as well. The container I linked does exactly that. If they find they want to automate processes in the future they can also trigger on the syslog stream as well. But that complexity is only there if they want it. Otherwise it’s just a stream they can parse and trigger an alert, just like SNMP. So OP could have an extensible solution if they want to expand. Also Grafana/Prometheus will take in syslog natively with a couple standard YAML configs if they choose that they want to look at that solution again in the future.

/Rant