Failed to renew Let's Encrypt certificate

[spyjoshx]

I just got an email a few days ago from Let's Encrypt that my certificate is expiring, so I tried forcing a renewal on the admin dashboard. It failed with this error: "sandcats request failed: 403: {"error":"Not authorized."} [couldnt_renew_cert]"
I'm also including the log.
sandstorm log.txt

[ocdtrekkie]

I'm wondering if this could be something specific with the Sandcats server, but I guess the first question would be whether you've changed anything with your Sandstorm server in the past 90 days or so.

[spyjoshx]

Well, The only thing that I've changed would be the hardware. I had been attempting to move it to a new server, but that ended up not working (I was trying for a while and asking for help on IRC), but I ended up just moving the old hard drive into a different server and splitting off my sandstorm from everything else. Could that have something to do with it even though nothing changed on the disk?

[ocdtrekkie]

I am not sure. I am pretty sure it is just the file in the folder that keeps a token for Sandcats, but I am not positive. I can probably try to test fetching a Sandcats certificate myself later tonight to verify if the server is working correctly.

[ocdtrekkie]

I can confirm the process works normally still as of right now. I was able to fetch my new certificate early and I am also using Sandcats. So it definitely seems like the issue is with your server not having the correct information to send the request.

I feel like if you moved the drive over as-is, the permissions should be right, but I'm trying to imagine a failure mode here for this. You should have a handful of files with names including "id_rsa" in /opt/sandstorm/var/sandcats, correct? (Do NOT share these files, they contain private keys.)

As a reference for myself for later, I am reading some of the following:
https://github.com/sandstorm-io/sandstorm/blob/master/install.sh#L1703 which is where the install script code for recovering a Sandcats domain is, it's pretty much all simple curl commands it's doing.
https://github.com/sandstorm-io/sandstorm/blob/master/acme-dns-01-sandcats/index.js#L86 is the function containing the error you're getting, which I am guessing is because it does not like your existing keys.

[spyjoshx]

I don't have time to do any digging right now in my files, but I'm just spitballing here: could trying (and failing) to start Sandstorm on another system have changed something on Sandcats' side or Let's Encrypt's side that my current installation doesn't know about?

[ocdtrekkie]

I don't feel like it should. But I feel like if you're concerned about that we could mimic the installation script's token recovery process using curl.

I'd be happy to try to set up some time when you do have time for a call and maybe a screenshare? I could compare with my own setup where necessary.

[kentonv]

If you set up a new server and associated it with your Sandcats subdomain ("recovering" your domain in the installer, where it emails you a token), then that will mean the old server's credentials have been invalidated. That's probably what happened here.

To fix this, you'll need to copy the contents of /opt/sandstorm/var/sandcats from your new server back to your old one (making sure to get the permissions set right), and then restart the old one. If you deleted this from the new server, you may need to set up another new server and do the recovery flow again, in order to get these files which you can then copy back to the old server.

[frigginglorious]

Wow, am having problems getting SSL back up and running too!

Tho in my case, it seems to be hitting the letsencrypt service properly, just... doesn't want to apply correctly?

Log says:

Certificate was successfully renewed!
Exception in changedobserve callback: Error: Unknown id for changed: tlsKeys
    at applyChange.changed (packages/minimongo/local_collection.js:725:15)
    at runWithEnvironment (packages/meteor.js:1286:24)
    at packages/meteor.js:1299:14
    at packages/mongo/observe_multiplex.js:178:30
    at Array.forEach (<anonymous>)
    at Function._.each._.forEach (packages/underscore.js:139:11)
    at Object.task (packages/mongo/observe_multiplex.js:172:9)
    at Meteor._SynchronousQueue.SQp._run (packages/meteor.js:917:16)
    at packages/meteor.js:894:12
sandstorm/gateway.c++:966: info: Loading TLS key into Gateway

And the admin panel doesn't know about the cert.

tlsfail.log

[ocdtrekkie]

@frigginglorious That's very strange. I assume you had no recent changes to your network or setup? I would say maybe reboot your machine and retry in a few hours? Maybe a service issue with LE?

[frigginglorious]

Thank you @ocdtrekkie sir. I have restarted it, https still down. Went to renew the cert from the admin panel. Seems successful, but https still doesn't work.

I probably won't get a chance to really dig into this til the new year. But I'll report back when I figure it out!

[ocdtrekkie]

I'm really pondering, I bet the valid cert is in your Sandstorm server's files somewhere, since the error appears to just be loading it into the database. I'm wondering if so if we can use the manual import command to apply it manually.

[frigginglorious]

That's where I was planning on digging. The stuff I need SSL for ain't mission critical. Cheers!

…

On Sun, Nov 12, 2023, 4:58 PM Jacob Weisz ***@***.***> wrote: I'm really pondering, I bet the valid cert is in your Sandstorm server's files somewhere, since the error appears to just be loading it into the database. I'm wondering if so if we can use the manual import command to apply it manually. — Reply to this email directly, view it on GitHub <#3701 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6MG6FLRL6N3DV5TP3FX5DYEFIDDAVCNFSM6AAAAAA4PHNSUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBXGI3TANZXGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>