Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Self hosted, Self signed SSL certificates - gRPC token refresh errors #1029

Open
grepson opened this issue Jan 6, 2025 · 7 comments
Open

Self hosted, Self signed SSL certificates - gRPC token refresh errors #1029

grepson opened this issue Jan 6, 2025 · 7 comments

Comments

@grepson
Copy link

grepson commented Jan 6, 2025

I'm having a hard time implementing a custom SSL certificates to dyrectorio. I was able to change some trafeik config in docker-compose file and also add my custom CA to the trusted ones. To bypass dagent I needed to mount volume to dagent using my script on github gist

I'm able to login, deploy, and add node but after token expiry it crashes because on start I cannot get the token fetch.

{"level":"info","time":"2025-01-06T14:26:30Z","message":"Checking key file: /srv/dagent/secret.key"}
{"level":"info","time":"2025-01-06T14:26:30Z","message":"Looking for file: /srv/dagent/token.jwt"}
{"level":"info","time":"2025-01-06T14:26:30Z","message":"Looking for file: /srv/dagent/token-nonce.blacklist"}
{"level":"info","time":"2025-01-06T14:26:30Z","message":"Configuration loaded."}
{"level":"info","time":"2025-01-06T14:26:30Z","message":"Starting dyrector.io DAgent service"}
{"level":"info","time":"2025-01-06T14:26:30Z","message":"Spinning up gRPC Agent client..."}
{"level":"info","time":"2025-01-06T14:26:30Z","message":"Retrieving certificate"}
{"level":"info","address":"dyrectorio.home.arpa:443","time":"2025-01-06T14:26:30Z","message":"Dialing to address."}
{"level":"info","hostname":"e7ee640225ec","time":"2025-01-06T14:26:30Z","message":"Getting self by hostname"}
{"level":"info","time":"2025-01-06T14:26:31Z","message":"Stream connection is up"}
{"level":"debug","time":"2025-01-06T14:26:31Z","message":"Replace token requested"}
{"level":"error","error":"rpc error: code = Internal desc = server closed the stream without sending trailers","time":"2025-01-06T14:26:31Z","message":"Failed to report token replacement, falling back to the old token"}
{"level":"error","error":"rpc error: code = Internal desc = server closed the stream without sending trailers","time":"2025-01-06T14:26:31Z","message":"Token replacement failed"}
{"level":"info","prefix":"","time":"2025-01-06T14:26:31Z","message":"Opening container status channel"}
{"level":"info","prefix":"","time":"2025-01-06T14:28:01Z","message":"Container status channel closed"}
{"level":"error","error":"rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR","time":"2025-01-06T14:41:31Z","message":"Cannot receive stream"}
{"level":"info","hostname":"e7ee640225ec","time":"2025-01-06T14:41:36Z","message":"Getting self by hostname"}
{"level":"info","time":"2025-01-06T14:41:36Z","message":"Stream connection is up"}
{"level":"error","error":"rpc error: code = Unauthenticated desc = {\"message\":\"Invalid token.\",\"details\":{}}","time":"2025-01-06T14:41:36Z","message":"Invalid install token. Blacklisting nonce"}
{"level":"info","path":"/srv/dagent/token-nonce.blacklist","time":"2025-01-06T14:41:36Z","message":"File saved"}

Where should I look to fix it? I wanna make PR later when I'll clean up the cloned repository to enable other users of dyrectorio to also deploy self signed certificates. I would appreciate if someone has already did repo with those changes.
@nandor-magyar
Copy link
Contributor

Just to double check, you would like to use system-wide CA in all containers and want your containers to trust your custom CA, right?

@grepson
Copy link
Author

grepson commented Jan 15, 2025

@nandor-magyar Yes exactly! I was able to somehow make it work by making modifications to docker-compose with trafeik ssl (the default one) plus injecting the certificate to dagent by making new dockerfile for it. But still this setup lacks grpc token refreshes (denied).

@nandor-magyar
Copy link
Contributor

nandor-magyar commented Jan 22, 2025 •
edited
Loading

This case the TLS termination is done by Traefik. It seems that is correct and agent already trusts the CA. Then the culprit should be somewhere else, did you check the logs of crux?
Note: I'll extend our docs with a custom CA section today.

https://docs.dyrector.io/self-managed/self-signed-certificates

Edit: added the link to the related docs page.

@grepson
Copy link
Author

grepson commented Jan 30, 2025

@nandor-magyar Thank you for this section! I'll check it today and if no problems occurs I'll close this issue!!

I really appreciate your work on that one <3

@grepson
Copy link
Author

grepson commented Jan 30, 2025

@nandor-magyar Unfortunately this documentation lacks a crucial amount of steps in order to make it work in 80%. Still I'm facing problem with refreshing token. No matter what I did was not able to get refresh token to work. In order to save time I'm including what I've tried to do and noticed:

The token is 0 bytes long in persistence:
-rw------- 1 root root 0 Jan 30 06:23 token.jwt

  • Tried to remove persistence folders
  • Manually add token.jwt from GRPC_TOKEN
  • Seeing if maybe I missed adding my certificate to trusted system wide (I had it installed)

To debug this issue further please can you include some branch which has entire setup for self signed certificates working as a example?

I'm willing to help resolve this issue as well as include PR for setting up self signed certificates more automatically. I've created fork and branch that has some of my changes implemented like separate docker-compose file for self signed ssl and script that automatically adds required cert to dagent at runtime: Branch

Some more logs from different attempts:

Making fresh install and trying to fetch token

infra@infra:~/dyrectorio$ docker logs dagent
{"level":"info","time":"2025-01-30T05:38:58Z","message":"Checking key file: /srv/dagent/secret.key"}
{"level":"debug","time":"2025-01-30T05:38:58Z","message":"Key file does not exist: /srv/dagent/secret.key"}
{"level":"info","time":"2025-01-30T05:38:58Z","message":"Generating new key file..."}
{"level":"info","time":"2025-01-30T05:38:58Z","message":"New key is generated and saved"}
{"level":"info","time":"2025-01-30T05:38:58Z","message":"Looking for file: /srv/dagent/token.jwt"}
{"level":"info","time":"2025-01-30T05:38:58Z","message":"Looking for file: /srv/dagent/token-nonce.blacklist"}
{"level":"info","time":"2025-01-30T05:38:58Z","message":"Configuration loaded."}
{"level":"info","time":"2025-01-30T05:38:58Z","message":"Starting dyrector.io DAgent service"}
{"level":"info","time":"2025-01-30T05:38:58Z","message":"Spinning up gRPC Agent client..."}
{"level":"info","time":"2025-01-30T05:38:58Z","message":"Retrieving certificate"}
{"level":"info","address":"dyrectorio.home.arpa:443","time":"2025-01-30T05:38:58Z","message":"Dialing to address."}
{"level":"info","hostname":"925bdf2458ac","time":"2025-01-30T05:38:58Z","message":"Getting self by hostname"}
{"level":"info","time":"2025-01-30T05:38:59Z","message":"Stream connection is up"}
{"level":"debug","time":"2025-01-30T05:38:59Z","message":"Replace token requested"}
{"level":"error","error":"rpc error: code = Internal desc = server closed the stream without sending trailers","time":"2025-01-30T05:38:59Z","message":"Failed to report token replacement, falling back to the old token"}
{"level":"error","error":"rpc error: code = Internal desc = server closed the stream without sending trailers","time":"2025-01-30T05:38:59Z","message":"Token replacement failed"}
{"level":"info","prefix":"","time":"2025-01-30T05:38:59Z","message":"Opening container status channel"}

Certificate looks valid:

openssl s_client -connect dyrectorio.home.arpa:443 -CAfile /home/infra/certificates/GHCA.crt
CONNECTED(00000003)
depth=1 CN = GHCA-CA, C = PL, O = Yeet Software INC., OU = Security Department
verify return:1
depth=0 CN = dyrectorio.home.arpa, C = PL, O = Yeet Software INC., OU = Security Department
verify return:1
---
Certificate chain
 0 s:CN = dyrectorio.home.arpa, C = PL, O = Yeet Software INC., OU = Security Department
   i:CN = GHCA-CA, C = PL, O = Yeet Software INC., OU = Security Department
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec  4 14:51:24 2024 GMT; NotAfter: Dec  4 14:51:24 2025 GMT
 1 s:CN = GHCA-CA, C = PL, O = Yeet Software INC., OU = Security Department
   i:CN = GHCA-CA, C = PL, O = Yeet Software INC., OU = Security Department
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA512
   v:NotBefore: Oct 12 21:11:28 2024 GMT; NotAfter: Oct 12 21:11:28 2026 GMT

Adding token manually and trying to execute dagent:

infra@infra:~/dyrectorio$ docker logs dagent 
{"level":"info","time":"2025-01-30T05:31:40Z","message":"Checking key file: /srv/dagent/secret.key"}
{"level":"debug","time":"2025-01-30T05:31:40Z","message":"Key file does not exist: /srv/dagent/secret.key"}
{"level":"info","time":"2025-01-30T05:31:40Z","message":"Generating new key file..."}
{"level":"info","time":"2025-01-30T05:31:40Z","message":"New key is generated and saved"}
{"level":"info","time":"2025-01-30T05:31:40Z","message":"Looking for file: /srv/dagent/token.jwt"}
{"level":"info","time":"2025-01-30T05:31:40Z","message":"Looking for file: /srv/dagent/token-nonce.blacklist"}
{"level":"info","time":"2025-01-30T05:31:40Z","message":"Looking for file: /srv/dagent/token-nonce.blacklist"}
{"level":"info","time":"2025-01-30T05:31:40Z","message":"Configuration loaded."}
{"level":"info","time":"2025-01-30T05:31:40Z","message":"Starting dyrector.io DAgent service"}
{"level":"info","time":"2025-01-30T05:31:40Z","message":"Spinning up gRPC Agent client..."}
{"level":"info","time":"2025-01-30T05:31:40Z","message":"Retrieving certificate"}
{"level":"info","address":"dyrectorio.home.arpa:443","time":"2025-01-30T05:31:40Z","message":"Dialing to address."}
{"level":"info","hostname":"28f868c447d5","time":"2025-01-30T05:31:40Z","message":"Getting self by hostname"}
{"level":"info","time":"2025-01-30T05:31:41Z","message":"Stream connection is up"}
{"level":"debug","time":"2025-01-30T05:31:41Z","message":"Replace token requested"}
{"level":"error","error":"rpc error: code = Internal desc = server closed the stream without sending trailers","time":"2025-01-30T05:31:41Z","message":"Failed to report token replacement, falling back to the old token"}
{"level":"error","error":"rpc error: code = Internal desc = server closed the stream without sending trailers","time":"2025-01-30T05:31:41Z","message":"Token replacement failed"}
{"level":"info","prefix":"","time":"2025-01-30T05:31:41Z","message":"Opening container status channel"}
infra@infra:~/dyrectorio$ docker exec dagent /agent verify-cert
{"level":"info","time":"2025-01-30T05:32:13Z","message":"Checking key file: /srv/dagent/secret.key"}
{"level":"info","time":"2025-01-30T05:32:13Z","message":"Looking for file: /srv/dagent/token.jwt"}
{"level":"info","time":"2025-01-30T05:32:13Z","message":"Looking for file: /srv/dagent/token-nonce.blacklist"}
{"level":"info","time":"2025-01-30T05:32:13Z","message":"Looking for file: /srv/dagent/token-nonce.blacklist"}
{"level":"info","time":"2025-01-30T05:32:13Z","message":"Configuration loaded."}
{"level":"info","time":"2025-01-30T05:32:14Z","message":"Starting dyrector.io DAgent service"}
{"level":"info","time":"2025-01-30T05:32:14Z","message":"Spinning up gRPC Agent client..."}
{"level":"info","time":"2025-01-30T05:32:14Z","message":"Retrieving certificate"}
{"level":"info","address":"dyrectorio.home.arpa:443","time":"2025-01-30T05:32:14Z","message":"Dialing to address."}
{"level":"info","hostname":"28f868c447d5","time":"2025-01-30T05:32:14Z","message":"Getting self by hostname"}
{"level":"info","time":"2025-01-30T05:32:14Z","message":"Stream connection is up"}
{"level":"error","error":"rpc error: code = Unauthenticated desc = {\"message\":\"Invalid token.\",\"details\":{}}","time":"2025-01-30T05:32:14Z","message":"Invalid install token. Blacklisting nonce"}
{"level":"info","path":"/srv/dagent/token-nonce.blacklist","time":"2025-01-30T05:32:14Z","message":"File saved"}
{"level":"warn","time":"2025-01-30T05:32:14Z","message":"Connection failed, trying fallback token"}
{"level":"info","time":"2025-01-30T05:32:14Z","message":"Retrieving certificate"}
{"level":"info","address":"dyrectorio.home.arpa:443","time":"2025-01-30T05:32:14Z","message":"Dialing to address."}
{"level":"info","hostname":"28f868c447d5","time":"2025-01-30T05:32:14Z","message":"Getting self by hostname"}
{"level":"info","time":"2025-01-30T05:32:14Z","message":"Stream connection is up"}
{"level":"error","error":"rpc error: code = Unauthenticated desc = {\"message\":\"Invalid token.\",\"details\":{}}","time":"2025-01-30T05:32:14Z","message":"Invalid install token. Blacklisting nonce"}
{"level":"info","path":"/srv/dagent/token-nonce.blacklist","time":"2025-01-30T05:32:14Z","message":"File saved"}
{"level":"panic","error":"server refused connection","time":"2025-01-30T05:32:14Z","message":"Connection refused with fallback token"}
panic: Connection refused with fallback token

goroutine 1 [running]:
github.com/rs/zerolog/log.Panic.(*Logger).Panic.func1({0x1108e40?, 0x0?})
	/go/pkg/mod/github.com/rs/[email protected]/log.go:376 +0x27
github.com/rs/zerolog.(*Event).msg(0xc0004b5080, {0x1108e40, 0x26})
	/go/pkg/mod/github.com/rs/[email protected]/event.go:156 +0x2b9
github.com/rs/zerolog.(*Event).Msg(...)
	/go/pkg/mod/github.com/rs/[email protected]/event.go:108
github.com/dyrector-io/dyrectorio/golang/internal/grpc.Init({0x12a0060, 0xc000370480}, 0xc0001eec60, {0x12a2dc8, 0xc0001eec00}, 0xc0000bbb10)
	/__w/dyrectorio/dyrectorio/golang/internal/grpc/grpc.go:443 +0x25e
github.com/dyrector-io/dyrectorio/golang/pkg/dagent.Serve(0xc0001eec00)
	/__w/dyrectorio/dyrectorio/golang/pkg/dagent/dagent.go:37 +0x269
main.serve(0xc00036ddc8?)
	/__w/dyrectorio/dyrectorio/golang/cmd/dagent/dagent.go:37 +0x171
github.com/urfave/cli/v2.(*Command).Run(0xc00038c840, 0xc0004dd740, {0xc000188060, 0x2, 0x2})
	/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:274 +0x93f
github.com/urfave/cli/v2.(*App).RunContext(0xc0000d4000, {0x129fea0, 0x1b4f220}, {0xc000188060, 0x2, 0x2})
	/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:332 +0x566
github.com/urfave/cli/v2.(*App).Run(...)
	/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:309
main.main()
	/__w/dyrectorio/dyrectorio/golang/cmd/dagent/dagent.go:75 +0x287
infra@infra:~/dyrectorio$

@nandor-magyar
Copy link
Contributor

Could you please attach some logs from crux when token this happens?

@grepson
Copy link
Author

grepson commented Jan 31, 2025

Sure thing, there you go! Crux logs after removing all persistence storage and revoking token:

{"level":30,"time":1738312522109,"pid":17,"hostname":"2117fc93bbbf","req":{"id":12,"method":"GET","url":"/api/dev/deployments","query":{},"params":{"0":"dev/deployments"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"node","accept":"*/*","accept-encoding":"br, gzip, deflate","accept-language":"*","sec-fetch-mode":"cors","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"589"}},"responseTime":39,"msg":"request completed"}
{"level":30,"time":1738312522160,"pid":17,"hostname":"2117fc93bbbf","req":{"id":13,"method":"POST","url":"/api/users/me","query":{},"params":{"0":"users/me"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","content-length":"0","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/deployments","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","msg":"200 POST /api/users/me"}
{"level":30,"time":1738312522161,"pid":17,"hostname":"2117fc93bbbf","req":{"id":13,"method":"POST","url":"/api/users/me","query":{},"params":{"0":"users/me"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","content-length":"0","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/deployments","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"190"}},"responseTime":13,"msg":"request completed"}
{"level":30,"time":1738312526579,"pid":17,"hostname":"2117fc93bbbf","req":{"id":14,"method":"GET","url":"/api/dev/registries","query":{},"params":{"0":"dev/registries"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"node","accept":"*/*","accept-encoding":"br, gzip, deflate","accept-language":"*","sec-fetch-mode":"cors","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","msg":"200 GET /api/dev/registries"}
{"level":30,"time":1738312526580,"pid":17,"hostname":"2117fc93bbbf","req":{"id":14,"method":"GET","url":"/api/dev/registries","query":{},"params":{"0":"dev/registries"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"node","accept":"*/*","accept-encoding":"br, gzip, deflate","accept-language":"*","sec-fetch-mode":"cors","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"1001"}},"responseTime":32,"msg":"request completed"}
{"level":30,"time":1738312526643,"pid":17,"hostname":"2117fc93bbbf","req":{"id":15,"method":"POST","url":"/api/users/me","query":{},"params":{"0":"users/me"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","content-length":"0","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/registries","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","msg":"200 POST /api/users/me"}
{"level":30,"time":1738312526643,"pid":17,"hostname":"2117fc93bbbf","req":{"id":15,"method":"POST","url":"/api/users/me","query":{},"params":{"0":"users/me"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","content-length":"0","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/registries","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"190"}},"responseTime":15,"msg":"request completed"}
{"level":30,"time":1738312527721,"pid":17,"hostname":"2117fc93bbbf","req":{"id":16,"method":"GET","url":"/api/dev/nodes","query":{},"params":{"0":"dev/nodes"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"node","accept":"*/*","accept-encoding":"br, gzip, deflate","accept-language":"*","sec-fetch-mode":"cors","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","msg":"200 GET /api/dev/nodes"}
{"level":30,"time":1738312527722,"pid":17,"hostname":"2117fc93bbbf","req":{"id":16,"method":"GET","url":"/api/dev/nodes","query":{},"params":{"0":"dev/nodes"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"node","accept":"*/*","accept-encoding":"br, gzip, deflate","accept-language":"*","sec-fetch-mode":"cors","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"188"}},"responseTime":30,"msg":"request completed"}
{"level":20,"time":1738312527790,"pid":17,"hostname":"2117fc93bbbf","context":"DyoWsAdapter","msg":"Connection authorized - ::ffff:172.18.0.5"}
{"level":30,"time":1738312527797,"pid":17,"hostname":"2117fc93bbbf","context":"DyoWsAdapter","msg":"Connected c2e511f9-6cf1-47db-ae55-5d1188aa3d7d clients: 1"}
{"level":30,"time":1738312529193,"pid":17,"hostname":"2117fc93bbbf","req":{"id":17,"method":"GET","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"node","accept":"*/*","accept-encoding":"br, gzip, deflate","accept-language":"*","sec-fetch-mode":"cors","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","msg":"200 GET /api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f"}
{"level":30,"time":1738312529194,"pid":17,"hostname":"2117fc93bbbf","req":{"id":17,"method":"GET","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"node","accept":"*/*","accept-encoding":"br, gzip, deflate","accept-language":"*","sec-fetch-mode":"cors","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"230"}},"responseTime":23,"msg":"request completed"}
{"level":30,"time":1738312529224,"pid":17,"hostname":"2117fc93bbbf","req":{"id":18,"method":"GET","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/deployments","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/deployments"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"node","accept":"*/*","accept-encoding":"br, gzip, deflate","accept-language":"*","sec-fetch-mode":"cors","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","msg":"200 GET /api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/deployments"}
{"level":30,"time":1738312529224,"pid":17,"hostname":"2117fc93bbbf","req":{"id":18,"method":"GET","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/deployments","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/deployments"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"node","accept":"*/*","accept-encoding":"br, gzip, deflate","accept-language":"*","sec-fetch-mode":"cors","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"589"}},"responseTime":22,"msg":"request completed"}
{"level":30,"time":1738312529282,"pid":17,"hostname":"2117fc93bbbf","req":{"id":19,"method":"POST","url":"/api/users/me","query":{},"params":{"0":"users/me"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","content-length":"0","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","msg":"200 POST /api/users/me"}
{"level":30,"time":1738312529282,"pid":17,"hostname":"2117fc93bbbf","req":{"id":19,"method":"POST","url":"/api/users/me","query":{},"params":{"0":"users/me"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","content-length":"0","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"190"}},"responseTime":14,"msg":"request completed"}
{"level":30,"time":1738312533275,"pid":17,"hostname":"2117fc93bbbf","req":{"id":20,"method":"DELETE","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/token","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/token"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","msg":"204 DELETE /api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/token"}
{"level":30,"time":1738312533276,"pid":17,"hostname":"2117fc93bbbf","req":{"id":20,"method":"DELETE","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/token","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/token"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":204,"headers":{"x-powered-by":"Express"}},"responseTime":46,"msg":"request completed"}
{"level":30,"time":1738312537230,"pid":17,"hostname":"2117fc93bbbf","req":{"id":21,"method":"POST","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","content-length":"73","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","content-type":"application/json","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","msg":"201 POST /api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script"}
{"level":30,"time":1738312537231,"pid":17,"hostname":"2117fc93bbbf","req":{"id":21,"method":"POST","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36","content-length":"73","accept":"*/*","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-GB,en;q=0.8","content-type":"application/json","origin":"https://dyrectorio.home.arpa","priority":"u=1, i","referer":"https://dyrectorio.home.arpa/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f","sec-ch-ua":"\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Brave\";v=\"132\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1"}},"context":"HTTP","res":{"statusCode":201,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"5421"}},"responseTime":59,"msg":"request completed"}
{"level":30,"time":1738312544952,"pid":17,"hostname":"2117fc93bbbf","req":{"id":22,"method":"GET","url":"/api/health","query":{},"params":{"0":"health"},"headers":{"host":"localhost:1848","user-agent":"Wget","connection":"close"}},"context":"HTTP","msg":"200 GET /api/health"}
{"level":30,"time":1738312544952,"pid":17,"hostname":"2117fc93bbbf","req":{"id":22,"method":"GET","url":"/api/health","query":{},"params":{"0":"health"},"headers":{"host":"localhost:1848","user-agent":"Wget","connection":"close"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"93"}},"responseTime":4,"msg":"request completed"}
{"level":30,"time":1738312575035,"pid":17,"hostname":"2117fc93bbbf","req":{"id":23,"method":"GET","url":"/api/health","query":{},"params":{"0":"health"},"headers":{"host":"localhost:1848","user-agent":"Wget","connection":"close"}},"context":"HTTP","msg":"200 GET /api/health"}
{"level":30,"time":1738312575036,"pid":17,"hostname":"2117fc93bbbf","req":{"id":23,"method":"GET","url":"/api/health","query":{},"params":{"0":"health"},"headers":{"host":"localhost:1848","user-agent":"Wget","connection":"close"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"93"}},"responseTime":5,"msg":"request completed"}
{"level":30,"time":1738312598841,"pid":17,"hostname":"2117fc93bbbf","req":{"id":24,"method":"GET","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"curl/7.88.1","accept":"*/*","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1","accept-encoding":"gzip"}},"context":"HTTP","msg":"200 GET /api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script"}
{"level":30,"time":1738312598841,"pid":17,"hostname":"2117fc93bbbf","req":{"id":24,"method":"GET","url":"/api/dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script","query":{},"params":{"0":"dev/nodes/3ba12632-a62a-44eb-9ff3-1dc195c8ee1f/script"},"headers":{"host":"dyrectorio.home.arpa","user-agent":"curl/7.88.1","accept":"*/*","x-forwarded-for":"172.18.0.1","x-forwarded-host":"dyrectorio.home.arpa","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"f6bb80d92027","x-real-ip":"172.18.0.1","accept-encoding":"gzip"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"text/plain; charset=utf-8","content-length":"4979"}},"responseTime":8,"msg":"request completed"}
{"level":30,"time":1738312605015,"pid":17,"hostname":"2117fc93bbbf","context":"AgentService","msg":"Agent joined with id: 3ba12632-a62a-44eb-9ff3-1dc195c8ee1f, version: 0.14.1-deb0354 (2025-01-22T11:19:16Z) key: true"}
{"level":20,"time":1738312605042,"pid":17,"hostname":"2117fc93bbbf","context":"NodeService","msg":"Opening container state stream for node - prefix: 3ba12632-a62a-44eb-9ff3-1dc195c8ee1f - "}
{"level":30,"time":1738312605104,"pid":17,"hostname":"2117fc93bbbf","req":{"id":25,"method":"GET","url":"/api/health","query":{},"params":{"0":"health"},"headers":{"host":"localhost:1848","user-agent":"Wget","connection":"close"}},"context":"HTTP","msg":"200 GET /api/health"}
{"level":30,"time":1738312605105,"pid":17,"hostname":"2117fc93bbbf","req":{"id":25,"method":"GET","url":"/api/health","query":{},"params":{"0":"health"},"headers":{"host":"localhost:1848","user-agent":"Wget","connection":"close"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"93"}},"responseTime":5,"msg":"request completed"}
{"level":30,"time":1738312635176,"pid":17,"hostname":"2117fc93bbbf","req":{"id":26,"method":"GET","url":"/api/health","query":{},"params":{"0":"health"},"headers":{"host":"localhost:1848","user-agent":"Wget","connection":"close"}},"context":"HTTP","msg":"200 GET /api/health"}
{"level":30,"time":1738312635176,"pid":17,"hostname":"2117fc93bbbf","req":{"id":26,"method":"GET","url":"/api/health","query":{},"params":{"0":"health"},"headers":{"host":"localhost:1848","user-agent":"Wget","connection":"close"}},"context":"HTTP","res":{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"application/json; charset=utf-8","content-length":"93"}},"responseTime":5,"msg":"request completed"}

dagent logs after that:

{"level":"info","time":"2025-01-31T08:36:42Z","message":"Retrieving certificate"}
{"level":"info","address":"dyrectorio.home.arpa:443","time":"2025-01-31T08:36:43Z","message":"Dialing to address."}
{"level":"info","hostname":"d01a6e0e1456","time":"2025-01-31T08:36:43Z","message":"Getting self by hostname"}
{"level":"info","time":"2025-01-31T08:36:44Z","message":"Stream connection is up"}
{"level":"debug","time":"2025-01-31T08:36:45Z","message":"Replace token requested"}
{"level":"error","error":"rpc error: code = Internal desc = server closed the stream without sending trailers","time":"2025-01-31T08:36:45Z","message":"Failed to report token replacement, falling back to the old token"}
{"level":"error","error":"rpc error: code = Internal desc = server closed the stream without sending trailers","time":"2025-01-31T08:36:45Z","message":"Token replacement failed"}
{"level":"info","prefix":"","time":"2025-01-31T08:36:45Z","message":"Opening container status channel"}
{"level":"info","prefix":"","time":"2025-01-31T08:38:15Z","message":"Container status channel closed"}

Just to be sure I've tried to pass .key .crt or full .pem certificate to see if the issue lies in the certiificate handling by dagent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nandor-magyar @grepson