Hi all, can anyone help me debug / find a reason why can’t I reach self-hosted atuin server deployed via docker compose on a VPS behind nginx’s reverse proxy? I’ve spent a couple of days on troubleshooting this and I’m out of ideas.
Problem:
curl or atuin register commands time out with 504 code
What works?
- starting atuin server and connecting to postgres db
- connecting to atuin server directly via IP over HTTP (checked with curl)
- other docker images with above nginx proxy config
Configs, logs and command outputs:
# /etc/nginx/sites-enabled/atuin.my-domain.com.conf
server {
server_name atuin.my-domain.com;
location / {
proxy_pass http://127.0.0.1:8888;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/my-domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/my-domain.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host ~ ^[^.]+\.my-domain\.com$) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name atuin.my-domain.com;
listen 80;
return 404; # managed by Certbot
}
# .env
ATUIN_DB_NAME=atuin
ATUIN_DB_USERNAME=atuin
ATUIN_DB_PASSWORD=db-pass
ATUIN_HOST="127.0.0.1"
ATUIN_PORT=8888
ATUIN_OPEN_REGISTRATION=true
ATUIN_DB_URI=postgres://$ATUIN_DB_USERNAME:$ATUIN_DB_PASSWORD@db/$ATUIN_DB_NAME
RUST_LOG=info,atuin_server=debug
# docker-compose.yml
services:
atuin:
restart: always
image: ghcr.io/atuinsh/atuin:v18.3.0
command: server start
volumes:
- "./config:/config"
links:
- postgresql:db
ports:
- "127.0.0.1:8888:$ATUIN_PORT"
environment:
ATUIN_HOST: $ATUIN_HOST
ATUIN_OPEN_REGISTRATION: $ATUIN_OPEN_REGISTRATION
ATUIN_DB_URI: postgres://$ATUIN_DB_USERNAME:$ATUIN_DB_PASSWORD@db/$ATUIN_DB_NAME
RUST_LOG: info,atuin_server=debug
postgresql:
image: postgres:14
restart: unless-stopped
volumes: # Don't remove permanent storage for index database files!
- "./database:/var/lib/postgresql/data/"
environment:
POSTGRES_USER: ${ATUIN_DB_USERNAME}
POSTGRES_PASSWORD: ${ATUIN_DB_PASSWORD}
POSTGRES_DB: ${ATUIN_DB_NAME}
Output for ATUIN_LOG=debug atuin register -u myuser -e atuin@my-domain.com
:
# some db messages...
[2024-09-06T20:33:47Z DEBUG reqwest::connect] starting new connection: https://atuin.my-domain.com/
[2024-09-06T20:33:47Z DEBUG hyper::client::connect::dns] resolving host="atuin.my-domain.com"
[2024-09-06T20:33:47Z DEBUG hyper::client::connect::http] connecting to <vps_ip>:443
[2024-09-06T20:33:47Z DEBUG hyper::client::connect::http] connected to <vps_ip>:443
[2024-09-06T20:33:47Z DEBUG rustls::client::hs] No cached session for DnsName("atuin.my-domain.com")
[2024-09-06T20:33:47Z DEBUG rustls::client::hs] Not resuming any session
[2024-09-06T20:33:47Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_256_GCM_SHA384
[2024-09-06T20:33:47Z DEBUG rustls::client::tls13] Not resuming
[2024-09-06T20:33:47Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]
[2024-09-06T20:33:47Z DEBUG rustls::client::hs] ALPN protocol is Some(b"http/1.1")
[2024-09-06T20:33:47Z DEBUG hyper::proto::h1::io] flushed 64 bytes
[2024-09-06T20:34:47Z DEBUG hyper::proto::h1::io] parsed 5 headers
[2024-09-06T20:34:47Z DEBUG hyper::proto::h1::conn] incoming body is content-length (160 bytes)
[2024-09-06T20:34:47Z DEBUG hyper::proto::h1::conn] incoming body completed
[2024-09-06T20:34:47Z DEBUG hyper::client::pool] pooling idle connection for ("https", atuin.my-domain.com)
[2024-09-06T20:34:47Z DEBUG reqwest::connect] starting new connection: https://atuin.my-domain.com/
[2024-09-06T20:34:47Z DEBUG rustls::common_state] Sending warning alert CloseNotify
[2024-09-06T20:34:47Z DEBUG hyper::client::connect::dns] resolving host="atuin.my-domain.com"
[2024-09-06T20:34:47Z DEBUG hyper::client::connect::http] connecting to <vps_ip>:443
[2024-09-06T20:34:47Z DEBUG hyper::client::connect::http] connected to <vps_ip>:443
[2024-09-06T20:34:47Z DEBUG rustls::client::hs] No cached session for DnsName("atuin.my-domain.com")
[2024-09-06T20:34:47Z DEBUG rustls::client::hs] Not resuming any session
[2024-09-06T20:34:47Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_256_GCM_SHA384
[2024-09-06T20:34:47Z DEBUG rustls::client::tls13] Not resuming
[2024-09-06T20:34:47Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]
[2024-09-06T20:34:47Z DEBUG rustls::client::hs] ALPN protocol is Some(b"http/1.1")
[2024-09-06T20:34:47Z DEBUG hyper::proto::h1::io] flushed 207 bytes
[2024-09-06T20:35:48Z DEBUG hyper::proto::h1::io] parsed 5 headers
[2024-09-06T20:35:48Z DEBUG hyper::proto::h1::conn] incoming body is content-length (160 bytes)
[2024-09-06T20:35:48Z DEBUG hyper::proto::h1::conn] incoming body completed
[2024-09-06T20:35:48Z DEBUG hyper::client::pool] pooling idle connection for ("https", atuin.my-domain.com)
Error: error decoding response body: expected value at line 1 column 1
Caused by:
expected value at line 1 column 1
Location:
/home/runner/work/atuin/atuin/atuin-client/src/api_client.rs:60:21
# /var/log/nginx/access.log
89.64.74.65 - - [06/Sep/2024:20:34:41 +0000] "GET /user/myuser HTTP/1.1" 504 160 "-" "-"
89.64.74.65 - - [06/Sep/2024:20:35:41 +0000] "POST /register HTTP/1.1" 504 160 "-" "atuin/17.0.0"
# docker ps
a23c3e888ffa ghcr.io/atuinsh/atuin:v18.3.0 "/usr/local/bin/atui…" 28 minutes ago Up 28 minutes 127.0.0.1:8888->8888/tcp atuin-atuin-1
079edb86e6c0 postgres:14 "docker-entrypoint.s…" 28 minutes ago Up 28 minutes 5432/tcp atuin-postgresql-1
# docker logs a23c3e888ffa
2024-09-06T21:10:35.269550Z INFO sqlx::postgres::notice: relation "_sqlx_migrations" already exists, skipping
# docker logs 079edb86e6c0
PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-09-06 21:10:35.116 UTC [1] LOG: starting PostgreSQL 14.13 (Debian 14.13-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-09-06 21:10:35.117 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2024-09-06 21:10:35.117 UTC [1] LOG: listening on IPv6 address "::", port 5432
2024-09-06 21:10:35.119 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-09-06 21:10:35.124 UTC [26] LOG: database system was shut down at 2024-09-06 21:10:30 UTC
2024-09-06 21:10:35.137 UTC [1] LOG: database system is ready to accept connections
I’d appreciate any tips or ideas. I’m close to giving up on sync altogether.