Cannot reach atuin server via nginx reverse proxy

Help
1

Hi all, can anyone help me debug / find a reason why can’t I reach self-hosted atuin server deployed via docker compose on a VPS behind nginx’s reverse proxy? I’ve spent a couple of days on troubleshooting this and I’m out of ideas.

Problem:
curl or atuin register commands time out with 504 code

What works?

  • starting atuin server and connecting to postgres db
  • connecting to atuin server directly via IP over HTTP (checked with curl)
  • other docker images with above nginx proxy config

Configs, logs and command outputs:

# /etc/nginx/sites-enabled/atuin.my-domain.com.conf
server {
        server_name atuin.my-domain.com;
        location / {
                proxy_pass http://127.0.0.1:8888;
				proxy_set_header Host $http_host;
				proxy_set_header X-Real-IP $remote_addr;
				proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
				proxy_set_header X-Forwarded-Proto $scheme;
        }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/my-domain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/my-domain.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host ~ ^[^.]+\.my-domain\.com$) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    server_name atuin.my-domain.com;
    listen 80;
    return 404; # managed by Certbot
}

# .env
ATUIN_DB_NAME=atuin
ATUIN_DB_USERNAME=atuin
ATUIN_DB_PASSWORD=db-pass
ATUIN_HOST="127.0.0.1"
ATUIN_PORT=8888
ATUIN_OPEN_REGISTRATION=true
ATUIN_DB_URI=postgres://$ATUIN_DB_USERNAME:$ATUIN_DB_PASSWORD@db/$ATUIN_DB_NAME
RUST_LOG=info,atuin_server=debug

# docker-compose.yml
services:
  atuin:
    restart: always
    image: ghcr.io/atuinsh/atuin:v18.3.0
    command: server start
    volumes:
      - "./config:/config"
    links:
      - postgresql:db
    ports:
      - "127.0.0.1:8888:$ATUIN_PORT"
    environment:
      ATUIN_HOST: $ATUIN_HOST
      ATUIN_OPEN_REGISTRATION: $ATUIN_OPEN_REGISTRATION
      ATUIN_DB_URI: postgres://$ATUIN_DB_USERNAME:$ATUIN_DB_PASSWORD@db/$ATUIN_DB_NAME
      RUST_LOG: info,atuin_server=debug
  postgresql:
    image: postgres:14
    restart: unless-stopped
    volumes: # Don't remove permanent storage for index database files!
      - "./database:/var/lib/postgresql/data/"
    environment:
      POSTGRES_USER: ${ATUIN_DB_USERNAME}
      POSTGRES_PASSWORD: ${ATUIN_DB_PASSWORD}
      POSTGRES_DB: ${ATUIN_DB_NAME}

Output for ATUIN_LOG=debug atuin register -u myuser -e atuin@my-domain.com:

# some db messages...
[2024-09-06T20:33:47Z DEBUG reqwest::connect] starting new connection: https://atuin.my-domain.com/
[2024-09-06T20:33:47Z DEBUG hyper::client::connect::dns] resolving host="atuin.my-domain.com"
[2024-09-06T20:33:47Z DEBUG hyper::client::connect::http] connecting to <vps_ip>:443
[2024-09-06T20:33:47Z DEBUG hyper::client::connect::http] connected to <vps_ip>:443
[2024-09-06T20:33:47Z DEBUG rustls::client::hs] No cached session for DnsName("atuin.my-domain.com")
[2024-09-06T20:33:47Z DEBUG rustls::client::hs] Not resuming any session
[2024-09-06T20:33:47Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_256_GCM_SHA384
[2024-09-06T20:33:47Z DEBUG rustls::client::tls13] Not resuming
[2024-09-06T20:33:47Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]
[2024-09-06T20:33:47Z DEBUG rustls::client::hs] ALPN protocol is Some(b"http/1.1")
[2024-09-06T20:33:47Z DEBUG hyper::proto::h1::io] flushed 64 bytes
[2024-09-06T20:34:47Z DEBUG hyper::proto::h1::io] parsed 5 headers
[2024-09-06T20:34:47Z DEBUG hyper::proto::h1::conn] incoming body is content-length (160 bytes)
[2024-09-06T20:34:47Z DEBUG hyper::proto::h1::conn] incoming body completed
[2024-09-06T20:34:47Z DEBUG hyper::client::pool] pooling idle connection for ("https", atuin.my-domain.com)
[2024-09-06T20:34:47Z DEBUG reqwest::connect] starting new connection: https://atuin.my-domain.com/
[2024-09-06T20:34:47Z DEBUG rustls::common_state] Sending warning alert CloseNotify
[2024-09-06T20:34:47Z DEBUG hyper::client::connect::dns] resolving host="atuin.my-domain.com"
[2024-09-06T20:34:47Z DEBUG hyper::client::connect::http] connecting to <vps_ip>:443
[2024-09-06T20:34:47Z DEBUG hyper::client::connect::http] connected to <vps_ip>:443
[2024-09-06T20:34:47Z DEBUG rustls::client::hs] No cached session for DnsName("atuin.my-domain.com")
[2024-09-06T20:34:47Z DEBUG rustls::client::hs] Not resuming any session
[2024-09-06T20:34:47Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_256_GCM_SHA384
[2024-09-06T20:34:47Z DEBUG rustls::client::tls13] Not resuming
[2024-09-06T20:34:47Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]
[2024-09-06T20:34:47Z DEBUG rustls::client::hs] ALPN protocol is Some(b"http/1.1")
[2024-09-06T20:34:47Z DEBUG hyper::proto::h1::io] flushed 207 bytes
[2024-09-06T20:35:48Z DEBUG hyper::proto::h1::io] parsed 5 headers
[2024-09-06T20:35:48Z DEBUG hyper::proto::h1::conn] incoming body is content-length (160 bytes)
[2024-09-06T20:35:48Z DEBUG hyper::proto::h1::conn] incoming body completed
[2024-09-06T20:35:48Z DEBUG hyper::client::pool] pooling idle connection for ("https", atuin.my-domain.com)
Error: error decoding response body: expected value at line 1 column 1

Caused by:
    expected value at line 1 column 1

Location:
    /home/runner/work/atuin/atuin/atuin-client/src/api_client.rs:60:21

# /var/log/nginx/access.log
89.64.74.65 - - [06/Sep/2024:20:34:41 +0000] "GET /user/myuser HTTP/1.1" 504 160 "-" "-"
89.64.74.65 - - [06/Sep/2024:20:35:41 +0000] "POST /register HTTP/1.1" 504 160 "-" "atuin/17.0.0"

# docker ps
a23c3e888ffa   ghcr.io/atuinsh/atuin:v18.3.0   "/usr/local/bin/atui…"   28 minutes ago   Up 28 minutes             127.0.0.1:8888->8888/tcp   atuin-atuin-1
079edb86e6c0   postgres:14                   "docker-entrypoint.s…"   28 minutes ago   Up 28 minutes             5432/tcp                   atuin-postgresql-1

# docker logs a23c3e888ffa
2024-09-06T21:10:35.269550Z  INFO sqlx::postgres::notice: relation "_sqlx_migrations" already exists, skipping

# docker logs 079edb86e6c0
PostgreSQL Database directory appears to contain a database; Skipping initialization

2024-09-06 21:10:35.116 UTC [1] LOG:  starting PostgreSQL 14.13 (Debian 14.13-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-09-06 21:10:35.117 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-09-06 21:10:35.117 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-09-06 21:10:35.119 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-09-06 21:10:35.124 UTC [26] LOG:  database system was shut down at 2024-09-06 21:10:30 UTC
2024-09-06 21:10:35.137 UTC [1] LOG:  database system is ready to accept connections

I’d appreciate any tips or ideas. I’m close to giving up on sync altogether.

2

Hello. Don’t Give up! I had a heck of a time getting atuin going on my VPS. I’m glad to see you’re using a subdomain that was my big hurdle figuring that one out.

I have a slash after the ip in my proxy pass directive or w/e they call the setting there.

Do you have SELinux on the host? If you have existing other subdomains working that are also in a container can you compare the dockerfiles of the two?

server {

        server_name atuin.example.com;

        ignore_invalid_headers off;
        underscores_in_headers on;

        location / {
            proxy_pass http://127.0.0.1:1983/;
            proxy_pass_request_headers      on;
        }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/atuin.example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/atuin.example.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = atuin.example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot



    server_name atuin.example.com;
    listen 80;
    return 404; # managed by Certbot
3

Thanks for encouraging words <3 So thanks to you, I’ve sat down to it once again and managed to find out the timeout was caused by ufw blocking a request to container’s IP. Why this does not happen on other docker apps I’ve deployed, I have no idea.

Now I’m stuck with 502 bad gateway though.

4

Have you tried setting the ATUIN_HOST env var to 0.0.0.0?

I run a very similar setup and it works fine, the only main difference is that envvar.

  services:
    atuin:
      restart: always
      image: ghcr.io/atuinsh/atuin:18.3.0
      command: server start
      volumes:
        - "/mnt/remotedata:/config"
      ports:
        - 8888:8888
      environment:
        ATUIN_HOST: "0.0.0.0"
        ATUIN_OPEN_REGISTRATION: "true"
        ATUIN_DB_URI: postgres://$ATUIN_DB_USERNAME:$ATUIN_DB_PASSWORD@192.168.202.192/atuin
        RUST_LOG: info,atuin_server=debug
      <<: *logging
1 Like
5

Yes, that was it. THANK YOU! I was focused on nginx so much that I’ve never thought to try that. I’m sooo happy :slight_smile: