Packaging Atuin for Void Linux

Void Linux package PR: New package: atuin-17.0.1 by tranzystorekk · Pull Request #40600 · void-linux/void-packages · GitHub

As discussed previously on Discord, I am trying to get through issues preventing Atuin from dynamically linking a system-installed libsqlite, which is an important requirement for managing a distro package.

Part of the solution has been done via chore: update to sqlx 0.7.3 by ellie · Pull Request #1416 · atuinsh/atuin · GitHub

Atuin now at least seems to successfully link sqlite on non-cross platforms (Void notoriously manages a lot of targets, including 64 and 32 bit, as well as both glibc and musl :face_with_diagonal_mouth:),

but there are database tests failing that I don’t want to immediately disable: New package: atuin-17.0.1 · void-linux/void-packages@44c71ce · GitHub

as well as failures to correctly link on some (suspiciously cross-compiled) targets: New package: atuin-17.0.1 · void-linux/void-packages@44c71ce · GitHub

That’s so weird. Are you able to run the logs with ATUIN_LOG=debug? It’s strange that they’re all because a table doesn’t exist, hopefully that will illuminate why

We established on Discord that test failure only occurs when sqlite is dynlinked, with void using 3.44.2.

I’ll see if I can replicate the same test failures outside of the Void test env

After installing SQLite 3.44.2 on my mac, and setting

export LIBSQLITE3_SYS_USE_PKG_CONFIG=1
export LDFLAGS="-L/opt/homebrew/opt/sqlite/lib"
export CPPFLAGS="-I/opt/homebrew/opt/sqlite/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/sqlite/lib/pkgconfig"

I can confirm it has dynamically linked with

🦄 otool -L target/debug/atuin | grep sqlite
	/opt/homebrew/opt/sqlite/lib/libsqlite3.0.dylib (compatibility version 9.0.0, current version 9.6.0)

And all tests are currently passing :confused:

Managed to reproduce the test failure in an atuin repository cloned into a Void Linux system with a LIBSQLITE3_SYS_USE_PKG_CONFIG=1 cargo test --release --locked -- --skip sync --skip registration.

The system has pkg-config and sqlite-devel installed.

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/release/deps/atuin_client-81fe75a31e2170bb)

running 38 tests
test encryption::test::key_encodings ... ok
test encryption::test::test_decode ... ok
test encryption::test::test_decode_deleted ... ok
test encryption::test::test_decode_old ... ok
test encryption::test::test_encrypt_decrypt ... ok
test import::bash::test::parse_no_timestamps ... ok
test import::bash::test::parse_with_partial_timestamps ... ok
test import::bash::test::parse_with_timestamps ... ok
test history::tests::disable_secrets ... ok
test import::fish::test::parse_complex ... ok
test import::zsh::test::test_parse_extended_simple ... ok
test import::zsh::test::test_parse_file ... ok
test import::zsh_histdb::test::test_env_vars ... ok
test history::tests::privacy_test ... ok
test kv::tests::encode_decode ... ok
test record::encryption::tests::cannot_decrypt_different_id ... ok
test record::encryption::tests::cannot_decrypt_different_key ... ok
test record::encryption::tests::full_record_round_trip ... ok
test record::encryption::tests::full_record_round_trip_fail ... ok
test record::encryption::tests::re_encrypt_round_trip ... ok
test record::encryption::tests::round_trip ... ok
test record::encryption::tests::same_entry_different_output ... ok
test kv::tests::build_kv ... FAILED
test import::zsh_histdb::test::test_import ... FAILED
test database::test::test_search_fuzzy ... FAILED
test record::sqlite_store::tests::create_db ... ok
test database::test::test_search_fulltext ... FAILED
test record::sqlite_store::tests::append_a_bunch ... FAILED
test database::test::test_search_bench_dupes ... FAILED
test database::test::test_search_reordered_fuzzy ... FAILED
test database::test::test_search_prefix ... FAILED
test secrets::tests::test_secrets ... ok
test record::sqlite_store::tests::get_record ... FAILED
test record::sqlite_store::tests::len ... FAILED
test record::sqlite_store::tests::push_record ... FAILED
test record::sqlite_store::tests::len_different_tags ... FAILED
test record::sqlite_store::tests::test_chain ... FAILED
test record::sqlite_store::tests::append_a_big_bunch ... FAILED

failures:

---- kv::tests::build_kv stdout ----
thread 'kv::tests::build_kv' panicked at atuin-client/src/kv.rs:255:14:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:173:19

---- import::zsh_histdb::test::test_import stdout ----
thread 'import::zsh_histdb::test::test_import' panicked at atuin-client/src/import/zsh_histdb.rs:221:56:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: history

Caused by:
    (code: 1) no such table: history

Location:
    atuin-client/src/import/zsh_histdb.rs:103:40
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- database::test::test_search_fuzzy stdout ----
thread 'database::test::test_search_fuzzy' panicked at atuin-client/src/database.rs:674:59:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- database::test::test_search_fulltext stdout ----
thread 'database::test::test_search_fulltext' panicked at atuin-client/src/database.rs:658:59:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- record::sqlite_store::tests::append_a_bunch stdout ----
thread 'record::sqlite_store::tests::append_a_bunch' panicked at atuin-client/src/record/sqlite_store.rs:305:30:
failed to push record: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- database::test::test_search_bench_dupes stdout ----
thread 'database::test::test_search_bench_dupes' panicked at atuin-client/src/database.rs:801:18:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- database::test::test_search_reordered_fuzzy stdout ----
thread 'database::test::test_search_reordered_fuzzy' panicked at atuin-client/src/database.rs:766:49:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- database::test::test_search_prefix stdout ----
thread 'database::test::test_search_prefix' panicked at atuin-client/src/database.rs:642:59:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- record::sqlite_store::tests::get_record stdout ----
thread 'record::sqlite_store::tests::get_record' panicked at atuin-client/src/record/sqlite_store.rs:259:32:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::len stdout ----
thread 'record::sqlite_store::tests::len' panicked at atuin-client/src/record/sqlite_store.rs:270:32:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::push_record stdout ----
thread 'record::sqlite_store::tests::push_record' panicked at atuin-client/src/record/sqlite_store.rs:252:32:
failed to insert record: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::len_different_tags stdout ----
thread 'record::sqlite_store::tests::len_different_tags' panicked at atuin-client/src/record/sqlite_store.rs:290:31:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::test_chain stdout ----
thread 'record::sqlite_store::tests::test_chain' panicked at atuin-client/src/record/sqlite_store.rs:358:45:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::append_a_big_bunch stdout ----
thread 'record::sqlite_store::tests::append_a_big_bunch' panicked at atuin-client/src/record/sqlite_store.rs:335:45:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9


failures:
    database::test::test_search_bench_dupes
    database::test::test_search_fulltext
    database::test::test_search_fuzzy
    database::test::test_search_prefix
    database::test::test_search_reordered_fuzzy
    import::zsh_histdb::test::test_import
    kv::tests::build_kv
    record::sqlite_store::tests::append_a_big_bunch
    record::sqlite_store::tests::append_a_bunch
    record::sqlite_store::tests::get_record
    record::sqlite_store::tests::len
    record::sqlite_store::tests::len_different_tags
    record::sqlite_store::tests::push_record
    record::sqlite_store::tests::test_chain

test result: FAILED. 24 passed; 14 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.15s

error: test failed, to rerun pass `-p atuin-client --lib`

I haven’t been able to get that command to fail on mac, or on Ubuntu server.

Do you think you’d be able to build a Void docker image that demonstrates the failure? I’m not sure what Void could be doing that’s causing this issue

With the following Dockerfile atuin can be cloned and the repro command should fail:

FROM ghcr.io/void-linux/void-glibc

RUN xbps-install -Suy xbps
RUN xbps-install -Suy git bash cargo sqlite-devel

The culprit seems to be the SQLITE_USE_URI option. It’s enabled on the bundled rusqlite build, but currently disabled in Void’s sqlite package.

Uniform Resource Identifiers the sqlite docs state that the setting can be overriden in runtime, but in atuin’s case sqlx doesn’t seem to provide any API to set the SQLITE_OPEN_URI flag, even though rusqlite underneath does.

Looks like it automatically enables the flag

Maybe changing the test to be sqlite::memory: would sort it out?

docs: SqliteConnectOptions in sqlx::sqlite - Rust

The tests already seem to be sqlite::memory: unless I’m understanding sth wrong?

Looking closely at the surrounding code, sqlx seems to only use SQLITE_OPEN_URI to set the vfs or immutable params if they were enabled by the corresponding SqliteConnectOptions methods.

Yep you’re right

Options seem to be

  1. Get a change into sqlx allowing us to enable that flag
  2. Void enable SQLITE_OPEN_URI
  3. rewrite atuin to use rusqlite

I’d prefer (2), and would also like to know how the void sqlite is configured to build. In future we may use a bunch more sqlite features and need to be able to ensure void hasn’t built sqlite in a way that it will break.

(2) may even be enabled by default in the future by sqlite

URI filename interpretation is turned off by default, but future releases of SQLite might enable URI filename interpretation by default. See “URI filenames” for additional information.

The set of features currently looks like this: void-packages/srcpkgs/sqlite/template at a0d0ef829ba9aac93dbfa2eda8bf802ebada30f8 · void-linux/void-packages · GitHub

There weren’t any objections when I discussed setting URI support by default, so we’ll probably go with that

Fantastic! :raised_hands: Let me know if there’s anything else we need to change

:tada::tada::tada:

1 Like

Woooo! :partying_face: :partying_face: :partying_face:

Thank you so much for the perseverance on this one, awesome to get Atuin on Void :raised_hands:

3 Likes

Atuin is now proudly packaged in Void, thank you for all the support and discussion, I think we can close this topic

2 Likes

Amazing news <3 Thanks again!