Packaging Atuin for Void Linux

tranzystorekk · November 29, 2023, 10:01am

Void Linux package PR: New package: atuin-17.0.1 by tranzystorekk · Pull Request #40600 · void-linux/void-packages · GitHub

As discussed previously on Discord, I am trying to get through issues preventing Atuin from dynamically linking a system-installed libsqlite, which is an important requirement for managing a distro package.

tranzystorekk · November 29, 2023, 10:07am

Part of the solution has been done via chore: update to sqlx 0.7.3 by ellie · Pull Request #1416 · atuinsh/atuin · GitHub

Atuin now at least seems to successfully link sqlite on non-cross platforms (Void notoriously manages a lot of targets, including 64 and 32 bit, as well as both glibc and musl ),

but there are database tests failing that I don’t want to immediately disable: New package: atuin-17.0.1 · void-linux/void-packages@44c71ce · GitHub

as well as failures to correctly link on some (suspiciously cross-compiled) targets: New package: atuin-17.0.1 · void-linux/void-packages@44c71ce · GitHub

ellie · November 29, 2023, 8:19pm

That’s so weird. Are you able to run the logs with ATUIN_LOG=debug? It’s strange that they’re all because a table doesn’t exist, hopefully that will illuminate why

ellie · December 2, 2023, 9:05am

We established on Discord that test failure only occurs when sqlite is dynlinked, with void using 3.44.2.

I’ll see if I can replicate the same test failures outside of the Void test env

ellie · December 2, 2023, 9:31am

After installing SQLite 3.44.2 on my mac, and setting

export LIBSQLITE3_SYS_USE_PKG_CONFIG=1
export LDFLAGS="-L/opt/homebrew/opt/sqlite/lib"
export CPPFLAGS="-I/opt/homebrew/opt/sqlite/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/sqlite/lib/pkgconfig"

I can confirm it has dynamically linked with

🦄 otool -L target/debug/atuin | grep sqlite
	/opt/homebrew/opt/sqlite/lib/libsqlite3.0.dylib (compatibility version 9.0.0, current version 9.6.0)

And all tests are currently passing

tranzystorekk · December 3, 2023, 3:48pm

Managed to reproduce the test failure in an atuin repository cloned into a Void Linux system with a LIBSQLITE3_SYS_USE_PKG_CONFIG=1 cargo test --release --locked -- --skip sync --skip registration.

The system has pkg-config and sqlite-devel installed.

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/release/deps/atuin_client-81fe75a31e2170bb)

running 38 tests
test encryption::test::key_encodings ... ok
test encryption::test::test_decode ... ok
test encryption::test::test_decode_deleted ... ok
test encryption::test::test_decode_old ... ok
test encryption::test::test_encrypt_decrypt ... ok
test import::bash::test::parse_no_timestamps ... ok
test import::bash::test::parse_with_partial_timestamps ... ok
test import::bash::test::parse_with_timestamps ... ok
test history::tests::disable_secrets ... ok
test import::fish::test::parse_complex ... ok
test import::zsh::test::test_parse_extended_simple ... ok
test import::zsh::test::test_parse_file ... ok
test import::zsh_histdb::test::test_env_vars ... ok
test history::tests::privacy_test ... ok
test kv::tests::encode_decode ... ok
test record::encryption::tests::cannot_decrypt_different_id ... ok
test record::encryption::tests::cannot_decrypt_different_key ... ok
test record::encryption::tests::full_record_round_trip ... ok
test record::encryption::tests::full_record_round_trip_fail ... ok
test record::encryption::tests::re_encrypt_round_trip ... ok
test record::encryption::tests::round_trip ... ok
test record::encryption::tests::same_entry_different_output ... ok
test kv::tests::build_kv ... FAILED
test import::zsh_histdb::test::test_import ... FAILED
test database::test::test_search_fuzzy ... FAILED
test record::sqlite_store::tests::create_db ... ok
test database::test::test_search_fulltext ... FAILED
test record::sqlite_store::tests::append_a_bunch ... FAILED
test database::test::test_search_bench_dupes ... FAILED
test database::test::test_search_reordered_fuzzy ... FAILED
test database::test::test_search_prefix ... FAILED
test secrets::tests::test_secrets ... ok
test record::sqlite_store::tests::get_record ... FAILED
test record::sqlite_store::tests::len ... FAILED
test record::sqlite_store::tests::push_record ... FAILED
test record::sqlite_store::tests::len_different_tags ... FAILED
test record::sqlite_store::tests::test_chain ... FAILED
test record::sqlite_store::tests::append_a_big_bunch ... FAILED

failures:

---- kv::tests::build_kv stdout ----
thread 'kv::tests::build_kv' panicked at atuin-client/src/kv.rs:255:14:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:173:19

---- import::zsh_histdb::test::test_import stdout ----
thread 'import::zsh_histdb::test::test_import' panicked at atuin-client/src/import/zsh_histdb.rs:221:56:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: history

Caused by:
    (code: 1) no such table: history

Location:
    atuin-client/src/import/zsh_histdb.rs:103:40
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- database::test::test_search_fuzzy stdout ----
thread 'database::test::test_search_fuzzy' panicked at atuin-client/src/database.rs:674:59:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- database::test::test_search_fulltext stdout ----
thread 'database::test::test_search_fulltext' panicked at atuin-client/src/database.rs:658:59:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- record::sqlite_store::tests::append_a_bunch stdout ----
thread 'record::sqlite_store::tests::append_a_bunch' panicked at atuin-client/src/record/sqlite_store.rs:305:30:
failed to push record: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- database::test::test_search_bench_dupes stdout ----
thread 'database::test::test_search_bench_dupes' panicked at atuin-client/src/database.rs:801:18:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- database::test::test_search_reordered_fuzzy stdout ----
thread 'database::test::test_search_reordered_fuzzy' panicked at atuin-client/src/database.rs:766:49:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- database::test::test_search_prefix stdout ----
thread 'database::test::test_search_prefix' panicked at atuin-client/src/database.rs:642:59:
called `Result::unwrap()` on an `Err` value: Database(SqliteError { code: 1, message: "no such table: history" })

---- record::sqlite_store::tests::get_record stdout ----
thread 'record::sqlite_store::tests::get_record' panicked at atuin-client/src/record/sqlite_store.rs:259:32:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::len stdout ----
thread 'record::sqlite_store::tests::len' panicked at atuin-client/src/record/sqlite_store.rs:270:32:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::push_record stdout ----
thread 'record::sqlite_store::tests::push_record' panicked at atuin-client/src/record/sqlite_store.rs:252:32:
failed to insert record: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::len_different_tags stdout ----
thread 'record::sqlite_store::tests::len_different_tags' panicked at atuin-client/src/record/sqlite_store.rs:290:31:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::test_chain stdout ----
thread 'record::sqlite_store::tests::test_chain' panicked at atuin-client/src/record/sqlite_store.rs:358:45:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9

---- record::sqlite_store::tests::append_a_big_bunch stdout ----
thread 'record::sqlite_store::tests::append_a_big_bunch' panicked at atuin-client/src/record/sqlite_store.rs:335:45:
called `Result::unwrap()` on an `Err` value: error returned from database: (code: 1) no such table: records

Caused by:
    (code: 1) no such table: records

Location:
    atuin-client/src/record/sqlite_store.rs:63:9


failures:
    database::test::test_search_bench_dupes
    database::test::test_search_fulltext
    database::test::test_search_fuzzy
    database::test::test_search_prefix
    database::test::test_search_reordered_fuzzy
    import::zsh_histdb::test::test_import
    kv::tests::build_kv
    record::sqlite_store::tests::append_a_big_bunch
    record::sqlite_store::tests::append_a_bunch
    record::sqlite_store::tests::get_record
    record::sqlite_store::tests::len
    record::sqlite_store::tests::len_different_tags
    record::sqlite_store::tests::push_record
    record::sqlite_store::tests::test_chain

test result: FAILED. 24 passed; 14 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.15s

error: test failed, to rerun pass `-p atuin-client --lib`

ellie · December 12, 2023, 9:37am

I haven’t been able to get that command to fail on mac, or on Ubuntu server.

Do you think you’d be able to build a Void docker image that demonstrates the failure? I’m not sure what Void could be doing that’s causing this issue

tranzystorekk · December 12, 2023, 10:58am

With the following Dockerfile atuin can be cloned and the repro command should fail:

FROM ghcr.io/void-linux/void-glibc

RUN xbps-install -Suy xbps
RUN xbps-install -Suy git bash cargo sqlite-devel

tranzystorekk · December 12, 2023, 2:57pm

The culprit seems to be the SQLITE_USE_URI option. It’s enabled on the bundled rusqlite build, but currently disabled in Void’s sqlite package.

Uniform Resource Identifiers the sqlite docs state that the setting can be overriden in runtime, but in atuin’s case sqlx doesn’t seem to provide any API to set the SQLITE_OPEN_URI flag, even though rusqlite underneath does.

ellie · December 12, 2023, 4:07pm

Looks like it automatically enables the flag

github.com

launchbadge/sqlx/blob/929af41745a9434ae83417dcf2571685cecca6f0/sqlx-sqlite/src/connection/establish.rs#L105-L111


      
          if !query_params.is_empty() {
              filename = format!(
                  "file:{}?{}",
                  urlencoding::encode(&filename),
                  query_params.join("&")
              );
              flags |= libsqlite3_sys::SQLITE_OPEN_URI;

Maybe changing the test to be sqlite::memory: would sort it out?

docs: SqliteConnectOptions in sqlx::sqlite - Rust

tranzystorekk · December 12, 2023, 4:53pm

The tests already seem to be sqlite::memory: unless I’m understanding sth wrong?

tranzystorekk · December 12, 2023, 6:34pm

Looking closely at the surrounding code, sqlx seems to only use SQLITE_OPEN_URI to set the vfs or immutable params if they were enabled by the corresponding SqliteConnectOptions methods.

ellie · December 13, 2023, 8:35am

Yep you’re right

Options seem to be

Get a change into sqlx allowing us to enable that flag
Void enable SQLITE_OPEN_URI
rewrite atuin to use rusqlite

I’d prefer (2), and would also like to know how the void sqlite is configured to build. In future we may use a bunch more sqlite features and need to be able to ensure void hasn’t built sqlite in a way that it will break.

(2) may even be enabled by default in the future by sqlite

URI filename interpretation is turned off by default, but future releases of SQLite might enable URI filename interpretation by default. See “URI filenames” for additional information.

tranzystorekk · December 13, 2023, 9:10am

The set of features currently looks like this: void-packages/srcpkgs/sqlite/template at a0d0ef829ba9aac93dbfa2eda8bf802ebada30f8 · void-linux/void-packages · GitHub

There weren’t any objections when I discussed setting URI support by default, so we’ll probably go with that

ellie · December 13, 2023, 9:20am

Fantastic! Let me know if there’s anything else we need to change

tranzystorekk · December 13, 2023, 11:06am

ellie · December 13, 2023, 11:56am

Woooo!

Thank you so much for the perseverance on this one, awesome to get Atuin on Void

tranzystorekk · December 14, 2023, 1:46pm

Atuin is now proudly packaged in Void, thank you for all the support and discussion, I think we can close this topic

ellie · December 15, 2023, 7:06pm

Amazing news <3 Thanks again!