Understanding atuin history search

offby1 · June 25, 2024, 8:43pm

I’d like to understand what atuin search is doing, because it’s definitely not doing what I expect.

I have a lot of curl commands in my history. If I pull up atuin’s history TUI and type curl I get the most recent curl command. So far so good. However, suppose I want to find the last time I curl’d my self-hosted atuin endpoint.

I type curl atuin in the search history, and what I see has no apparent connection to it:

  4 328ms 5mo ago curl -v https://api.github.com/repos/offbyone/ideas/commits/main
  3 6s    5mo ago be rubocop --autocorrect-all --only Lint/ParenthesesAsGroupedExpression
  2 23ms  5mo ago export bash <(curl --proto '=https' --tlsv1.2 -sSf https://setup.atuin.sh)
  1 5s    5mo ago be rubocop --autocorrect --only Layout/ArgumentAlignment
  > 519ms 3mo ago curl http://shop-data-s-4vcpu-16gb-320gb-intel-nyc3-01.tail9a8e5.ts.net:9242/
 [    GLOBAL    ] curl atuin

I expect to see curl https://atuin.camel-tortoise.ts.net/ there – it’s the history entry where curl and atuin occur closest to the start of the string and to each other.

For comparison, if I dump out history into fzf (atuin history list --print0 --cmd-only | fzf --read0 --no-mouse):

  curl -H "Host: atuin" http://localhost:3123/                                                                                               │
  curl -H 'Host: atuin' 100.87.119.29:80                                                                                                     │
  curl atuin.camel-tortoise.ts.net:80                                                                                                        │
  curl atuin.camel-tortoise.ts.net                                                                                                           │
> curl atuin                                                                                                                                 │
  1008/16815 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
> curl atuin

That gets me much better results.

Is there an atuin configuration I can tweak to improve these? Am I misusing search somehow?

ellie · June 25, 2024, 9:02pm

Thanks for bringing this up! It’s currently sorting by minspan, which can sometimes feel wrong depending on the query. I’ve been wanting to improve on it for a while, but doing so without regressions, including performance impact is harder than I’d like.

It’ll probably be because

curl http://shop-data-s-4vcpu-16gb-320gb-intel-nyc3-0b

isn’t actually that bad by that scoring, but obviously looks wrong to a user. We’re not doing any tokenisation either, so cannot prioritise terms properly, which makes that harder.

First, our minspan sorting is here;

github.com

atuinsh/atuin/blob/fd3aca7cb3394d895d27e8cea26a02948b06d6f1/crates/atuin-client/src/ordering.rs

use minspan::minspan;

use super::{history::History, settings::SearchMode};

pub fn reorder_fuzzy(mode: SearchMode, query: &str, res: Vec<History>) -> Vec<History> {
    match mode {
        SearchMode::Fuzzy => reorder(query, |x| &x.command, res),
        _ => res,
    }
}

fn reorder<F, A>(query: &str, f: F, res: Vec<A>) -> Vec<A>
where
    F: Fn(&A) -> &String,
    A: Clone,
{
    let mut r = res.clone();
    let qvec = &query.chars().collect();
    r.sort_by_cached_key(|h| {
        // TODO for fzf search we should sum up scores for each matched term

This file has been truncated. show original

We’ve also experimented with Skim (similar to fzf, but usable as a library). The issue there is that the startup time is noticeable to users, as we need to read and index all shell history (same issues with atuin list | fzf)

I’d like to make some changes really, but doing so without a set of examples of what most people think feels good is tricky. There’s an initial attempt at that hiding behind

smart_sort = true

In config. Implementation here:

github.com

atuinsh/atuin/blob/main/crates/atuin/src/command/client/search/sort.rs

use atuin_client::history::History;

type ScoredHistory = (f64, History);

// Fuzzy search already comes sorted by minspan
// This sorting should be applicable to all search modes, and solve the more "obvious" issues
// first.
// Later on, we can pass in context and do some boosts there too.
pub fn sort(query: &str, input: Vec<History>) -> Vec<History> {
    // This can totally be extended. We need to be _careful_ that it's not slow.
    // We also need to balance sorting db-side with sorting here. SQLite can do a lot,
    // but some things are just much easier/more doable in Rust.

    let mut scored = input
        .into_iter()
        .map(|h| {
            // If history is _prefixed_ with the query, score it more highly
            let score = if h.command.starts_with(query) {
                2.0
            } else if h.command.contains(query) {

This file has been truncated. show original

It’ll basically prefer newer commands, and commands that more literally contain the query. If you give it a go and lmk how it works for you, that would be great!

The other limitations are because we’re trying to approximate fzf in sql, which doesn’t work as well as I’d like (implementation here)

Otherwise, some other search modes may feel better to you: Config | Atuin Docs

Maybe more detail than you wanted, but I’m very happy to discuss further

offby1 · June 25, 2024, 9:08pm

There’s no such thing as too much detail when it comes to digging into features like this one; the scoring of my result was exactly what I expected, I just didn’t want to assume a cause without asking.

(You may recall I introduced --print0 to the history command… this is a large part of why )

I’ll give smart search a try and see how I feel about it; I’ve been trying to live with atuin’s built in UI for a while, because it’s got more useful info than the fzf one ever will, but it’s been a challenge with anything other than the simplest search terms.

Thanks!

offby1 · June 25, 2024, 9:10pm

Also, skim gets me a lot closer; thanks for the tip!

ellie · June 26, 2024, 11:43am

Ah yes! I thought so, totally makes sense

Did it work any better for you?

No worries!