Sync v2 testing

I have only one permanent VM that I keep around (with my slowly being worked on NixOS configuration, so Atuin on Nix, but built manually from the latest master). I have used Atuin in the past with about 3 other VMs when I was testing stuff.

1 Like

I have almost 150k entries in my history and the conversion with atuin history init-store is estimated to take almost 10 minutes. Can this be speedup with some parallism?

It should be sped up in the latest weekly release - Weekly release 2024.08

Mostly a SQL transaction batching issue, it was a single transaction per record beforehand :woman_facepalming:

Ellie, I’m super impressed. Atuin is one of my absolute must-have tools.

Did the migration today on 3 of my regular work machines - flawless. Love the atuin store status confirmation.

A quick question - what happens to the existing “old” blob on your end now that I’ve moved over to the new method?

I’d imagine you purge these after a time?

1 Like

Glad to hear it, thank you!

The old blobs are still hanging around, and will do for a while - until sync v2 is the default I imagine. In the next release it will be the default for new users only, then I’ll hit the switch for everyone.

Also need to figure out a neat way of slowing the rollout so I don’t end up with a lot of upload at the same time :laughing:

Hi everyone

I noticed that although everything was getting stored remotely correctly, the up arrow wasn’t showing the command from both computers as it used to.

It used to work fine, I found this thread and made sure I followed the steps.

Here are the outputs

❯ atuin store status
host: 018b2c2b-7664-7a06-8b36-3bcf9a1c84c9
        store: history
                idx: 891
                first: 018e6798-c1f7-7693-b7d0-f6a6aa22c12d
                        created: 2024-03-22 19:17:28.183391732 +00:00:00
                last: 018fa121-442f-7c06-8d6d-a2db466b6393
                        created: 2024-05-22 16:27:43.023387245 +00:00:00

host: 018b2c28-45bc-7c6d-9b71-b28588746193 <- CURRENT HOST
        store: history
                idx: 14195
                first: 018e2ea9-bab2-7e02-87ab-2c3d94e4f847
                        created: 2024-03-11 17:57:39.122259192 +00:00:00
                last: 018fa121-defe-71f8-9615-010b479c495d
                        created: 2024-05-22 16:28:22.654603473 +00:00:00
atuin store status
host: 018b2c2b-7664-7a06-8b36-3bcf9a1c84c9 <- CURRENT HOST
        store: history
                idx: 893
                first: 018e6798-c1f7-7693-b7d0-f6a6aa22c12d
                        created: 2024-03-22 19:17:28.183391732 +00:00:00
                last: 018fa121-9038-75e2-a308-eae85e24b64a
                        created: 2024-05-22 16:28:02.48868615 +00:00:00

host: 018b2c28-45bc-7c6d-9b71-b28588746193
        store: history
                idx: 14187
                first: 018e2ea9-bab2-7e02-87ab-2c3d94e4f847
                        created: 2024-03-11 17:57:39.122259192 +00:00:00
                last: 018fa11d-6227-7ab9-a630-f313114437b0
                        created: 2024-05-22 16:23:28.55117354 +00:00:00

Is there anything else I have to do? Maybe another parameter?

Edit
I think this fixed it, but I just had to wait for a few minutes. It seems to be working now.

1 Like

Glad it’s sorted! Lmk if you have any other issues

Was the delay after switching to sync v2?

I am in a strange situation. I have some records I cannot decrypt (~1500) and when I run atuin store purge, they are removed but as soon as I run atuin sync, they get downloaded again.

What to do now?

Also unrelated to that: How can I remove the history store for a specific host?

So, first I’d disable auto_sync to make sure no hosts push bad history

  1. atuin store purge - this will delete all records in the store that cannot be decrypted with the current key
  2. atuin store verify - verify that the previous operation was successful
  3. atuin store push --force - this will delete all records stored remotely, and then push up local data. Run this on the machine that has been purged
  4. atuin store pull --force - this does the opposite to (3). Delete all local data in the store, and pull from the remote
  5. atuin store rebuild history - ensure your history.db is up to date after all these operations

Currently not possible, though with a similar set of steps to the above it could be. Due to the nature of sync, ensuring that those steps propagate properly is difficult

2 Likes

I have made several edits and figured it out. When I was logging into each host I was generating a new key instead of using the same key for the same user. The result was that each host could only decrypt their own entries and it would spit out the error message about encryption. This wasn’t actually related to the new sync v2 it was just me making a newbie mistake. Ooops!

Have a great day!

1 Like

Thanks for letting us know what it was! I’ll see if we can make it harder to make this mistake in the future :smiley:

1 Like

I have three nodes and get different amount of history when I use this:

atuin sync --force ; atuin history list --format "{host}" | sort --ignore-case | uniq -c --ignore-case | sort -n

Output examples from two nodes:

# on FV:
   5425 sa
  23942 ar
  75240 FV

# on sa:
   5470 sa
  23942 ar
  75229 fv

I’d expect a little skew of < 10 commands due to syncing/running stuff on one machine after another, but this is a little much?

Storage status:

# sa
host: 018b1f22-f498-7bd1-9573-ddf4c61970dd
	store: history
		idx: 27138
		first: 018d9ce7-2feb-7c7b-acd9-fef5d519e6a8
			created: 2024-02-12 10:40:13.291778 +00:00:00
		last: 01907d6f-f0e6-7449-8816-2e1fedb97f62
			created: 2024-07-04 11:10:06.566287 +00:00:00

host: 018a975e-e082-7198-a370-7362d00f20f9
	store: history
		idx: 87289
		first: 018eed8c-00ba-7838-9bdf-848888a9569a
			created: 2024-04-17 19:32:39.226825384 +00:00:00
		last: 01907a3c-7705-7809-8251-2e73c252489f
			created: 2024-07-03 20:15:01.38110538 +00:00:00

host: 018a98ec-8d19-7c04-8751-a3781bc6ce3e <- CURRENT HOST
	store: history
		idx: 1046
		first: 018d9f6a-6e9a-7587-aabb-8faae3c63b58
			created: 2024-02-12 22:22:48.986073096 +00:00:00
		last: 01907d76-3780-72fb-ba90-afde09db9eef
			created: 2024-07-04 11:16:57.856668815 +00:00:00

# FV
host: 018a975e-e082-7198-a370-7362d00f20f9
	store: history
		idx: 87289
		first: 018eed8c-00ba-7838-9bdf-848888a9569a
			created: 2024-04-17 19:32:39.226825384 +00:00:00
		last: 01907a3c-7705-7809-8251-2e73c252489f
			created: 2024-07-03 20:15:01.38110538 +00:00:00

host: 018a98ec-8d19-7c04-8751-a3781bc6ce3e
	store: history
		idx: 1045
		first: 018d9f6a-6e9a-7587-aabb-8faae3c63b58
			created: 2024-02-12 22:22:48.986073096 +00:00:00
		last: 01907d75-5c5d-7a2b-9a40-718aaef6972e
			created: 2024-07-04 11:16:01.757814519 +00:00:00

host: 018b1f22-f498-7bd1-9573-ddf4c61970dd <- CURRENT HOST
	store: history
		idx: 27151
		first: 018d9ce7-2feb-7c7b-acd9-fef5d519e6a8
			created: 2024-02-12 10:40:13.291778 +00:00:00
		last: 01907d7c-6711-7ee1-8cc9-0a97399246c6
			created: 2024-07-04 11:23:43.249684 +00:00:00

I’m not dependent on this, but it would be interesting to see why there are these differences…

Either way: thanks for your work

1 Like

Looks like a skew of around 20 records? Not super implausible, depending on when sync has been ran. --force doesn’t actually do anything on sync v2, but what’s the output of atuin sync on each machine?

Otherwise, how about atuin doctor?

Either way: thanks for your work

<3

# FV, mac with gnu utils:

atuin sync ; atuin history list --format "{host}" | gsort --ignore-case | guniq -c --ignore-case | gsort -n
Uploading 3 records to 018b1f22f4987bd19573ddf4c61970dd/history
  [00:00:00] [###################################################################################################################################] 3/3 (0.0s)Downloading 1 records from 018a98ec8d197c048751a3781bc6ce3e/history
  [00:00:00] [###################################################################################################################################] 1/1 (0.0s)4/2 up/down to record store
Sync complete! 104729 items in history database, force: false
   5432 sa
  23942 ar
  75355 FV
# sa:

atuin sync ;  atuin history list --format "{host}" | sort --ignore-case | uniq -c --ignore-case | sort -n

Uploading 2 records to 018a98ec8d197c048751a3781bc6ce3e/history
  [00:00:00] [###################################################################################################################################] 2/2 (0.0s)Downloading 3 records from 018b1f22f4987bd19573ddf4c61970dd/history
  [00:00:00] [###################################################################################################################################] 3/3 (0.0s)3/4 up/down to record store
Sync complete! 104775 items in history database, force: false
   5479 sa
  23942 ar
  75354 fv

doctor on sa:

[Filesystem] ZFS is known to have some issues with SQLite. Atuin uses SQLite heavily. If you are having poor performance, there are some workarounds here: Pool timed out while waiting for an open connection, ZFS · Issue #952 · atuinsh/atuin · GitHub

{
  "atuin": {
    "version": "18.3.0",
    "sync": {
      "cloud": false,
      "records": true,
      "auto_sync": true,
      "last_sync": "2024-07-04 20:00:20.878997476 +00:00:00"
    },
    "sqlite_version": "3.44.0"
  },
  "shell": {
    "name": "zsh",
    "default": "zsh",
    "plugins": [
      "atuin"
    ],
    "preexec": "built-in"
  },
  "system": {
    "os": "Arch Linux",
    "arch": "x86_64",
    "version": "rolling",
    "disks": [
    
[... much zfs and docker stuff ...]

FV:

{
  "atuin": {
    "version": "18.3.0",
    "sync": {
      "cloud": false,
      "records": true,
      "auto_sync": true,
      "last_sync": "2024-07-04 19:58:24.02087 +00:00:00"
    },
    "sqlite_version": "3.44.0"
  },
  "shell": {
    "name": "zsh",
    "default": "zsh",
    "plugins": [
      "atuin"
    ],
    "preexec": "built-in"
  },
  "system": {
    "os": "Darwin",
    "arch": "arm64",
    "version": "14.5",
    "disks": [
      {
        "name": "Macintosh HD",
        "filesystem": "apfs"

For the sa host ~50 records are missing :thinking: