Sync v2 testing

General
1

As of v18, sync v2 is now available to opt-in

Hey! I think we’re finally ready :slight_smile:

I’ve mentioned it a whole bunch over the past few months, but I’ve been working on replacing Atuin’s sync with something much more flexible, resilient, and performance (both in terms of time, and network usage).

You may have seen references to “records” in Atuin’s sync output. These are generic syncable-things. Currently used by the kv store, and now by history too!

How to use

All machines must run the same version of sync - so if you enable record sync on one machine, but not another, they will not sync with each other.

Setup

  1. Ensure you’re running at least atuin v18
  2. Add this to your ~/.config/atuin/config.toml:
[sync]
records = true
  1. Run atuin sync.
  2. If prompted by (3), run atuin history init-store. This will copy all of your history from history.db into records.db. Then run atuin sync again. This is only required when you’ve used the old sync, to ensure sync v2 is all caught up with old data.

At any time, run atuin store status to see what’s going on with your stores.

More details

records.db store an encrypted (on disk) “log” of records. For history, this is Create(History), or Delete(HistoryId). This is then “built” into history.db, which turns it into a searchable format that can be used by the history TUI, search, etc. Records works great for sync and long term storage, history is great for actual usage. This makes things much simpler + more flexible.

You can trigger a “rebuild” manually with

atuin store rebuild history

One issue that has appeared a bunch with the “old” sync is when users have mixed keys. Maybe they paste it wrong, forget it, whatever. Their remote account ends up with history encrypted with a variety of keys, which causes sync to fail. The only solution here was to delete your account and start afresh.

This worked fine, as history was store unencrypted locally.

Now that we are storing it encrypted locally and syncing from there, it’s a little more complicated.

The new sync provides new operations to both check the state of the local store, and to recovery. I’ll provide more details on those later!

5 Likes
2

I’d like to test, but I have a few questions.

My history has not been fully synced from the start (maybe because they already have similar local histories before I start syncing). Is there anything special I should do for this test? Will this v2 solve the desync for me?

1 Like
3

Hey. I will try to set it up later today. Is there anything special you would like us to tests? Any test cases to try and experiment with?

1 Like
4

Nothing special! I’m hoping it will “just work”, once it’s gotten going

Without knowing more about the reason it’s not syncing for you atm I cannot say for sure, but it’s very likely to solve the problem. If it doesn’t, I’d definitely like to have a look and figure out why

Thank you for testing!

5

Thank you!

Nothing in particular - honestly just building some confidence that the sync will work without issues, end to end, on a variety of systems and setups.

If you and @lilydjwg are able to share

  1. How many machines you’re syncing
  2. The output of atuin store status

Then that would be extra helpful :slight_smile: Just to check that it’s working optimally internally too.

1 Like
6

I’m syncing two machines.

host: 018cb9fc-a161-7f64-ae47-b38a5db0d123
	store: history
		idx: 59241
		first: 018ce771-5dc4-7d26-8abe-a2fdf9d2ef38
			created: 2024-01-08 5:00:15.684612767 +00:00:00
		last: 018d7ef9-7429-7c18-a679-cd3fd19f0c01
			created: 2024-02-06 15:11:33.929096321 +00:00:00

host: 018ccdf9-4437-713e-85e6-fae9d64c6b6f <- CURRENT HOST
	store: history
		idx: 57030
		first: 018cf369-0d6c-7837-b3f9-5bfeb908687a
			created: 2024-01-10 12:46:37.420039931 +00:00:00
		last: 018d7ef9-903d-79a6-a0c7-2d4283a72ec6
			created: 2024-02-06 15:11:41.117508519 +00:00:00

host: 018cb9fc-a161-7f64-ae47-b38a5db0d123 <- CURRENT HOST
        store: history
                idx: 59240
                first: 018ce771-5dc4-7d26-8abe-a2fdf9d2ef38
                        created: 2024-01-08 5:00:15.684612767 +00:00:00
                last: 018d7ef9-5a4a-7cae-8195-23e9392b2576
                        created: 2024-02-06 15:11:27.306777929 +00:00:00

host: 018ccdf9-4437-713e-85e6-fae9d64c6b6f
        store: history
                idx: 57029
                first: 018cf369-0d6c-7837-b3f9-5bfeb908687a
                        created: 2024-01-10 12:46:37.420039931 +00:00:00
                last: 018d7edc-4878-7ff9-85af-04d97def1e64
                        created: 2024-02-06 14:39:42.200366345 +00:00:00

They seem to be fully synced now!

1 Like
7

I have another set of atuin consisting of four instances , and three of them has run into trouble:

$ atuin history init-store
Importing all history.db data into records.db
Fetching history from old database
Fetching history already in store
Error: attempting to decrypt with incorrect key. currently using AAA, expecting BBB

Location:
    atuin-client/src/record/encryption.rs:132:9
8

Ah, ok. At some point, those machines were probably running either

  1. not being logged in, so with some generated key
  2. with the incorrect key specified

There’s a few workarounds. First, we can check that this is definitely the case (mixed keys in store) with

atuin store verify

If that reports mixed keys, you can run

atuin store purge

To delete records from the store that we no longer have the key for. Those are essentially noise, anyway.

Let me know if this machine has been syncing prior to this, and we can sort the remote storage out too.

(tldr of my life is that e2e encryption and key management makes things really annoying sometimes :joy:)

1 Like
9

This should be the case. I might have imported history before logging in.

The issue has been resolved after running atuin store purge, thanks!

1 Like
10

I use mainly one machine (the first; history backed-up on the server) and synchronize with experimental VM (second machine). Everything seems to work as expected. It runs beautifully.

host: 018a79f8-ed46-7d93-965d-c241a1aa624e <- CURRENT HOST
        store: history
                idx: 21685
                first: 018d7f64-8cde-7f62-b1e6-b187ec0173de
                        created: 2024-02-06 17:08:32.606837568 +00:00:00
                last: 018d7fba-ddbc-7b8f-8413-e222ef77b7f9
                        created: 2024-02-06 18:42:49.404281554 +00:00:00

host: 018d44f9-b2af-7525-89a0-0974a2c6491b
        store: history
                idx: 47
                first: 018d7faf-53ce-78f3-afd7-f6aca20caf55
                        created: 2024-02-06 18:30:13.198771468 +00:00:00
                last: 018d7fba-b9d6-72fb-b37a-7cfbc91ccba2
                        created: 2024-02-06 18:42:40.214356124 +00:00:00

host: 018a79f8-ed46-7d93-965d-c241a1aa624e
        store: history
                idx: 21684
                first: 018d7f64-8cde-7f62-b1e6-b187ec0173de
                        created: 2024-02-06 17:08:32.606837568 +00:00:00
                last: 018d7fb8-e606-70d7-9e64-75482f58a085
                        created: 2024-02-06 18:40:40.454934974 +00:00:00

host: 018d44f9-b2af-7525-89a0-0974a2c6491b <- CURRENT HOST
        store: history
                idx: 46
                first: 018d7faf-53ce-78f3-afd7-f6aca20caf55
                        created: 2024-02-06 18:30:13.198771468 +00:00:00
                last: 018d7fba-b1bf-7837-9777-840c702b01ba
                        created: 2024-02-06 18:42:38.143468074 +00:00:00
11

Glad to hear it! :smiling_face: thanks for testing

12

Doing this seems to have sorted out my laptop and desktop not sharing history correctly. Thanks!

1 Like
13

Great! Glad to hear that. Welcome to the forum :blush:

14

I’ve had issues for a while now with three hosts being out of sync, but upgrading them all to 18.0.1 and switching to sync v2 has everything happy now, all hosts have all records from all other hosts. I’ll close my GitHub issue for this now too.

15

Awesome! Glad that sorted it all out for you :blush:

16

saw Ellie’s post on mastodon and decided to give it a try…the migration process is really smooth and the instructions worked fine…i dont have multiple machines so it’s just this one for now

the timeout was a but small though, i had to move my laptop to basically in front of the router (but that’s probs just shitty internet)

1 Like
17

Glad to hear it went smoothly! Curious as to the biggest reason you want to sync, if you’re using one machine - is it for backups? (totally valid either way)

What sort of error were you getting? Also was this a self hosted server or the public one?

18

yeahh the backups are the main thing…and for eventually when i move over from this
also the ui looks greattttttt :star_struck:
(and i get to see history entries from other currently opened shells – normally, i think my zsh .histfile only gets written to when i close the shell, so if there’s multiple sessions open, you can’t see something you just did somewhere else)

the public one, and it was something along the lines of “something api.atuin.sh timeout reached”, but it was happening like only after 3 seconds

like i said, probably terrible wifi…i was just surprised it didn’t try for longer

19

I found out that using Atuin to sync history between the host machine and numerous VMs is also a great use for the sync feature, even if it is still just to share history on a single machine, technically. It helps immensely.

20

Makes sense! Haha glad to hear it :smiley:

We have some timeout config here, if this happens often for you

https://docs.atuin.sh/configuration/config/#network_timeout

I’ve heard this a few times! I never thought of that with the initial development, but it’s awesome that Atuin helps there too. Roughly how many VMs have you used it with?

1 Like