Deletion with daemon is slow

Feedback
1

It now takes several seconds to delete a single history.

In the past the daemon was lightweight and provided in-time syncing, but now it does too many things including rebuilding a search index over and over again of which I don’t know what use is.

Sorry, the log file I am trying to upload is not authorized.

I’ve deleted the reload_search_index call and it becomes fast again.

2

There are two ways to avoid the problem:

  1. Finish the response without waiting for the rebuild to complete. The disk reads and computation still happens.
  2. Don’t build the index immediately if search_mode is set to fulltext in the settings. Wait until it is actually used.
3

Hi @lilydjwg, thanks for the heads up. I believe I have improved on this in perf: make the daemon's large deletes and CI's integration tests fast by markovejnovic · Pull Request #4189 · atuinsh/atuin · GitHub. Have you had a chance to try this branch?

The reason we rebuild the index is because any delete without a completed index rebuild would result in inaccurate search results – the index would tell us that results exist which are not actually available in storage.

I think this suggestion makes sense:

  1. Finish the response without waiting for the rebuild to complete. The disk reads and computation still happens.

I think the correct solution here is to “kick off” index rebuilding in the background and raise a semaphore, immediately return from the delete call. Subsequent searches would block on the semaphore.

Also, how are you deleting your entries? If you are trying to delete multiple entries, then the Daemon RPC for deleting history entries should enable you to do them all in one batch:

// One chunk of a client-streamed delete.
message DeleteHistoryRequest {
  repeated common.HistoryId ids = 1;
}

message DeleteHistoryReply {
  // Number of entries processed for deletion.
  uint64 deleted = 1;
  string version = 2;
  uint32 protocol = 3;
}

service History {
  // Delete history entries by id.
  //
  // Clients may want to delete millions of rows and tonic will limit a request at 4MB, so this
  // stream serves to circumvent this limitation.
  //
  // The server will accumulate up to 10M history entries and then delete them all in one go.
  //
  // TODO(markovejnovic): It would be really good if we could stream the reply back so that the
  //                      user can see a progress bar.
  rpc DeleteHistory(stream DeleteHistoryRequest) returns (DeleteHistoryReply);
}
4

I have another idea: remember a small amount (e.g. 50) of the ids of deleted entries and exclude them from the index search result, and only rebuild the index when the amount becomes large. This can avoid rebuilding it repeatedly when I delete entries. Also you don’t need to block search while rebuilding.

I delete interactively when I see unneeded entries, or intentionally run an interactive search, find and delete some “bad” ones after I’ve figured out the “good” one. So it is to delete several entries one-by-one in a short time window.

5

Understood! Thank you. Those are good ideas.

There’s currently a relatively nasty bug in the form of Search Index Race Conditions in the Daemon · Issue #4052 · atuinsh/atuin · GitHub that we have to fix before we can work on more search improvements.

I’ll try to finish that this week, but it’s a nasty concurrency bug so it might take a bit more.