After rebasing on the show_tabs change, I tidied up a little and put together some ideas based on that feedback:
The thinking is that the original inspected command (the one highlighted when Ctrl+o is pressed) gives a baseline for time. Moreover, the offsets help highlight movement (especially if a series of commands are re-run). In the first example, I switched from the inspector on cargo run search -i and pressed Up once (so the “next command” is now “cargo run search -i”, and I can see that I ran the same command three times, the first 1m5s before the second, and the final time 21m later). For the previous and next commands, the time from the middle command is given, and from the original inspected command in parentheses. I did think it might make sense to switch to Left/Right in the full view? It’s probably not a great example given it has the same command text three times, but I promise that the stats are being shown for whatever is currently in the middle block 
Still a PoC (Comparing atuinsh:main...philtweir:compact-inspector · atuinsh/atuin · GitHub for code) but happy to tidy up for PR if aligned with what you were thinking, or just enjoy by myself otherwise 
I did note in the code there was mention of a refactor to give the inspector its own state - I have tried to encapsulate the relevant state, but not separated it, so if it would help to do that first, makes sense.