Use workspace for up-key, or directory when not in a git workspace?

I would like to keep ⌃R to use global search mode, but make the ↑ key to use workspace when possible, or directory when not.

I tried these settings:

filters = [ "workspace", "directory", "global", "host", "session" ]
filter_mode = "global"
filter_mode_shell_up_key_binding = "workspace"
workspaces = true

I was hoping that the filter mode would use the filters list to determine the next mode to try, but it doesn’t seem to be working (it uses global when outside a workspace). Is there a way to configure this?