How to run command in Rust

nohup and & are shell-specific constructs that tell your shell not to wait for the command to finish. They are not really part of your command at all, so what you want is:

let handle = Command::new("gdx-open")
                    .args(&["~/Documents"])
                    .spawn()
                    .unwrap();

If instead you wanted to wait for the command to finish before proceeding, you would use:

let result = Command::new("gdx-open")
                    .args(&["~/Documents"])
                    .status()
                    .unwrap();

or

let output = Command::new("gdx-open")
                    .args(&["~/Documents"])
                    .output()
                    .unwrap();

Moreover, the ~ is also a shell-specific construct which may or may not be understood by the command you're trying to run. It means "this user's home directory", which you can get from the HOME environment variable using std::env::var:

let path = format!(
    "{}/Documents", 
    std::env::var ("HOME").unwrap_or_else (|_| "".into()));
let handle = Command::new("gdx-open")
                    .args(&[&path])
                    .spawn()
                    .unwrap();

And BTW, ~/Documents is a special path, part of the XDG user directories specification. Its actual location is platform-dependent and language-dependent (i.e. it gets translated depending on the user's configuration). It can also be overridden by the user. You might want to look at the directories crate to get the correct value:

use directories::UserDirs;
let dirs = UserDirs::new();
let path = dirs
    .and_then (|u| u.document_dir())
    .unwrap();
let handle = Command::new("gdx-open")
                    .args(&[path])
                    .spawn()
                    .unwrap();