Framework 2.0 and Python subprocess

[BigBill]

Hi,

It somehow turned out that I had to call a command in bash (subprocess) directly from the custom coded plugin, something like this:

result = subprocess.run("superuser-command", 
             shell=True,
             stdin=subprocess.PIPE,
             stdout=subprocess.PIPE,
             stderr=subprocess.PIPE,
             bufsize=0)

The problem is that this command can only be executed by the root. I know I can work some magic with the sudoers file, that's OK, but this is only half of the issue. The biggest problem is that the user dtuser is not assigned a shell other than /bin/false, which virtually eliminates the ability to run commands. Does changing the shell from /bin/false to /bin/bash have any serious consequences for the user dtuser?

I wonder what your recommendations are for when a subprocess needs to be called from extensions?

Cheers,

BB

[vagiz_duseev · Accepted solution]

There are multiple ways to address this. Some depend on available features in the certain Linux distribution more than others.

Long story short, on Linux, we need to consider several impacting factors:

(optional) What the PATH environment variable looks like for `dtuser`.

This impacts whether dtuser can find the superuser-command executable by its name. Even if PATH is empty (it's not), we can still instead provide a full path to the executable: /opt/some-software/superuser-command

(mandatory) How the `shell` argument in the `subprocess` module works in Python.

The parameters that we use are important. https://docs.python.org/3.10/library/subprocess.html
We should only specify `shell=True` when we want to use shell-only syntax in our command. For the purposes of this question, we can set shell=False.

This is especially important, because, as you have correctly mentioned, `dtuser` has no shell, so we can't use it. There is a famous workaround though. Call `bash` executable, and pass your shell command as an argument. Examples:

Example 1

cat my_file.txt | grep "my-value" &> my_output.log

In this example piping `|` and output redirection `&>` are pure shell syntax. We can only use them inside command if `shell=True`.

Example 2

/opt/soft/executable first_arg second_arg --some-flag

This command contains no shell syntax. We can execute it as-is with `shell=False`.

Example 3

bash -c "cat my_file.txt | grep "my-value" &> my_output.log"

This example uses shell-only syntax but passes it to bash executable as an argument. This can be successfully executed with `shell=False`, because we don't really use shell directly. Bash does. 

(optional) How the command we give to subprocess is structured.

The `subprocess.run()` method is a light wrapper around the `subprocess.Popen` method. Unlike `Popen`, it can accept both a string representing a command ("/opt/soft/executable first_arg second_arg --some-flag") and an array of strings (["/opt/soft/executable", "first_arg", "second_arg", "--some-flag"]). `Popen` only accepts arrays.

You can be more explicit and pass an array for more granular control over how command is passed to child process. Under the hood, the `subprocess.run` method converts the string you give it to an array anyway, but it does it using its own heuristics.

(optional) How slashes and quotes are escaped inside the command.

If you have convoluted double quote, whitespaces, or slash symbols inside the command they have to be carefully escaped, if needed.

(mandatory) Permission elevation.

If some command can only be executed by root, then, of course, `dtuser` can't run it as-is – it has no permission to do so. Thus, we need to grant the user such permission. If your Linux distribution uses sudo, we can use the sudoers file to explicitly allow dtuser to run certain executables. You can add the following to the sudoers file

dtuser ALL=(ALL:ALL) superuser-command​

This will allow `dtuser` to run the superuser-command on ALL hosts, as ALL users and ALL groups.

Good human-readable explanation of this syntax can be found here: https://www.digitalocean.com/community/tutorials/how-to-edit-the-sudoers-file

If the sudo requires a password on your system, then, of course, `dtuser` will not be able to interactively provide it and the password input must be disabled. This changes the sudoers file as follows:

dtuser NOPASSWD: ALL=(ALL:ALL) superuser-command

Or, alternatively, grant dtuser the sudo permissions, if you don't really care about the granularity:

sudo usermod -aG sudo dtuser

This will allow `dtuser` to execute commands as root. You can then use the syntax of `sudo superuser-command` when invoking subprocess module.

(optional) Do we intend to detach the process to run in the background independently of the extension?

This is a narrow use case, I must admit. In this situation, we must remember, that we operate within a process tree. Extension that kicks-off a subprocess is itself a child process of Extension execution controller module, which is a child process of ActiveGate (I simplify things here a lot). So, when we spawn another process within the extension, it becomes a child of the root extension Python process.

If we want this child process to continue to run even if the extension restarts, we must detach the input/output pipes. Example:

subprocess.Popen(
    ["/opt/software/superuser-command"],
    stdout=subprocess.DEVNULL,
    stderr=subprocess.DEVNULL,
)​

Detaching a process on Windows is a bit more tricky. Example with invoking a powershell script using `dtuser` on Windows and letting it run independently.

subprocess.Popen(
    ["powershell.exe", "-ExecutionPolicy", "RemoteSigned", "-File", "my_script.ps1"],
    stdout=subprocess.DEVNULL,
    stderr=subprocess.DEVNULL,
    creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,
)

Permission elevation on Windows is a whole different story and a mess. There is a reason Python doesn't provide an out of the box method to do that – it's hard and not standardized. The way to solve it with Python on Windows is to essentially create your own subprocess module, which will rely on system calls from win32 API to properly elevate permissions for a child process it creates.

Hope this helps!

[BigBill]

Thank you @vagiz_duseev for such a comprehensive and informative explanation of how to deal with subprocesses in extensions. Currently my custom coded plugin is working as intended.

Cheers,

BB