Python - Get command output cannot be decoded

Note: The following are Python 3+ solutions, but there is a caveat:

• With the first solution below and also with the second one - but only if UTF-8 data must be sent to PowerShell's stdin stream - due to a bug in powershell.exe, the Windows PowerShell CLI, the current console window switches to a raster font (potentially with a different font size), which does not support most non-extended-ASCII-range Unicode characters. While visually jarring, this is merely a display (rendering) problem; the data is handled correctly; switching back to a Unicode-aware font such as Consolas reveals the correct output.

• By contrast, pwsh.exe, the PowerShell (Core) (v6+) CLI does not exhibit this problem.

Option A: Configure both the console and Python to use UTF-8 character encoding before executing your script:

• Configure the console to use UTF-8:

  • From cmd.exe, by switching the active OEM code page to 65001 (UTF-8); note that this change potentially affects all later calls to console applications in the session, independently of Python, unless you restore the original code page (see Option B below):

    chcp 65001
    

  • From PowerShell:

    $OutputEncoding = [Console]::InputEncoding = [Console]::OutputEncoding = [System.Text.UTF8Encoding]::new()
    

• And configure Python (v3+) to use UTF-8 consistently:^[1]

  • Set environment variable PYTHONUTF8 to 1, possibly persistently, via the registry; to do it ad hoc:

    • From cmd.exe:

      Set PYTHONUTF8=1
      

    • From PowerShell:

      $env:PYTHONUTF8=1
      

  • Alternatively, for an individual call (v3.7+): Pass command-line option -X utf8 to the python interpreter (note: case matters):

      python -X utf8 somefile.py ...
    

  • Both options enable Python UTF-8 Mode, which will become the default in Python 3.15.

Now, your original code should work as-is (except for the display bug).

Note:

• A simpler alternative via a one-time configuration step is to configure your system to use UTF-8 system-wide, in which case both the OEM and the ANSI code pages are set to 65001. However, this has far-reaching consequences - see this answer.

Option B: (Temporarily) switch to UTF-8 for the PowerShell call:

import sys, ctypes, subprocess

# Switch Python's own encoding to UTF-8, if necessary
# This is the in-script equivalent of setting environment var. 
# PYTHONUTF8 to 1 *before* calling the script.
sys.stdin.reconfigure(encoding='utf-8'); sys.stdout.reconfigure(encoding='utf-8'); sys.stderr.reconfigure(encoding='utf-8')

# Save the current console output code page and switch to 65001 (UTF-8)
previousCp = windll.kernel32.GetConsoleOutputCP()
windll.kernel32.SetConsoleOutputCP(65001)

# PowerShell now emits UTF-8-encoded output; decode it as such.
command = "powershell.exe ls ~/Desktop"
print(subprocess.run(command, stdout=subprocess.PIPE).stdout.decode())

# Restore the previous output console code page.
windll.kernel32.SetConsoleOutputCP(previousCp)

Note:

• Due to setting only the output console page, the Windows PowerShell display bug is avoided.
• If you also wanted to send input to PowerShell's stdin stream, you'd have to set the input console page too, via windll.kernel32.SetConsoleCP(65001) (which would then again surface the display bug).

^{[1] This isn't strictly necessary just for correctly decoding PowerShell's output, but matters if you want to pass that output on from Python: Python 3.x defaults to the active ANSI(!) code page for encoding non-console output, which means that Hebrew characters, for instance, cannot be represented in non-console output (e.g., when redirecting to a file), and cause the script to break.}