sed find and replace html tag

This will work:

sed -i -e "s@<span class=\"symbol\">&#58771;</span>@[\&#58771;]{custom-style=\"symbol\"}@g" myfile.md

You need to use \ before &, then it will work fine because & is special in the replacement text: it means “the whole part of the input that was matched by the pattern”.

From man sed:

s/regexp/replacement/

The replacement may contain the special character & to refer to that portion of the pattern space which matched, and the special escapes \1 through \9 to refer to the corresponding matching sub-expressions in the regexp.

This seems to work for me

# echo '<span class="symbol">&#58771;</span>' > test 
# sed -i -e 's@<span class="symbol">\&#58771;<\/span>@[\&#58771;]{custom- 
style="symbol"}@g' test
# cat test 
[&#58771;]{custom-style="symbol"}

With awk

echo "<span class="symbol">&#58771;</span>" |awk -F'=|>|<' 'BEGIN { OFS=""; }{ print "["$4"]","{cumtom-style=""\""$3"\"""}"}'
[&#58771;]{cumtom-style="symbol"}

There are some issues with the code you tried.

First, things are easier if you choose a character that shows up nowhere in your text as the delimiter for the s/// command. Alternatively, you have to escape with a \ any occurrence of the delimiter in your text.

In your first command,

sed -i -e 's/\(<span class=\"symbol\">\)&#58771;\(<\/span>\)/\[/&#58771;\]/\{custom-style=\"symbol\"\}/g'myfile.md

the error you get is due to the / in [/\], that is not escaped.

Also, to put a literal & in the output text, you have to escape it with a \ in your replacement expression. Otherwise & is interpreted as special and replaced with the matched portion of the pattern space.

To keep things simple, here I removed (possibly?) unnecessary escape characters and a couple of (possibly?) unneeded / (from [/\]/\{custom-style=\"symbol\"\}). I also chose | as the delimiter for the s/// command, since it does not appear in your input text. The command becomes:

sed -e 's|<span class="symbol">&#58771;</span>|[\&#58771;]{custom-style="symbol"}|g'

And this is what it does:

$ echo '<span class="symbol">&#58771;</span>' | sed -e 's|<span class="symbol">&#58771;</span>|[\&#58771;]{custom-style="symbol"}|g'
[&#58771;]{custom-style="symbol"}

If the string  is not static and you want to replace <span class="symbol"> with [ and </span> with ]{custom-style="symbol"} around it, whatever its value, you can use a capturing group (()) and a backreference (here, \1):

sed -e 's|<span class="symbol">\(.*\)</span>|[\1]{custom-style="symbol"}|g'

What this command does:

$ echo '<span class="symbol">whatever is here</span>' | sed -e 's|<span class="symbol">\(.*\)</span>|[\1]{custom-style="symbol"}|g'
[whatever is here]{custom-style="symbol"}

Finally, you may generally prefer to use single quotes (') instead of double quotes (") around sed scripts, to protect them from the shell - parameter expansion triggered by $, escaping effects of \...