Export-Csv will always put all fields in double quotes, so you have to remove the undesired quotes the hard way. Something like this might work:
$csv = 'C:\path\to\your.csv'
(Get-Content $csv) -replace '^"(.*?)",(.*?),"(.*?)"$', '$1,$2,$3' |
Set-Content $csv
Regular expression breakdown:
^ and $ match the beginning and end of a string respectively (Get-Content returns an array with the lines from the file).
"(.*?)" matches text between two double quotes and captures the match (without the double quotes) in a group.
,(.*?), matches text between two commas and captures the match (including double quotes) in a group.
$1,$2,$3 replaces a matching string with the comma-separated first, second and third group from the match.