1

I want to retrieve content from web page. However, I tried above method but the error still come when the query string contain Chinese character.

code


$json = Get-Content -Encoding utf8 -Path "./kw.json" | ConvertFrom-Json

$kw = $json.name
$numPages = $json.page
# $encodingKW = [System.Text.Encoding]::UTF8.GetString([System.Text.Encoding]::GetEncoding("UTF-8").GetBytes($kw))
Write-Output $encodingKW

for ($i = 0; $i -lt $numPages; $i ++) {
    $pn = $i * 50
    
    [string]$url = "https://tieba.baidu.com/f?kw=${kw}&ie=utf-8&pn=${pn}"
    # https://tieba.baidu.com/f?kw=%E5%88%80%E5%8A%8D%E7%A5%9E%E5%9F%9F&ie=utf-8&pn=0
    
    Write-Output $url
    $html = Invoke-WebRequest -Method Get -Uri $url
    # $content = [System.Text.Encoding]::UTF8.GetString([System.Text.Encoding]::GetEncoding("iso-8859-1").GetBytes($html.Content))

    Write-Output $html
    break;
}
# [string]$url = "https://tieba.baidu.com/f?kw=${kw}"

# Write-Output $json.name
# Write-Output $json.page
# Write-Output $url

The url is

https://tieba.baidu.com/f?kw=地理&ie=utf-8&pn=0

It give me Invoke-WebRequest error. I think the url is successfully encoding since the url is in encoding form ,backurl:'https://tieba.baidu.com/f?kw=%E5%9C%B0%E7%90%86&ie=utf-8&pn=0',.

But the error is

$html = Invoke-WebRequest -Method Get -Uri $url CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest] WebException FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

And Is there alternative method as python url = baseurl+"f?kw="+parse.quote(kw) + "&ie=utf-8&pn=" + str(i * 50)?

3
  • Your primary problem is one of lack of authentication, not encoding; the response is a 403 (Forbidden)` error response. Commented Nov 14 at 17:58
  • I see. So I need to set cookies? Commented Nov 15 at 2:29
  • The Invoke-WebRequest cmdlet has authentication-related parameters, driven by the -Authentication parameter. Commented Nov 15 at 2:40

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.