JSON string parsing error in Python

Question

I am sending a request to an API and it returns this JSON response

'{"Reply":{"Header":{"Method":"mGSSCBetHistory","ErrorCode": "0","MerchantID":"BETSTARtest","MessageID": "H140201152657m6k3f"},"Param":{"TotalRecord":"1","BetDatas":"[{"Column1":""}]","ErrorDesc": ""}}}'

when I try to convert it to dictionary whether ast.literal_eval or json.loads it returns this error:

Traceback (most recent call last):
  File "/Users/deanchristianarmada/Desktop/projects/asian_gaming/radar/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/deanchristianarmada/Desktop/projects/asian_gaming/radar/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/deanchristianarmada/Desktop/projects/asian_gaming/radar/provider/KENO/tasks.py", line 127, in run
    ['Reply', 'Param', 'BetDatas'], 'post')
  File "/Users/deanchristianarmada/Desktop/projects/asian_gaming/radar/core/classes.py", line 217, in check_records
    self.result = response.json()
  File "/Users/deanchristianarmada/Desktop/projects/asian_gaming/radar/lib/python2.7/site-packages/requests/models.py", line 826, in json
    return complexjson.loads(self.text, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting ',' delimiter: line 1 column 169 (char 168)

The reason for this is because of the value of the BetDatas which has a double quote inside a double quote.. Is there an easy workaround for this? Because the only proper solution that I can think of is to tell the third party that I've been sending request to change their JSON response.

yeah its invalid json.

Craicerjack
– Craicerjack

2017-03-01 10:49:07 +00:00
Commented Mar 1, 2017 at 10:49 — Craicerjack
– Craicerjack, Commented Mar 1, 2017 at 10:49

Crispin · Accepted Answer · 2017-03-01 12:30:33Z

3

Use a regular expression to match the offending value and use re.sub to escape the match.

import re
import json

data = '{"Reply":{"Header":{"Method":"mGSSCBetHistory","ErrorCode": "0","MerchantID":"BETSTARtest","MessageID": "H140201152657m6k3f"},"Param":{"TotalRecord":"1","BetDatas":"[{"Column1":""}]","ErrorDesc": ""}}}'

def escape(match_obj):
    print(match_obj.group(1))
    return match_obj.group(1).replace('"','\"')
REGEX = '(?<="BetDatas":")(\S+)(?=",)'

data = re.sub(REGEX, escape, data)
print(data)

edited Mar 1, 2017 at 12:30

answered Mar 1, 2017 at 11:11

Crispin

2,1201 gold badge15 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

sxn Over a year ago

While this might function as a workaround, the safer solution would be to contact the third party and tell them they are returning invalid JSON. Since they don't escape special characters in this particular instance, it's likely that they don't do it in general, potentially requiring you to add more hacks in place in the future.

Crispin Over a year ago

@sxn agreed. a new request could easily break this hack

Dean Christian Armada Over a year ago

Wow thanks! I agree with you, I should report this to the third party than doing hacks

Meghdeep Ray · Accepted Answer · 2017-03-01 10:39:29Z

2

If you exclude "BetDatas" as you correctly guessed then it parses properly

import json
json.loads('{"Reply":{"Header":{"Method":"mGSSCBetHistory","ErrorCode": "0","MerchantID":"BETSTARtest","MessageID": "H140201152657m6k3f"},"Param":{"TotalRecord":"1","ErrorDesc": ""}}}')

{u'Reply': {u'Header': {u'ErrorCode': u'0', u'MessageID': u'H140201152657m6k3f', u'Method': u'mGSSCBetHistory', u'MerchantID': u'BETSTARtest'}, u'Param': {u'ErrorDesc': u'', u'TotalRecord': u'1'}}}

There is no obvious way to handle this, it depends on the API provider rectifying their JSONs.

answered Mar 1, 2017 at 10:39

Meghdeep Ray

5,6254 gold badges38 silver badges62 bronze badges

Collectives™ on Stack Overflow

JSON string parsing error in Python

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related