Decoding JSON using json.Unmarshal vs json.NewDecoder.Decode

Question

I'm developing an API client where I need to encode a JSON payload on request and decode a JSON body from the response.

I've read the source code from several libraries and from what I have seen, I have basically two possibilities for encoding and decoding a JSON string.

Use json.Unmarshal passing the entire response string

data, err := ioutil.ReadAll(resp.Body)
if err == nil && data != nil {
    err = json.Unmarshal(data, value)
}

or using json.NewDecoder.Decode

err = json.NewDecoder(resp.Body).Decode(value)

In my case, when dealing with HTTP responses that implements io.Reader, the second version seems to be require less code, but since I've seen both I wonder if there is any preference whether I should use a solution rather than the other.

Moreover, the accepted answer from this question says

Please use json.Decoder instead of json.Unmarshal.

but it didn't mention the reason. Should I really avoid using json.Unmarshal?

This pull request on GitHub replaced a call to Unmarshal with json.NewDecoder to "remove buffer in JSON decoding." — Matt
– Matt, Commented Jan 17, 2014 at 22:46
It just depends on what input is more convenient for you to use. blog.golang.org/json-and-go gives examples of using both techniques. — rexposadas
– rexposadas, Commented Jan 17, 2014 at 23:00
IMO, ioutil.ReadAll is almost always the wrong thing to do. It's not related to your goal, but requires you to have enough contiguous memory to store whatever might be coming down the pipe, even if the last 20TB of response is after the last } in your JSON. — Dustin
– Dustin, Commented Jan 17, 2014 at 23:40
@Dustin "ReadAll is almost always the wrong" I think if JSON is small and about few Kb then it might be better to read it all to release connection. And this is very often so it looks like ReadAll and then Unmarshall is better solution in most cases — Sergey Ponomarev
– Sergey Ponomarev, Commented Sep 7, 2021 at 12:32

Seth · Accepted Answer · 2016-07-14 01:50:31Z

396

It really depends on what your input is. If you look at the implementation of the Decode method of json.Decoder, it buffers the entire JSON value in memory before unmarshalling it into a Go value. So in most cases it won't be any more memory efficient (although this could easily change in a future version of the language).

So a better rule of thumb is this:

Use json.Decoder if your data is coming from an io.Reader stream, or you need to decode multiple values from a stream of data.
Use json.Unmarshal if you already have the JSON data in memory.

For the case of reading from an HTTP request, I'd pick json.Decoder since you're obviously reading from a stream.

edited Jul 14, 2016 at 1:50

Seth

10.5k10 gold badges50 silver badges69 bronze badges

answered Jan 18, 2014 at 0:40

James Henstridge

44.2k9 gold badges139 silver badges119 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Aktau Over a year ago

Also: by inspecting the Go 1.3 source code, we can also learn that for encoding, if you use a json.Encoder, it will reuse a global buffer pool (backed by the new sync.Pool), which should decrease buffer churn a lot if you're encoding a lot of json. There's only one global pool so different json.Encoder's share it. The reason this couldn't be done for the json.Marshal interface is because the bytes are returned to the user and the user doesn't have a way to "return" the bytes to the pool. So if you're doing a lot of encoding, json.Marshal always has quite a bit buffer churn.

James Henstridge Over a year ago

@Flimzy: are you sure? The source code still says it reads the entire value into the buffer before decoding: github.com/golang/go/blob/master/src/encoding/json/…. The Buffered method is there to let you see any extra data that was read into the internal buffer after the value.

Jonathan Hall Over a year ago

@JamesHenstridge: No, you're probably right. I was just interpreting your statement differently than you intended. Apologies for the confusion.

Samir Kape · Accepted Answer · 2022-01-10 05:43:39Z

15

I found this paragraph in the Go web programming book. But there is no explanation given

So when do we use Decoder versus Unmarshal?

That depends on the input. If your data is coming from an io.Reader stream, like the Body of an http.Request, use Decoder. If you have the data in a string or somewhere in memory, use Unmarshal.

edited Jan 10, 2022 at 5:43

answered Aug 6, 2021 at 2:23

Samir Kape

2,1332 gold badges19 silver badges21 bronze badges

Collectives™ on Stack Overflow

Decoding JSON using json.Unmarshal vs json.NewDecoder.Decode

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related