Unicode encode string

Question

I'm json_encoding some strings. Sometimes they contain binary data. This causes the encoding to fail with error code JSON_ERROR_UTF8. Running the strings through utf8_encode gets around this error. However, ✓ (a unicode checkmark) gets encoded as \u00e2\u009c\u0093 which when interpreted by JavaScript and rendered in your browser actually looks like â.

How can I fix this? Is there another encoding I can use?

echo json_encode(utf8_encode('✓')); // "\u00e2\u009c\u0093"

Now press F12 and paste that into your JavaScript console (quotes included). It should output â.

Please note that

echo json_encode('✓'); // "\u2713"

Works as intended. The issue is that sometimes the string will contain binary data which json_encode can't handle, so I need to sanitize every string without breaking the strings it can handle.

More examples:

json_encode(chr(200));              // false (bad)
json_encode(utf8_encode(chr(200)))  // "\u00c8" (good)
json_encode('✓');                   // "\u2713" (good)
json_encode(utf8_encode(chr(200)))  // "\u00e2\u009c\u0093" (bad)

So you see, encoding it works well for some strings and breaks others.

This is strictly for logging. I don't care if the binary data comes out weird, I just don't want it to mess with valid strings.

Maybe the problem relies in the document charset. Did you tried to add <meta charset="UTF-8">in the head of the HTML document? — collimarco
– collimarco, Commented Aug 21, 2014 at 19:20
@hek2mgl I pretty much gave it to you, but nevertheless, I updated the question. — mpen
– mpen, Commented Aug 21, 2014 at 19:20
This question is unanswerable. Encoding arbitrary binary data is one thing, keeping UTF-8 characters intact is something completely separate. What's to stop 0xe29c93 from being interpreted as ✓ when it shows up in your binary data? — amphetamachine
– amphetamachine, Commented Aug 21, 2014 at 19:27

Community · Accepted Answer · 2017-05-23 10:25:19Z

1

Running strings through this function

function _utf8($str) {
    if(!mb_check_encoding($str, 'UTF-8')) {
        return utf8_encode($str);
    }
    return $str;
}

(taken and modified from here)

Seems to give the results I'm after.

Checkmarks are left alone, but chr(200) and other weirdness is encoded:

json_encode(utf8_encode(chr(200))) // "\u00c8"

edited May 23, 2017 at 10:25

CommunityBot

11 silver badge

answered Aug 21, 2014 at 19:41

mpen

285k290 gold badges896 silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

amphetamachine · Accepted Answer · 2014-08-21 19:31:06Z

0

EDIT: This question is unanswerable. Encoding arbitrary binary data is one thing, keeping UTF-8 characters intact is something completely separate. What's to stop the byte sequence 0xe29c93 from being interpreted as ✓ when it shows up in your binary data?

According to the json_encode PHP reference page, you can use the following syntax to encode Unicode characters:

json_encode($data, JSON_UNESCAPED_UNICODE);

It should make it pass unicode characters through unescaped.

edited Aug 21, 2014 at 19:31

answered Aug 21, 2014 at 19:24

amphetamachine

31k12 gold badges68 silver badges74 bronze badges

4 Comments

mpen Over a year ago

Tried it already. Doesn't work: json_encode(chr(200),JSON_UNESCAPED_UNICODE) yields false.

mpen Over a year ago

re: "What's to stop..." I don't actually care if that shows up in my binary data. I just need it not break (return false) for data it can't handle.

amphetamachine Over a year ago

@Mark Then transfer it into an encoding it will always be able to handle. For example, base64 encode it.

mpen Over a year ago

That would make valid strings illegible. It's for logging. I want to be able to read the valid strings. I'll visually ignore any binary data.

Collectives™ on Stack Overflow

Unicode encode string

2 Answers 2

Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related