Removing invalid characters in JavaScript

Question

Can someone provide a regular expression to search and replace illegal characters found

Example, removing �

I am not sure how many types of 'illegal' characters exist but I think this will be a good start.

Many thanks

edit - I have no control over the data, we're trying to create a catch for the potentially bad data we're receiving.

I think first you should see why they're getting there. What's the encoding? — Vivin Paliath
– Vivin Paliath, Commented Oct 5, 2012 at 21:29
I think it may be better to include only those characters which are legal, which is probably really easy. Then again I don't know how many characters are legal to you. — Isaac Fife
– Isaac Fife, Commented Oct 5, 2012 at 21:30
We're receiving bad data, trying to push for the vendor to make sure the strings are encoded correctly, but we're trying to setup a catch for it. — Chris
– Chris, Commented Oct 5, 2012 at 21:30
I'd recommend only removing the characters that the string decoder throws up, which are replaced with 0xFFFD as I suggested below. — saml
– saml, Commented Oct 5, 2012 at 21:55

saml · Accepted Answer · 2012-10-06 00:53:29Z

35

Invalid characters get converted to 0xFFFD on parsing, so any invalid character codes would get replaced with:

myString = myString.replace(/\uFFFD/g, '')

You can get all types of invalid sorts of chars here

edited Oct 6, 2012 at 0:53

answered Oct 5, 2012 at 21:31

saml

6,8321 gold badge37 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Chris Over a year ago

Thank you for the info, I won't be able to check again for a few days but this didn't work on the first attempt. I think this is the way forward though so I'll just check to see whether we implemented it correctly, it's hardly a lot of code :-)

saml Over a year ago

did you reassign the string? replace isn't destructive, so you need to reassign the replaced string.

Chris Over a year ago

Yes it was a return myString.replace(/\uFFFD/g, '') we'll re-review it during the working week I wouldn't be surprised if something was overlooked

Javier González de Lope Over a year ago

In my case, I had to replace the \uFFFD chars from my string, not the unicode symbol. So I solved the problem by using myString.replace(/\\uFFFD/g, '')

allyourcode · Accepted Answer · 2012-10-06 01:56:16Z

21

Instead of having a blacklist, you could use a whitelist. e.g. If you want to only accept letters, numbers, space, and a few punctuation characters, you could do

myString.replace(/[^a-z0-9 ,.?!]/ig, '')

answered Oct 6, 2012 at 1:56

allyourcode

22.8k20 gold badges86 silver badges108 bronze badges

3 Comments

saml Over a year ago

An invalid character in this context is clearly malformed UTF-8, not non-ASCII.

allyourcode Over a year ago

You're reading more into the question than what's actually stated. The OP may be having a problem with encodings, but that's not what his question actually says.

saml Over a year ago

I don't think I am: "Example, removing �" I think he makes it pretty clear the types of invalid characters he wants to remove are invalid characters in the sense that a string decoder can't, not that he doesn't like them. As I previously stated, limiting valid utf-8 to ascii is appalling advice.

Kapil Thakkar · Accepted Answer · 2018-02-14 13:27:35Z

4

Try this, it will work for all unexpected character like ♫ ◘ etc...

dataStr.replace(/[\u{0080}-\u{FFFF}]/gu,"");

answered Feb 14, 2018 at 13:27

Kapil Thakkar

87010 silver badges18 bronze badges

1 Comment

Cesar Over a year ago

That replaces accents

Collectives™ on Stack Overflow

Removing invalid characters in JavaScript

3 Answers 3

4 Comments

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related