Convert string matrix to numpy

Question

I have matrices of arbitrary dimension, formatted similarly to the example below. They come from an external source and the formatting cannot be changed.

[[[1.65 0.53 0 1][0.99 1.41 0 1][0.38 1.37 0 1][0 0 1 1][1.10 0.69 0 1][0 0 1 1][0.60 1.21 0 1][0.99 1.04 0 1][1.86 1.20 0 1][0 0 1 1][1.66 0.68 0 1][0.96 0.75 0 1][0.86 0.80 0 1][1.13 0.97 0 1][1.86 1.48 0 1][0 0 1 1][0.71 1.10 0 1][1.43 0.58 0 1][1.34 0.63 0 1][1.37 1.45 0 1][0.36 1.08 0 1][0 0 1 1][0.60 1.18 0 1][1.08 0.64 0 1][0.99 0.58 0 1][1.57 1.16 0 1][0.87 1.39 0 1][0.48 1.21 0 1][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0]][[1.52 1.01 0 1][0.93 0.62 0 1][1.41 0.52 0 1][1.66 0.83 0 1][0 0 1 1][1.02 1.03 0 1][0.98 0.92 0 1][0 0 1 1][0.65 0.90 0 1][0 0 1 1][1.27 0.61 0 1][0.41 0.79 0 1][1.23 1.04 0 1][0.56 0.70 0 1][0 0 1 1][1.81 0.90 0 1][0 0 1 1][1.71 0.57 0 1][1.53 1.06 0 1][1.28 1.42 0 1][1.50 0.91 0 1][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0]][[0 0 1 1][0.53 1.17 0 1][0.24 0.54 0 1][1.88 0.68 0 1][0 0 1 1][1.33 0.68 0 1][0.32 0.55 0 1][1.28 0.73 0 1][0.49 1.13 0 1][1.45 1.28 0 1][0.66 1.47 0 1][0 0 1 1][0.76 1.10 0 1][1.95 0.78 0 1][0 0 1 1][0.56 0.61 0 1][0.84 1.05 0 1][1.07 0.59 0 1][1.79 0.95 0 1][1.93 1.02 0 1][1.93 1.16 0 1][0 0 1 1][0.55 0.58 0 1][0.29 1.13 0 1][1.46 0.50 0 1][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0]][[1.71 0.50 0 1][0.70 1.35 0 1][0 0 1 1][0.90 0.83 0 1][1.81 0.97 0 1][1.64 1.35 0 1][1.21 1.15 0 1][0.54 0.50 0 1][0 0 1 1][0.62 0.72 0 1][0.86 1.38 0 1][0 0 1 1][1.76 1.15 0 1][1.83 1.43 0 1][0.20 0.51 0 1][0.81 0.65 0 1][0 0 1 1][0.51 0.79 0 1][1.09 1.43 0 1][1.65 1.03 0 1][1.47 1.49 0 1][0 0 1 1][1.57 0.97 0 1][0.99 0.93 0 1][1.82 0.66 0 1][1.84 1.01 0 1][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0]][[0 0 1 1][1.36 0.94 0 1][1.61 0.64 0 1][0.99 1.03 0 1][1.43 1.12 0 1][1.09 1.16 0 1][0.40 1.40 0 1][0 0 1 1][0.86 0.56 0 1][0.54 0.80 0 1][0.77 1.04 0 1][0 0 1 1][1.38 0.61 0 1][0.37 1.38 0 1][1.12 1.28 0 1][0 0 1 1][1.87 0.67 0 1][1.75 0.52 0 1][0.31 0.52 0 1][0.99 0.88 0 1][0 0 1 1][1.38 1.30 0 1][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0][0 0 0 0]]]

How can I convert this into NumPy form? This answer suggests using fromstring; however, the documentation says this only works for 1D matrices.

In the future, try to use a more portable string format. As you can see from the answers this style requires editing. The string format of arr.tolist() is better. csv format is a good portable format for 2d arrays. np.save/load is best for general arrays. — hpaulj
– hpaulj, Commented Nov 8, 2019 at 2:41
@hpaulj: I do not have control over the format and anticipate that my arrays, like the one in the example I show, will often have >2 dimensions. — Richard
– Richard, Commented Nov 8, 2019 at 3:59
If the arrays are large enough, that print display will be truncated, making recovery impossible. — hpaulj
– hpaulj, Commented Nov 8, 2019 at 4:13
@hpaulj: You may be working under the belief that the matrices are being pretty-printed by NumPy. That is not so. Regardless, this is the form the matrices I want to process are in and it cannot be changed. There's no need to suggest otherwise or to imply limitations which may not be relevant. — Richard
– Richard, Commented Nov 8, 2019 at 4:17

Max Kaha · Accepted Answer · 2019-11-07 22:24:07Z

2

Depends, if it is already a list you could just use np.array() to convert it directly, if it is a string you probably need to insert , first to make it valid Python list. Here would be an example incase it is a string

import ast
import numpy as np

myString = "<YOUR STRING ABOVE>"
myString = myString.replace(" ", ",") # Replace [0 0 0 0] with [0,0,0,0]
myString = myString.replace("][", "],[") # Replace [0,0,0,0][0,0,0,0] with [0,0,0,0],[0,0,0,0] 
myList = ast.literal_eval(s) # Turn string into a list
myArr = np.array(myList) # Turn list into np_array

Hope it helps, if your data already is a list you can just skip straight to np.array(myList)

Dimensions of the array I created from the data above:

myArr.shape
(5, 33, 4)

Edit: Changed eval() to ast.literal_eval as suggested by @b_c

edited Nov 7, 2019 at 22:24

answered Nov 7, 2019 at 22:14

Max Kaha

9325 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

b_c Over a year ago

You should use ast.literal_eval instead of eval.

jsbueno Over a year ago

ast.literal.eval is good but won't make much of a difference if the data is hardcoded - whoever, yes. I will change my answer to use that.

Max Kaha Over a year ago

@b_c I have edited my answer, thanks for the suggestion.

Richard · Accepted Answer · 2019-11-07 22:35:33Z

0

I ended up with this:

import ast
import numpy as np

def StringToMatrix(txtmat):
    txtmat = txtmat.replace(" ",",").replace("][","],[")
    try:
        ret = np.array(ast.literal_eval(txtmat))
    except:
        ret = None
    return ret

answered Nov 7, 2019 at 22:35

Richard

62.8k40 gold badges198 silver badges277 bronze badges

Comments

jsbueno · Accepted Answer · 2019-11-07 22:59:28Z

Without line breaks, that is anything but "pretty".

I think the fastest way to go there is to use regular expressions search/replace to add a , between space-separated digits and between ][. this however won't be very smart and would break on corner cases (like numbers ending in .), so you may have to fine-tune the idea:

Now, on trying it here, the old saying proves itself once more as correct: "if you have one problem that needs regular expressions, you have two problems" (unknown author).

The problem is that as some of the numbers are composed by a single digit, when this number is matched for the comma placing for the number before it, it is not available for the regex engine to match it before the number after it.

So we have to go after "look ahead matches" and "look behind matches" using the (?=...) syntax - which enables one to match just the place where we want to add the ",".

after that you have a string where you can use "eval" to have a nested list structure which can be passed directly to numpy.array.

import numpy as np
import re
from ast import literal_eval

b = re.sub(r"((?<=\d)\s+(?=\d)|(?<=\])\s*?(?=\[))", ", " , a) 
c = np.array(literal_eval(b))

Of course, if you always have one space between numbers, and no-space or breaks between "][", simple string replacing, with no regexps is much easier to go. Use the regexp if you have loose spacing in your input data.

Collectives™ on Stack Overflow

Convert string matrix to numpy

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related