Importing an irregular shaped array into Python

Question

I have some data generated in Mathematica that I need imported into Python. The way the data is generated relies on symbolic calculations so simply generating it in Python is out of the question. The data is an array of dimensions (126,2) but, where the first position in each element is an integer, the second position is a list of lists and the dimensions are not constant from element to element, for example:

`
    {
    {-9,{{4,2},{5,6},{8,10}}},
    {-2,{{3,6},{6,1}}}
    {4,{{3,6},{6,1},{3,6},{6,1},{3,6},{6,1},{3,6},{6,1}}}
    }
`

would be the first three elements. The second position in each element will always a list of 2-D lists. The goal here is to have this data imported as a numpy array such that I can call each element, no matter it's position.

I had some success with numpy.genfromtxt("data.txt",delimiters="}}}") which gives me the correct shape (126,2) but each element is simply "nan".

I had some more success with

`
with open("data.csv") as csvfile:
     reader = csv.reader(csvfile, delimiter=' ')
     for element in reader:
         print(np.asarray(element)[0])
`

This gives me the integer values as an array, which is great! For the second position in each element i tried:

`
def replace_all(text, dic):
    for i, j in dic.items():
        text = text.replace(i, j)
    return text
d={"{":"[","}":"]"}
with open("spinweights.csv") as csvfile:
     reader = csv.reader(csvfile, delimiter=',')
     it=0
     for element in reader:
         while it<1:
             curlToSq=replace_all(str(element[1]),d)
             print(np.asarray(curlToSq))
`

Where the replace_all function is changing all curly brackets in square brackets (the thinking here was this would make it easier to convert into a numpy array). The final line there does return an array...of shape () with none of it's objects subscriptable, which is what I need!

Any help is appreciated.

hpaulj · Accepted Answer · 2019-03-09 00:36:15Z

I think converting this to a list structure will be easiest.

I added a , to the following:

In [22]: astr=""" { 
    ...:     {-9,{{4,2},{5,6},{8,10}}}, 
    ...:     {-2,{{3,6},{6,1}}}, 
    ...:     {4,{{3,6},{6,1},{3,6},{6,1},{3,6},{6,1},{3,6},{6,1}}} 
    ...:     }"""                                                               
In [23]: astr1=astr.replace('{','[').replace('}',']').replace('\n','')          
In [24]: astr1                                                                  
Out[24]: ' [    [-9,[[4,2],[5,6],[8,10]]],    [-2,[[3,6],[6,1]]],    [4,[[3,6],[6,1],[3,6],[6,1],[3,6],[6,1],[3,6],[6,1]]]    ]'

The are various string evaluators. eval is always available. ast is a bit safer. json.loads(astr1) also works.

In [25]: alist= eval(astr1)                                                     
In [26]: alist                                                                  
Out[26]: 
[[-9, [[4, 2], [5, 6], [8, 10]]],
 [-2, [[3, 6], [6, 1]]],
 [4, [[3, 6], [6, 1], [3, 6], [6, 1], [3, 6], [6, 1], [3, 6], [6, 1]]]]

If it must be an array, do:

In [27]: arr = np.array(alist)                                                  
In [28]: arr                                                                    
Out[28]: 
array([[-9, list([[4, 2], [5, 6], [8, 10]])],
       [-2, list([[3, 6], [6, 1]])],
       [4,
        list([[3, 6], [6, 1], [3, 6], [6, 1], [3, 6], [6, 1], [3, 6], [6, 1]])]],
      dtype=object)

This is (3,2). arr[:,0] is an array of integers, but arr[:,1] is an array of lists.

genfromtxt as a default tries to convert the input to floats (what it can convert becomes nan). It is intended for csv - a neat table of numbers of strings, with equal number of columns for each row.

Edmund · Accepted Answer · 2019-03-09 01:58:32Z

1

You may use Export with either "JSON" or "PythonExpression" to export the data to a file. Either of these can be directly read by Python.

Below I use ExportString to demonstrate.

With

x =
 {
  {-9, {{4, 2}, {5, 6}, {8, 10}}},
  {-2, {{3, 6}, {6, 1}}},
  {4, {{3, 6}, {6, 1}, {3, 6}, {6, 1}, {3, 6}, {6, 1}, {3, 6}, {6, 1}}}
  };

Then

ExportString[x, "JSON", "Compact" -> True]

[[-9,[[4,2],[5,6],[8,10]]],[-2,[[3,6],[6,1]]],[4,[[3,6],[6,1],[3,6],[6,1],[3,6],[6,1],[3,6],[6,1]]]]

or

ExportString[x, "PythonExpression"]

[[-9, [[4, 2], [5, 6], [8, 10]]], [-2, [[3, 6], [6, 1]]], [4, [[3, 6], [6, 1], [3, 6], [6, 1], [3, 6], [6, 1], [3, 6], [6, 1]]]]

Wolfram Language (a.k.a. Mathematica) can import and export many formats with the two above being some of the Basic Formats it supports. There is generally no need to custom code consumption of output as there is normally a format available that the other platform can naively (or has a library to) read.

Hope this helps.

answered Mar 9, 2019 at 1:58

Edmund

5385 silver badges23 bronze badges

2 Comments

Brendan Reid Over a year ago

This is very helpful! Unfortunately I'm still having issues getting it into Python as an array with subscriptable elements. I have it imported as a numpy nd array (type checked) but its shape is (). Any ideas? Copying and pasting would work but...

Edmund Over a year ago

@BrendanReid Perhaps try exporting in the "MAT" format and see if there is a Python library that will import that. Otherwise ask a question on how to import a JSON array into a Python array. You should get more answers to that question.

Collectives™ on Stack Overflow

Importing an irregular shaped array into Python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related