Strange basic computation issue with numpy.int64 and/or numpy.float64?

Question

I am getting a very weird and unexpected ERROR from python-numpy

I am working with the following libraries:

import os, glob, string, math, csv, json
import datetime as dt
import numpy as np
import scipy as sci
import pandas as pd
import matplotlib.pyplot as plt
import feedparser as fp
import cPickle as pickle
import networkx as nx
from urllib2 import urlopen
import statsmodels.formula.api as sm
import patsy

The following code:

n,k = 2643605051, 648128.068241
print n,type(n)
print k, type(k)
nkvar = (k + 1)*(n + 2)/( (n+2) * (n+1)**2 )
print nkvar

n = np.int64(n)
k = np.float64(k)
print n,type(n)
print k, type(k)
nkvar = (k + 1)*(n + 2)/( (n+2) * (n+1)**2 )
print nkvar

Yields:

2643605051 <type 'int'>
648128.068241 <type 'float'>
9.27402694708e-14
2643605051 <type 'numpy.int64'>
648128.068241 <type 'numpy.float64'>
-0.00383719008751

The second answer is OBVIOUSLY wrong! Could someone please help me understand what is going on?

Ok, sorry for blaming pandas. The issue seems to be with numpy! Here is an example: n,k = 2643605051, 648128.068241 print n,type(n) print k, type(k) nkvar = (k + 1)*(n + 2)/( (n+2) * (n+1)**2 ) print nkvar n = np.int64(n) k = np.float64(k) print n,type(n) print k, type(k) nkvar = (k + 1)*(n + 2)/( (n+2) * (n+1)**2 ) print nkvar which yields: 2643605051 <type 'int'> 648128.068241 <type 'float'> 9.27402694708e-14 2643605051 <type 'numpy.int64'> 648128.068241 <type 'numpy.float64'> -0.00383719008751 — tipanverella
– tipanverella, Commented Sep 11, 2013 at 18:16
Can you put that example in your question? It's near impossible to follow in a comment. — Phillip Cloud
– Phillip Cloud, Commented Sep 11, 2013 at 18:18
int64 is 64-bit. Operations on it are restricted to 64 bits. int produces arbitrary-precision longs if the result doesn't fit into an int. — user2357112
– user2357112, Commented Sep 11, 2013 at 18:30
is there a way to set the default precision to int64 and float128 in NUMPY? — tipanverella
– tipanverella, Commented Sep 11, 2013 at 18:36
Is there a reason that you haven't canceled the common factor (n+2) in the numerator and denominator of your expression? — Warren Weckesser
– Warren Weckesser, Commented Sep 12, 2013 at 2:28

unutbu · Accepted Answer · 2013-09-11 18:46:09Z

4

You are suffering from arithmetic overflow. With NumPy, for the sake of speed, most operations do not check for arithmetic overflow. The onus is on you to choose the proper dtype to avoid overflow.

import numpy as np

n,k = 2643605051, 648128.068241
nkvar = (k + 1)*(n + 2)/((n+1)**2 * (n+2))
print "In foo nkvar = ", nkvar, "  from (n,k) = ", (n,k)
# In foo nkvar =  9.27402694708e-14   from (n,k) =  (2643605051L, 648128.068241)       

n,k = np.int64(2643605051), np.float32(648128.068241)
nkvar = (k + 1)*(n + 2)/((n+1)**2 * (n+2))
print "In foo nkvar = ", nkvar, "  from (n,k) = ", (n,k)
# In foo nkvar =  -0.00383719005352   from (n,k) =  (2643605051, 648128.06)

A workaround: Since there is no NumPy integer dtype large enough to perform the computation without overflow, you'll need to convert n to a Python int first:

n = int(w.sum())

Another alternative is to change the dtype of n to float64:

n,k = np.float64(2643605051), np.float64(648128.068241)
nkvar = (k + 1)*(n + 2)/((n+1)**2 * (n+2))
print "In foo nkvar = ", nkvar, "  from (n,k) = ", (n,k)
# In foo nkvar =  9.27402694708e-14   from (n,k) =  (2643605051.0, 648128.06824099994)

edited Sep 11, 2013 at 18:46

answered Sep 11, 2013 at 18:20

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

tipanverella Over a year ago

is there a way to set the default precision to int64 and float128 in NUMPY?

unutbu Over a year ago

When you define your arrays, you can supply the desired dtype. Or, after the fact, the dtype can be changed with arr = arr.astype('int64'). However, even int64 is not big enough to avoid overflow in this case.

Collectives™ on Stack Overflow

Strange basic computation issue with numpy.int64 and/or numpy.float64?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related