Pandas - if all values of dataFrame are NaN


math student

How to create an if statement that does the following:

 if all values in dataframe are nan:
     do something 
 else: 
     do something else

According to this post , it is possible to check if all values ​​of a DataFrame are NaN. I know that one cannot do:

if df.isnull().all():
    do something

It returns the following error:

ValueError: The truth value of the series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Jesler

Another one is needed allbecause another one is allreturned Seriesfirst scalar:

if df.isnull().all().all():
    do something

sample:

df = pd.DataFrame(index=range(5), columns=list('abcde'))
print (df)
     a    b    c    d    e
0  NaN  NaN  NaN  NaN  NaN
1  NaN  NaN  NaN  NaN  NaN
2  NaN  NaN  NaN  NaN  NaN
3  NaN  NaN  NaN  NaN  NaN
4  NaN  NaN  NaN  NaN  NaN

print (df.isnull())
      a     b     c     d     e
0  True  True  True  True  True
1  True  True  True  True  True
2  True  True  True  True  True
3  True  True  True  True  True
4  True  True  True  True  True

print (df.isnull().all())
a    True
b    True
c    True
d    True
e    True
dtype: bool

print (df.isnull().all().all())
True

if df.isnull().all().all():
    print ('do something')

If you need a faster solution - there are , but first convert all values :numpy.isnannumpy.allnumpy arrayvalues

print (np.isnan(df.values).all())
True

Time :

df = pd.DataFrame(np.full((1000,1000), np.nan))
print (df)

In [232]: %timeit (np.isnan(df.values).all())
1000 loops, best of 3: 1.23 ms per loop

In [233]: %timeit (df.isnull().all().all())
100 loops, best of 3: 10 ms per loop

In [234]: %timeit (df.isnull().values.all())
1000 loops, best of 3: 1.46 ms per loop

Related


Pandas - if all values of dataFrame are NaN

math student How to create an if statement that does the following: if all values in dataframe are nan: do something else: do something else According to this post , it is possible to check if all values of a DataFrame are NaN. I know that one c

Pandas - if all values of dataFrame are NaN

math student How to create an if statement that does the following: if all values in dataframe are nan: do something else: do something else According to this post , it is possible to check if all values of a DataFrame are NaN. I know that one c

Pandas - if all values of dataFrame are NaN

math student How to create an if statement that does the following: if all values in dataframe are nan: do something else: do something else According to this post , it is possible to check if all values of a DataFrame are NaN. I know that one c

Pandas - if all values of dataFrame are NaN

math student How to create an if statement that does the following: if all values in dataframe are nan: do something else: do something else According to this post , it is possible to check if all values of a DataFrame are NaN. I know that one c

Pandas - if all values of dataFrame are NaN

math student How to create an if statement that does the following: if all values in dataframe are nan: do something else: do something else According to this post , it is possible to check if all values of a DataFrame are NaN. I know that one c

pandas.DataFrame set all string values to nan

Fahawa I have a pandas.DataFrametype that contains strings, floats and ints. Is there a way to set all strings that cannot be converted to floats to NaN? E.g: A B C D 0 1 2 5 7 1 0 4 NaN 15 2 4 8 9 10 3 11 5 8 0 4

Pandas melt dataframe based on datetime and fill all values NaN

Tony Brand I have a dataframe with all countries and datetimes ranging from "1/22/20" to "2/22/20". Here is my dataframe column as shown below. Country 1/22/20 1/23/20 1/24/20 1/25/20 1/26/20 1/27/20 1/28/20 1/29/20 1/30/20... I try to fuse a dataframe to get

pandas.DataFrame set all string values to nan

Fahawa I have a pandas.DataFrametype that contains strings, floats and ints. Is there a way to set all strings that cannot be converted to floats to NaN? E.g: A B C D 0 1 2 5 7 1 0 4 NaN 15 2 4 8 9 10 3 11 5 8 0 4

pandas.DataFrame set all string values to nan

Fahawa I have a pandas.DataFrametype that contains strings, floats and ints. Is there a way to set all strings that cannot be converted to floats to NaN? E.g: A B C D 0 1 2 5 7 1 0 4 NaN 15 2 4 8 9 10 3 11 5 8 0 4

All NAN values when filtering Pandas DataFrame to one column

Taylor I am importing data from a .csv file which is stored in a dataframe. It looks good there: After that I try to store only one column of the dataframe elsewhere. However, it returns all NaN values: The exact same code works for the previous .xls file in t

pandas.DataFrame set all string values to nan

Fahawa I have one pandas.DataFramethat contains string, float and integer types. Is there a way to set all strings that cannot be converted to floats to NaN? E.g: A B C D 0 1 2 5 7 1 0 4 NaN 15 2 4 8 9 10 3 11 5 8

Pandas melt dataframe based on datetime and fill all values NaN

Tony Brand I have a dataframe with all countries and datetimes ranging from "1/22/20" to "2/22/20". Here is my dataframe column as shown below. Country 1/22/20 1/23/20 1/24/20 1/25/20 1/26/20 1/27/20 1/28/20 1/29/20 1/30/20... I try to fuse a dataframe to get

Pandas melt dataframe based on datetime and fill all values NaN

Tony Brand I have a dataframe with all countries and datetimes ranging from "1/22/20" to "2/22/20". Here is my dataframe column as shown below. Country 1/22/20 1/23/20 1/24/20 1/25/20 1/26/20 1/27/20 1/28/20 1/29/20 1/30/20... I try to fuse a dataframe to get

Pandas melt dataframe based on datetime and fill all values NaN

Tony Brand I have a dataframe with all countries and datetimes ranging from "1/22/20" to "2/22/20". Here is my dataframe column as shown below. Country 1/22/20 1/23/20 1/24/20 1/25/20 1/26/20 1/27/20 1/28/20 1/29/20 1/30/20... I try to fuse a dataframe to get

pandas.DataFrame set all string values to nan

Fahawa I have a pandas.DataFrametype that contains strings, floats and ints. Is there a way to set all strings that cannot be converted to floats to NaN? E.g: A B C D 0 1 2 5 7 1 0 4 NaN 15 2 4 8 9 10 3 11 5 8 0 4

pandas.DataFrame set all string values to nan

Fahawa I have a pandas.DataFrametype that contains strings, floats and ints. Is there a way to set all strings that cannot be converted to floats to NaN? E.g: A B C D 0 1 2 5 7 1 0 4 NaN 15 2 4 8 9 10 3 11 5 8 0 4

Pandas melt dataframe based on datetime and fill all values NaN

Tony Brand I have a dataframe with all countries and datetimes ranging from "1/22/20" to "2/22/20". Here is my dataframe column as shown below. Country 1/22/20 1/23/20 1/24/20 1/25/20 1/26/20 1/27/20 1/28/20 1/29/20 1/30/20... I try to fuse a dataframe to get

Replacing NaN values in dataframe with pandas

ho_howdy I want to create a function that takes a dataframe and replaces NaNs with patterns in categorical columns and NaNs in numeric columns with the mean of that column. If there are multiple modes in a categorical column, the first mode should be used. I m

Replacing NaN values in dataframe with pandas

ho_howdy I want to create a function that takes a dataframe and replaces NaNs with patterns in categorical columns and NaNs in numeric columns with the mean of that column. If there are multiple modes in a categorical column, the first mode should be used. I m

Replacing NaN values in dataframe with pandas

ho_howdy I want to create a function that takes a dataframe and replaces NaNs with patterns in categorical columns and NaNs in numeric columns with the mean of that column. If there are multiple modes in a categorical column, the first mode should be used. I m

Replacing NaN values in dataframe with pandas

ho_howdy I want to create a function that takes a dataframe and replaces NaNs with patterns in categorical columns and NaNs in numeric columns with the mean of that column. If there are multiple modes in a categorical column, the first mode should be used. I m

Replacing NaN values in dataframe with pandas

ho_howdy I want to create a function that takes a dataframe and replaces NaNs with patterns in categorical columns and NaNs in numeric columns with the mean of that column. If there are multiple modes in a categorical column, the first mode should be used. I m

Replacing NaN values in dataframe with pandas

ho_howdy I want to create a function that takes a dataframe and replaces NaNs with patterns in categorical columns and NaNs in numeric columns with the mean of that column. If there are multiple modes in a categorical column, the first mode should be used. I m