Set _FillValue attribute with its default value

[dopplershift]

From bstde...@gmail.com on July 22, 2011 12:08:31

If the 'fill_value' keyword is not used when creating a variable, the default NetCDF _FillValue is used, but the _FillValue attribute is not explicitely written to the NetCDF file (and it can't be written to afterwards). I would like to have the ability to make the _FillValue attribute appear with its default value.

Is there any way I can get the default NetCDF _FillValue for a given data type?

e.g.:
ncdtype = 'i2'
x = # some function that returns the default NetCDF _FillValue of ncdtype
nc.createVariable(...,dtype=ncdtype,fill_value=x)

I assume that would output the _FillValue attribute to the NetCDF file, even if it seems redundant... Perhaps there is some other way to achieve this?

Original issue: http://code.google.com/p/netcdf4-python/issues/detail?id=94

[dopplershift]

From whitaker.jeffrey@gmail.com on July 22, 2011 12:54:19

The default fill values are in netcdf.h as described here http://mailman.unidata.ucar.edu/software/netcdf/docs/netcdf-c/Fill-Values.html The netcdf library does not create a user visible attribute unless the default is over-ridden. You can get them from python if you know where to look

[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import netCDF4
netCDF4._default_fillvals
{'i8': -9223372036854775806L, 'f4': 9.969209968386869e+36, 'u8': 18446744073709551614L, 'i1': -127, 'u4': 4294967295L, 'S1': '\x00', 'i2': -32767, 'u1': 255, 'i4': -2147483647, 'u2': 65535, 'f8': 9.969209968386869e+36}

I can easily make a function to retrieve these, if you think it would be useful.

[dopplershift]

From bstde...@gmail.com on July 22, 2011 13:30:14

netCDF4._default_fillvals is exactly what I need.

Maybe mention it in the createVariable documentation for people like me that are not good at searching!
e.g.
The optional keyword fill_value can be used to override the default netCDF _FillValue (NetCDF4._default_fillvals, the value that the variable gets filled with before any data is written to it). If fill_value is set to False, then the variable is not pre-filled.

Thanks

[dopplershift]

From whitaker.jeffrey@gmail.com on July 22, 2011 14:14:57

It's already there in the docstring for createVariable:

fill_value - If specified, the default netCDF _FillValue (the value that the variable gets filled with before any data is written to it) is replaced with this value. If fill_value is set to False, then the variable is not pre-filled.

[dopplershift]

From whitaker.jeffrey@gmail.com on July 22, 2011 14:17:39

Oh, I see you mean I should mention _default_fillvals in the docstring. Good idea.

[dopplershift]

From chaoyue...@gmail.com on September 15, 2012 09:11:23

Then I am confused by how netCDF module determines the "fill_value" attribute of numpy (masked) arrays which are retrieved by rootgrp.variables[varname][:].

I tried a couple of tests:

1. when I write the nc file, I set the "missing_value" or "_FillValue" attribute of variables by using "setncattr" (without using fill_value= argument within the creatVariables function) to a value other than 9.96921e+36 (default float _FillValue). But when I read variable value to a numpy masked array, the "fill_value" attribute is still 9.96921e+36. In this case, when I use ncdump to check the data, it reads:

float var(lat, lon) ;
    var:missing_value = 1.e+20f ;
    var:_FillValue = 1.e+20f ;

}

1. when I write the nc file, I use the "fill_value" argument within the creatVariable function.(rootgrp.createVariable('var','f4',dimensions=['lat','lon'],fill_value=0.005)). Then the "_FillValue" attribute automatically appears when I use ncdump to check the data. and when I read the variable value into a numpy masked array, the "fill_value" of masked array is 0.005.
   the output of ncdump -h:

   float var(lat, lon) ;
   var:_FillValue = 0.005f ;
   }

My point is, in both cases, the result (and format) of _FillValue by ncdump is the same by only differing in the values, but in the first case, the 1.e+20f is not taken by the netCDF4 when reading the data, but in the second case, it really takes the value as specified in the "_FillValue" attribute.

[dopplershift]

From whitaker.jeffrey@gmail.com on September 15, 2012 09:23:25

You have to set the fill value at variable creation (using the fill_value keyword to createVariable), as in your second example. The underscore in front means that it is a private attribute, that should not be modified by the user. It works this way in all netCDF clients.

[dopplershift]

From whitaker.jeffrey@gmail.com on September 15, 2012 09:26:19

You have to set the fill value at variable creation (using the fill_value keyword to createVariable), as in your second example. The underscore in front means that it is a private attribute, that should not be modified by the user. It works this way in all netCDF clients.

Note that missing_value and _FillValue are very different. _FillValue is used by the C library to fill in values in variables that have not yet been written to. missing_value is meant to represent values set by the user to represent data points that do not exist.

[dopplershift]

From chaoyue...@gmail.com on September 20, 2012 12:45:55

Thanks Jeff. It's strange I didn't receive email prompt when you replied. It's more clear now.

[dopplershift]

From whitaker.jeffrey@gmail.com on February 25, 2014 18:04:10

Status: Fixed

[gerritholl]

In case someone else is looking for this like me:

Since version 0.9.7:

• changed netCDF4._default_fillvals to netCDF4.default_fillvals (to make part
  of public API). Added to docs (issue 94).

which is documented with the Dataset.create_variable static method:

The optional keyword fill_value can be used to override the default netCDF _FillValue (the value that the variable gets filled with before any data is written to it, defaults given in netCDF4.default_fillvals). If fill_value is set to False, then the variable is not pre-filled.

This was fixed in commit 36f6544.

[lukemorrill]

Just another update on the url for Dataset.create_variable. It is now at: https://unidata.github.io/netcdf4-python/#Dataset.createVariable.