Skip to content

Commit 520d135

Browse files
trexfeathersbjlittlepp-mostephenworsleytkknight
authored
Mesh API Merge-Back (#4027)
* add ugrid mesh-api stubs (#4001) * add additional mesh stubs (#4005) * Update mesh-data-model branch (#4009) (#4011) * Add abstract cube summary (#3987) Co-authored-by: stephen.worsley <[email protected]> * add nox session conda list (#3990) * Added text to state the Python version used to build the docs. (#3989) * Added text to state the Python version used to build the docs. * Added footer template that includes the Python version used to build. * added new line * Review actions * added whatsnew * Iris py38 (#3976) * support for py38 * update CI and noxfile * enforce alphabetical xml element attribute order * full tests for py38 + fix docs-tests * add whatsnew entry * update doc-strings + review actions * Alternate xml handling routine (#29) * all xml tests pass for nox tests-3.8 * restored docstrings * move sort_xml_attrs * make sort_xml_attrs a classmethod * update sort_xml_attr doc-string Co-authored-by: Bill Little <[email protected]> * add jamesp to whatsnew + minor tweak Co-authored-by: James Penn <[email protected]> * normalise version to implicit development release number (#3991) * Gallery: update COP maps example (#3934) * update cop maps example * comment tweaks * minor comment tweak + whatsnew * reinstate whatsnew addition * remove duplicate whatsnew * don't support mpl v1.2 (#3941) * Cubesummary tidy (#3988) * Extra tests; fix for array attributes. * Docstring for CubeSummary, and remove some unused parts. * Fix section name capitalisation, in line with existing cube summary. * Handle array differences; quote strings in extras and if 'awkward'-printing. * Ensure scalar string coord 'content' prints on one line. * update intersphinx mapping and matplotlib urls (#4003) * update intersphinx mapping and matplotlib urls * use matplotlib intersphinx where possible * review actions * review actions * update readme badges (#4004) * update readme badges * pimp twitter badge * update readme logo img src and href (#4006) * update setuptools description (#4008) Co-authored-by: Patrick Peglar <[email protected]> Co-authored-by: stephen.worsley <[email protected]> Co-authored-by: tkknight <[email protected]> Co-authored-by: James Penn <[email protected]> Co-authored-by: Ruth Comer <[email protected]> Co-authored-by: Patrick Peglar <[email protected]> Co-authored-by: stephen.worsley <[email protected]> Co-authored-by: tkknight <[email protected]> Co-authored-by: James Penn <[email protected]> Co-authored-by: Ruth Comer <[email protected]> * MeshMetadata class. (#4002) * MeshMetadata class. * MeshMetadata extra members for dim names. * Comment for BaseMetadata refactoring. * add meshmetadata services (#4012) * Mesh api coord manager (#4015) * add mesh coordinate manager * wip * make shape methods private + reorganise method order * review actions * partial mesh * wip * Mesh data model to ng vat mesh api (#4023) * Update mesh-data-model branch (#4009) * Add abstract cube summary (#3987) Co-authored-by: stephen.worsley <[email protected]> * add nox session conda list (#3990) * Added text to state the Python version used to build the docs. (#3989) * Added text to state the Python version used to build the docs. * Added footer template that includes the Python version used to build. * added new line * Review actions * added whatsnew * Iris py38 (#3976) * support for py38 * update CI and noxfile * enforce alphabetical xml element attribute order * full tests for py38 + fix docs-tests * add whatsnew entry * update doc-strings + review actions * Alternate xml handling routine (#29) * all xml tests pass for nox tests-3.8 * restored docstrings * move sort_xml_attrs * make sort_xml_attrs a classmethod * update sort_xml_attr doc-string Co-authored-by: Bill Little <[email protected]> * add jamesp to whatsnew + minor tweak Co-authored-by: James Penn <[email protected]> * normalise version to implicit development release number (#3991) * Gallery: update COP maps example (#3934) * update cop maps example * comment tweaks * minor comment tweak + whatsnew * reinstate whatsnew addition * remove duplicate whatsnew * don't support mpl v1.2 (#3941) * Cubesummary tidy (#3988) * Extra tests; fix for array attributes. * Docstring for CubeSummary, and remove some unused parts. * Fix section name capitalisation, in line with existing cube summary. * Handle array differences; quote strings in extras and if 'awkward'-printing. * Ensure scalar string coord 'content' prints on one line. * update intersphinx mapping and matplotlib urls (#4003) * update intersphinx mapping and matplotlib urls * use matplotlib intersphinx where possible * review actions * review actions * update readme badges (#4004) * update readme badges * pimp twitter badge * update readme logo img src and href (#4006) * update setuptools description (#4008) Co-authored-by: Patrick Peglar <[email protected]> Co-authored-by: stephen.worsley <[email protected]> Co-authored-by: tkknight <[email protected]> Co-authored-by: James Penn <[email protected]> Co-authored-by: Ruth Comer <[email protected]> * Master to mesh data model (#4022) * Add abstract cube summary (#3987) Co-authored-by: stephen.worsley <[email protected]> * add nox session conda list (#3990) * Added text to state the Python version used to build the docs. (#3989) * Added text to state the Python version used to build the docs. * Added footer template that includes the Python version used to build. * added new line * Review actions * added whatsnew * Iris py38 (#3976) * support for py38 * update CI and noxfile * enforce alphabetical xml element attribute order * full tests for py38 + fix docs-tests * add whatsnew entry * update doc-strings + review actions * Alternate xml handling routine (#29) * all xml tests pass for nox tests-3.8 * restored docstrings * move sort_xml_attrs * make sort_xml_attrs a classmethod * update sort_xml_attr doc-string Co-authored-by: Bill Little <[email protected]> * add jamesp to whatsnew + minor tweak Co-authored-by: James Penn <[email protected]> * normalise version to implicit development release number (#3991) * Gallery: update COP maps example (#3934) * update cop maps example * comment tweaks * minor comment tweak + whatsnew * reinstate whatsnew addition * remove duplicate whatsnew * don't support mpl v1.2 (#3941) * Cubesummary tidy (#3988) * Extra tests; fix for array attributes. * Docstring for CubeSummary, and remove some unused parts. * Fix section name capitalisation, in line with existing cube summary. * Handle array differences; quote strings in extras and if 'awkward'-printing. * Ensure scalar string coord 'content' prints on one line. * update intersphinx mapping and matplotlib urls (#4003) * update intersphinx mapping and matplotlib urls * use matplotlib intersphinx where possible * review actions * review actions * update readme badges (#4004) * update readme badges * pimp twitter badge * update readme logo img src and href (#4006) * update setuptools description (#4008) * cirrus-ci compute credits (#4007) * update release process (#4010) * Stop using deprecated aliases of builtin types (#3997) * Stopped using deprecated aliases of builtin types. This is required to avoid warnings starting with NumPy 1.20.0. * Update lib/iris/tests/test_cell.py Co-authored-by: Bill Little <[email protected]> * Update lib/iris/tests/test_cell.py Co-authored-by: Bill Little <[email protected]> * Updated whatsnew. Co-authored-by: Bill Little <[email protected]> * celebrate first time iris contributors (#4013) * Docs unreleased banner (#3999) * baseline * removed debug comments * reverted * remove line * Testing * testing extensions * testing rtd_version * fixed if * removed line * tidy up * tidy comments * debug of pre-existing rtd variables * added reminder * testing * testing still * updated comments * added whatsnew * expanded the if conditiion * review actions * Update layout.html Remove alternative banner that used the RestructuredText notation. * review actions * drop __unicode__ method usage (#4018) * cirrus-ci conditional tasks (#4019) * cirrus-ci conditional tasks * use bc for bash arithmetic * revert back to sed * use expr * reword * minor documentation changes * review actions * make iris.common.metadata._hexdigest public (#4020) Co-authored-by: Patrick Peglar <[email protected]> Co-authored-by: stephen.worsley <[email protected]> Co-authored-by: tkknight <[email protected]> Co-authored-by: James Penn <[email protected]> Co-authored-by: Ruth Comer <[email protected]> Co-authored-by: Alexander Kuhn-Regnier <[email protected]> Co-authored-by: Patrick Peglar <[email protected]> Co-authored-by: stephen.worsley <[email protected]> Co-authored-by: tkknight <[email protected]> Co-authored-by: James Penn <[email protected]> Co-authored-by: Ruth Comer <[email protected]> Co-authored-by: Alexander Kuhn-Regnier <[email protected]> * Connectivity manager (#4017) * ConnectivityManager first pass. * ConnectivityManager align with proposed CoordManager. * Connectivity Manager review actions. * Connectivity Manager more review changes. * Use metadata_manager for Mesh location dimension. * Mesh dimension name abstraction. * Align Cooord and Connectivity Managers filters methods. * Completed Mesh class. * filter_cf improvements. * Moved filter_cf. * Mesh connectivity manager namedtuples comment. * Mesh removed trailing underscores. * Mesh _set_dimension_names improvements. * Mesh import rationalisation. * Mesh connectivity manager remove NDIM. * Connectivity manager use lazy indices_by_src(). * Connectivity manager clearer removal syntax. * Connectivity manager don't override __init__. * Connectivity manager correct base class syntax. * Metadata filter hexdigest reference fix. * test_MeshMetadata fix. * Rename filter to metadata_filter. * minor fixes (#4025) * minor fixes * wip * add mesh pickle support (#4026) Co-authored-by: Bill Little <[email protected]> Co-authored-by: Patrick Peglar <[email protected]> Co-authored-by: stephen.worsley <[email protected]> Co-authored-by: tkknight <[email protected]> Co-authored-by: James Penn <[email protected]> Co-authored-by: Ruth Comer <[email protected]> Co-authored-by: Alexander Kuhn-Regnier <[email protected]>
1 parent 59fa3f1 commit 520d135

File tree

7 files changed

+2525
-135
lines changed

7 files changed

+2525
-135
lines changed

docs/src/userguide/cube_statistics.rst

Lines changed: 38 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,9 @@ Collapsing Entire Data Dimensions
2323

2424
In the :doc:`subsetting_a_cube` section we saw how to extract a subset of a
2525
cube in order to reduce either its dimensionality or its resolution.
26-
Instead of simply extracting a sub-region of the data,
27-
we can produce statistical functions of the data values
28-
across a particular dimension,
26+
Instead of simply extracting a sub-region of the data,
27+
we can produce statistical functions of the data values
28+
across a particular dimension,
2929
such as a 'mean over time' or 'minimum over latitude'.
3030

3131
.. _cube-statistics_forecast_printout:
@@ -57,9 +57,9 @@ For instance, suppose we have a cube:
5757
um_version: 7.3
5858

5959

60-
In this case we have a 4 dimensional cube;
61-
to mean the vertical (z) dimension down to a single valued extent
62-
we can pass the coordinate name and the aggregation definition to the
60+
In this case we have a 4 dimensional cube;
61+
to mean the vertical (z) dimension down to a single valued extent
62+
we can pass the coordinate name and the aggregation definition to the
6363
:meth:`Cube.collapsed() <iris.cube.Cube.collapsed>` method:
6464

6565
>>> import iris.analysis
@@ -88,8 +88,8 @@ we can pass the coordinate name and the aggregation definition to the
8888
mean: model_level_number
8989

9090

91-
Similarly other analysis operators such as ``MAX``, ``MIN`` and ``STD_DEV``
92-
can be used instead of ``MEAN``, see :mod:`iris.analysis` for a full list
91+
Similarly other analysis operators such as ``MAX``, ``MIN`` and ``STD_DEV``
92+
can be used instead of ``MEAN``, see :mod:`iris.analysis` for a full list
9393
of currently supported operators.
9494

9595
For an example of using this functionality, the
@@ -103,14 +103,14 @@ in the gallery takes a zonal mean of an ``XYT`` cube by using the
103103
Area Averaging
104104
^^^^^^^^^^^^^^
105105

106-
Some operators support additional keywords to the ``cube.collapsed`` method.
107-
For example, :func:`iris.analysis.MEAN <iris.analysis.MEAN>` supports
108-
a weights keyword which can be combined with
106+
Some operators support additional keywords to the ``cube.collapsed`` method.
107+
For example, :func:`iris.analysis.MEAN <iris.analysis.MEAN>` supports
108+
a weights keyword which can be combined with
109109
:func:`iris.analysis.cartography.area_weights` to calculate an area average.
110110

111-
Let's use the same data as was loaded in the previous example.
112-
Since ``grid_latitude`` and ``grid_longitude`` were both point coordinates
113-
we must guess bound positions for them
111+
Let's use the same data as was loaded in the previous example.
112+
Since ``grid_latitude`` and ``grid_longitude`` were both point coordinates
113+
we must guess bound positions for them
114114
in order to calculate the area of the grid boxes::
115115

116116
import iris.analysis.cartography
@@ -155,24 +155,24 @@ including an example on taking a :ref:`global area-weighted mean
155155
Partially Reducing Data Dimensions
156156
----------------------------------
157157

158-
Instead of completely collapsing a dimension, other methods can be applied
159-
to reduce or filter the number of data points of a particular dimension.
158+
Instead of completely collapsing a dimension, other methods can be applied
159+
to reduce or filter the number of data points of a particular dimension.
160160

161161

162162
Aggregation of Grouped Data
163163
^^^^^^^^^^^^^^^^^^^^^^^^^^^
164164

165-
The :meth:`Cube.aggregated_by <iris.cube.Cube.aggregated_by>` operation
166-
combines data for all points with the same value of a given coordinate.
167-
To do this, you need a coordinate whose points take on only a limited set
168-
of different values -- the *number* of these then determines the size of the
165+
The :meth:`Cube.aggregated_by <iris.cube.Cube.aggregated_by>` operation
166+
combines data for all points with the same value of a given coordinate.
167+
To do this, you need a coordinate whose points take on only a limited set
168+
of different values -- the *number* of these then determines the size of the
169169
reduced dimension.
170-
The :mod:`iris.coord_categorisation` module can be used to make such
171-
'categorical' coordinates out of ordinary ones: The most common use is
172-
to aggregate data over regular *time intervals*,
170+
The :mod:`iris.coord_categorisation` module can be used to make such
171+
'categorical' coordinates out of ordinary ones: The most common use is
172+
to aggregate data over regular *time intervals*,
173173
such as by calendar month or day of the week.
174174

175-
For example, let's create two new coordinates on the cube
175+
For example, let's create two new coordinates on the cube
176176
to represent the climatological seasons and the season year respectively::
177177

178178
import iris
@@ -188,8 +188,8 @@ to represent the climatological seasons and the season year respectively::
188188

189189
.. note::
190190

191-
The 'season year' is not the same as year number, because (e.g.) the months
192-
Dec11, Jan12 + Feb12 all belong to 'DJF-12'.
191+
The 'season year' is not the same as year number, because (e.g.) the months
192+
Dec11, Jan12 + Feb12 all belong to 'DJF-12'.
193193
See :meth:`iris.coord_categorisation.add_season_year`.
194194

195195

@@ -206,10 +206,10 @@ to represent the climatological seasons and the season year respectively::
206206
iris.coord_categorisation.add_season_year(cube, 'time', name='season_year')
207207

208208
annual_seasonal_mean = cube.aggregated_by(
209-
['clim_season', 'season_year'],
209+
['clim_season', 'season_year'],
210210
iris.analysis.MEAN)
211211

212-
212+
213213
Printing this cube now shows that two extra coordinates exist on the cube:
214214

215215
.. doctest:: aggregation
@@ -238,20 +238,20 @@ These two coordinates can now be used to aggregate by season and climate-year:
238238
.. doctest:: aggregation
239239

240240
>>> annual_seasonal_mean = cube.aggregated_by(
241-
... ['clim_season', 'season_year'],
241+
... ['clim_season', 'season_year'],
242242
... iris.analysis.MEAN)
243243
>>> print(repr(annual_seasonal_mean))
244244
<iris 'Cube' of surface_temperature / (K) (time: 19; latitude: 18; longitude: 432)>
245-
246-
The primary change in the cube is that the cube's data has been
247-
reduced in the 'time' dimension by aggregation (taking means, in this case).
248-
This has collected together all data points with the same values of season and
245+
246+
The primary change in the cube is that the cube's data has been
247+
reduced in the 'time' dimension by aggregation (taking means, in this case).
248+
This has collected together all data points with the same values of season and
249249
season-year.
250250
The results are now indexed by the 19 different possible values of season and
251251
season-year in a new, reduced 'time' dimension.
252252

253-
We can see this by printing the first 10 values of season+year
254-
from the original cube: These points are individual months,
253+
We can see this by printing the first 10 values of season+year
254+
from the original cube: These points are individual months,
255255
so adjacent ones are often in the same season:
256256

257257
.. doctest:: aggregation
@@ -271,7 +271,7 @@ so adjacent ones are often in the same season:
271271
djf 2007
272272
djf 2007
273273

274-
Compare this with the first 10 values of the new cube's coordinates:
274+
Compare this with the first 10 values of the new cube's coordinates:
275275
All the points now have distinct season+year values:
276276

277277
.. doctest:: aggregation
@@ -294,7 +294,7 @@ All the points now have distinct season+year values:
294294

295295
Because the original data started in April 2006 we have some incomplete seasons
296296
(e.g. there were only two months worth of data for 'mam-2006').
297-
In this case we can fix this by removing all of the resultant 'times' which
297+
In this case we can fix this by removing all of the resultant 'times' which
298298
do not cover a three month period (note: judged here as > 3*28 days):
299299

300300
.. doctest:: aggregation
@@ -306,7 +306,7 @@ do not cover a three month period (note: judged here as > 3*28 days):
306306
>>> full_season_means
307307
<iris 'Cube' of surface_temperature / (K) (time: 17; latitude: 18; longitude: 432)>
308308

309-
The final result now represents the seasonal mean temperature for 17 seasons
309+
The final result now represents the seasonal mean temperature for 17 seasons
310310
from jja-2006 to jja-2010:
311311

312312
.. doctest:: aggregation

lib/iris/common/metadata.py

Lines changed: 159 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,22 +27,26 @@
2727

2828

2929
__all__ = [
30-
"SERVICES_COMBINE",
31-
"SERVICES_DIFFERENCE",
32-
"SERVICES_EQUAL",
33-
"SERVICES",
3430
"AncillaryVariableMetadata",
3531
"BaseMetadata",
3632
"CellMeasureMetadata",
3733
"CoordMetadata",
3834
"CubeMetadata",
3935
"DimCoordMetadata",
4036
"hexdigest",
37+
"metadata_filter",
4138
"metadata_manager_factory",
39+
"SERVICES",
40+
"SERVICES_COMBINE",
41+
"SERVICES_DIFFERENCE",
42+
"SERVICES_EQUAL",
4243
]
4344

4445

4546
# https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_data_set_components.html#object_name
47+
48+
from ..util import guess_coord_axis
49+
4650
_TOKEN_PARSE = re.compile(r"""^[a-zA-Z0-9][\w\.\+\-@]*$""")
4751

4852
# Configure the logger.
@@ -194,9 +198,18 @@ def func(field):
194198
return result
195199

196200
# Note that, for strict we use "_fields" not "_members".
197-
# The "circular" and "src_dim" members do not participate in strict equivalence.
201+
# TODO: refactor so that 'non-participants' can be held in their specific subclasses.
202+
# Certain members never participate in strict equivalence, so
203+
# are filtered out.
198204
fields = filter(
199-
lambda field: field not in ("circular", "src_dim"),
205+
lambda field: field
206+
not in (
207+
"circular",
208+
"src_dim",
209+
"node_dimension",
210+
"edge_dimension",
211+
"face_dimension",
212+
),
200213
self._fields,
201214
)
202215
result = all([func(field) for field in fields])
@@ -1330,6 +1343,146 @@ def equal(self, other, lenient=None):
13301343
return super().equal(other, lenient=lenient)
13311344

13321345

1346+
def metadata_filter(
1347+
instances,
1348+
item=None,
1349+
standard_name=None,
1350+
long_name=None,
1351+
var_name=None,
1352+
attributes=None,
1353+
axis=None,
1354+
):
1355+
"""
1356+
Filter a collection of objects by their metadata to fit the given metadata
1357+
criteria. Criteria can be one or both of: specific properties / other objects
1358+
carrying metadata to be matched.
1359+
1360+
Args:
1361+
1362+
* instances
1363+
One or more objects to be filtered.
1364+
1365+
Kwargs:
1366+
1367+
* item
1368+
Either
1369+
1370+
(a) a :attr:`standard_name`, :attr:`long_name`, or
1371+
:attr:`var_name`. Defaults to value of `default`
1372+
(which itself defaults to `unknown`) as defined in
1373+
:class:`~iris.common.CFVariableMixin`.
1374+
1375+
(b) a 'coordinate' instance with metadata equal to that of
1376+
the desired coordinates. Accepts either a
1377+
:class:`~iris.coords.DimCoord`, :class:`~iris.coords.AuxCoord`,
1378+
:class:`~iris.aux_factory.AuxCoordFactory`,
1379+
:class:`~iris.common.CoordMetadata` or
1380+
:class:`~iris.common.DimCoordMetadata` or
1381+
:class:`~iris.experimental.ugrid.ConnectivityMetadata`.
1382+
* standard_name
1383+
The CF standard name of the desired coordinate. If None, does not
1384+
check for standard name.
1385+
* long_name
1386+
An unconstrained description of the coordinate. If None, does not
1387+
check for long_name.
1388+
* var_name
1389+
The netCDF variable name of the desired coordinate. If None, does
1390+
not check for var_name.
1391+
* attributes
1392+
A dictionary of attributes desired on the coordinates. If None,
1393+
does not check for attributes.
1394+
* axis
1395+
The desired coordinate axis, see
1396+
:func:`~iris.util.guess_coord_axis`. If None, does not check for
1397+
axis. Accepts the values 'X', 'Y', 'Z' and 'T' (case-insensitive).
1398+
1399+
Returns:
1400+
A list of the objects supplied in the ``instances`` argument, limited
1401+
to only those that matched the given criteria.
1402+
1403+
"""
1404+
name = None
1405+
obj = None
1406+
1407+
if isinstance(item, str):
1408+
name = item
1409+
else:
1410+
obj = item
1411+
1412+
# apply de morgan's law for one less logical operation
1413+
if not (isinstance(instances, str) or isinstance(instances, Iterable)):
1414+
instances = [instances]
1415+
1416+
result = instances
1417+
1418+
if name is not None:
1419+
result = [instance for instance in result if instance.name() == name]
1420+
1421+
if standard_name is not None:
1422+
result = [
1423+
instance
1424+
for instance in result
1425+
if instance.standard_name == standard_name
1426+
]
1427+
1428+
if long_name is not None:
1429+
result = [
1430+
instance for instance in result if instance.long_name == long_name
1431+
]
1432+
1433+
if var_name is not None:
1434+
result = [
1435+
instance for instance in result if instance.var_name == var_name
1436+
]
1437+
1438+
if attributes is not None:
1439+
if not isinstance(attributes, Mapping):
1440+
msg = (
1441+
"The attributes keyword was expecting a dictionary "
1442+
"type, but got a %s instead." % type(attributes)
1443+
)
1444+
raise ValueError(msg)
1445+
1446+
def attr_filter(instance):
1447+
return all(
1448+
k in instance.attributes
1449+
and hexdigest(instance.attributes[k]) == hexdigest(v)
1450+
for k, v in attributes.items()
1451+
)
1452+
1453+
result = [instance for instance in result if attr_filter(instance)]
1454+
1455+
if axis is not None:
1456+
axis = axis.upper()
1457+
1458+
def get_axis(instance):
1459+
if hasattr(instance, "axis"):
1460+
axis = instance.axis.upper()
1461+
else:
1462+
axis = guess_coord_axis(instance)
1463+
return axis
1464+
1465+
result = [
1466+
instance for instance in result if get_axis(instance) == axis
1467+
]
1468+
1469+
if obj is not None:
1470+
if hasattr(obj, "__class__") and issubclass(
1471+
obj.__class__, BaseMetadata
1472+
):
1473+
target_metadata = obj
1474+
else:
1475+
target_metadata = obj.metadata
1476+
1477+
result = [
1478+
instance
1479+
for instance in result
1480+
if instance.metadata == target_metadata
1481+
]
1482+
1483+
return result
1484+
1485+
13331486
def metadata_manager_factory(cls, **kwargs):
13341487
"""
13351488
A class instance factory function responsible for manufacturing

0 commit comments

Comments
 (0)