From f571ef07349c126c885b49e736452e22710ca34a Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Mon, 23 Jun 2025 17:51:47 +0100 Subject: [PATCH 1/7] Detailed documentation on Load Problems. --- docs/src/userguide/loading_iris_cubes.rst | 112 ++++++++++++++++++++++ docs/src/whatsnew/latest.rst | 5 +- 2 files changed, 115 insertions(+), 2 deletions(-) diff --git a/docs/src/userguide/loading_iris_cubes.rst b/docs/src/userguide/loading_iris_cubes.rst index cbba4da39f..9ebcaa5857 100644 --- a/docs/src/userguide/loading_iris_cubes.rst +++ b/docs/src/userguide/loading_iris_cubes.rst @@ -298,3 +298,115 @@ these two cubes into separate variables. >>> print(number_two) 2 +.. _load-problems: + +Load Problems +------------- + +The Iris data model - see :ref:`iris_data_structures` - is highly flexible, but +there are many examples of file content that will not be loaded into the data +model. These fall into two categories: + +1. Malformations in the file. + - For example: a variable that is referenced, but is missing. +2. Content not conformant with the standard for that file type. + - Most commonly :term:`NetCDF` file content that is not + compliant with the :term:`CF conventions` - the basis for the Iris + data model. But Iris also relies on standards for other file + formats such as :term:`GRIB Format` and :term:`Post Processing (PP) Format`. + - Content in a non-NetCDF file that Iris does not know how to map onto CF + concepts. + - :term:`CF conventions` concepts that Iris does not support yet. + +.. note:: + + The below approach was introduced in Iris 3.12, and widely used in + CF-NetCDF loading by Iris 3.13. We hope to continue spreading it to + other file formats, and overlooked corner cases, in future releases. + +When Iris encounters problem content in a file, it will not make 'best efforts' +to parse the content, but will instead redirect it to +:data:`iris.loading.LOAD_PROBLEMS`, as well as issuing a warning to the user. +The user is then free to add any operations to their script(s) for +incorporating :data:`~iris.loading.LOAD_PROBLEMS` content into the Iris data +model, as they see fit. + +.. todo:: flesh out with examples/code. + +Why this approach? +^^^^^^^^^^^^^^^^^^ + +.. todo:: figure out the best order for these headings. + +In many cases, a sensible workaround for loading 'problem content' would be +obvious, especially given the flexibility of the Iris data model. But instead, +this stricter approach from Iris on file quality has several benefits: + +Diversity +""""""""" + +Several less 'opinionated' libraries are already available for those users that +want to load all content from their file, regardless of quality or meaning. +These libraries give the user the freedom to customise the handling of their +files as they see fit, but also put the onus on the user to understand the file +content and write code to handle it. Iris would be adding little new to the +ecosystem if it had an identical philosophy. + +Examples include: :term:`netCDF4`, :term:`Xarray`, `ecCodes`_. + +Instead, when working with the Iris data model, users can be confident in +the validity, and precise meaning (from the :term:`CF conventions`) of this +information. + +Maintainability +""""""""""""""" + +Well written standards allow the loading code to be written with assumptions +about what file content to expect. This code is much simpler than either fully +'agnostic' code which can load anything, or code which embeds various +workarounds for known problems. Simpler code takes less resource/expertise to +maintain, increasing the long-term sustainability of Iris. + +User Discretion +""""""""""""""" + +File malformations/non-conformances are by-definition not covered by any +standard for that file type - there is no consensus on the correct way to +represent this information. By avoiding encoding workarounds into Iris' +codebase, we avoid imposing one party's opinion onto other Iris users, who may +believe the problem should be handled differently. + +Raised Awareness +"""""""""""""""" + +The Iris developers are keen for a world with maximum file compatibility - where +files can be correctly parsed by different parties and even different +software, without the need for caveats, notes, or workarounds. This is why Iris +conforms to file standards wherever they are available: +:term:`CF conventions`, :ref:`UGRID`, :term:`GRIB Format`, +etcetera. + +Iris makes users aware of any non-compliance with a file standard by not +loading it directly into the data model (instead redirecting to +:data:`iris.loading.LOAD_PROBLEMS`). Awareness gives data consumers and +providers the opportunity to collaborate on improving file quality, increasing +the ease with which the file can be loaded by ANY appropriate software. + +(Any workarounds or 'agnosticism' would only increase the ease of *Iris* +loading the file, hiding the fact that other software and other collaborators +might not understand it). + +Robustness +"""""""""" + +Redirecting problem content to :data:`iris.loading.LOAD_PROBLEMS` occurs in +places where Iris would otherwise raise an exception. This means that +Iris can continue to load all the valid parts of the file, and the user has +a way to fix problems **within Iris**, rather than learning a NetCDF tool or +similar. + +This will not only handle file problems, but also any current or future bugs in +the Iris codebase, until they are fixed in the next release. + + +.. _ecCodes: https://github.com/ecmwf/eccodes diff --git a/docs/src/whatsnew/latest.rst b/docs/src/whatsnew/latest.rst index 6923db7179..58bfe13a20 100644 --- a/docs/src/whatsnew/latest.rst +++ b/docs/src/whatsnew/latest.rst @@ -32,12 +32,13 @@ This document explains the changes made to Iris for this release =========== #. `@trexfeathers`_ and `@ukmo-ccbunney`_ extended the - :data:`iris.loading.LOAD_PROBLEMS` capturing to _all_ NetCDF objects that are + :data:`iris.loading.LOAD_PROBLEMS` capturing to *all* NetCDF objects that are added to a :class:`~iris.cube.Cube` during loading, as well as a selection of other objects such as :class:`~iris.coord_systems.CoordSystem`. Note this includes an improvement to how :class:`~iris.coords.DimCoord` is 'gracefully' converted to :class:`~iris.coords.AuxCoord` if it is masked - the mask is - now preserved when it was not previously. + now preserved when it was not previously. See also: :ref:`load-problems`. + (:pull:`6465`, pull:`6529`) 🐛 Bugs Fixed From 31433d7a36ea3f524cc8aebe4e1ca5c4369022f1 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Tue, 24 Jun 2025 08:56:00 +0100 Subject: [PATCH 2/7] Correct What's New. --- docs/src/whatsnew/latest.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/whatsnew/latest.rst b/docs/src/whatsnew/latest.rst index 58bfe13a20..3d41420017 100644 --- a/docs/src/whatsnew/latest.rst +++ b/docs/src/whatsnew/latest.rst @@ -38,7 +38,7 @@ This document explains the changes made to Iris for this release includes an improvement to how :class:`~iris.coords.DimCoord` is 'gracefully' converted to :class:`~iris.coords.AuxCoord` if it is masked - the mask is now preserved when it was not previously. See also: :ref:`load-problems`. - (:pull:`6465`, pull:`6529`) + (:pull:`6465`, :pull:`6529`) 🐛 Bugs Fixed From 32ec7bd0149cd29e16df7647cd1e1a5db7777751 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Wed, 25 Jun 2025 16:44:14 +0100 Subject: [PATCH 3/7] Include a light example. --- docs/src/userguide/loading_iris_cubes.rst | 76 ++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/docs/src/userguide/loading_iris_cubes.rst b/docs/src/userguide/loading_iris_cubes.rst index 9ebcaa5857..516f1c78ba 100644 --- a/docs/src/userguide/loading_iris_cubes.rst +++ b/docs/src/userguide/loading_iris_cubes.rst @@ -331,7 +331,81 @@ The user is then free to add any operations to their script(s) for incorporating :data:`~iris.loading.LOAD_PROBLEMS` content into the Iris data model, as they see fit. -.. todo:: flesh out with examples/code. +The below example has 'booby trapped' the Iris loading process, to give an +impression of the user experience when loading problems are encountered. The +example shows typical :data:`~iris.loading.LOAD_PROBLEMS` content, and a +deeper inspection of one redirected object. **Much more detail is in the +API documentation for:** :class:`iris.loading.LoadProblems`. + +.. testsetup:: load-problems + + from pathlib import Path + from pprint import pprint + import sys + import warnings + + import iris + import iris.common + from iris.fileformats._nc_load_rules import helpers + import iris.loading + from iris import std_names + + # Ensure doctests actually see Warnings that are raised, and that + # they have a relative path (so a test pass is not machine-dependent). + showwarning_original = warnings.showwarning + warnings.filterwarnings("default") + IRIS_FILE = Path(iris.__file__) + def custom_warn(message, category, filename, lineno, file=None, line=None): + filepath = Path(filename) + filename = str(filepath.relative_to(IRIS_FILE.parents[1])) + sys.stdout.write(warnings.formatwarning(message, category, filename, lineno)) + warnings.showwarning = custom_warn + + get_names_original = helpers.get_names + + def raise_example_error_names(cf_coord_var, coord_name, attributes): + if cf_coord_var.cf_name == "time": + raise ValueError("Example coordinate error") + else: + return get_names_original( + cf_coord_var, coord_name, attributes + ) + + helpers.get_names = raise_example_error_names + air_temperature = std_names.STD_NAMES.pop("air_temperature") + iris.FUTURE.date_microseconds = True + +.. doctest:: load-problems + + >>> cube_a1b = iris.load_cube(iris.sample_data_path("A1B_north_america.nc")) + iris/...IrisLoadWarning: Not all file objects were parsed correctly. See iris.loading.LOAD_PROBLEMS for details. + warnings.warn(message, category=IrisLoadWarning) + >>> print(iris.loading.LOAD_PROBLEMS) + : + .../A1B_north_america.nc: "'air_temperature' is not a valid standard_name", {'standard_name': 'air_temperature'} + .../A1B_north_america.nc: "Example coordinate error", unknown / (unknown) (-- : 240) + >>> last_problem = iris.loading.LOAD_PROBLEMS.problems[-1] + >>> print(last_problem.loaded) + unknown / (unknown) (-- : 240) + Attributes:... + IRIS_RAW {'axis': 'T', ...} + >>> attributes = last_problem.loaded.attributes[ + ... iris.common.LimitedAttributeDict.IRIS_RAW + ... ] + >>> pprint(attributes) + {'axis': 'T', + 'bounds': 'time_bnds', + 'calendar': '360_day', + 'standard_name': 'time', + 'units': 'hours since 1970-01-01 00:00:00', + 'var_name': 'time'} + +.. testcleanup:: load-problems + + warnings.showwarning = showwarning_original + warnings.filterwarnings("ignore") + helpers.get_names = get_names_original + std_names.STD_NAMES["air_temperature"] = air_temperature Why this approach? ^^^^^^^^^^^^^^^^^^ From 2947e4a56ca8402a7c3502c8cb6e80662d20438c Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Wed, 25 Jun 2025 16:46:31 +0100 Subject: [PATCH 4/7] Re-ordered explanation headings. --- docs/src/userguide/loading_iris_cubes.rst | 69 +++++++++++------------ 1 file changed, 33 insertions(+), 36 deletions(-) diff --git a/docs/src/userguide/loading_iris_cubes.rst b/docs/src/userguide/loading_iris_cubes.rst index 516f1c78ba..4ff5f4e561 100644 --- a/docs/src/userguide/loading_iris_cubes.rst +++ b/docs/src/userguide/loading_iris_cubes.rst @@ -410,46 +410,10 @@ API documentation for:** :class:`iris.loading.LoadProblems`. Why this approach? ^^^^^^^^^^^^^^^^^^ -.. todo:: figure out the best order for these headings. - In many cases, a sensible workaround for loading 'problem content' would be obvious, especially given the flexibility of the Iris data model. But instead, this stricter approach from Iris on file quality has several benefits: -Diversity -""""""""" - -Several less 'opinionated' libraries are already available for those users that -want to load all content from their file, regardless of quality or meaning. -These libraries give the user the freedom to customise the handling of their -files as they see fit, but also put the onus on the user to understand the file -content and write code to handle it. Iris would be adding little new to the -ecosystem if it had an identical philosophy. - -Examples include: :term:`netCDF4`, :term:`Xarray`, `ecCodes`_. - -Instead, when working with the Iris data model, users can be confident in -the validity, and precise meaning (from the :term:`CF conventions`) of this -information. - -Maintainability -""""""""""""""" - -Well written standards allow the loading code to be written with assumptions -about what file content to expect. This code is much simpler than either fully -'agnostic' code which can load anything, or code which embeds various -workarounds for known problems. Simpler code takes less resource/expertise to -maintain, increasing the long-term sustainability of Iris. - -User Discretion -""""""""""""""" - -File malformations/non-conformances are by-definition not covered by any -standard for that file type - there is no consensus on the correct way to -represent this information. By avoiding encoding workarounds into Iris' -codebase, we avoid imposing one party's opinion onto other Iris users, who may -believe the problem should be handled differently. - Raised Awareness """""""""""""""" @@ -470,6 +434,15 @@ the ease with which the file can be loaded by ANY appropriate software. loading the file, hiding the fact that other software and other collaborators might not understand it). +Maintainability +""""""""""""""" + +Well written standards allow the loading code to be written with assumptions +about what file content to expect. This code is much simpler than either fully +'agnostic' code which can load anything, or code which embeds various +workarounds for known problems. Simpler code takes less resource/expertise to +maintain, increasing the long-term sustainability of Iris. + Robustness """""""""" @@ -482,5 +455,29 @@ similar. This will not only handle file problems, but also any current or future bugs in the Iris codebase, until they are fixed in the next release. +User Discretion +""""""""""""""" + +File malformations/non-conformances are by-definition not covered by any +standard for that file type - there is no consensus on the correct way to +represent this information. By avoiding encoding workarounds into Iris' +codebase, we avoid imposing one party's opinion onto other Iris users, who may +believe the problem should be handled differently. + +Diversity +""""""""" + +Several less 'opinionated' libraries are already available for those users that +want to load all content from their file, regardless of quality or meaning. +These libraries give the user the freedom to customise the handling of their +files as they see fit, but also put the onus on the user to understand the file +content and write code to handle it. Iris would be adding little new to the +ecosystem if it had an identical philosophy. + +Examples include: :term:`netCDF4`, :term:`Xarray`, `ecCodes`_. + +Instead, when working with the Iris data model, users can be confident in +the validity, and precise meaning (from the :term:`CF conventions`) of this +information. .. _ecCodes: https://github.com/ecmwf/eccodes From 39ea2f78c6b39feea9f36070787bd4fbed3d1849 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Wed, 25 Jun 2025 17:23:31 +0100 Subject: [PATCH 5/7] Move explanatory docs into Iris Philosophy page. --- .../src/further_topics/filtering_warnings.rst | 34 +--- docs/src/userguide/code_maintenance.rst | 60 ------ docs/src/userguide/index.rst | 2 +- docs/src/userguide/iris_philosophy.rst | 191 ++++++++++++++++++ docs/src/userguide/loading_iris_cubes.rst | 77 +------ docs/src/whatsnew/latest.rst | 9 +- 6 files changed, 204 insertions(+), 169 deletions(-) delete mode 100644 docs/src/userguide/code_maintenance.rst create mode 100644 docs/src/userguide/iris_philosophy.rst diff --git a/docs/src/further_topics/filtering_warnings.rst b/docs/src/further_topics/filtering_warnings.rst index ef8701f951..204049942b 100644 --- a/docs/src/further_topics/filtering_warnings.rst +++ b/docs/src/further_topics/filtering_warnings.rst @@ -8,6 +8,8 @@ Since Iris cannot predict your specific needs, it by default raises Warnings for anything that might be a problem for **any** user, and is designed to work with you to ``ignore`` Warnings which you do not find helpful. +Find out more about *why* we chose this approach: :ref:`filtering-warnings-explanation`. + .. testsetup:: filtering_warnings from pathlib import Path @@ -231,38 +233,6 @@ viewing the :mod:`iris.warnings` module. :ref:`get in touch`! The categories exist to make your life easier, and it is simple to make modifications. - -More Detail ------------ - -Different people use Iris for very different purposes, from quick file -visualisation to extract-transform-load to statistical analysis. These -contrasting priorities mean disagreement on which Iris problems can be ignored -and which are critically important. - -For problems that prevent Iris functioning: **Concrete Exceptions** are raised, which -stop code from running any further - no debate here. For less catastrophic -problems: **Warnings** are raised, -which notify you (in ``stderr``) but allow code to continue running. The Warnings are -there because Iris may **OR may not** function in the way you expect, -depending on what you need - e.g. a problem might prevent data being saved to -NetCDF, but statistical analysis will still work fine. - -Examples of Iris Warnings -~~~~~~~~~~~~~~~~~~~~~~~~~ - -- If you attempt to plot un-bounded point data as a ``pcolormesh``: Iris will - guess appropriate bounds around each point so that quadrilaterals can be - plotted. This permanently modifies the relevant coordinates, so the you are - warned in case downstream operations assume un-bounded coordinates. -- If you load a NetCDF file where a CF variable references another variable - - e.g. ``my_var:coordinates = "depth_var" ;`` - but the referenced variable - (``depth_var``) is not in the file: Iris will still construct - its data model, but without this reference relationship. You are warned since - the file includes an error and the loaded result might therefore not be as - expected. - - .. testcleanup:: filtering_warnings warnings.filterwarnings("ignore") diff --git a/docs/src/userguide/code_maintenance.rst b/docs/src/userguide/code_maintenance.rst deleted file mode 100644 index c01c1975a7..0000000000 --- a/docs/src/userguide/code_maintenance.rst +++ /dev/null @@ -1,60 +0,0 @@ -Code Maintenance -================ - -From a user point of view "code maintenance" means ensuring that your existing -working code stays working, in the face of changes to Iris. - - -Stability and Change ---------------------- - -In practice, as Iris develops, most users will want to periodically upgrade -their installed version to access new features or at least bug fixes. - -This is obvious if you are still developing other code that uses Iris, or using -code from other sources. -However, even if you have only legacy code that remains untouched, some code -maintenance effort is probably still necessary: - -* On the one hand, *in principle*, working code will go on working, as long - as you don't change anything else. - -* However, such "version stasis" can easily become a growing burden, if you - are simply waiting until an update becomes unavoidable, often that will - eventually occur when you need to update some other software component, - for some completely unconnected reason. - - -Principles of Change Management -------------------------------- - -When you upgrade software to a new version, you often find that you need to -rewrite your legacy code, simply to keep it working. - -In Iris, however, we aim to reduce code maintenance problems to an absolute -minimum by following defined change management rules. -These ensure that, *within a major release number* : - -* you can be confident that your code will still work with subsequent minor - releases - -* you will be aware of future incompatibility problems in advance - -* you can defer making code compatibility changes for some time, until it - suits you - -The above applies to minor version upgrades : e.g. code that works with version -"1.4.2" should still work with a subsequent minor release such as "1.5.0" or -"1.7.2". - -A *major* release however, e.g. "v2.0.0" or "v3.0.0", can include more -significant changes, including so-called "breaking" changes: This means that -existing code may need to be modified to make it work with the new version. - -Since breaking change can only occur at major releases, these are the *only* -times we can alter or remove existing behaviours (even deprecated -ones). This is what a major release is for : it enables the removal and -replacement of old features. - -Of course, even at a major release, we do still aim to keep breaking changes to -a minimum. diff --git a/docs/src/userguide/index.rst b/docs/src/userguide/index.rst index d986a986ad..2b77129a4e 100644 --- a/docs/src/userguide/index.rst +++ b/docs/src/userguide/index.rst @@ -40,6 +40,6 @@ they may serve as a useful reference for future exploration. cube_statistics cube_maths citation - code_maintenance + iris_philosophy glossary ../further_topics/index diff --git a/docs/src/userguide/iris_philosophy.rst b/docs/src/userguide/iris_philosophy.rst new file mode 100644 index 0000000000..1d1148fcaf --- /dev/null +++ b/docs/src/userguide/iris_philosophy.rst @@ -0,0 +1,191 @@ +.. _iris-philosophy: + +**************** +Iris' Philosophy +**************** + +.. todo:: https://github.com/SciTools/iris/issues/6511; this page belongs in 'Explanation' + +.. _code-maintenance: + +Code Maintenance +================ + +From a user point of view "code maintenance" means ensuring that your existing +working code stays working, in the face of changes to Iris. + + +Stability and Change +--------------------- + +In practice, as Iris develops, most users will want to periodically upgrade +their installed version to access new features or at least bug fixes. + +This is obvious if you are still developing other code that uses Iris, or using +code from other sources. +However, even if you have only legacy code that remains untouched, some code +maintenance effort is probably still necessary: + +* On the one hand, *in principle*, working code will go on working, as long + as you don't change anything else. + +* However, such "version stasis" can easily become a growing burden, if you + are simply waiting until an update becomes unavoidable, often that will + eventually occur when you need to update some other software component, + for some completely unconnected reason. + + +Principles of Change Management +------------------------------- + +When you upgrade software to a new version, you often find that you need to +rewrite your legacy code, simply to keep it working. + +In Iris, however, we aim to reduce code maintenance problems to an absolute +minimum by following defined change management rules. +These ensure that, *within a major release number* : + +* you can be confident that your code will still work with subsequent minor + releases + +* you will be aware of future incompatibility problems in advance + +* you can defer making code compatibility changes for some time, until it + suits you + +The above applies to minor version upgrades : e.g. code that works with version +"1.4.2" should still work with a subsequent minor release such as "1.5.0" or +"1.7.2". + +A *major* release however, e.g. "v2.0.0" or "v3.0.0", can include more +significant changes, including so-called "breaking" changes: This means that +existing code may need to be modified to make it work with the new version. + +Since breaking change can only occur at major releases, these are the *only* +times we can alter or remove existing behaviours (even deprecated +ones). This is what a major release is for : it enables the removal and +replacement of old features. + +Of course, even at a major release, we do still aim to keep breaking changes to +a minimum. + +.. _load-problems-explanation: + +Loading Invalid File Content +============================ + +As discussed in :ref:`load-problems`, Iris will not attempt to load file content +that is malformed or non-conformant with relevant standards, instead redirecting +the content to :data:`iris.loading.LOAD_PROBLEMS`. +In many cases, a sensible workaround for loading 'problem content' would be +obvious, especially given the flexibility of the Iris data model. But instead, +this stricter approach from Iris on file quality has several benefits: + +(See also: :issue:`5165`). + +Raised Awareness +---------------- + +The Iris developers are keen for a world with maximum file compatibility - where +files can be correctly parsed by different parties and even different +software, without the need for caveats, notes, or workarounds. This is why Iris +conforms to file standards wherever they are available: +:term:`CF conventions`, :ref:`UGRID`, :term:`GRIB Format`, +etcetera. + +Iris makes users aware of any non-compliance with a file standard by not +loading it directly into the data model (instead redirecting to +:data:`iris.loading.LOAD_PROBLEMS`). Awareness gives data consumers and +providers the opportunity to collaborate on improving file quality, increasing +the ease with which the file can be loaded by ANY appropriate software. + +(Any workarounds or 'agnosticism' would only increase the ease of *Iris* +loading the file, hiding the fact that other software and other collaborators +might not understand it). + +Maintainability +--------------- + +Well written standards allow the loading code to be written with assumptions +about what file content to expect. This code is much simpler than either fully +'agnostic' code which can load anything, or code which embeds various +workarounds for known problems. Simpler code takes less resource/expertise to +maintain, increasing the long-term sustainability of Iris. + +Robustness +---------- + +Redirecting problem content to :data:`iris.loading.LOAD_PROBLEMS` occurs in +places where Iris would otherwise raise an exception. This means that +Iris can continue to load all the valid parts of the file, and the user has +a way to fix problems **within Iris**, rather than learning a NetCDF tool or +similar. + +This will not only handle file problems, but also any current or future bugs in +the Iris codebase, until they are fixed in the next release. + +User Discretion +--------------- + +File malformations/non-conformances are by-definition not covered by any +standard for that file type - there is no consensus on the correct way to +represent this information. By avoiding encoding workarounds into Iris' +codebase, we avoid imposing one party's opinion onto other Iris users, who may +believe the problem should be handled differently. + +Diversity +--------- + +Several less 'opinionated' libraries are already available for those users that +want to load all content from their file, regardless of quality or meaning. +These libraries give the user the freedom to customise the handling of their +files as they see fit, but also put the onus on the user to understand the file +content and write code to handle it. Iris would be adding little new to the +ecosystem if it had an identical philosophy. + +Examples include: :term:`netCDF4`, :term:`Xarray`, `ecCodes`_. + +Instead, when working with the Iris data model, users can be confident in +the validity, and precise meaning (from the :term:`CF conventions`) of this +information. + +.. _filtering-warnings-explanation: + +Verbose Warnings +================ + +Different people use Iris for very different purposes, from quick file +visualisation to extract-transform-load to statistical analysis. These +contrasting priorities mean disagreement on which Iris problems can be ignored +and which are critically important. + +For problems that prevent Iris functioning: **Concrete Exceptions** are raised, which +stop code from running any further - no debate here. For less catastrophic +problems: **Warnings** are raised, +which notify you (in ``stderr``) but allow code to continue running. The Warnings are +there because Iris may **OR may not** function in the way you expect, +depending on what you need - e.g. a problem might prevent data being saved to +NetCDF, but statistical analysis will still work fine. + +This means that Iris' default behaviour is to raise Warnings +for anything that might be a problem for **any** user, since it cannot predict +specific user needs. It is designed to work with the user to ``ignore`` Warnings +which are not considered helpful in their specific use case. + +**How to ignore unwanted warnings:** :ref:`filtering-warnings` + +Examples of Iris Warnings +------------------------- + +- If you attempt to plot un-bounded point data as a ``pcolormesh``: Iris will + guess appropriate bounds around each point so that quadrilaterals can be + plotted. This permanently modifies the relevant coordinates, so the you are + warned in case downstream operations assume un-bounded coordinates. +- If you load a NetCDF file where a CF variable references another variable - + e.g. ``my_var:coordinates = "depth_var" ;`` - but the referenced variable + (``depth_var``) is not in the file: Iris will still construct + its data model, but without this reference relationship. You are warned since + the file includes an error and the loaded result might therefore not be as + expected. + +.. _ecCodes: https://github.com/ecmwf/eccodes diff --git a/docs/src/userguide/loading_iris_cubes.rst b/docs/src/userguide/loading_iris_cubes.rst index 4ff5f4e561..ac6b370466 100644 --- a/docs/src/userguide/loading_iris_cubes.rst +++ b/docs/src/userguide/loading_iris_cubes.rst @@ -331,6 +331,8 @@ The user is then free to add any operations to their script(s) for incorporating :data:`~iris.loading.LOAD_PROBLEMS` content into the Iris data model, as they see fit. +Find out more about *why* we chose this approach: :ref:`load-problems-explanation` + The below example has 'booby trapped' the Iris loading process, to give an impression of the user experience when loading problems are encountered. The example shows typical :data:`~iris.loading.LOAD_PROBLEMS` content, and a @@ -406,78 +408,3 @@ API documentation for:** :class:`iris.loading.LoadProblems`. warnings.filterwarnings("ignore") helpers.get_names = get_names_original std_names.STD_NAMES["air_temperature"] = air_temperature - -Why this approach? -^^^^^^^^^^^^^^^^^^ - -In many cases, a sensible workaround for loading 'problem content' would be -obvious, especially given the flexibility of the Iris data model. But instead, -this stricter approach from Iris on file quality has several benefits: - -Raised Awareness -"""""""""""""""" - -The Iris developers are keen for a world with maximum file compatibility - where -files can be correctly parsed by different parties and even different -software, without the need for caveats, notes, or workarounds. This is why Iris -conforms to file standards wherever they are available: -:term:`CF conventions`, :ref:`UGRID`, :term:`GRIB Format`, -etcetera. - -Iris makes users aware of any non-compliance with a file standard by not -loading it directly into the data model (instead redirecting to -:data:`iris.loading.LOAD_PROBLEMS`). Awareness gives data consumers and -providers the opportunity to collaborate on improving file quality, increasing -the ease with which the file can be loaded by ANY appropriate software. - -(Any workarounds or 'agnosticism' would only increase the ease of *Iris* -loading the file, hiding the fact that other software and other collaborators -might not understand it). - -Maintainability -""""""""""""""" - -Well written standards allow the loading code to be written with assumptions -about what file content to expect. This code is much simpler than either fully -'agnostic' code which can load anything, or code which embeds various -workarounds for known problems. Simpler code takes less resource/expertise to -maintain, increasing the long-term sustainability of Iris. - -Robustness -"""""""""" - -Redirecting problem content to :data:`iris.loading.LOAD_PROBLEMS` occurs in -places where Iris would otherwise raise an exception. This means that -Iris can continue to load all the valid parts of the file, and the user has -a way to fix problems **within Iris**, rather than learning a NetCDF tool or -similar. - -This will not only handle file problems, but also any current or future bugs in -the Iris codebase, until they are fixed in the next release. - -User Discretion -""""""""""""""" - -File malformations/non-conformances are by-definition not covered by any -standard for that file type - there is no consensus on the correct way to -represent this information. By avoiding encoding workarounds into Iris' -codebase, we avoid imposing one party's opinion onto other Iris users, who may -believe the problem should be handled differently. - -Diversity -""""""""" - -Several less 'opinionated' libraries are already available for those users that -want to load all content from their file, regardless of quality or meaning. -These libraries give the user the freedom to customise the handling of their -files as they see fit, but also put the onus on the user to understand the file -content and write code to handle it. Iris would be adding little new to the -ecosystem if it had an identical philosophy. - -Examples include: :term:`netCDF4`, :term:`Xarray`, `ecCodes`_. - -Instead, when working with the Iris data model, users can be confident in -the validity, and precise meaning (from the :term:`CF conventions`) of this -information. - -.. _ecCodes: https://github.com/ecmwf/eccodes diff --git a/docs/src/whatsnew/latest.rst b/docs/src/whatsnew/latest.rst index 3d41420017..ce5003958a 100644 --- a/docs/src/whatsnew/latest.rst +++ b/docs/src/whatsnew/latest.rst @@ -95,7 +95,14 @@ This document explains the changes made to Iris for this release 📚 Documentation ================ -#. N/A +#. `@trexfeathers`_ and `@ukmo-ccbunney`_ added :ref:`load-problems` to the user + guide. (:pull:`6529`) + +#. `@trexfeathers`_ and `@ukmo-ccbunney`_ added a new user guide page: + :ref:`iris-philosophy`, for readers who are interested in why Iris is + designed/maintained the way it is. Initial content: :ref:`code-maintenance`, + :ref:`load-problems-explanation`, :ref:`filtering-warnings-explanation`. + (:pull:`6529`) 💼 Internal From 824bda2c93dacbf961f4ba504d53b5809528b8dd Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Fri, 27 Jun 2025 12:20:10 +0100 Subject: [PATCH 6/7] Grammer fixes to the Code Maintenance section. --- docs/src/userguide/iris_philosophy.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/src/userguide/iris_philosophy.rst b/docs/src/userguide/iris_philosophy.rst index 1d1148fcaf..519bbec4d5 100644 --- a/docs/src/userguide/iris_philosophy.rst +++ b/docs/src/userguide/iris_philosophy.rst @@ -29,9 +29,9 @@ maintenance effort is probably still necessary: * On the one hand, *in principle*, working code will go on working, as long as you don't change anything else. -* However, such "version stasis" can easily become a growing burden, if you +* However, such "version stasis" can easily become a growing burden; if you are simply waiting until an update becomes unavoidable, often that will - eventually occur when you need to update some other software component, + eventually occur when you need to update some other software component for some completely unconnected reason. @@ -53,7 +53,7 @@ These ensure that, *within a major release number* : * you can defer making code compatibility changes for some time, until it suits you -The above applies to minor version upgrades : e.g. code that works with version +The above applies to *minor version upgrades* : e.g. code that works with version "1.4.2" should still work with a subsequent minor release such as "1.5.0" or "1.7.2". @@ -63,7 +63,7 @@ existing code may need to be modified to make it work with the new version. Since breaking change can only occur at major releases, these are the *only* times we can alter or remove existing behaviours (even deprecated -ones). This is what a major release is for : it enables the removal and +ones). This is what a major release is for: it enables the removal and replacement of old features. Of course, even at a major release, we do still aim to keep breaking changes to From 405f2ccd6c31bbd11cb7013e7da69ea9c6c1eb06 Mon Sep 17 00:00:00 2001 From: Martin Yeo Date: Fri, 27 Jun 2025 12:21:58 +0100 Subject: [PATCH 7/7] More inspiring wording. --- docs/src/userguide/iris_philosophy.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/userguide/iris_philosophy.rst b/docs/src/userguide/iris_philosophy.rst index 519bbec4d5..f2b699edc2 100644 --- a/docs/src/userguide/iris_philosophy.rst +++ b/docs/src/userguide/iris_philosophy.rst @@ -86,7 +86,7 @@ this stricter approach from Iris on file quality has several benefits: Raised Awareness ---------------- -The Iris developers are keen for a world with maximum file compatibility - where +The Iris developers aspire to a world with maximum file compatibility - where files can be correctly parsed by different parties and even different software, without the need for caveats, notes, or workarounds. This is why Iris conforms to file standards wherever they are available: