diff --git a/docs/src/further_topics/filtering_warnings.rst b/docs/src/further_topics/filtering_warnings.rst index ef8701f951..204049942b 100644 --- a/docs/src/further_topics/filtering_warnings.rst +++ b/docs/src/further_topics/filtering_warnings.rst @@ -8,6 +8,8 @@ Since Iris cannot predict your specific needs, it by default raises Warnings for anything that might be a problem for **any** user, and is designed to work with you to ``ignore`` Warnings which you do not find helpful. +Find out more about *why* we chose this approach: :ref:`filtering-warnings-explanation`. + .. testsetup:: filtering_warnings from pathlib import Path @@ -231,38 +233,6 @@ viewing the :mod:`iris.warnings` module. :ref:`get in touch`! The categories exist to make your life easier, and it is simple to make modifications. - -More Detail ------------ - -Different people use Iris for very different purposes, from quick file -visualisation to extract-transform-load to statistical analysis. These -contrasting priorities mean disagreement on which Iris problems can be ignored -and which are critically important. - -For problems that prevent Iris functioning: **Concrete Exceptions** are raised, which -stop code from running any further - no debate here. For less catastrophic -problems: **Warnings** are raised, -which notify you (in ``stderr``) but allow code to continue running. The Warnings are -there because Iris may **OR may not** function in the way you expect, -depending on what you need - e.g. a problem might prevent data being saved to -NetCDF, but statistical analysis will still work fine. - -Examples of Iris Warnings -~~~~~~~~~~~~~~~~~~~~~~~~~ - -- If you attempt to plot un-bounded point data as a ``pcolormesh``: Iris will - guess appropriate bounds around each point so that quadrilaterals can be - plotted. This permanently modifies the relevant coordinates, so the you are - warned in case downstream operations assume un-bounded coordinates. -- If you load a NetCDF file where a CF variable references another variable - - e.g. ``my_var:coordinates = "depth_var" ;`` - but the referenced variable - (``depth_var``) is not in the file: Iris will still construct - its data model, but without this reference relationship. You are warned since - the file includes an error and the loaded result might therefore not be as - expected. - - .. testcleanup:: filtering_warnings warnings.filterwarnings("ignore") diff --git a/docs/src/userguide/code_maintenance.rst b/docs/src/userguide/code_maintenance.rst deleted file mode 100644 index c01c1975a7..0000000000 --- a/docs/src/userguide/code_maintenance.rst +++ /dev/null @@ -1,60 +0,0 @@ -Code Maintenance -================ - -From a user point of view "code maintenance" means ensuring that your existing -working code stays working, in the face of changes to Iris. - - -Stability and Change ---------------------- - -In practice, as Iris develops, most users will want to periodically upgrade -their installed version to access new features or at least bug fixes. - -This is obvious if you are still developing other code that uses Iris, or using -code from other sources. -However, even if you have only legacy code that remains untouched, some code -maintenance effort is probably still necessary: - -* On the one hand, *in principle*, working code will go on working, as long - as you don't change anything else. - -* However, such "version stasis" can easily become a growing burden, if you - are simply waiting until an update becomes unavoidable, often that will - eventually occur when you need to update some other software component, - for some completely unconnected reason. - - -Principles of Change Management -------------------------------- - -When you upgrade software to a new version, you often find that you need to -rewrite your legacy code, simply to keep it working. - -In Iris, however, we aim to reduce code maintenance problems to an absolute -minimum by following defined change management rules. -These ensure that, *within a major release number* : - -* you can be confident that your code will still work with subsequent minor - releases - -* you will be aware of future incompatibility problems in advance - -* you can defer making code compatibility changes for some time, until it - suits you - -The above applies to minor version upgrades : e.g. code that works with version -"1.4.2" should still work with a subsequent minor release such as "1.5.0" or -"1.7.2". - -A *major* release however, e.g. "v2.0.0" or "v3.0.0", can include more -significant changes, including so-called "breaking" changes: This means that -existing code may need to be modified to make it work with the new version. - -Since breaking change can only occur at major releases, these are the *only* -times we can alter or remove existing behaviours (even deprecated -ones). This is what a major release is for : it enables the removal and -replacement of old features. - -Of course, even at a major release, we do still aim to keep breaking changes to -a minimum. diff --git a/docs/src/userguide/index.rst b/docs/src/userguide/index.rst index d986a986ad..2b77129a4e 100644 --- a/docs/src/userguide/index.rst +++ b/docs/src/userguide/index.rst @@ -40,6 +40,6 @@ they may serve as a useful reference for future exploration. cube_statistics cube_maths citation - code_maintenance + iris_philosophy glossary ../further_topics/index diff --git a/docs/src/userguide/iris_philosophy.rst b/docs/src/userguide/iris_philosophy.rst new file mode 100644 index 0000000000..f2b699edc2 --- /dev/null +++ b/docs/src/userguide/iris_philosophy.rst @@ -0,0 +1,191 @@ +.. _iris-philosophy: + +**************** +Iris' Philosophy +**************** + +.. todo:: https://github.com/SciTools/iris/issues/6511; this page belongs in 'Explanation' + +.. _code-maintenance: + +Code Maintenance +================ + +From a user point of view "code maintenance" means ensuring that your existing +working code stays working, in the face of changes to Iris. + + +Stability and Change +--------------------- + +In practice, as Iris develops, most users will want to periodically upgrade +their installed version to access new features or at least bug fixes. + +This is obvious if you are still developing other code that uses Iris, or using +code from other sources. +However, even if you have only legacy code that remains untouched, some code +maintenance effort is probably still necessary: + +* On the one hand, *in principle*, working code will go on working, as long + as you don't change anything else. + +* However, such "version stasis" can easily become a growing burden; if you + are simply waiting until an update becomes unavoidable, often that will + eventually occur when you need to update some other software component + for some completely unconnected reason. + + +Principles of Change Management +------------------------------- + +When you upgrade software to a new version, you often find that you need to +rewrite your legacy code, simply to keep it working. + +In Iris, however, we aim to reduce code maintenance problems to an absolute +minimum by following defined change management rules. +These ensure that, *within a major release number* : + +* you can be confident that your code will still work with subsequent minor + releases + +* you will be aware of future incompatibility problems in advance + +* you can defer making code compatibility changes for some time, until it + suits you + +The above applies to *minor version upgrades* : e.g. code that works with version +"1.4.2" should still work with a subsequent minor release such as "1.5.0" or +"1.7.2". + +A *major* release however, e.g. "v2.0.0" or "v3.0.0", can include more +significant changes, including so-called "breaking" changes: This means that +existing code may need to be modified to make it work with the new version. + +Since breaking change can only occur at major releases, these are the *only* +times we can alter or remove existing behaviours (even deprecated +ones). This is what a major release is for: it enables the removal and +replacement of old features. + +Of course, even at a major release, we do still aim to keep breaking changes to +a minimum. + +.. _load-problems-explanation: + +Loading Invalid File Content +============================ + +As discussed in :ref:`load-problems`, Iris will not attempt to load file content +that is malformed or non-conformant with relevant standards, instead redirecting +the content to :data:`iris.loading.LOAD_PROBLEMS`. +In many cases, a sensible workaround for loading 'problem content' would be +obvious, especially given the flexibility of the Iris data model. But instead, +this stricter approach from Iris on file quality has several benefits: + +(See also: :issue:`5165`). + +Raised Awareness +---------------- + +The Iris developers aspire to a world with maximum file compatibility - where +files can be correctly parsed by different parties and even different +software, without the need for caveats, notes, or workarounds. This is why Iris +conforms to file standards wherever they are available: +:term:`CF conventions`, :ref:`UGRID`, :term:`GRIB Format`, +etcetera. + +Iris makes users aware of any non-compliance with a file standard by not +loading it directly into the data model (instead redirecting to +:data:`iris.loading.LOAD_PROBLEMS`). Awareness gives data consumers and +providers the opportunity to collaborate on improving file quality, increasing +the ease with which the file can be loaded by ANY appropriate software. + +(Any workarounds or 'agnosticism' would only increase the ease of *Iris* +loading the file, hiding the fact that other software and other collaborators +might not understand it). + +Maintainability +--------------- + +Well written standards allow the loading code to be written with assumptions +about what file content to expect. This code is much simpler than either fully +'agnostic' code which can load anything, or code which embeds various +workarounds for known problems. Simpler code takes less resource/expertise to +maintain, increasing the long-term sustainability of Iris. + +Robustness +---------- + +Redirecting problem content to :data:`iris.loading.LOAD_PROBLEMS` occurs in +places where Iris would otherwise raise an exception. This means that +Iris can continue to load all the valid parts of the file, and the user has +a way to fix problems **within Iris**, rather than learning a NetCDF tool or +similar. + +This will not only handle file problems, but also any current or future bugs in +the Iris codebase, until they are fixed in the next release. + +User Discretion +--------------- + +File malformations/non-conformances are by-definition not covered by any +standard for that file type - there is no consensus on the correct way to +represent this information. By avoiding encoding workarounds into Iris' +codebase, we avoid imposing one party's opinion onto other Iris users, who may +believe the problem should be handled differently. + +Diversity +--------- + +Several less 'opinionated' libraries are already available for those users that +want to load all content from their file, regardless of quality or meaning. +These libraries give the user the freedom to customise the handling of their +files as they see fit, but also put the onus on the user to understand the file +content and write code to handle it. Iris would be adding little new to the +ecosystem if it had an identical philosophy. + +Examples include: :term:`netCDF4`, :term:`Xarray`, `ecCodes`_. + +Instead, when working with the Iris data model, users can be confident in +the validity, and precise meaning (from the :term:`CF conventions`) of this +information. + +.. _filtering-warnings-explanation: + +Verbose Warnings +================ + +Different people use Iris for very different purposes, from quick file +visualisation to extract-transform-load to statistical analysis. These +contrasting priorities mean disagreement on which Iris problems can be ignored +and which are critically important. + +For problems that prevent Iris functioning: **Concrete Exceptions** are raised, which +stop code from running any further - no debate here. For less catastrophic +problems: **Warnings** are raised, +which notify you (in ``stderr``) but allow code to continue running. The Warnings are +there because Iris may **OR may not** function in the way you expect, +depending on what you need - e.g. a problem might prevent data being saved to +NetCDF, but statistical analysis will still work fine. + +This means that Iris' default behaviour is to raise Warnings +for anything that might be a problem for **any** user, since it cannot predict +specific user needs. It is designed to work with the user to ``ignore`` Warnings +which are not considered helpful in their specific use case. + +**How to ignore unwanted warnings:** :ref:`filtering-warnings` + +Examples of Iris Warnings +------------------------- + +- If you attempt to plot un-bounded point data as a ``pcolormesh``: Iris will + guess appropriate bounds around each point so that quadrilaterals can be + plotted. This permanently modifies the relevant coordinates, so the you are + warned in case downstream operations assume un-bounded coordinates. +- If you load a NetCDF file where a CF variable references another variable - + e.g. ``my_var:coordinates = "depth_var" ;`` - but the referenced variable + (``depth_var``) is not in the file: Iris will still construct + its data model, but without this reference relationship. You are warned since + the file includes an error and the loaded result might therefore not be as + expected. + +.. _ecCodes: https://github.com/ecmwf/eccodes diff --git a/docs/src/userguide/loading_iris_cubes.rst b/docs/src/userguide/loading_iris_cubes.rst index cbba4da39f..ac6b370466 100644 --- a/docs/src/userguide/loading_iris_cubes.rst +++ b/docs/src/userguide/loading_iris_cubes.rst @@ -298,3 +298,113 @@ these two cubes into separate variables. >>> print(number_two) 2 +.. _load-problems: + +Load Problems +------------- + +The Iris data model - see :ref:`iris_data_structures` - is highly flexible, but +there are many examples of file content that will not be loaded into the data +model. These fall into two categories: + +1. Malformations in the file. + - For example: a variable that is referenced, but is missing. +2. Content not conformant with the standard for that file type. + - Most commonly :term:`NetCDF` file content that is not + compliant with the :term:`CF conventions` - the basis for the Iris + data model. But Iris also relies on standards for other file + formats such as :term:`GRIB Format` and :term:`Post Processing (PP) Format`. + - Content in a non-NetCDF file that Iris does not know how to map onto CF + concepts. + - :term:`CF conventions` concepts that Iris does not support yet. + +.. note:: + + The below approach was introduced in Iris 3.12, and widely used in + CF-NetCDF loading by Iris 3.13. We hope to continue spreading it to + other file formats, and overlooked corner cases, in future releases. + +When Iris encounters problem content in a file, it will not make 'best efforts' +to parse the content, but will instead redirect it to +:data:`iris.loading.LOAD_PROBLEMS`, as well as issuing a warning to the user. +The user is then free to add any operations to their script(s) for +incorporating :data:`~iris.loading.LOAD_PROBLEMS` content into the Iris data +model, as they see fit. + +Find out more about *why* we chose this approach: :ref:`load-problems-explanation` + +The below example has 'booby trapped' the Iris loading process, to give an +impression of the user experience when loading problems are encountered. The +example shows typical :data:`~iris.loading.LOAD_PROBLEMS` content, and a +deeper inspection of one redirected object. **Much more detail is in the +API documentation for:** :class:`iris.loading.LoadProblems`. + +.. testsetup:: load-problems + + from pathlib import Path + from pprint import pprint + import sys + import warnings + + import iris + import iris.common + from iris.fileformats._nc_load_rules import helpers + import iris.loading + from iris import std_names + + # Ensure doctests actually see Warnings that are raised, and that + # they have a relative path (so a test pass is not machine-dependent). + showwarning_original = warnings.showwarning + warnings.filterwarnings("default") + IRIS_FILE = Path(iris.__file__) + def custom_warn(message, category, filename, lineno, file=None, line=None): + filepath = Path(filename) + filename = str(filepath.relative_to(IRIS_FILE.parents[1])) + sys.stdout.write(warnings.formatwarning(message, category, filename, lineno)) + warnings.showwarning = custom_warn + + get_names_original = helpers.get_names + + def raise_example_error_names(cf_coord_var, coord_name, attributes): + if cf_coord_var.cf_name == "time": + raise ValueError("Example coordinate error") + else: + return get_names_original( + cf_coord_var, coord_name, attributes + ) + + helpers.get_names = raise_example_error_names + air_temperature = std_names.STD_NAMES.pop("air_temperature") + iris.FUTURE.date_microseconds = True + +.. doctest:: load-problems + + >>> cube_a1b = iris.load_cube(iris.sample_data_path("A1B_north_america.nc")) + iris/...IrisLoadWarning: Not all file objects were parsed correctly. See iris.loading.LOAD_PROBLEMS for details. + warnings.warn(message, category=IrisLoadWarning) + >>> print(iris.loading.LOAD_PROBLEMS) + : + .../A1B_north_america.nc: "'air_temperature' is not a valid standard_name", {'standard_name': 'air_temperature'} + .../A1B_north_america.nc: "Example coordinate error", unknown / (unknown) (-- : 240) + >>> last_problem = iris.loading.LOAD_PROBLEMS.problems[-1] + >>> print(last_problem.loaded) + unknown / (unknown) (-- : 240) + Attributes:... + IRIS_RAW {'axis': 'T', ...} + >>> attributes = last_problem.loaded.attributes[ + ... iris.common.LimitedAttributeDict.IRIS_RAW + ... ] + >>> pprint(attributes) + {'axis': 'T', + 'bounds': 'time_bnds', + 'calendar': '360_day', + 'standard_name': 'time', + 'units': 'hours since 1970-01-01 00:00:00', + 'var_name': 'time'} + +.. testcleanup:: load-problems + + warnings.showwarning = showwarning_original + warnings.filterwarnings("ignore") + helpers.get_names = get_names_original + std_names.STD_NAMES["air_temperature"] = air_temperature diff --git a/docs/src/whatsnew/latest.rst b/docs/src/whatsnew/latest.rst index 6923db7179..ce5003958a 100644 --- a/docs/src/whatsnew/latest.rst +++ b/docs/src/whatsnew/latest.rst @@ -32,12 +32,13 @@ This document explains the changes made to Iris for this release =========== #. `@trexfeathers`_ and `@ukmo-ccbunney`_ extended the - :data:`iris.loading.LOAD_PROBLEMS` capturing to _all_ NetCDF objects that are + :data:`iris.loading.LOAD_PROBLEMS` capturing to *all* NetCDF objects that are added to a :class:`~iris.cube.Cube` during loading, as well as a selection of other objects such as :class:`~iris.coord_systems.CoordSystem`. Note this includes an improvement to how :class:`~iris.coords.DimCoord` is 'gracefully' converted to :class:`~iris.coords.AuxCoord` if it is masked - the mask is - now preserved when it was not previously. + now preserved when it was not previously. See also: :ref:`load-problems`. + (:pull:`6465`, :pull:`6529`) 🐛 Bugs Fixed @@ -94,7 +95,14 @@ This document explains the changes made to Iris for this release 📚 Documentation ================ -#. N/A +#. `@trexfeathers`_ and `@ukmo-ccbunney`_ added :ref:`load-problems` to the user + guide. (:pull:`6529`) + +#. `@trexfeathers`_ and `@ukmo-ccbunney`_ added a new user guide page: + :ref:`iris-philosophy`, for readers who are interested in why Iris is + designed/maintained the way it is. Initial content: :ref:`code-maintenance`, + :ref:`load-problems-explanation`, :ref:`filtering-warnings-explanation`. + (:pull:`6529`) 💼 Internal