diff --git a/doc/fig/EarthByte_logo_small.png b/doc/fig/EarthByte_logo_small.png new file mode 100644 index 00000000000..99e01de5977 Binary files /dev/null and b/doc/fig/EarthByte_logo_small.png differ diff --git a/doc/fig/igpp.png b/doc/fig/igpp.png new file mode 100644 index 00000000000..c347a041f5e Binary files /dev/null and b/doc/fig/igpp.png differ diff --git a/doc/fig/nasa-logo-web-rgb.png b/doc/fig/nasa-logo-web-rgb.png new file mode 100644 index 00000000000..511e80812e8 Binary files /dev/null and b/doc/fig/nasa-logo-web-rgb.png differ diff --git a/doc/rst/source/datasets.rst b/doc/rst/source/datasets.rst index 8399f92d1c8..bd33c877d2b 100644 --- a/doc/rst/source/datasets.rst +++ b/doc/rst/source/datasets.rst @@ -3,9 +3,6 @@ Datasets .. toctree:: + datasets/remote-data datasets/gshhg datasets/dcw - datasets/earth-relief - datasets/earth-daynight - datasets/earth-masks - datasets/earth-age diff --git a/doc/rst/source/datasets/earth-age.rst b/doc/rst/source/datasets/earth-age.rst_ similarity index 73% rename from doc/rst/source/datasets/earth-age.rst rename to doc/rst/source/datasets/earth-age.rst_ index a75ed5aa816..d84f8ecf6da 100644 --- a/doc/rst/source/datasets/earth-age.rst +++ b/doc/rst/source/datasets/earth-age.rst_ @@ -1,5 +1,8 @@ Global Earth Seafloor Crustal Age Grids -======================================= +--------------------------------------- +.. figure:: /_images/EarthByte_logo_small.png + :align: right + :scale: 20 % .. figure:: /_images/GMT_agefig.* :width: 710 px @@ -11,24 +14,14 @@ the creation of crustal age grids since 1997 and we offer their latest version f remote use in GMT. Usage ------ +~~~~~ -We have processed and reformatted the EarthByte crustal age grids -and standardized their file names. In GMT, you may access a global crustal age grid -(or a subset only by using the **-R** option) by specifying the special name +You access a global crustal age grid by specifying the special name @earth_age_\ *rr*\ *u*\ [_\ *reg*\ ] -where *rr* is a 2-digit integer specifying the grid resolution in the unit *u*, where -*u* is either **d**, **m** or **s** for arc degree, arc minute or arc second, respectively. -Optionally, you can append _\ **g** or _\ **p** to specifically get the gridline-registered or -pixel-registered version (if they both exist). If *reg* is not specified we will return -the pixel-registered version unless only the gridline-registered file is available. The following codes for *rr*\ *u* and the optional *reg* are supported (dimensions are listed -for pixel-registered grids; gridline-registered grids increment dimensions by one). -The full (6 sigma) filter-widths used to downsample the 1 min resolution are indicated in -parenthesis. **Note**: Like the Earth relief data, grids of 5 arc minute and smaller resolutions -are tiled for faster download. +for pixel-registered grids; gridline-registered grids increment dimensions by one): .. _tbl-earth_age: @@ -53,7 +46,7 @@ the earth_age files being placed in an ``earth/earth_age`` sub-directory. If you specify a CPT, the default CPT for this data set will be used (*@age_chrons_GTS2012_2020.cpt*) Technical Information ---------------------- +~~~~~~~~~~~~~~~~~~~~~ We scale and reformat the original data to take up very little space so that downloads from the servers are as fast as possible. For the seafloor crustal age grid this means @@ -63,6 +56,6 @@ by netCDF lossless compression and chunking. The data are reported in Myr relat to the 2012 Geological Time Scale. Data References ---------------- +~~~~~~~~~~~~~~~ #. Seton et al. [2020, in review] [https://www.earthbyte.org/category/resources/data-models/seafloor-age/]. diff --git a/doc/rst/source/datasets/earth-daynight.rst b/doc/rst/source/datasets/earth-daynight.rst_ similarity index 84% rename from doc/rst/source/datasets/earth-daynight.rst rename to doc/rst/source/datasets/earth-daynight.rst_ index 0601e8d7783..3f68c6bdc54 100644 --- a/doc/rst/source/datasets/earth-daynight.rst +++ b/doc/rst/source/datasets/earth-daynight.rst_ @@ -1,5 +1,8 @@ Global Earth Day/Night Images -============================= +----------------------------- +.. figure:: /_images/nasa-logo-web-rgb.png + :align: right + :scale: 20 % .. figure:: /_images/daynight.jpg :height: 888 px @@ -15,11 +18,9 @@ by :doc:`/grdmix`. The above example mixes both images according to a day-night mask and adds illumination from a corresponding Earth DEM. Usage ------ +~~~~~ -We have processed and reformatted this publicly available global imagery -and standardized their file names. In GMT, you may access a global daytime image -by specifying the special names +You access a global daytime image by specifying the special names @earth_day_\ *rr*\ *u* @@ -27,12 +28,10 @@ Similarly for the nighttime view: @earth_night_\ *rr*\ *u* -Here, *rr* is a 2-digit integer specifying the image resolution in the unit *u*, where -*u* is either **d**, **m** or **s** for arc degree, arc minute or arc second, respectively. -Note there is no file extension specified. -The following codes for *rr*\ *u* are supported: +The following codes for *rr*\ *u* and the optional *reg* are supported (dimensions are listed +for pixel-registered grids; gridline-registered grids increment dimensions by one): -.. _tbl-earth_relief: +.. _tbl-earth_daynight: ==== ================= ======= =========================================================== Code Dimensions Size Description @@ -55,7 +54,7 @@ All of these images will, when downloaded, be placed in your ~/.gmt/server direc the ``earth/earth_day`` and ``earth/earth_night`` sub-directories. Technical Information ---------------------- +~~~~~~~~~~~~~~~~~~~~~ The 01m and lower resolution images are derivatives of NASA's Blue and Black marble image mosaics. We have downsampled them via Cartesian Gaussian filtering to prevent aliasing while preserving @@ -66,10 +65,8 @@ to the geotiff files on the remote server. **Note**: This data set is experiment format and delivery is likely to change in the future (e.g., via image tiles). To make the files as small as possible we have also downgraded them from 24-bit to 8-bit indexed images. -.. include:: ../data-updating.rst_ - Data References ---------------- +~~~~~~~~~~~~~~~ #. Blue Marble [https://visibleearth.nasa.gov/images/57752/blue-marble-land-surface-shallow-water-and-shaded-topography]. #. Black Marble: [https://earthobservatory.nasa.gov/features/NightLights/page3.php]. diff --git a/doc/rst/source/datasets/earth-masks.rst b/doc/rst/source/datasets/earth-masks.rst_ similarity index 77% rename from doc/rst/source/datasets/earth-masks.rst rename to doc/rst/source/datasets/earth-masks.rst_ index 4e745b2be60..a150eae29f9 100644 --- a/doc/rst/source/datasets/earth-masks.rst +++ b/doc/rst/source/datasets/earth-masks.rst_ @@ -1,5 +1,5 @@ Global Earth Mask Grids -======================= +----------------------- .. figure:: /_images/GMT_earthmask.* :height: 888 px @@ -13,20 +13,17 @@ via the clipping in :doc:`/coast`, you may still want to perform calculations on gridded data and apply masks via :doc:`/grdmath`. Such mask grids can be computed via :doc:`/grdlandmask` but they can take a long time to compute for large regions, the full GSHHG resolution, and small grid spacings. For these reasons we offer -precalculated mask grids via the remote server mechanism. In GMT, you may access a -global mask grid (or a subset only by using the **-R** option) by specifying the -special name +precalculated mask grids via the remote server mechanism. + +Usage +~~~~~ + +You access a global mask grid by specifying the special name @earth_mask_\ *rr*\ *u*\ [_\ *reg*\ ] -where *rr* is a 2-digit integer specifying the grid resolution in the unit *u*, where -*u* is either **d**, **m** or **s** for arc degree, arc minute or arc second, respectively. -Optionally, you can append _\ **g** or _\ **p** to specifically get the gridline-registered or -pixel-registered version (if they both exist). If *reg* is not specified we will return -the pixel-registered version. All masks are stored as compressed byte-grids and take very -little space, hence no tiling is implemented. The following codes for *rr*\ *u* and the -optional *reg* are supported (dimensions are for the pixel-registered grid; add one to -each dimension for gridline-registered grids): +The following codes for *rr*\ *u* and the optional *reg* are supported (dimensions are listed +for pixel-registered grids; gridline-registered grids increment dimensions by one): .. _tbl-earth_masks: @@ -52,7 +49,7 @@ All of these data will, when downloaded, be placed in your ~/.gmt/server directo the Earth mask files being placed in an ``earth/earth_mask`` sub-directory. Technical Information ---------------------- +~~~~~~~~~~~~~~~~~~~~~ Given that GSHHG contains 5 levels (ocean, land, lake, island-in-lake, pond-in-island-in-lake) corresponding to levels 0-4, the mask grids were computed to reflect those 5 levels. This @@ -70,5 +67,3 @@ To make a pixel-registered mask that has NaNs on land and 1 in the oceans, use:: Finally, to make a pixel-registered mask that is 1 for lakes and NaN elsewhere, try:: gmt grdmath @earth_mask_01m 2 EQ 0 NAN = lakes_and_NaN.grd - -.. include:: ../data-updating.rst_ diff --git a/doc/rst/source/datasets/earth-relief.rst b/doc/rst/source/datasets/earth-relief.rst deleted file mode 100644 index 191e27b345d..00000000000 --- a/doc/rst/source/datasets/earth-relief.rst +++ /dev/null @@ -1,137 +0,0 @@ -Global Earth Relief Grids -========================= - -.. figure:: /_images/dem.jpg - :height: 888 px - :width: 1774 px - :align: center - :scale: 40 % - -In addition to the GSHHG coastlines, rivers, and borders data built into some -modules (:doc:`/coast`, :doc:`/gmtselect`, :doc:`/grdlandmask`), we offer several -remote global data grids that you can access via our *remote file* mechanism. -The first time you access one of these files we will download the file from -the GMT server and save it to the *server* directory under your GMT user directory (~/.gmt). -We then read the file from there. Should you have limited space then there -are ways to control how much is downloaded (see below). - -Usage ------ - -We have processed and reformatted publicly available global relief -grids and standardized their file names. In GMT, you may access a global relief grid -(or a subset only by using the **-R** option) by specifying the special name - - @earth_relief_\ *rr*\ *u*\ [_\ *reg*\ ] - -where *rr* is a 2-digit integer specifying the grid resolution in the unit *u*, where -*u* is either **d**, **m** or **s** for arc degree, arc minute or arc second, respectively. -Optionally, you can append _\ **g** or _\ **p** to specifically get the gridline-registered or -pixel-registered version (if they both exist). If *reg* is not specified we will return -the pixel-registered version unless only the gridline-registered file is available. -The following codes for *rr*\ *u* and the optional *reg* are supported (dimensions are listed -for pixel-registered grids; gridline-registered grids increment dimensions by one): - -.. _tbl-earth_relief: - -==== ================= === ======= ================================================== -Code Dimensions Reg Size Description -==== ================= === ======= ================================================== -01d 360 x 180 g,p 128 KB 1 arc degree global relief (SRTM15+V2.1 @ 111 km) -30m 720 x 360 g,p 435 KB 30 arc minute global relief (SRTM15+V2.1 @ 55 km) -20m 1080 x 540 g,p 918 KB 20 arc minute global relief (SRTM15+V2.1 @ 37 km) -15m 1440 x 720 g,p 1.6 MB 15 arc minute global relief (SRTM15+V2.1 @ 28 km) -10m 2160 x 1080 g,p 3.4 MB 10 arc minute global relief (SRTM15+V2.1 @ 18 km) -06m 3600 x 1800 g,p 8.8 MB 6 arc minute global relief (SRTM15+V2.1 @ 10 km) -05m 4320 x 2160 g,p 13 MB 5 arc minute global relief (SRTM15+V2.1 @ 9 km) -04m 5400 x 2700 g,p 19 MB 4 arc minute global relief (SRTM15+V2.1 @ 7.5 km) -03m 7200 x 3600 g,p 33 MB 3 arc minute global relief (SRTM15+V2.1 @ 5.6 km) -02m 10800 x 5400 g,p 71 MB 2 arc minute global relief (SRTM15+V2.1 @ 3.7 km) -01m 21600 x 10800 g,p 258 MB 1 arc minute global relief (SRTM15+V2.1 @ 1.9 km) -30s 43200 x 21600 g,p 935 MB 30 arc second global relief (SRTM15+V2.1 @ 1.0 km) -15s 86400 x 43200 p 3.2 GB 15 arc second global relief (SRTM15+V2.1) -03s 432000 x 216000 g 6.8 GB 3 arc second global relief (SRTM3S) -01s 1296000 x 432000 g 41 GB 1 arc second global relief (SRTM1S) -==== ================= === ======= ================================================== - -All of these data will, when downloaded, be placed in your ~/.gmt/server directory, with -the earth_relief files being placed in an ``earth/earth_relief`` sub-directory. If you -do not specify a CPT then this dataset default to the GMT master *geo*. - -Technical Information ---------------------- - -As you see, the 30s and lower resolutions are all derivatives of Scripps' SRTM15+V2.1 grid -(Tozer et al., 2019). We have downsampled it via Cartesian Gaussian filtering to prevent -aliasing while preserving the latitude-dependent resolution in the original 15 arc sec grid. -The full (6 sigma) filter-widths are indicated in parenthesis. The 3 and 1 arc second data -are the SRTM 1x1 degree tiles from NASA. To improve responsiveness, the larger files (i.e., -for grid spacings 05m and smaller) have been tiled as well. When the 06m or lower resolution -grids are accessed the first time we download the entire file, regardless of your selected region (**-R**). -However, for the tiled data sets we only download the tiles that are inside your selected region -the first time they are referenced. **Note**: The 3 and 1 arc second grids only extend -to latitudes ±60˚ and are only available over land. When these grids are accessed as -@earth_relief_01s or @earth_relief_03s we will automatically up-sample the relevant @earth_relief_15s -tiles to fill in the missing ocean values. If you just want the original land-only SRTM tiles -you may use the special names @srtm_relief_03s or @srtm_relief_01s instead. Almost all grids -are available in both gridline- and pixel-registered formats except the original pixel-registered -SRTM15+V2.1 (here called @earth_relief_15s) and the gridline-registered SRTM tiles. - -The dimensions above reflect the number of nodes covered by the global grids and the sizes refer -to the files on the remote server. For single grids, these are already in the final netCDF-4 -compressed short int grids, making the files much smaller than their original source files without -any loss of precision. To minimize download speed, the dataset tiles are all stored as JPEG2000 -images on the GMT server due to superior compression, but once downloaded to your server directory -they are converted to the same short int compressed netCDF4 format for easier access. This step -uses our GDAL bridge and thus requires that you have built GMT with GDAL support -*and* that your GDAL distribution was built with openjpeg support. - - -.. _jp2_compression: - -.. figure:: /_images/srtm1.* - :width: 500 px - :align: center - - Histogram of compression rates for the SRTM 1x1 arc second tiles. 100% reflects the full short integer - size of an uncompressed tile (~25 Mb). As can be seen, on average a JPEG2000 tile is only half the - size of the corresponding fully compressed (level 9) netCDF short int grid. This is why we - have chosen the JP2 format for tiles on the server. - -Data Space Concerns -------------------- - -There are several ways you can control the amount of space taken up by your own server directory: - -#. You can set an upper file size limit for download via the GMT default setting - :term:`GMT_DATA_SERVER_LIMIT`; the default is unlimited. -#. You can remove the entire server directory via gmt :doc:`/clear` data. -#. You can be clever and set up a crontab job that deletes data files you have not - touched in, say, 6 months (or some other interval). - -Offline Usage -------------- - -If you anticipate to be without an Internet connection (or a very slow one), you can download -all the remote files prior to losing connection, using the module :doc:`/gmtget`. You can choose -which data to download and limit it to grid spacings larger or equal to a limit, and you can -minimize space on your computer by requesting the JPEG2000 tiles not be converted until GMT -is accessing them. Here are some examples of usage. Download the entire cache directory used -in examples and tests:: - - gmt get -Dcache - -To get all the data for Earth but only for 30 arc sec and coarser, and leave as JPEG2000 tiles:: - - gmt get -Ddata=earth -I30s -N - -As shown in the table, the largest dataset may take some time to download the data from GMT server. Be patient! - -.. include:: ../data-updating.rst_ - -Data References ---------------- - -#. SRTM15+V2.1 [http://dx.doi.org/10.1029/2019EA000658]. -#. SRTMGL3 tiles: [https://lpdaac.usgs.gov/products/srtmgl3v003]. -#. SRTMGL1 tiles: [https://lpdaac.usgs.gov/products/srtmgl1v003]. diff --git a/doc/rst/source/datasets/earth-relief.rst_ b/doc/rst/source/datasets/earth-relief.rst_ new file mode 100644 index 00000000000..2cf8667fea5 --- /dev/null +++ b/doc/rst/source/datasets/earth-relief.rst_ @@ -0,0 +1,70 @@ +Global Earth Relief Grids +------------------------- + +.. figure:: /_images/igpp.png + :align: right + :scale: 20 % + +.. figure:: /_images/dem.jpg + :height: 888 px + :width: 1774 px + :align: center + :scale: 40 % + +Usage +~~~~~ + +You access a global relief grid by specifying the special name + + @earth_relief_\ *rr*\ *u*\ [_\ *reg*\ ] + +The following codes for *rr*\ *u* and the optional *reg* are supported (dimensions are listed +for pixel-registered grids; gridline-registered grids increment dimensions by one): + +.. _tbl-earth_relief: + +==== ================= === ======= ================================================== +Code Dimensions Reg Size Description +==== ================= === ======= ================================================== +01d 360 x 180 g,p 128 KB 1 arc degree global relief (SRTM15+V2.1 @ 111 km) +30m 720 x 360 g,p 435 KB 30 arc minute global relief (SRTM15+V2.1 @ 55 km) +20m 1080 x 540 g,p 918 KB 20 arc minute global relief (SRTM15+V2.1 @ 37 km) +15m 1440 x 720 g,p 1.6 MB 15 arc minute global relief (SRTM15+V2.1 @ 28 km) +10m 2160 x 1080 g,p 3.4 MB 10 arc minute global relief (SRTM15+V2.1 @ 18 km) +06m 3600 x 1800 g,p 8.8 MB 6 arc minute global relief (SRTM15+V2.1 @ 10 km) +05m 4320 x 2160 g,p 13 MB 5 arc minute global relief (SRTM15+V2.1 @ 9 km) +04m 5400 x 2700 g,p 19 MB 4 arc minute global relief (SRTM15+V2.1 @ 7.5 km) +03m 7200 x 3600 g,p 33 MB 3 arc minute global relief (SRTM15+V2.1 @ 5.6 km) +02m 10800 x 5400 g,p 71 MB 2 arc minute global relief (SRTM15+V2.1 @ 3.7 km) +01m 21600 x 10800 g,p 258 MB 1 arc minute global relief (SRTM15+V2.1 @ 1.9 km) +30s 43200 x 21600 g,p 935 MB 30 arc second global relief (SRTM15+V2.1 @ 1.0 km) +15s 86400 x 43200 p 3.2 GB 15 arc second global relief (SRTM15+V2.1) +03s 432000 x 216000 g 6.8 GB 3 arc second global relief (SRTM3S) +01s 1296000 x 432000 g 41 GB 1 arc second global relief (SRTM1S) +==== ================= === ======= ================================================== + +All of these data will, when downloaded, be placed in your ~/.gmt/server directory, with +the earth_relief files being placed in an ``earth/earth_relief`` sub-directory. If you +do not specify a CPT then this dataset default to the GMT master *geo*. + +Technical Information +~~~~~~~~~~~~~~~~~~~~~ + +As you see, the 30s and lower resolutions are all derivatives of Scripps' SRTM15+V2.1 grid +(Tozer et al., 2019). We have downsampled it via Cartesian Gaussian filtering to prevent +aliasing while preserving the latitude-dependent resolution in the original 15 arc sec grid. +The full (6 sigma) filter-widths are indicated in parenthesis. The 3 and 1 arc second data +are the SRTM 1x1 degree tiles from NASA. **Note**: The 3 and 1 arc second grids only extend +to latitudes ±60˚ and are only available over land. When these grids are accessed as +@earth_relief_01s or @earth_relief_03s we will automatically up-sample the relevant @earth_relief_15s +tiles to fill in the missing ocean values. If you just want the original land-only SRTM tiles +you may use the special names @srtm_relief_03s or @srtm_relief_01s instead. Almost all grids +are available in both gridline- and pixel-registered formats except the original pixel-registered +SRTM15+V2.1 (here called @earth_relief_15s) and the gridline-registered SRTM tiles. + +Data References +~~~~~~~~~~~~~~~ + +#. SRTM15+V2.1 [http://dx.doi.org/10.1029/2019EA000658]. +#. SRTMGL3 tiles: [https://lpdaac.usgs.gov/products/srtmgl3v003]. +#. SRTMGL1 tiles: [https://lpdaac.usgs.gov/products/srtmgl1v003]. diff --git a/doc/rst/source/datasets/remote-data.rst b/doc/rst/source/datasets/remote-data.rst new file mode 100644 index 00000000000..d94332f890f --- /dev/null +++ b/doc/rst/source/datasets/remote-data.rst @@ -0,0 +1,151 @@ +Remote Data Sets +================ + +A *remote data set* is a data set that is stored on one or more remote servers. It may +be a single grid file or a collection of subset tiles making up a larger grid. They +are not distributed with GMT or installed during the installation procedures. +GMT offers several remote global data grids that you can access via our *remote file* mechanism. +The first time you access one of these files, GMT will download the file (or a subset tile) from +the selected GMT server and save it to the *server* directory under your GMT user directory [~/.gmt]. +From then on we read the local file from there. + +By using the remote file mechanism you should know that these files, on the server, will change +from time to time (i.e., new versions are released, a problem in one file is fixed, or a dataset +becomes obsolete), and GMT will take actions accordingly. It is our policy to only supply the *latest* +version of any dataset that undergoes revisions. If you require previous versions for your work you +will need to get those data from the data provider separately. Unless you deactivate the remote data service, +GMT will do the following when you request a remote file in a GMT command: + +#. We check if the locally cached catalog with information about the data available from the server + is up-to-date or if it needs to be refreshed. If the file is older that the :term:`GMT_DATA_UPDATE_INTERVAL` + limit then we refresh the catalog. +#. When the catalog is refreshed, we determine the publication date for each dataset on the server, + and if any local copies you may have are now obsolete we will remove them to force a re-download from the server. + +Usage +----- + +We have processed and reformatted publicly available global data sets (grids and images) +and standardized their file names. In GMT, you may access such data +(or a subset only by using the **-R** option) by specifying the special name + + @remote_name_\ *rr*\ *u*\ [_\ *reg*\ ] + +where the leading @ symbol identifies the file as a remote data set, the *remote_name_* is specific +to the dataset and the *rr* code is a 2-digit integer specifying the grid/image +resolution in the unit *u*, where *u* is either **d**, **m** or **s** for arc degree, arc minute or +arc second, respectively. Optionally, you can append _\ **g** or _\ **p** to specifically get the +gridline-registered or pixel-registered version (if they both exist). If *reg* is not specified we +will return the pixel-registered version unless only the gridline-registered file is available. If you +do specify a specific registration and that version is not available you will get an error message. +The codes for *rr*\ *u* and the optional *reg* that are supported will be listed in the sections +below describing each of the available data sets. + + +Currently, GMT provides the following datasets (with their special names in parentheses) + +- `Global Earth Relief Grids`_ (``earth_relief``) +- `Global Earth Seafloor Crustal Age Grids`_ (``earth_age``) +- `Global Earth Day/Night Images`_ (``earth_day`` and ``earth_night``) +- `Global Earth Mask Grids`_ (``earth_mask``) + +Many of the remote datasets have a preferred, default color table that will be used unless you +override that default by giving your desired CPT information. + +Controlling the Process +----------------------- + +There are several ways you can control the remote data process and the amount of space taken up by your +own server directory: + +#. You can select the GMT data server closest to you to minimize download time [:term:`GMT_DATA_SERVER`]. +#. You can set an upper limit on the file sizes that may be downloaded [:term:`GMT_DATA_SERVER_LIMIT`]. +#. You can turn of the automatic download temporarily [:term:`GMT_AUTO_DOWNLOAD`]. +#. You can control how often GMT will refresh the catalog of information on your computer + [:term:`GMT_DATA_UPDATE_INTERVAL`] +#. You can clear the *server* directory, or perhaps just some subsets, any time via gmt :doc:`/clear`. + +Offline Usage +------------- + +If you anticipate to be without an Internet connection (or have a very slow one), you can download +all (or some) of the remote files prior to losing connection with the module :doc:`/gmtget`. You +can choose which data to download and limit it to node spacings larger or equal to a limit, and you +can minimize space on your computer by requesting that any JPEG2000 tiles *not* be converted until GMT +is accessing them. Here are some examples of usage. Download the entire cache directory used +in examples and tests:: + + gmt get -Dcache + +Get all the data for Earth but only for 1 arc minute and coarser, and leave tiles in JPEG2000 format:: + + gmt get -Ddata=earth -I1m -N + +As shown in the tables below, the largest datasets may take some time to download the data from GMT +server, so be patient! + +File Compression +---------------- + +Typically, a dataset is released by the data provider in a single, high-resolution format. +To optimize use of these data in GMT and to prevent download bottlenecks we have downsampled +them via Cartesian Gaussian filtering to prevent aliasing while preserving the latitude-dependent +resolution in the original grid or image. To improve responsiveness, the larger files (i.e., currently +for node spacings 05m and smaller) have been split into smaller tiles. When the 06m or lower resolution +files are accessed the first time we download the entire file, regardless of your selected region (**-R**). +However, for the tiled data sets we only download the tiles that intersect your selected region +the first time they are referenced. + +Single grids are provided as netCDF-4 maximum-lossless compressed short int grids, making the files +much smaller than their original source files without any loss of precision. To minimize download +speed, the dataset tiles are all stored as JPEG2000 images on the GMT server due to superior compression, +but once downloaded to your server directory they are converted to the same short int compressed netCDF4 +format for easier access. This step uses our GDAL bridge and thus requires that you have built GMT with +GDAL support *and* that your GDAL distribution was built with *openjpeg* support. + + +.. _jp2_compression: + +.. figure:: /_images/srtm1.* + :width: 500 px + :align: center + + Histogram of compression rates for the SRTM 1x1 arc second tiles. 100% reflects the full short integer + size of an uncompressed tile (~25 Mb). As can be seen, on average a JPEG2000 tile is only half the + size of the corresponding fully compressed (level 9) netCDF short int grid. This is why we + have chosen the JP2 format for tiles on the server. + +Cache File Updates +------------------ + +Remote cache files are our collection of miscellaneous files that are used throughout the GMT examples, +man pages, and test suite. There is no system nor catalog and files come and go as we need them. The cache +files are subject to similar rules as the remote data set when it comes to refreshing or deleting them. +If any of these files is precious to you we suggest you make a copy somewhere. + +Getting a single grid +--------------------- + +Should you need a single grid from any of our tiled dataset, e.g., to feed into other programs that do +not depend on GMT, you can create that via :doc:`/grdcut`. For instance, to make a global grid from the +eight tiles that make up the 2m x 2m gridline-registered data, try:: + + gmt grdcut @earth_relief_02m_g -Gearth_at_2m.grd + +---- + +.. include:: earth-relief.rst_ + +---- + +.. include:: earth-age.rst_ + +---- + +.. include:: earth-daynight.rst_ + +---- + +.. include:: earth-masks.rst_ + + diff --git a/doc/rst/source/gmt.conf.rst b/doc/rst/source/gmt.conf.rst index 1916291c1f7..2ccf6725ec2 100644 --- a/doc/rst/source/gmt.conf.rst +++ b/doc/rst/source/gmt.conf.rst @@ -320,6 +320,11 @@ GMT Miscellaneous Parameters the maximum file size in bytes, or append k, m, or g for kilo-, mega-, or giga-bytes. + **GMT_DATA_UPDATE_INTERVAL** + Specifies how often we update the local catalog of data available on + the remote server and pruning expired data sets [1d]. Allowable time + units are **d** (days), **w** (week), **o** (month, here 30 days). + **GMT_EXPORT_TYPE** This setting is only used by external interfaces and controls the data type used for table entries. Choose from double, diff --git a/src/gmt_defaults.h b/src/gmt_defaults.h index b6fa479dee2..9035e52436c 100644 --- a/src/gmt_defaults.h +++ b/src/gmt_defaults.h @@ -90,6 +90,7 @@ struct GMT_DEFAULTS { char format_time_stamp[GMT_LEN256]; /* Specify the format for writing time stamps (see strftime) */ /* GMT group */ size_t url_size_limit; + unsigned int refresh_time; /* Only refresh server catalog when the local copy is this old in days) */ unsigned int compatibility; /* Choose between 4 (GMT4) and up to latest version (5 for now) */ unsigned int auto_download; /* 0 (GMT_NO_DOWNLOAD) or 1 (GMT_YES_DOWNLOAD): For auto-downlaod of known files */ unsigned int interpolant; /* Choose between 0 (Linear), 1 (Akima), or 2 (Cubic spline) */ diff --git a/src/gmt_init.c b/src/gmt_init.c index eae6ebac907..7751376b334 100644 --- a/src/gmt_init.c +++ b/src/gmt_init.c @@ -175,6 +175,7 @@ static struct GMT5_params GMT5_keywords[]= { { 0, "GMT_AUTO_DOWNLOAD"}, { 0, "GMT_DATA_SERVER"}, { 0, "GMT_DATA_SERVER_LIMIT"}, + { 0, "GMT_DATA_UPDATE_INTERVAL"}, { 0, "GMT_COMPATIBILITY"}, { 0, "GMT_CUSTOM_LIBS"}, { 0, "GMT_EXPORT_TYPE"}, @@ -5994,6 +5995,8 @@ void gmt_conf (struct GMT_CTRL *GMT) { GMT->current.setting.auto_download = GMT_YES_DOWNLOAD; /* GMTCASE_GMT_DATA_SERVER_LIMIT */ GMT->current.setting.url_size_limit = 0; + /* GMTCASE_GMT_DATA_UPDATE_INTERVAL */ + GMT->current.setting.refresh_time = 1; /* GMT_CUSTOM_LIBS (default to none) */ /* GMT_EXPORT_TYPE */ GMT->current.setting.export_type = GMT_DOUBLE; @@ -10552,6 +10555,23 @@ unsigned int gmtlib_setparameter (struct GMT_CTRL *GMT, const char *keyword, cha } break; + case GMTCASE_GMT_DATA_UPDATE_INTERVAL: + if (lower_value[0]) { + size_t f, k = len - 1; + switch (lower_value[k]) { + case 'd': f = 1; break; + case 'w': f = 7; break; + case 'o': f = 30; break; + default: f = 1; break; + } + GMT->current.setting.refresh_time = atoi (lower_value) * f; + if (GMT->current.setting.refresh_time == 0) + error++; + } + else + error++; + break; + case GMTCASE_GMT_CUSTOM_LIBS: if (*value) { if (GMT->session.CUSTOM_LIBS) { @@ -11823,6 +11843,15 @@ char *gmtlib_putparameter (struct GMT_CTRL *GMT, const char *keyword) { snprintf (value, GMT_BUFSIZ, "%" PRIu64 "Gb", (uint64_t)GMT->current.setting.url_size_limit/(1024*1024*1024)); break; + case GMTCASE_GMT_DATA_UPDATE_INTERVAL: + if ((GMT->current.setting.refresh_time % 30) == 0) /* Whole "months" = 30 days */ + snprintf (value, GMT_BUFSIZ, "%do", GMT->current.setting.refresh_time / 30); + else if ((GMT->current.setting.refresh_time % 7) == 0) /* Whole weeks */ + snprintf (value, GMT_BUFSIZ, "%dw", GMT->current.setting.refresh_time / 7); + else /* Number of days */ + snprintf (value, GMT_BUFSIZ, "%dd", GMT->current.setting.refresh_time); + break; + case GMTCASE_GMT_CUSTOM_LIBS: strncpy (value, (GMT->session.CUSTOM_LIBS) ? GMT->session.CUSTOM_LIBS : "", GMT_BUFSIZ-1); break; @@ -13618,7 +13647,7 @@ struct GMT_CTRL *gmt_init_module (struct GMTAPI_CTRL *API, const char *lib_name, if (options) { for (opt = *options; opt; opt = opt->next) { /* Loop over all options */ if (remote_first && gmtinit_might_be_remotefile (opt->arg)) { - gmt_refresh_server (GMT); /* Refresh hash and info tables as needed */ + gmt_refresh_server (API); /* Refresh hash and info tables as needed */ remote_first = false; } if (opt->arg[0] != '@') continue; /* No remote file argument given */ diff --git a/src/gmt_keywords.txt b/src/gmt_keywords.txt index 1ff74a7e8c5..774e0d42d30 100644 --- a/src/gmt_keywords.txt +++ b/src/gmt_keywords.txt @@ -62,6 +62,7 @@ GMT_COMPATIBILITY # Oldest GMT version syntax to parse without error [4 and up] GMT_CUSTOM_LIBS # List of custom GMT-compliant shared libraries GMT_DATA_SERVER # Where to download from [SOEST] GMT_DATA_SERVER_LIMIT # Max file size to download remotely +GMT_DATA_UPDATE_INTERVAL # Frequency for refreshing local data catalog GMT_EXPORT_TYPE # Default type when GMT exports data to external interfaces GMT_EXTRAPOLATE_VAL # How to deal with 1D extrapolated values GMT_FFT # Which FFT algorithm to use diff --git a/src/gmt_prototypes.h b/src/gmt_prototypes.h index 82674a4552f..f460d14cbd3 100644 --- a/src/gmt_prototypes.h +++ b/src/gmt_prototypes.h @@ -170,7 +170,7 @@ EXTERN_MSC double gmt_fft_any_wave (uint64_t k, unsigned int mode, struct GMT_FF /* gmt_remote.c: */ -EXTERN_MSC void gmt_refresh_server (struct GMT_CTRL *GMT); +EXTERN_MSC void gmt_refresh_server (struct GMTAPI_CTRL *API); EXTERN_MSC int gmt_download_file (struct GMT_CTRL *GMT, const char *name, char *url, char *localfile, bool mode); EXTERN_MSC int gmt_set_remote_and_local_filenames (struct GMT_CTRL *GMT, const char* file, char *local_path, char *remote_path, unsigned int mode); EXTERN_MSC int gmt_remote_dataset_id (struct GMTAPI_CTRL *API, const char *file); diff --git a/src/gmt_remote.c b/src/gmt_remote.c index 2cc908a283f..e3ae4ac2a6e 100644 --- a/src/gmt_remote.c +++ b/src/gmt_remote.c @@ -178,54 +178,55 @@ GMT_LOCAL int gmtremote_remove_item (struct GMTAPI_CTRL *API, char *path, bool d return error; } -GMT_LOCAL struct GMT_DATA_INFO *gmtremote_data_load (struct GMT_CTRL *GMT, int *n) { +GMT_LOCAL struct GMT_DATA_INFO *gmtremote_data_load (struct GMTAPI_CTRL *API, int *n) { /* Read contents of the info file into an array of structs */ int k = 0, nr; FILE *fp = NULL; struct GMT_DATA_INFO *I = NULL; char unit, line[GMT_LEN512] = {""}, file[PATH_MAX] = {""}, *c = NULL; + struct GMT_CTRL *GMT = API->GMT; snprintf (file, PATH_MAX, "%s/server/%s", GMT->session.USERDIR, GMT_INFO_SERVER_FILE); - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "Load contents from %s\n", file); + GMT_Report (API, GMT_MSG_DEBUG, "Load contents from %s\n", file); *n = 0; if ((fp = fopen (file, "r")) == NULL) { - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Unable to open file %s\n", file); + GMT_Report (API, GMT_MSG_ERROR, "Unable to open file %s\n", file); return NULL; } if (fgets (line, GMT_LEN256, fp) == NULL) { /* Try to get first record */ fclose (fp); - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Read error first record in file %s\n", file); + GMT_Report (API, GMT_MSG_ERROR, "Read error first record in file %s\n", file); return NULL; } *n = atoi (line); /* Number of non-commented records to follow */ if (*n <= 0 || *n > GMT_BIG_CHUNK) { /* Probably not a good value */ fclose (fp); - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Bad record counter in file %s\n", file); + GMT_Report (API, GMT_MSG_ERROR, "Bad record counter in file %s\n", file); return NULL; } if (fgets (line, GMT_LEN256, fp) == NULL) { /* Try to get second record */ fclose (fp); - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Read error second record in file %s\n", file); + GMT_Report (API, GMT_MSG_ERROR, "Read error second record in file %s\n", file); return NULL; } if ((k = gmtremote_parse_version (line))) { fclose (fp); if (k == 2) - GMT_Report (GMT->parent, GMT_MSG_NOTICE, "Your GMT version too old to use the remote data mechanism - please upgrade to %s or later\n", line); + GMT_Report (API, GMT_MSG_NOTICE, "Your GMT version too old to use the remote data mechanism - please upgrade to %s or later\n", line); else - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Unable to parse \"%s\" to extract GMT version\n", line); + GMT_Report (API, GMT_MSG_ERROR, "Unable to parse \"%s\" to extract GMT version\n", line); return NULL; } if ((I = gmt_M_memory (GMT, NULL, *n, struct GMT_DATA_INFO)) == NULL) { - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Unable to allocated %d GMT_DATA_INFO structures!\n", *n); + GMT_Report (API, GMT_MSG_ERROR, "Unable to allocated %d GMT_DATA_INFO structures!\n", *n); return NULL; } while (fgets (line, GMT_LEN512, fp) != NULL) { if (line[0] == '#') continue; /* Skip any comments */ if ((nr = sscanf (line, "%s %s %s %c %lg %lg %s %lg %s %s %s %s %[^\n]", I[k].dir, I[k].file, I[k].inc, &I[k].reg, &I[k].scale, &I[k].offset, I[k].size, &I[k].tile_size, I[k].date, I[k].coverage, I[k].filler, I[k].CPT, I[k].remark)) != 13) { - GMT_Report (GMT->parent, GMT_MSG_WARNING, "File %s should have 13 fields but only %d read for record %d - download error???\n", file, nr, k); + GMT_Report (API, GMT_MSG_WARNING, "File %s should have 13 fields but only %d read for record %d - download error???\n", file, nr, k); gmt_M_free (GMT, I); fclose (fp); return NULL; @@ -245,8 +246,8 @@ GMT_LOCAL struct GMT_DATA_INFO *gmtremote_data_load (struct GMT_CTRL *GMT, int * fclose (fp); if (k != *n) { - GMT_Report (GMT->parent, GMT_MSG_WARNING, "File %s said it has %d records but only found %d - download error???\n", file, *n, k); - GMT_Report (GMT->parent, GMT_MSG_WARNING, "File %s should be deleted. Please try again\n", file); + GMT_Report (API, GMT_MSG_WARNING, "File %s said it has %d records but only found %d - download error???\n", file, *n, k); + GMT_Report (API, GMT_MSG_WARNING, "File %s should be deleted. Please try again\n", file); *n = 0; /* Flag that excrement hit the fan */ } /* Soft alphabetically on file names */ @@ -271,7 +272,7 @@ GMT_LOCAL struct GMT_DATA_INFO *gmtremote_data_load (struct GMT_CTRL *GMT, int * if (access (file, R_OK)) continue; /* No such file or directory yet */ /* Here we have a local copy of this remote file or directory - we examine its creation date */ if (stat (file, &buf)) { - GMT_Report (GMT->parent, GMT_MSG_WARNING, "Unable to get information about %s - skip\n", file); + GMT_Report (API, GMT_MSG_WARNING, "Unable to get information about %s - skip\n", file); continue; } /* Get its modification (creation) time */ @@ -293,7 +294,7 @@ GMT_LOCAL struct GMT_DATA_INFO *gmtremote_data_load (struct GMT_CTRL *GMT, int * } } /* If we get here we need to remove the outdated file or directory so we may download the latest on next try */ - if (gmtremote_remove_item (GMT->parent, file, I[k].tile_size > 0.0)) { + if (gmtremote_remove_item (API, file, I[k].tile_size > 0.0)) { GMT_Report (GMT->parent, GMT_MSG_WARNING, "Unable to remove %s \n", file); } } @@ -593,13 +594,14 @@ GMT_LOCAL struct GMT_DATA_HASH *gmtremote_hash_load (struct GMT_CTRL *GMT, char return (L); }; -GMT_LOCAL int gmtremote_refresh (struct GMT_CTRL *GMT, unsigned int index) { +GMT_LOCAL int gmtremote_refresh (struct GMTAPI_CTRL *API, unsigned int index) { /* This function is called every time we are about to access a @remotefile. * It is called twice: Once for the hash table and once for the info table. * First we check that we have the GMT_HASH_SERVER_FILE in the server directory. * If we don't then we download it and return since no old file to compare to. * If we do find the hash file then we get its creation time [st_mtime] as - * well as the current system time. If the file is < 1 day old we are done. + * well as the current system time. If the file is < GMT->current.setting.refresh_time + * days old we are done. * If the file is older we rename it to *.old and download the latest hash file. * This is the same for both values of index (hash and info). For hash, we do more: * Next, we load the contents of both files and do a double loop to find the @@ -614,6 +616,7 @@ GMT_LOCAL int gmtremote_refresh (struct GMT_CTRL *GMT, unsigned int index) { time_t mod_time, right_now = time (NULL); /* Unix time right now */ char indexpath[PATH_MAX] = {""}, old_indexpath[PATH_MAX] = {""}, new_indexpath[PATH_MAX] = {""}, url[PATH_MAX] = {""}; const char *index_file = (index == GMT_HASH_INDEX) ? GMT_HASH_SERVER_FILE : GMT_INFO_SERVER_FILE; + struct GMT_CTRL *GMT = API->GMT; /* Short hand */ if (GMT->current.io.refreshed[index]) return GMT_NOERROR; /* Already been here */ @@ -623,13 +626,13 @@ GMT_LOCAL int gmtremote_refresh (struct GMT_CTRL *GMT, unsigned int index) { char serverdir[PATH_MAX] = {""}; snprintf (serverdir, PATH_MAX, "%s/server", GMT->session.USERDIR); if (access (serverdir, R_OK) && gmt_mkdir (serverdir)) { - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Unable to create GMT server directory : %s\n", serverdir); + GMT_Report (API, GMT_MSG_ERROR, "Unable to create GMT server directory : %s\n", serverdir); return 1; } snprintf (url, PATH_MAX, "%s/%s", GMT->session.DATASERVER, index_file); - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "Download remote file %s for the first time\n", url); + GMT_Report (API, GMT_MSG_DEBUG, "Download remote file %s for the first time\n", url); if (gmtremote_get_url (GMT, url, indexpath, NULL, index)) { - GMT_Report (GMT->parent, GMT_MSG_INFORMATION, "Failed to get remote file %s\n", url); + GMT_Report (API, GMT_MSG_INFORMATION, "Failed to get remote file %s\n", url); if (!access (indexpath, F_OK)) gmt_remove_file (GMT, indexpath); /* Remove index file just in case it got corrupted or zero size */ GMT->current.setting.auto_download = GMT_NO_DOWNLOAD; /* Temporarily turn off auto download in this session only */ GMT->current.io.internet_error = true; /* No point trying again */ @@ -639,14 +642,14 @@ GMT_LOCAL int gmtremote_refresh (struct GMT_CTRL *GMT, unsigned int index) { return GMT_NOERROR; } else - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "Local file %s found\n", indexpath); + GMT_Report (API, GMT_MSG_DEBUG, "Local file %s found\n", indexpath); GMT->current.io.refreshed[index] = true; /* Done our job */ /* Here we have the existing index file and its path is in indexpath. Check how old it is */ if (stat (indexpath, &buf)) { - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Unable to get information about %s - abort\n", indexpath); + GMT_Report (API, GMT_MSG_ERROR, "Unable to get information about %s - abort\n", indexpath); return 1; } /* Get its modification (creation) time */ @@ -656,28 +659,28 @@ GMT_LOCAL int gmtremote_refresh (struct GMT_CTRL *GMT, unsigned int index) { mod_time = buf.st_mtime; #endif - if ((right_now - mod_time) > GMT_DAY2SEC_I) { /* Older than 1 day; Time to get a new index file */ - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "File %s older than 24 hours, get latest from server.\n", indexpath); + if ((right_now - mod_time) > (GMT_DAY2SEC_I * GMT->current.setting.refresh_time)) { /* Older than selected number of days; Time to get a new index file */ + GMT_Report (API, GMT_MSG_DEBUG, "File %s older than 24 hours, get latest from server.\n", indexpath); strcpy (new_indexpath, indexpath); /* Duplicate path name */ strcat (new_indexpath, ".new"); /* Append .new to the copied path */ strcpy (old_indexpath, indexpath); /* Duplicate path name */ strcat (old_indexpath, ".old"); /* Append .old to the copied path */ snprintf (url, PATH_MAX, "%s/%s", GMT->session.DATASERVER, index_file); /* Set remote path to new index file */ if (gmtremote_get_url (GMT, url, new_indexpath, indexpath, index)) { /* Get the new index file from server */ - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "Failed to download %s - Internet troubles?\n", url); + GMT_Report (API, GMT_MSG_DEBUG, "Failed to download %s - Internet troubles?\n", url); if (!access (new_indexpath, F_OK)) gmt_remove_file (GMT, new_indexpath); /* Remove index file just in case it got corrupted or zero size */ return 1; /* Unable to update the file (no Internet?) - skip the tests */ } if (!access (old_indexpath, F_OK)) remove (old_indexpath); /* Remove old index file if it exists */ - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "Rename %s to %s\n", indexpath, old_indexpath); + GMT_Report (API, GMT_MSG_DEBUG, "Rename %s to %s\n", indexpath, old_indexpath); if (gmt_rename_file (GMT, indexpath, old_indexpath, GMT_RENAME_FILE)) { /* Rename existing file to .old */ - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Failed to rename %s to %s.\n", indexpath, old_indexpath); + GMT_Report (API, GMT_MSG_ERROR, "Failed to rename %s to %s.\n", indexpath, old_indexpath); return 1; } - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "Rename %s to %s\n", new_indexpath, indexpath); + GMT_Report (API, GMT_MSG_DEBUG, "Rename %s to %s\n", new_indexpath, indexpath); if (gmt_rename_file (GMT, new_indexpath, indexpath, GMT_RENAME_FILE)) { /* Rename newly copied file to existing file */ - GMT_Report (GMT->parent, GMT_MSG_ERROR, "Failed to rename %s to %s.\n", new_indexpath, indexpath); + GMT_Report (API, GMT_MSG_ERROR, "Failed to rename %s to %s.\n", new_indexpath, indexpath); return 1; } @@ -700,26 +703,26 @@ GMT_LOCAL int gmtremote_refresh (struct GMT_CTRL *GMT, unsigned int index) { if (!strcmp (N[n].name, O[o].name)) { /* File is in the current hash table */ found = true; /* We will exit this loop regardless of what happens next below */ if (strcmp (N[n].hash, O[o].hash)) { /* New hash differs from entry in hash old file */ - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "Server and cache versions of %s have different hash codes - must download new copy.\n", N[n].name); + GMT_Report (API, GMT_MSG_DEBUG, "Server and cache versions of %s have different hash codes - must download new copy.\n", N[n].name); gmt_remove_file (GMT, url); /* Need to re-download so be gone with it */ } else { /* Do size check */ struct stat buf; if (stat (url, &buf)) { - GMT_Report (GMT->parent, GMT_MSG_WARNING, "Could not determine size of file %s.\n", url); + GMT_Report (API, GMT_MSG_WARNING, "Could not determine size of file %s.\n", url); continue; } if (N[n].size != (size_t)buf.st_size) { /* Downloaded file size differ - need to re-download */ - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "Server and cache versions of %s have different byte sizes (%" PRIuS " versus %" PRIuS ") - must download new copy.\n", N[n].name, N[n].size, (size_t)buf.st_size); + GMT_Report (API, GMT_MSG_DEBUG, "Server and cache versions of %s have different byte sizes (%" PRIuS " versus %" PRIuS ") - must download new copy.\n", N[n].name, N[n].size, (size_t)buf.st_size); gmt_remove_file (GMT, url); /* Need to re-download so be gone with it */ } else - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "Server and cache versions of %s are identical - no need to download new file.\n", N[n].name); + GMT_Report (API, GMT_MSG_DEBUG, "Server and cache versions of %s are identical - no need to download new file.\n", N[n].name); } } } if (!found) { /* This file was present locally but is no longer part of files on the server and should be removed */ - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "File %s no longer supported on server - deleting local copy.\n", O[o].name); + GMT_Report (API, GMT_MSG_DEBUG, "File %s no longer supported on server - deleting local copy.\n", O[o].name); gmt_remove_file (GMT, url); } } @@ -733,11 +736,11 @@ GMT_LOCAL int gmtremote_refresh (struct GMT_CTRL *GMT, unsigned int index) { GMT->current.io.new_data_list = true; /* Flag that we wish to delete datasets older than entries in this file */ } else - GMT_Report (GMT->parent, GMT_MSG_DEBUG, "File %s less than 24 hours old, refresh is premature.\n", indexpath); + GMT_Report (API, GMT_MSG_DEBUG, "File %s less than 24 hours old, refresh is premature.\n", indexpath); return GMT_NOERROR; } -void gmt_refresh_server (struct GMT_CTRL *GMT) { +void gmt_refresh_server (struct GMTAPI_CTRL *API) { /* Called once in gmt_begin from GMT_Create_Session, The following actions take place: * * The data info table is refreshed if missing or older than 24 hours. @@ -748,16 +751,16 @@ void gmt_refresh_server (struct GMT_CTRL *GMT) { * user requests it, it forces a download of the updated file. */ - if (gmtremote_refresh (GMT, GMT_INFO_INDEX)) /* Watch out for changes on the server info once a day */ - GMT_Report (GMT->parent, GMT_MSG_INFORMATION, "Unable to obtain remote information file %s\n", GMT_INFO_SERVER_FILE); - else { /* Get server file attribution info */ - if ((GMT->parent->remote_info = gmtremote_data_load (GMT, &GMT->parent->n_remote_info)) == NULL) { /* Failed to load the info file */ - GMT_Report (GMT->parent, GMT_MSG_INFORMATION, "Unable to read server information file\n"); + if (gmtremote_refresh (API, GMT_INFO_INDEX)) /* Watch out for changes on the server info once a day */ + GMT_Report (API, GMT_MSG_INFORMATION, "Unable to obtain remote information file %s\n", GMT_INFO_SERVER_FILE); + else if (API->remote_info == NULL) { /* Get server file attribution info if not yet loaded */ + if ((API->remote_info = gmtremote_data_load (API, &API->n_remote_info)) == NULL) { /* Failed to load the info file */ + GMT_Report (API, GMT_MSG_INFORMATION, "Unable to read server information file\n"); } } - if (gmtremote_refresh (GMT, GMT_HASH_INDEX)) { /* Watch out for changes on the server hash once a day */ - GMT_Report (GMT->parent, GMT_MSG_INFORMATION, "Unable to obtain remote hash table %s\n", GMT_HASH_SERVER_FILE); + if (gmtremote_refresh (API, GMT_HASH_INDEX)) { /* Watch out for changes on the server hash once a day */ + GMT_Report (API, GMT_MSG_INFORMATION, "Unable to obtain remote hash table %s\n", GMT_HASH_SERVER_FILE); } } diff --git a/src/gmtget.c b/src/gmtget.c index efccb6c7128..586a5e5a773 100644 --- a/src/gmtget.c +++ b/src/gmtget.c @@ -212,7 +212,7 @@ EXTERN_MSC int GMT_gmtget (void *V_API, int mode, void *args) { double world[4] = {-180.0, +180.0, -90.0, +90.0}; struct GMT_RECORD *Out = NULL; - gmt_refresh_server (GMT); /* Refresh hash and info tables as needed since we need to know what is there */ + gmt_refresh_server (API); /* Refresh hash and info tables as needed since we need to know what is there */ if (Ctrl->Q.active) { /* Must activate data output machinery for a DATASET with no numerical columns */ Out = gmt_new_record (GMT, NULL, message);