Data Access Methods

Marty Hidas

Australian Ocean Data Network (AODN)

This tutorial demostrates several ways data can be accessed remotely and loaded into a Python environment, including

• OPeNDAP

• OGC Web Feature Service (WFS)

• direct access to files on cloud storage (AWS S3)

• cloud-optimised formats Zarr & Parquet

The examples here use data from Australia’s Integrated Marine Observing System (IMOS). These can be browsed and accessed via the AODN Portal, the IMOS Metadata Catalogue, or the IMOS THREDDS Server. Each data collection’s metadata record includes more specific links to the relevant THREDDS folders, and WFS services.

# Import all the tools we need...
import os

# For working with data
import pandas as pd
import xarray as xr

# For data that may be larger than the memory available on your computer...
import dask
import dask.dataframe as dd

# For accessing OGC Web Feature Service
from owslib.wfs import WebFeatureService

# For accessing AWS S3 cloud storage
import s3fs

# Plotting tools
import holoviews as hv
import hvplot.pandas
import hvplot.xarray

# For plotting geographic data & maps
import geoviews as gv
import geoviews.feature as gf
from geoviews import opts
from cartopy import crs

## Use Matplotlib backend for web preview of notebook
## Comment out these lines to get the default interactive plots using Bokeh
hvplot.extension('matplotlib', compatibility='bokeh')
gv.extension('matplotlib')
gv.output(size=200)

# Set up local data path
DATA_BASEPATH = "/home/jovyan/shared/IMOS"

The old school way

The old (and still common) way to access data is to first download it to your computer and read it from there. This is easy for small datasets, but not always ideal:

• What if the data is bigger than your hard disk?

• What if you only need a small fraction of a dataset?

• What if the dataset is routinely updated and you want to re-run your analysis on the latest data?

• What if you want to run your analysis on another computer or in the cloud?

These days it is often more convenient to have data managed in a central location and access it remotely. There are many ways this can be done. In this tutorial we will look at a few of the common ones, and some of the newer ones.

OPeNDAP

• OPeNDAP stands for “Open-source Project for a Network Data Access Protocol”

• Provides access to metadata and data subsets via the Web without downloading an entire dataset

• Many tools that can read NetCDF files can also talk to an OPeNDAP URL directly

In Python, we can simply open the URL with xarray, then proceed with our analysis using the resulgting Dataset object.

Here we use an example from the AODN THREDDS server.

opendap_url = ("https://thredds.aodn.org.au/thredds/dodsC/"
               "IMOS/ANMN/NSW/PH100/gridded_timeseries/"
               "IMOS_ANMN-NSW_TZ_20091029_PH100_FV02_TEMP-gridded-timeseries_END-20230316_C-20230520.nc")

# You can preview the file's metadata in your browser by adding ".html" to the above URL
print(opendap_url + ".html")

https://thredds.aodn.org.au/thredds/dodsC/IMOS/ANMN/NSW/PH100/gridded_timeseries/IMOS_ANMN-NSW_TZ_20091029_PH100_FV02_TEMP-gridded-timeseries_END-20230316_C-20230520.nc.html

## Uncomment the line below to access a local copy of data file, (in case server is overloaded)

# opendap_url = os.path.join(DATA_BASEPATH,
#                           "IMOS_ANMN-NSW_TZ_20091029_PH100_FV02_TEMP-gridded-timeseries_END-20230316_C-20230520.nc")

ds_mooring = xr.open_dataset(opendap_url)
ds_mooring

<xarray.Dataset>
Dimensions:     (TIME: 115050, DEPTH: 12)
Coordinates:
  * DEPTH       (DEPTH) float32 0.0 10.0 20.0 30.0 ... 80.0 90.0 100.0 110.0
  * TIME        (TIME) datetime64[ns] 2009-10-29T03:00:00 ... 2023-03-16T10:0...
    LONGITUDE   float64 151.2
    LATITUDE    float64 -34.12
Data variables:
    TEMP_count  (TIME) int16 ...
    TEMP        (TIME, DEPTH) float32 ...
Attributes: (12/42)
    Conventions:                   CF-1.6,IMOS-1.4
    abstract:                      Gridded Time Series Product: This file con...
    acknowledgement:               Any users of IMOS data are required to cle...
    author:                        Australian Ocean Data Network (AODN)
    author_email:                  [email protected]
    citation:                      The citation in a list of references is: "...
    ...                            ...
    source_file_download:          https://s3-ap-southeast-2.amazonaws.com/im...
    source_file_opendap:           http://thredds.aodn.org.au/thredds/dodsC/I...
    standard_name_vocabulary:      NetCDF Climate and Forecast (CF) Metadata ...
    time_coverage_end:             2023-03-16T10:00:00Z
    time_coverage_start:           2009-10-29T03:00:00Z
    title:                         Gridded Time Series Product: TEMP interpol...

xarray.Dataset

• Dimensions:
  • TIME: 115050
  • DEPTH: 12
• Coordinates: (4)
  • DEPTH
    (DEPTH)
    float32
    0.0 10.0 20.0 ... 90.0 100.0 110.0

    long_name :

    target depth

    positive :

    down

    reference_datum :

    sea surface

    standard_name :

    depth

    units :

    m

    valid_max :

    12000.0

    valid_min :

    -5.0

    _ChunkSizes :

    12

    array([  0.,  10.,  20.,  30.,  40.,  50.,  60.,  70.,  80.,  90., 100., 110.],
          dtype=float32)

  • TIME
    (TIME)
    datetime64[ns]
    2009-10-29T03:00:00 ... 2023-03-...

    axis :

    T

    comment :

    time stamp corresponds to the hour and represents binned data [30,30) minutes before and after the hour

    long_name :

    time

    standard_name :

    time

    valid_max :

    90000.0

    valid_min :

    0.0

    _ChunkSizes :

    115050

    array(['2009-10-29T03:00:00.000000000', '2009-10-29T04:00:00.000000000',
           '2009-10-29T05:00:00.000000000', ..., '2023-03-16T08:00:00.000000000',
           '2023-03-16T09:00:00.000000000', '2023-03-16T10:00:00.000000000'],
          dtype='datetime64[ns]')

  • LONGITUDE
    ()
    float64
    ...

    axis :

    X

    long_name :

    longitude

    reference_datum :

    WGS84 geographic coordinate system

    standard_name :

    longitude

    comment :

    mean of all values over time interval

    units :

    degrees_east

    valid_max :

    180.0

    valid_min :

    -180.0

    array(151.224452)

  • LATITUDE
    ()
    float64
    ...

    axis :

    Y

    long_name :

    latitude

    reference_datum :

    WGS84 geographic coordinate system

    standard_name :

    latitude

    comment :

    mean of all values over time interval

    units :

    degrees_north

    valid_max :

    90.0

    valid_min :

    -90.0

    array(-34.119776)

• Data variables: (2)
  • TEMP_count
    (TIME)
    int16
    ...

    long_name :

    number of observations in the water column

    cell_methods :

    DEPTH:count

    units :

    1

    _ChunkSizes :

    115050

    [115050 values with dtype=int16]

  • TEMP
    (TIME, DEPTH)
    float32
    ...

    long_name :

    sea_water_temperature

    standard_name :

    sea_water_temperature

    cell_methods :

    TIME:median DEPTH: point

    units :

    degrees_Celsius

    valid_max :

    40.0

    valid_min :

    -2.5

    _ChunkSizes :

    [57525 6]

    [1380600 values with dtype=float32]

• Attributes: (42)

  Conventions :

  CF-1.6,IMOS-1.4

  abstract :

  Gridded Time Series Product: This file contains TEMP readings from all instruments deployed at the PH100 mooring site. The source of the values is the Hourly Time Series Product where TIME is fixed to 1-hour interval. The variable values are interpolated to a fixed target depths using a linear interpolation between consecutive existing depths. Only values flagged as 1 or 2 are used in the interpolation.

  acknowledgement :

  Any users of IMOS data are required to clearly acknowledge the source of the material derived from IMOS in the format: "Data was sourced from the Integrated Marine Observing System (IMOS) - IMOS is a national collaborative research infrastructure, supported by the Australian Government." If relevant, also credit other organisations involved in collection of this particular datastream (as listed in 'credit' in the metadata record).

  author :

  Australian Ocean Data Network (AODN)

  author_email :

  [email protected]

  citation :

  The citation in a list of references is: "IMOS [year-of-data-download], [Title], [data-access-URL], accessed [date-of-access].".

  contributor_email :

  [email protected]; [email protected]; [email protected]

  contributor_name :

  Anderson, Zachary; Austin, Timothy; Roughan, Moninya

  contributor_role :

  author; author; principal_investigator

  data_centre :

  Australian Ocean Data Network (AODN)

  data_centre_email :

  [email protected]

  date_created :

  2023-05-20T21:07:41Z

  disclaimer :

  Data, products and services from IMOS are provided "as is" without any warranty as to fitness for a particular purpose.

  featureType :

  timeSeriesProfile

  file_version :

  Level 2 - Derived Products

  file_version_quality_control :

  Derived products require scientific and technical interpretation. Normally these will be defined by the community that collects or utilises the data.

  generating_code_version :

  1.6.5

  geospatial_lat_max :

  -34.1192

  geospatial_lat_min :

  -34.1206833333

  geospatial_lon_max :

  151.2252666667

  geospatial_lon_min :

  151.22405

  geospatial_vertical_max :

  110

  geospatial_vertical_min :

  0

  history :

  2023-05-20T20:39:19Z: Hourly aggregated file created. 2023-05-20T21:07:41Z: Gridded file created.

  included_values_flagged_as :

  Good_data, Probably_good_data

  institution_references :

  http://imos.org.au/facilities/aodn/

  keywords :

  TEMP, DEPTH, HOURLY, GRIDDED

  keywords_vocabulary :

  IMOS parameter names. See https://github.com/aodn/imos-toolbox/blob/master/IMOS/imosParameters.txt

  license :

  http://creativecommons.org/licenses/by/4.0/

  lineage :

  The Variable of Interest (VoI) is produced by sequentially interpolating linearly the individual values at pre-defined target depths using the 1-hour binned data from the Hourly Time Series Product. The number of instruments used for the interpolation at each timestamp is recorded in the VoI_count variable. If less that two instrument readings are present in the timestamp, fill values are stored in all depth bins. The resulting variable has dimensions TIME and DEPTH. This file was created using https://github.com/aodn/python-aodntools/blob/1.6.5/aodntools/timeseries_products/gridded_timeseries.py

  naming_authority :

  IMOS

  project :

  Integrated Marine Observing System (IMOS)

  references :

  http://www.imos.org.au

  site_code :

  PH100

  source :

  Mooring

  source_file :

  IMOS/ANMN/NSW/PH100/hourly_timeseries/IMOS_ANMN-NSW_BOSTUZ_20091029_PH100_FV02_hourly-timeseries_END-20230316_C-20230520.nc

  source_file_download :

  https://s3-ap-southeast-2.amazonaws.com/imos-data/IMOS/ANMN/NSW/PH100/hourly_timeseries/IMOS_ANMN-NSW_BOSTUZ_20091029_PH100_FV02_hourly-timeseries_END-20230316_C-20230520.nc

  source_file_opendap :

  http://thredds.aodn.org.au/thredds/dodsC/IMOS/ANMN/NSW/PH100/hourly_timeseries/IMOS_ANMN-NSW_BOSTUZ_20091029_PH100_FV02_hourly-timeseries_END-20230316_C-20230520.nc

  standard_name_vocabulary :

  NetCDF Climate and Forecast (CF) Metadata Convention Standard Name Table 45

  time_coverage_end :

  2023-03-16T10:00:00Z

  time_coverage_start :

  2009-10-29T03:00:00Z

  title :

  Gridded Time Series Product: TEMP interpolated at PH100 to fixed target depths at 1-hour time intervals, between 2009-10-29T03:00:00Z and 2023-03-16T10:00:00Z and 0 and 110 meters.

print(ds_mooring.title)

Gridded Time Series Product: TEMP interpolated at PH100 to fixed target depths at 1-hour time intervals, between 2009-10-29T03:00:00Z and 2023-03-16T10:00:00Z and 0 and 110 meters.

This dataset is derived from repeated deployments of moored temperature loggers, binned to hourly intervals and interpolated to a fixed set of target depths. See the file metadata, or the associated metadata record for more info.

# Hourly averages x 12 depths x 13+ yr = over a million points to plot!
# Let's just look at a year's worth to speed things up...
ds_mooring.sel(TIME="2022").hvplot.scatter(x="TIME", y="DEPTH", c="TEMP",
                                           cmap="coolwarm", alpha=0.2,
                                           flip_yaxis=True, hover=False)

# ... or we can look at the full timeseries of temperature at a single depth
ds_mooring.TEMP.sel(DEPTH=30).hvplot()

Another example

See also the OPeNDAP example in last year’s data access tutorial, or explore that dataset in the OPeNDAP Access Form.

Web Feature Service (WFS)

• A standard of the Open Geospatial Consortium (OGC)

• Allows tabular geospatial data to be accessed via the Web.

• A feature has a geometry (e.g. a point/line/polygon) indicating a geographic location, and a set of properties (e.g. temperature)

• WFS allows filtering based on geometry or properties.

• In Python WFS and other OGC Web Services (OWS) can be accessed using the owslib library

For example, most of the tabular hosted by the AODN is available via WFS.

wfs = WebFeatureService(url="https://geoserver-123.aodn.org.au/geoserver/wfs",
                        version="1.1.0")
wfs.identification.title

'AODN Web Feature Service (WFS)'

# Each dataset is served as a separate "feature type":
print(f"There are {len(wfs.contents)} fature types, e.g.")
list(wfs.contents)[:10]

There are 397 fature types, e.g.

['imos:anmn_ctd_profiles_data',
 'imos:anmn_ctd_profiles_map',
 'imos:anmn_velocity_timeseries_map',
 'imos:anmn_nrs_rt_meteo_timeseries_data',
 'imos:anmn_nrs_rt_meteo_timeseries_map',
 'imos:anmn_nrs_rt_bio_timeseries_data',
 'imos:anmn_nrs_rt_bio_timeseries_map',
 'imos:anmn_nrs_rt_wave_timeseries_data',
 'imos:anmn_nrs_rt_wave_timeseries_map',
 'imos:anmn_acoustics_map']

For now we’ll assume we already know which featuretype we want. In this example we’ll look at a dataset containing condicutivity-temperature-depth (CTD) profiles obtained at the National Reference Stations around Australia (here’s a detailed metadata record)

typename = 'imos:anmn_ctd_profiles_data'
wfs.get_schema(typename)['properties']

{'file_id': 'int',
 'site_code': 'string',
 'cruise_id': 'string',
 'time_coverage_start': 'dateTime',
 'time_coverage_end': 'dateTime',
 'TIME': 'dateTime',
 'INSTANCE': 'int',
 'DIRECTION': 'string',
 'TIME_quality_control': 'string',
 'LATITUDE': 'double',
 'LATITUDE_quality_control': 'string',
 'LONGITUDE': 'double',
 'LONGITUDE_quality_control': 'string',
 'DEPTH': 'float',
 'DEPTH_quality_control': 'string',
 'BOT_DEPTH': 'float',
 'BOT_DEPTH_quality_control': 'string',
 'PRES_REL': 'float',
 'PRES_REL_quality_control': 'string',
 'TEMP': 'float',
 'TEMP_quality_control': 'string',
 'PSAL': 'float',
 'PSAL_quality_control': 'string',
 'DOX2': 'float',
 'DOX2_quality_control': 'string',
 'TURB': 'float',
 'TURB_quality_control': 'string',
 'CHLF': 'float',
 'CHLF_quality_control': 'string',
 'CHLU': 'float',
 'CHLU_quality_control': 'string',
 'CPHL': 'float',
 'CPHL_quality_control': 'string',
 'CNDC': 'float',
 'CNDC_quality_control': 'string',
 'DESC': 'float',
 'DESC_quality_control': 'string',
 'DENS': 'float',
 'DENS_quality_control': 'string'}

We can read in a subset of the data by specifying a bounding box (in this case near Sydney, Australia). We’ll get the result in CSV format so it’s easy to read into a Pandas DataFrame.

First we’ll ask for just 10 features, for a quick look at the data.

xmin, xmax = 151.2, 151.25   # Port Hacking, near Sydney, NSW
ymin, ymax = -34.2, -34.1

response = wfs.getfeature(typename=typename,
                          bbox=(xmin, ymin, xmax, ymax),
                          maxfeatures=10,
                          outputFormat='csv')
df = pd.read_csv(response)
response.close()

df

	FID	file_id	site_code	cruise_id	time_coverage_start	time_coverage_end	TIME	INSTANCE	DIRECTION	TIME_quality_control	...	CHLU_quality_control	CPHL	CPHL_quality_control	CNDC	CNDC_quality_control	DESC	DESC_quality_control	DENS	DENS_quality_control	geom
0	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.6266	0	0.228	0	1025.8478	0	POINT (-34.1161666667 151.218)
1	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.6246	0	0.574	0	1025.8652	0	POINT (-34.1161666667 151.218)
2	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.6224	0	0.741	0	1025.8737	0	POINT (-34.1161666667 151.218)
3	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.6190	0	0.803	0	1025.8790	0	POINT (-34.1161666667 151.218)
4	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.6138	0	0.749	0	1025.8892	0	POINT (-34.1161666667 151.218)
5	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.6089	0	0.687	0	1025.9072	0	POINT (-34.1161666667 151.218)
6	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.6067	0	0.722	0	1025.9241	0	POINT (-34.1161666667 151.218)
7	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.6048	0	0.773	0	1025.9321	0	POINT (-34.1161666667 151.218)
8	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.6023	0	0.788	0	1025.9385	0	POINT (-34.1161666667 151.218)
9	anmn_ctd_profiles_data.fid--6fff0f0_189bed8a1b...	754	PH100	PHNRS_1108	2011-08-29T00:03:40	2011-08-29T00:03:40	2011-08-29T00:03:40	NaN	D	NaN	...	NaN	NaN	NaN	4.5982	0	0.846	0	1025.9432	0	POINT (-34.1161666667 151.218)

10 rows × 41 columns

## Load local copy of CSV file returned...

# local_csv = os.path.join(DATA_BASEPATH, 'wfs_response1.csv')
# df = pd.read_csv(local_csv)
# df

We can also filter the data based on the values in specified columns (properties) and ask for only a subset of the columns to be returned. The filters need to be provided in XML format, but the owslib library allows us to construct them in a more Pythonic way.

Here we select only the profiles associated with the Port Hacking 100m mooring site, and only the data points flagged as “good data” by automated quality-control procedures.

from owslib.etree import etree
from owslib.fes import PropertyIsEqualTo, And

filter = And([PropertyIsEqualTo(propertyname="site_code", literal="PH100"),
              PropertyIsEqualTo(propertyname="PRES_REL_quality_control", literal="1"),
              PropertyIsEqualTo(propertyname="TEMP_quality_control", literal="1"),
              PropertyIsEqualTo(propertyname="PSAL_quality_control", literal="1"),
              PropertyIsEqualTo(propertyname="CPHL_quality_control", literal="1")
             ])
filterxml = etree.tostring(filter.toXML(), encoding="unicode")

response = wfs.getfeature(typename=typename, filter=filterxml, outputFormat="csv",
                          propertyname=["TIME", "DEPTH", "TEMP", "PSAL", "CPHL"]
                         )
df = pd.read_csv(response, parse_dates=["TIME"])
response.close()

# the server adds a feature ID column we don't really need
df.drop(columns='FID', inplace=True)

## Load local copy of CSV file returned...

# local_csv = os.path.join(DATA_BASEPATH, 'wfs_response2.csv')
# df = pd.read_csv(local_csv, parse_dates=["TIME"]).drop(columns='FID')

df

	TIME	DEPTH	TEMP	PSAL	CPHL
0	2014-12-08 22:28:54	1.986	21.6432	35.5080	0.9365
1	2014-12-08 22:28:54	2.979	21.6441	35.5085	0.9560
2	2014-12-08 22:28:54	3.971	21.6417	35.5085	0.9644
3	2014-12-08 22:28:54	4.964	21.6314	35.5089	0.9963
4	2014-12-08 22:28:54	5.957	21.6077	35.5102	0.9844
...	...	...	...	...	...
11377	2023-05-15 22:08:05	82.398	18.0130	35.5832	0.1554
11378	2023-05-15 22:08:05	83.391	18.0008	35.5841	0.1417
11379	2023-05-15 22:08:05	84.384	17.9824	35.5843	0.1345
11380	2023-05-15 22:08:05	85.376	17.9343	35.5821	0.0937
11381	2023-05-15 22:08:05	86.368	17.8612	35.5799	0.0300

11382 rows × 5 columns

# We can explore the temperature, salinity and chlorophyll profiles
# Change "by" to "groupby" to view one profile at a time, with time selected interactively
temp_plot = df.hvplot(x="TEMP", y="DEPTH", by="TIME", flip_yaxis=True, legend=False, width=200)
psal_plot = df.hvplot(x="PSAL", y="DEPTH", by="TIME", flip_yaxis=True, legend=False, width=200)
cphl_plot = df.hvplot(x="CPHL", y="DEPTH", by="TIME", flip_yaxis=True, legend=False, width=200)

(temp_plot + psal_plot + cphl_plot).opts(tight=True)

# We can also extract the temperature measurements at a fixed depth
# and compare to the timeseries from the mooring 
comp_depth = 20  # metres

df_sub = df[df.DEPTH.round() == comp_depth]
ctd_plot = df_sub.hvplot.scatter(x="TIME", y="TEMP", c="red")

mooring_plot = ds_mooring.TEMP.sel(DEPTH=comp_depth).hvplot()

mooring_plot * ctd_plot

Direct access to files on cloud storage

Data files made available to the public on cloud storage such as Amazon S3 (Simple Storage Service) can be accessed over the web as if they were stored locally. You just need to find the exact URL for each file.

In Python, we can access S3 storage in a very similar way to a local filesystem using the s3fs library.

For example, all the public data files hosted by the Australian Ocean Data Network are stored in an S3 bucket called imos-data. You can browse the contents of the bucket and download individual files here.

Below we’ll look at a high-resolution regional SST product from IMOS (based on satellite and in-situ observations). This product is a collection of daily gridded NetCDF files covering the Australian region.

s3 = s3fs.S3FileSystem(anon=True)

# List the most recent files available
sst_files = s3.ls("imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023")
sst_files[-20:]

['imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230713120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230714120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230715120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230716120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230717120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230718120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230719120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230720120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230721120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230722120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230723120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230725120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230726120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230727120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230728120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230729120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230730120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230731120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230801120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc',
 'imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/20230802120000-ABOM-L4_GHRSST-SSTfnd-RAMSSA_09km-AUS-v02.0-fv01.0.nc']

# Open the latest file and look at its contents
ds = xr.open_dataset(s3.open(sst_files[-1]))
ds

<xarray.Dataset>
Dimensions:           (time: 1, lat: 1081, lon: 1561)
Coordinates:
  * time              (time) datetime64[ns] 2023-08-02T12:00:00
  * lat               (lat) float32 -70.0 -69.92 -69.83 ... 19.83 19.92 20.0
  * lon               (lon) float32 60.0 60.08 60.17 60.25 ... 189.8 189.9 190.0
Data variables:
    sea_ice_fraction  (time, lat, lon) float32 ...
    analysed_sst      (time, lat, lon) float32 ...
    analysis_error    (time, lat, lon) float32 ...
    mask              (time, lat, lon) float32 ...
    crs               int32 -2147483647
Attributes: (12/65)
    id:                         RAMSSA_09km-ABOM-L4-AUS-v01
    Conventions:                CF-1.6, ACDD-1.3, ISO 8601
    title:                      RAMSSA v1.1 Analysed high resolution foundati...
    summary:                    AMSR2-JAXA nobs=946601 obsesd: avg=0.693 min=...
    source:                     AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VII...
    references:                 Beggs H., A. Zhong, G. Warren, O. Alves, G. B...
    ...                         ...
    geospatial_lat_max:         20.0
    geospatial_lat_min:         -70.0
    geospatial_lon_max:         190.0
    geospatial_lon_min:         60.0
    geospatial_bounds:          POLYGON((-70 60, 20 60, 20 190, -70 190, -70 ...
    geospatial_bounds_crs:      EPSG:4326

xarray.Dataset

• Dimensions:
  • time: 1
  • lat: 1081
  • lon: 1561
• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2023-08-02T12:00:00

    long_name :

    reference time of SST field

    standard_name :

    time

    axis :

    T

    comment :

    Nominal time because observations are from different sources and are made at different times of the day

    coverage_content_type :

    coordinate

    array(['2023-08-02T12:00:00.000000000'], dtype='datetime64[ns]')

  • lat
    (lat)
    float32
    -70.0 -69.92 -69.83 ... 19.92 20.0

    long_name :

    latitude

    standard_name :

    latitude

    axis :

    Y

    comment :

    Latitudes for locating data

    valid_min :

    -90.0

    valid_max :

    90.0

    units :

    degrees_north

    coverage_content_type :

    coordinate

    array([-70.      , -69.916664, -69.833336, ...,  19.833334,  19.916666,
            20.      ], dtype=float32)

  • lon
    (lon)
    float32
    60.0 60.08 60.17 ... 189.9 190.0

    long_name :

    longitude

    standard_name :

    longitude

    axis :

    X

    comment :

    Longitudes for locating data

    units :

    degrees_east

    coverage_content_type :

    coordinate

    valid_min :

    0.0

    valid_max :

    360.0

    array([ 60.      ,  60.083332,  60.166668, ..., 189.83333 , 189.91667 ,
           190.      ], dtype=float32)

• Data variables: (5)
  • sea_ice_fraction
    (time, lat, lon)
    float32
    ...

    valid_min :

    0

    valid_max :

    100

    units :

    1

    long_name :

    sea ice area fraction

    standard_name :

    sea_ice_area_fraction

    comment :

    Fractional sea ice cover from NCEP-ICE product (Grumbine, 1996) .

    source :

    SSMI-NCEP-Analysis-ICE-5min

    coverage_content_type :

    auxiliaryInformation

    grid_mapping :

    crs

    [1687441 values with dtype=float32]

  • analysed_sst
    (time, lat, lon)
    float32
    ...

    valid_min :

    -300

    valid_max :

    4500

    units :

    kelvin

    long_name :

    analysed sea surface temperature

    standard_name :

    sea_surface_foundation_temperature

    comment :

    Optimally interpolated analysis of SST observations.

    source :

    AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VIIRS_N20_OSPO,IN_SITU-GTS_BUOYS,IN_SITU-GTS_BATHY,IN_SITU-GTS_SHIP,IN_SITU-GTS_TESAC,ACCESS-G-WSP,NCEP-ICE

    coverage_content_type :

    physicalMeasurement

    grid_mapping :

    crs

    [1687441 values with dtype=float32]

  • analysis_error
    (time, lat, lon)
    float32
    ...

    valid_min :

    0

    valid_max :

    32765

    units :

    kelvin

    long_name :

    estimated error standard deviation of analysed_sst

    comment :

    Calculated as analysis field error in OI analysis system and a function of the age and estimated standard deviation of all observations used to form analysed_sst value. See Beggs et al (2011) in http://www.bom.gov.au/jshess/papers.php?year=2011.

    coverage_content_type :

    auxiliaryInformation

    grid_mapping :

    crs

    [1687441 values with dtype=float32]

  • mask
    (time, lat, lon)
    float32
    ...

    valid_min :

    1

    valid_max :

    8

    long_name :

    sea/land/lake/ice field composite mask

    comment :

    b0: 1 = grid cell is open sea water, b1: 1 = land is present in this grid cell, b2: 1 = lake surface is present in this grid cell, b3: 1 = sea ice is present in this grid cell, b4-b7: reserved for future grid mask data

    source :

    NAVOCEANO 1/120 deg land-sea mask. See https://www.ghrsst.org/products-and-services/tools/navo-ghrsst-pp-land-sea-mask/.

    flag_meanings :

    water land optional_lake_surface sea-ice

    grid_mapping :

    crs

    flag_masks :

    [1 2 4 8]

    [1687441 values with dtype=float32]

  • crs
    ()
    int32
    ...

    long_name :

    coordinate reference system

    grid_mapping_name :

    latitude_longitude

    semi_major_axis :

    6379137.0

    inverse_flattening :

    298.257223563

    epsg_code :

    EPSG:4326

    array(-2147483647, dtype=int32)

• Attributes: (65)

  id :

  RAMSSA_09km-ABOM-L4-AUS-v01

  Conventions :

  CF-1.6, ACDD-1.3, ISO 8601

  title :

  RAMSSA v1.1 Analysed high resolution foundation sea surface temperature over Australian region

  summary :

  AMSR2-JAXA nobs=946601 obsesd: avg=0.693 min=0.700 max= 0.700, AVHRRMTB_G-NAVO nobs=109917 obsesd: avg=0.551 min=0.550 max= 0.550, VIIRS_NPP_OSPO nobs= 25877 obsesd: avg=0.400 min=0.400 max= 0.400, VIIRS_N20_OSPO nobs= 25772 obsesd: avg=0.380 min=0.380 max= 0.380, IN_SITU-GTS_BUOYS nobs= 4598 obsesd: avg=0.400 min=0.400 max= 0.400, IN_SITU-GTS_BATHY nobs= 10 obsesd: avg=1.200 min=1.200 max= 1.200, IN_SITU-GTS_SHIP nobs= 405 obsesd: avg=1.200 min=1.200 max= 1.200, IN_SITU-GTS_TESAC nobs= 84 obsesd: avg=1.200 min=1.200 max= 1.200, Univariate Statistical Interpolation Package, SIANAL v13, background correlation length scale = 50 km, observation correlation length scale = 12 km, observation correlation time scale = 0.5 days

  source :

  AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VIIRS_N20_OSPO,IN_SITU-GTS_BUOYS,IN_SITU-GTS_BATHY,IN_SITU-GTS_SHIP,IN_SITU-GTS_TESAC,ACCESS-G-WSP,NCEP-ICE

  references :

  Beggs H., A. Zhong, G. Warren, O. Alves, G. Brassington and T. Pugh (2011) RAMSSA An Operational, High-Resolution, Multi-Sensor Sea Surface Temperature Analysis over the Australian Region, Australian Meteorological and Oceanographic Journal, 61, 1-22. http://www.bom.gov.au/jshess/papers.php?year=2011

  institution :

  ABOM

  comment :

  This is a Bureau of Meteorology experimental research product originally developed for the BLUElink> Ocean Forecasting Australia Project. WARNING: some applications are unable to properly handle signed byte values. If byte values >127 are encountered, subtract 256 from this reported value.

  license :

  GHRSST protocol describes data use as free and open

  naming_authority :

  org.ghrsst

  product_version :

  1.0

  program :

  IMOS

  uuid :

  aa08b18a-70ef-4614-8497-2e30268150ac

  gds_version_id :

  2.0r5

  netcdf_version_id :

  4.3

  spatial_resolution :

  1/12 degree

  platform :

  GCOM-W,MetOpB,N20,NPP

  sensor :

  AMSR2,AVHRR_GAC,VIIRS

  Metadata_Link :

  https://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=RAMSSA_09km-ABOM-L4-AUS-v01

  keywords :

  Oceans > Ocean Temperature > Sea Surface Temperature

  platform_vocabulary :

  GHRSST Data Specification (GDS)2.0, https://www.ghrsst.org/documents/q/category/gds-documents/operational/

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Science Keywords

  standard_name_vocabulary :

  NetCDF Climate and Forecast (CF) Metadata Convention

  geospatial_lat_units :

  degrees_north

  geospatial_lon_units :

  degrees_east

  geospatial_lat_resolution :

  0.0833

  geospatial_lon_resolution :

  0.0833

  history :

  Modified format and attributes of 20230802-ABOM-L4fnd-AUS-v01-fv01.nc using file_convert_RAMSSA,version 1.0 to match the GDS2.0 rev 5 specification

  acknowledgment :

  Any use of these data requires the following acknowledgment: Experimental near real-time sea surface temperature analysis produced by the Australian Bureau of Meteorology and provided in GHRSST GDS2.0 format as a contribution to the Integrated Marine Observing System – IMOS is a national collaborative research infrastructure, supported by the Australian Government.

  creator_name :

  Australian Bureau of Meteorology

  creator_email :

  [email protected]

  creator_url :

  http://www.bom.gov.au

  creator_type :

  Institution

  creator_institution :

  Australian Bureau of Meteorology

  project :

  Integrated Marine Observing System

  publisher_name :

  Australian Bureau of Meteorology

  publisher_email :

  [email protected]

  publisher_url :

  http://www.bom.gov.au

  publisher_institution :

  Australian Bureau of Meteorology

  publisher_type :

  Institution

  processing_level :

  L4

  cdm_data_type :

  Grid

  date_created :

  20230803T120000Z

  date_modified :

  20230803T120000Z

  date_metadata_modified :

  20230803T120000Z

  date_issued :

  20230803T120000Z

  start_time :

  20230802T000000Z

  time_coverage_start :

  20230802T000000Z

  stop_time :

  20230803T000000Z

  time_coverage_end :

  20230803T000000Z

  time_coverage_duration :

  P1D

  time_coverage_resolution :

  P1D

  northernmost_latitude :

  20.0

  southernmost_latitude :

  -70.0

  easternmost_longitude :

  360.0

  westernmost_longitude :

  0.0

  file_quality_level :

  3

  contributor_name :

  Helen Beggs

  contributor_role :

  IMOS PI

  geospatial_lat_max :

  20.0

  geospatial_lat_min :

  -70.0

  geospatial_lon_max :

  190.0

  geospatial_lon_min :

  60.0

  geospatial_bounds :

  POLYGON((-70 60, 20 60, 20 190, -70 190, -70 60))

  geospatial_bounds_crs :

  EPSG:4326

# Plot a subset of the dataset around Australia
sst_var = 'analysed_sst'
gds = gv.Dataset(ds.sel(lat=slice(-50, 0), lon=slice(105, 175)),
                 kdims=['lon', 'lat'],
                 vdims=[sst_var]
                )
sst_plot = (gds.to(gv.Image)
               .opts(cmap='coolwarm', colorbar=True, aspect=1.4, title=ds.title))
sst_plot * gf.land

/opt/conda/lib/python3.9/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/110m_physical/ne_110m_land.zip
  warnings.warn(f'Downloading: {url}', DownloadWarning)

It’s worth understanding a little about how this works.

The above example only makes use of the metadata from the file, one of the 4 data variables, and the lon and lat coordinates. On a local filesystem, it would be easy to read only these specific parts of the file from disk.

However, on cloud storage services like S3 (also called “object storage”) the basic read/write functions operate on the entire file (object), so at least in the backend, the entire file is read**. If you only need a small subset of a large file, this can be a very inefficient way to get it.

** Note: it is possible to request only a subset of an S3 object to be read, but this is more advanced usage than what we’re doing here.

For example, if we wanted to plot a timeseries of the above satellite SST product at a given point, we would only need a single value out of each file (corresponding to one point in the timeseries), but the entire file would need to be read each time.

For a quick demo we’ll try this with last month’s files. xarray has a handy open_mfdataset function that can create a single Dataset object out of a series of files (with similar structure).

%%time
s3_objs = [s3.open(f)
           for f in s3.glob("imos-data/IMOS/SRS/SST/ghrsst/L4/RAMSSA/2023/202307*")
          ]
mds = xr.open_mfdataset(s3_objs, engine="h5netcdf")
mds

CPU times: user 4.78 s, sys: 119 ms, total: 4.9 s
Wall time: 15.4 s

<xarray.Dataset>
Dimensions:           (time: 30, lat: 1081, lon: 1561)
Coordinates:
  * time              (time) datetime64[ns] 2023-07-01T12:00:00 ... 2023-07-3...
  * lat               (lat) float32 -70.0 -69.92 -69.83 ... 19.83 19.92 20.0
  * lon               (lon) float32 60.0 60.08 60.17 60.25 ... 189.8 189.9 190.0
Data variables:
    sea_ice_fraction  (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>
    analysed_sst      (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>
    analysis_error    (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>
    mask              (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>
    crs               (time) int32 -2147483647 -2147483647 ... -2147483647
Attributes: (12/65)
    id:                         RAMSSA_09km-ABOM-L4-AUS-v01
    Conventions:                CF-1.6, ACDD-1.3, ISO 8601
    title:                      RAMSSA v1.1 Analysed high resolution foundati...
    summary:                    AMSR2-JAXA nobs=****** obsesd: avg=0.693 min=...
    source:                     AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VII...
    references:                 Beggs H., A. Zhong, G. Warren, O. Alves, G. B...
    ...                         ...
    geospatial_lat_max:         20.0
    geospatial_lat_min:         -70.0
    geospatial_lon_max:         190.0
    geospatial_lon_min:         60.0
    geospatial_bounds:          POLYGON((-70 60, 20 60, 20 190, -70 190, -70 ...
    geospatial_bounds_crs:      EPSG:4326

xarray.Dataset

• Dimensions:
  • time: 30
  • lat: 1081
  • lon: 1561
• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2023-07-01T12:00:00 ... 2023-07-...

    long_name :

    reference time of SST field

    standard_name :

    time

    axis :

    T

    comment :

    Nominal time because observations are from different sources and are made at different times of the day

    coverage_content_type :

    coordinate

    array(['2023-07-01T12:00:00.000000000', '2023-07-02T12:00:00.000000000',
           '2023-07-03T12:00:00.000000000', '2023-07-04T12:00:00.000000000',
           '2023-07-05T12:00:00.000000000', '2023-07-06T12:00:00.000000000',
           '2023-07-07T12:00:00.000000000', '2023-07-08T12:00:00.000000000',
           '2023-07-09T12:00:00.000000000', '2023-07-10T12:00:00.000000000',
           '2023-07-11T12:00:00.000000000', '2023-07-12T12:00:00.000000000',
           '2023-07-13T12:00:00.000000000', '2023-07-14T12:00:00.000000000',
           '2023-07-15T12:00:00.000000000', '2023-07-16T12:00:00.000000000',
           '2023-07-17T12:00:00.000000000', '2023-07-18T12:00:00.000000000',
           '2023-07-19T12:00:00.000000000', '2023-07-20T12:00:00.000000000',
           '2023-07-21T12:00:00.000000000', '2023-07-22T12:00:00.000000000',
           '2023-07-23T12:00:00.000000000', '2023-07-25T12:00:00.000000000',
           '2023-07-26T12:00:00.000000000', '2023-07-27T12:00:00.000000000',
           '2023-07-28T12:00:00.000000000', '2023-07-29T12:00:00.000000000',
           '2023-07-30T12:00:00.000000000', '2023-07-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • lat
    (lat)
    float32
    -70.0 -69.92 -69.83 ... 19.92 20.0

    long_name :

    latitude

    standard_name :

    latitude

    axis :

    Y

    comment :

    Latitudes for locating data

    valid_min :

    -90.0

    valid_max :

    90.0

    units :

    degrees_north

    coverage_content_type :

    coordinate

    array([-70.      , -69.916664, -69.833336, ...,  19.833334,  19.916666,
            20.      ], dtype=float32)

  • lon
    (lon)
    float32
    60.0 60.08 60.17 ... 189.9 190.0

    long_name :

    longitude

    standard_name :

    longitude

    axis :

    X

    comment :

    Longitudes for locating data

    units :

    degrees_east

    coverage_content_type :

    coordinate

    valid_min :

    0.0

    valid_max :

    360.0

    array([ 60.      ,  60.083332,  60.166668, ..., 189.83333 , 189.91667 ,
           190.      ], dtype=float32)

• Data variables: (5)
  • sea_ice_fraction
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>

    valid_min :

    0

    valid_max :

    100

    units :

    1

    long_name :

    sea ice area fraction

    standard_name :

    sea_ice_area_fraction

    comment :

    Fractional sea ice cover from NCEP-ICE product (Grumbine, 1996) .

    source :

    SSMI-NCEP-Analysis-ICE-5min

    coverage_content_type :

    auxiliaryInformation

    grid_mapping :

    crs

    Array Chunk Bytes 193.11 MiB 6.44 MiB Shape (30, 1081, 1561) (1, 1081, 1561) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

  • analysed_sst
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>

    valid_min :

    -300

    valid_max :

    4500

    clip_min :

    269.30999398231506

    clip_max :

    304.8399931881577

    units :

    kelvin

    long_name :

    analysed sea surface temperature

    standard_name :

    sea_surface_foundation_temperature

    comment :

    Optimally interpolated analysis of SST observations.

    source :

    AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VIIRS_N20_OSPO,IN_SITU-GTS_BUOYS,IN_SITU-GTS_SHIP,IN_SITU-GTS_TESAC,ACCESS-G-WSP,NCEP-ICE

    coverage_content_type :

    physicalMeasurement

    grid_mapping :

    crs

    Array Chunk Bytes 193.11 MiB 6.44 MiB Shape (30, 1081, 1561) (1, 1081, 1561) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

  • analysis_error
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>

    valid_min :

    0

    valid_max :

    32765

    units :

    kelvin

    long_name :

    estimated error standard deviation of analysed_sst

    comment :

    Calculated as analysis field error in OI analysis system and a function of the age and estimated standard deviation of all observations used to form analysed_sst value. See Beggs et al (2011) in http://www.bom.gov.au/jshess/papers.php?year=2011.

    coverage_content_type :

    auxiliaryInformation

    grid_mapping :

    crs

    Array Chunk Bytes 193.11 MiB 6.44 MiB Shape (30, 1081, 1561) (1, 1081, 1561) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

  • mask
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>

    valid_min :

    1

    valid_max :

    8

    long_name :

    sea/land/lake/ice field composite mask

    comment :

    b0: 1 = grid cell is open sea water, b1: 1 = land is present in this grid cell, b2: 1 = lake surface is present in this grid cell, b3: 1 = sea ice is present in this grid cell, b4-b7: reserved for future grid mask data

    source :

    NAVOCEANO 1/120 deg land-sea mask. See https://www.ghrsst.org/products-and-services/tools/navo-ghrsst-pp-land-sea-mask/.

    flag_meanings :

    water land optional_lake_surface sea-ice

    grid_mapping :

    crs

    flag_masks :

    [1 2 4 8]

    Array Chunk Bytes 193.11 MiB 6.44 MiB Shape (30, 1081, 1561) (1, 1081, 1561) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

  • crs
    (time)
    int32
    -2147483647 ... -2147483647

    long_name :

    coordinate reference system

    grid_mapping_name :

    latitude_longitude

    semi_major_axis :

    6379137.0

    inverse_flattening :

    298.257223563

    epsg_code :

    EPSG:4326

    array([-2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647],
          dtype=int32)

• Attributes: (65)

  id :

  RAMSSA_09km-ABOM-L4-AUS-v01

  Conventions :

  CF-1.6, ACDD-1.3, ISO 8601

  title :

  RAMSSA v1.1 Analysed high resolution foundation sea surface temperature over Australian region

  summary :

  AMSR2-JAXA nobs=****** obsesd: avg=0.693 min=0.700 max= 0.700, AVHRRMTB_G-NAVO nobs=144056 obsesd: avg=0.550 min=0.550 max= 0.550, VIIRS_NPP_OSPO nobs= 28034 obsesd: avg=0.400 min=0.400 max= 0.400, VIIRS_N20_OSPO nobs= 28487 obsesd: avg=0.380 min=0.380 max= 0.380, IN_SITU-GTS_BUOYS nobs= 5165 obsesd: avg=0.400 min=0.400 max= 0.400, IN_SITU-GTS_SHIP nobs= 483 obsesd: avg=1.200 min=1.200 max= 1.200, IN_SITU-GTS_TESAC nobs= 56 obsesd: avg=1.200 min=1.200 max= 1.200, Univariate Statistical Interpolation Package, SIANAL v13, background correlation length scale = 50 km, observation correlation length scale = 12 km, observation correlation time scale = 0.5 days

  source :

  AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VIIRS_N20_OSPO,IN_SITU-GTS_BUOYS,IN_SITU-GTS_SHIP,IN_SITU-GTS_TESAC,ACCESS-G-WSP,NCEP-ICE

  references :

  Beggs H., A. Zhong, G. Warren, O. Alves, G. Brassington and T. Pugh (2011) RAMSSA An Operational, High-Resolution, Multi-Sensor Sea Surface Temperature Analysis over the Australian Region, Australian Meteorological and Oceanographic Journal, 61, 1-22. http://www.bom.gov.au/jshess/papers.php?year=2011

  institution :

  ABOM

  comment :

  This is a Bureau of Meteorology experimental research product originally developed for the BLUElink> Ocean Forecasting Australia Project. WARNING: some applications are unable to properly handle signed byte values. If byte values >127 are encountered, subtract 256 from this reported value.

  license :

  GHRSST protocol describes data use as free and open

  naming_authority :

  org.ghrsst

  product_version :

  1.0

  program :

  IMOS

  uuid :

  3de67715-144b-450b-8d33-4e2551e485fa

  gds_version_id :

  2.0r5

  netcdf_version_id :

  4.3

  spatial_resolution :

  1/12 degree

  platform :

  GCOM-W,MetOpB,N20,NPP

  sensor :

  AMSR2,AVHRR_GAC,VIIRS

  Metadata_Link :

  https://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=RAMSSA_09km-ABOM-L4-AUS-v01

  keywords :

  Oceans > Ocean Temperature > Sea Surface Temperature

  platform_vocabulary :

  GHRSST Data Specification (GDS)2.0, https://www.ghrsst.org/documents/q/category/gds-documents/operational/

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Science Keywords

  standard_name_vocabulary :

  NetCDF Climate and Forecast (CF) Metadata Convention

  geospatial_lat_units :

  degrees_north

  geospatial_lon_units :

  degrees_east

  geospatial_lat_resolution :

  0.0833

  geospatial_lon_resolution :

  0.0833

  history :

  Modified format and attributes of 20230701-ABOM-L4fnd-AUS-v01-fv01.nc using file_convert_RAMSSA,version 1.0 to match the GDS2.0 rev 5 specification

  acknowledgment :

  Any use of these data requires the following acknowledgment: Experimental near real-time sea surface temperature analysis produced by the Australian Bureau of Meteorology and provided in GHRSST GDS2.0 format as a contribution to the Integrated Marine Observing System – IMOS is a national collaborative research infrastructure, supported by the Australian Government.

  creator_name :

  Australian Bureau of Meteorology

  creator_email :

  [email protected]

  creator_url :

  http://www.bom.gov.au

  creator_type :

  Institution

  creator_institution :

  Australian Bureau of Meteorology

  project :

  Integrated Marine Observing System

  publisher_name :

  Australian Bureau of Meteorology

  publisher_email :

  [email protected]

  publisher_url :

  http://www.bom.gov.au

  publisher_institution :

  Australian Bureau of Meteorology

  publisher_type :

  Institution

  processing_level :

  L4

  cdm_data_type :

  Grid

  date_created :

  20230702T120000Z

  date_modified :

  20230702T120000Z

  date_metadata_modified :

  20230702T120000Z

  date_issued :

  20230702T120000Z

  start_time :

  20230701T000000Z

  time_coverage_start :

  20230701T000000Z

  stop_time :

  20230702T000000Z

  time_coverage_end :

  20230702T000000Z

  time_coverage_duration :

  P1D

  time_coverage_resolution :

  P1D

  northernmost_latitude :

  20.0

  southernmost_latitude :

  -70.0

  easternmost_longitude :

  360.0

  westernmost_longitude :

  0.0

  file_quality_level :

  3

  contributor_name :

  Helen Beggs

  contributor_role :

  IMOS PI

  geospatial_lat_max :

  20.0

  geospatial_lat_min :

  -70.0

  geospatial_lon_max :

  190.0

  geospatial_lon_min :

  60.0

  geospatial_bounds :

  POLYGON((-70 60, 20 60, 20 190, -70 190, -70 60))

  geospatial_bounds_crs :

  EPSG:4326

The variables in the dataset are not loaded into memory (they’re still dask.arrays). However, in the background, each complete file had to be downloaded from S3 before the metadata needed by open_mfdataset could be read.

mds.analysed_sst

<xarray.DataArray 'analysed_sst' (time: 30, lat: 1081, lon: 1561)>
dask.array<concatenate, shape=(30, 1081, 1561), dtype=float32, chunksize=(1, 1081, 1561), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2023-07-01T12:00:00 ... 2023-07-31T12:00:00
  * lat      (lat) float32 -70.0 -69.92 -69.83 -69.75 ... 19.75 19.83 19.92 20.0
  * lon      (lon) float32 60.0 60.08 60.17 60.25 ... 189.8 189.8 189.9 190.0
Attributes:
    valid_min:              -300
    valid_max:              4500
    clip_min:               269.30999398231506
    clip_max:               304.8399931881577
    units:                  kelvin
    long_name:              analysed sea surface temperature
    standard_name:          sea_surface_foundation_temperature
    comment:                Optimally interpolated analysis of SST observations.
    source:                 AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VIIRS_N...
    coverage_content_type:  physicalMeasurement
    grid_mapping:           crs

xarray.DataArray

'analysed_sst'

• time: 30
• lat: 1081
• lon: 1561

• dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>

  Array Chunk Bytes 193.11 MiB 6.44 MiB Shape (30, 1081, 1561) (1, 1081, 1561) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2023-07-01T12:00:00 ... 2023-07-...

    long_name :

    reference time of SST field

    standard_name :

    time

    axis :

    T

    comment :

    Nominal time because observations are from different sources and are made at different times of the day

    coverage_content_type :

    coordinate

    array(['2023-07-01T12:00:00.000000000', '2023-07-02T12:00:00.000000000',
           '2023-07-03T12:00:00.000000000', '2023-07-04T12:00:00.000000000',
           '2023-07-05T12:00:00.000000000', '2023-07-06T12:00:00.000000000',
           '2023-07-07T12:00:00.000000000', '2023-07-08T12:00:00.000000000',
           '2023-07-09T12:00:00.000000000', '2023-07-10T12:00:00.000000000',
           '2023-07-11T12:00:00.000000000', '2023-07-12T12:00:00.000000000',
           '2023-07-13T12:00:00.000000000', '2023-07-14T12:00:00.000000000',
           '2023-07-15T12:00:00.000000000', '2023-07-16T12:00:00.000000000',
           '2023-07-17T12:00:00.000000000', '2023-07-18T12:00:00.000000000',
           '2023-07-19T12:00:00.000000000', '2023-07-20T12:00:00.000000000',
           '2023-07-21T12:00:00.000000000', '2023-07-22T12:00:00.000000000',
           '2023-07-23T12:00:00.000000000', '2023-07-25T12:00:00.000000000',
           '2023-07-26T12:00:00.000000000', '2023-07-27T12:00:00.000000000',
           '2023-07-28T12:00:00.000000000', '2023-07-29T12:00:00.000000000',
           '2023-07-30T12:00:00.000000000', '2023-07-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • lat
    (lat)
    float32
    -70.0 -69.92 -69.83 ... 19.92 20.0

    long_name :

    latitude

    standard_name :

    latitude

    axis :

    Y

    comment :

    Latitudes for locating data

    valid_min :

    -90.0

    valid_max :

    90.0

    units :

    degrees_north

    coverage_content_type :

    coordinate

    array([-70.      , -69.916664, -69.833336, ...,  19.833334,  19.916666,
            20.      ], dtype=float32)

  • lon
    (lon)
    float32
    60.0 60.08 60.17 ... 189.9 190.0

    long_name :

    longitude

    standard_name :

    longitude

    axis :

    X

    comment :

    Longitudes for locating data

    units :

    degrees_east

    coverage_content_type :

    coordinate

    valid_min :

    0.0

    valid_max :

    360.0

    array([ 60.      ,  60.083332,  60.166668, ..., 189.83333 , 189.91667 ,
           190.      ], dtype=float32)

• Attributes: (11)

  valid_min :

  -300

  valid_max :

  4500

  clip_min :

  269.30999398231506

  clip_max :

  304.8399931881577

  units :

  kelvin

  long_name :

  analysed sea surface temperature

  standard_name :

  sea_surface_foundation_temperature

  comment :

  Optimally interpolated analysis of SST observations.

  source :

  AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VIIRS_N20_OSPO,IN_SITU-GTS_BUOYS,IN_SITU-GTS_SHIP,IN_SITU-GTS_TESAC,ACCESS-G-WSP,NCEP-ICE

  coverage_content_type :

  physicalMeasurement

  grid_mapping :

  crs

Let’s compare this to reading the same files from a local filesystem…

%%time
from glob import glob
local_files = glob(os.path.join(DATA_BASEPATH, "RAMSSA", "*"))

mds = xr.open_mfdataset(local_files, engine="h5netcdf")
mds

CPU times: user 4.36 s, sys: 56.1 ms, total: 4.41 s
Wall time: 5.28 s

<xarray.Dataset>
Dimensions:           (time: 30, lat: 1081, lon: 1561)
Coordinates:
  * time              (time) datetime64[ns] 2023-07-01T12:00:00 ... 2023-07-3...
  * lat               (lat) float32 -70.0 -69.92 -69.83 ... 19.83 19.92 20.0
  * lon               (lon) float32 60.0 60.08 60.17 60.25 ... 189.8 189.9 190.0
Data variables:
    sea_ice_fraction  (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>
    analysed_sst      (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>
    analysis_error    (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>
    mask              (time, lat, lon) float32 dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>
    crs               (time) int32 -2147483647 -2147483647 ... -2147483647
Attributes: (12/65)
    id:                         RAMSSA_09km-ABOM-L4-AUS-v01
    Conventions:                CF-1.6, ACDD-1.3, ISO 8601
    title:                      RAMSSA v1.1 Analysed high resolution foundati...
    summary:                    AMSR2-JAXA nobs=****** obsesd: avg=0.693 min=...
    source:                     AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VII...
    references:                 Beggs H., A. Zhong, G. Warren, O. Alves, G. B...
    ...                         ...
    geospatial_lat_max:         20.0
    geospatial_lat_min:         -70.0
    geospatial_lon_max:         190.0
    geospatial_lon_min:         60.0
    geospatial_bounds:          POLYGON((-70 60, 20 60, 20 190, -70 190, -70 ...
    geospatial_bounds_crs:      EPSG:4326

xarray.Dataset

• Dimensions:
  • time: 30
  • lat: 1081
  • lon: 1561
• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2023-07-01T12:00:00 ... 2023-07-...

    long_name :

    reference time of SST field

    standard_name :

    time

    axis :

    T

    comment :

    Nominal time because observations are from different sources and are made at different times of the day

    coverage_content_type :

    coordinate

    array(['2023-07-01T12:00:00.000000000', '2023-07-02T12:00:00.000000000',
           '2023-07-03T12:00:00.000000000', '2023-07-04T12:00:00.000000000',
           '2023-07-05T12:00:00.000000000', '2023-07-06T12:00:00.000000000',
           '2023-07-07T12:00:00.000000000', '2023-07-08T12:00:00.000000000',
           '2023-07-09T12:00:00.000000000', '2023-07-10T12:00:00.000000000',
           '2023-07-11T12:00:00.000000000', '2023-07-12T12:00:00.000000000',
           '2023-07-13T12:00:00.000000000', '2023-07-14T12:00:00.000000000',
           '2023-07-15T12:00:00.000000000', '2023-07-16T12:00:00.000000000',
           '2023-07-17T12:00:00.000000000', '2023-07-18T12:00:00.000000000',
           '2023-07-19T12:00:00.000000000', '2023-07-20T12:00:00.000000000',
           '2023-07-21T12:00:00.000000000', '2023-07-22T12:00:00.000000000',
           '2023-07-23T12:00:00.000000000', '2023-07-25T12:00:00.000000000',
           '2023-07-26T12:00:00.000000000', '2023-07-27T12:00:00.000000000',
           '2023-07-28T12:00:00.000000000', '2023-07-29T12:00:00.000000000',
           '2023-07-30T12:00:00.000000000', '2023-07-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • lat
    (lat)
    float32
    -70.0 -69.92 -69.83 ... 19.92 20.0

    long_name :

    latitude

    standard_name :

    latitude

    axis :

    Y

    comment :

    Latitudes for locating data

    valid_min :

    -90.0

    valid_max :

    90.0

    units :

    degrees_north

    coverage_content_type :

    coordinate

    array([-70.      , -69.916664, -69.833336, ...,  19.833334,  19.916666,
            20.      ], dtype=float32)

  • lon
    (lon)
    float32
    60.0 60.08 60.17 ... 189.9 190.0

    long_name :

    longitude

    standard_name :

    longitude

    axis :

    X

    comment :

    Longitudes for locating data

    units :

    degrees_east

    coverage_content_type :

    coordinate

    valid_min :

    0.0

    valid_max :

    360.0

    array([ 60.      ,  60.083332,  60.166668, ..., 189.83333 , 189.91667 ,
           190.      ], dtype=float32)

• Data variables: (5)
  • sea_ice_fraction
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>

    valid_min :

    0

    valid_max :

    100

    units :

    1

    long_name :

    sea ice area fraction

    standard_name :

    sea_ice_area_fraction

    comment :

    Fractional sea ice cover from NCEP-ICE product (Grumbine, 1996) .

    source :

    SSMI-NCEP-Analysis-ICE-5min

    coverage_content_type :

    auxiliaryInformation

    grid_mapping :

    crs

    Array Chunk Bytes 193.11 MiB 6.44 MiB Shape (30, 1081, 1561) (1, 1081, 1561) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

  • analysed_sst
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>

    valid_min :

    -300

    valid_max :

    4500

    clip_min :

    269.30999398231506

    clip_max :

    304.8399931881577

    units :

    kelvin

    long_name :

    analysed sea surface temperature

    standard_name :

    sea_surface_foundation_temperature

    comment :

    Optimally interpolated analysis of SST observations.

    source :

    AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VIIRS_N20_OSPO,IN_SITU-GTS_BUOYS,IN_SITU-GTS_SHIP,IN_SITU-GTS_TESAC,ACCESS-G-WSP,NCEP-ICE

    coverage_content_type :

    physicalMeasurement

    grid_mapping :

    crs

    Array Chunk Bytes 193.11 MiB 6.44 MiB Shape (30, 1081, 1561) (1, 1081, 1561) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

  • analysis_error
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>

    valid_min :

    0

    valid_max :

    32765

    units :

    kelvin

    long_name :

    estimated error standard deviation of analysed_sst

    comment :

    Calculated as analysis field error in OI analysis system and a function of the age and estimated standard deviation of all observations used to form analysed_sst value. See Beggs et al (2011) in http://www.bom.gov.au/jshess/papers.php?year=2011.

    coverage_content_type :

    auxiliaryInformation

    grid_mapping :

    crs

    Array Chunk Bytes 193.11 MiB 6.44 MiB Shape (30, 1081, 1561) (1, 1081, 1561) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

  • mask
    (time, lat, lon)
    float32
    dask.array<chunksize=(1, 1081, 1561), meta=np.ndarray>

    valid_min :

    1

    valid_max :

    8

    long_name :

    sea/land/lake/ice field composite mask

    comment :

    b0: 1 = grid cell is open sea water, b1: 1 = land is present in this grid cell, b2: 1 = lake surface is present in this grid cell, b3: 1 = sea ice is present in this grid cell, b4-b7: reserved for future grid mask data

    source :

    NAVOCEANO 1/120 deg land-sea mask. See https://www.ghrsst.org/products-and-services/tools/navo-ghrsst-pp-land-sea-mask/.

    flag_meanings :

    water land optional_lake_surface sea-ice

    grid_mapping :

    crs

    flag_masks :

    [1 2 4 8]

    Array Chunk Bytes 193.11 MiB 6.44 MiB Shape (30, 1081, 1561) (1, 1081, 1561) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

  • crs
    (time)
    int32
    -2147483647 ... -2147483647

    long_name :

    coordinate reference system

    grid_mapping_name :

    latitude_longitude

    semi_major_axis :

    6379137.0

    inverse_flattening :

    298.257223563

    epsg_code :

    EPSG:4326

    array([-2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647,
           -2147483647, -2147483647, -2147483647, -2147483647, -2147483647],
          dtype=int32)

• Attributes: (65)

  id :

  RAMSSA_09km-ABOM-L4-AUS-v01

  Conventions :

  CF-1.6, ACDD-1.3, ISO 8601

  title :

  RAMSSA v1.1 Analysed high resolution foundation sea surface temperature over Australian region

  summary :

  AMSR2-JAXA nobs=****** obsesd: avg=0.693 min=0.700 max= 0.700, AVHRRMTB_G-NAVO nobs=144056 obsesd: avg=0.550 min=0.550 max= 0.550, VIIRS_NPP_OSPO nobs= 28034 obsesd: avg=0.400 min=0.400 max= 0.400, VIIRS_N20_OSPO nobs= 28487 obsesd: avg=0.380 min=0.380 max= 0.380, IN_SITU-GTS_BUOYS nobs= 5165 obsesd: avg=0.400 min=0.400 max= 0.400, IN_SITU-GTS_SHIP nobs= 483 obsesd: avg=1.200 min=1.200 max= 1.200, IN_SITU-GTS_TESAC nobs= 56 obsesd: avg=1.200 min=1.200 max= 1.200, Univariate Statistical Interpolation Package, SIANAL v13, background correlation length scale = 50 km, observation correlation length scale = 12 km, observation correlation time scale = 0.5 days

  source :

  AMSR2-JAXA,AVHRRMTB_G-NAVO,VIIRS_NPP_OSPO,VIIRS_N20_OSPO,IN_SITU-GTS_BUOYS,IN_SITU-GTS_SHIP,IN_SITU-GTS_TESAC,ACCESS-G-WSP,NCEP-ICE

  references :

  Beggs H., A. Zhong, G. Warren, O. Alves, G. Brassington and T. Pugh (2011) RAMSSA An Operational, High-Resolution, Multi-Sensor Sea Surface Temperature Analysis over the Australian Region, Australian Meteorological and Oceanographic Journal, 61, 1-22. http://www.bom.gov.au/jshess/papers.php?year=2011

  institution :

  ABOM

  comment :

  This is a Bureau of Meteorology experimental research product originally developed for the BLUElink> Ocean Forecasting Australia Project. WARNING: some applications are unable to properly handle signed byte values. If byte values >127 are encountered, subtract 256 from this reported value.

  license :

  GHRSST protocol describes data use as free and open

  naming_authority :

  org.ghrsst

  product_version :

  1.0

  program :

  IMOS

  uuid :

  3de67715-144b-450b-8d33-4e2551e485fa

  gds_version_id :

  2.0r5

  netcdf_version_id :

  4.3

  spatial_resolution :

  1/12 degree

  platform :

  GCOM-W,MetOpB,N20,NPP

  sensor :

  AMSR2,AVHRR_GAC,VIIRS

  Metadata_Link :

  https://podaac.jpl.nasa.gov/ws/metadata/dataset/?format=iso&shortName=RAMSSA_09km-ABOM-L4-AUS-v01

  keywords :

  Oceans > Ocean Temperature > Sea Surface Temperature

  platform_vocabulary :

  GHRSST Data Specification (GDS)2.0, https://www.ghrsst.org/documents/q/category/gds-documents/operational/

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Science Keywords

  standard_name_vocabulary :

  NetCDF Climate and Forecast (CF) Metadata Convention

  geospatial_lat_units :

  degrees_north

  geospatial_lon_units :

  degrees_east

  geospatial_lat_resolution :

  0.0833

  geospatial_lon_resolution :

  0.0833

  history :

  Modified format and attributes of 20230701-ABOM-L4fnd-AUS-v01-fv01.nc using file_convert_RAMSSA,version 1.0 to match the GDS2.0 rev 5 specification

  acknowledgment :

  Any use of these data requires the following acknowledgment: Experimental near real-time sea surface temperature analysis produced by the Australian Bureau of Meteorology and provided in GHRSST GDS2.0 format as a contribution to the Integrated Marine Observing System – IMOS is a national collaborative research infrastructure, supported by the Australian Government.

  creator_name :

  Australian Bureau of Meteorology

  creator_email :

  [email protected]

  creator_url :

  http://www.bom.gov.au

  creator_type :

  Institution

  creator_institution :

  Australian Bureau of Meteorology

  project :

  Integrated Marine Observing System

  publisher_name :

  Australian Bureau of Meteorology

  publisher_email :

  [email protected]

  publisher_url :

  http://www.bom.gov.au

  publisher_institution :

  Australian Bureau of Meteorology

  publisher_type :

  Institution

  processing_level :

  L4

  cdm_data_type :

  Grid

  date_created :

  20230702T120000Z

  date_modified :

  20230702T120000Z

  date_metadata_modified :

  20230702T120000Z

  date_issued :

  20230702T120000Z

  start_time :

  20230701T000000Z

  time_coverage_start :

  20230701T000000Z

  stop_time :

  20230702T000000Z

  time_coverage_end :

  20230702T000000Z

  time_coverage_duration :

  P1D

  time_coverage_resolution :

  P1D

  northernmost_latitude :

  20.0

  southernmost_latitude :

  -70.0

  easternmost_longitude :

  360.0

  westernmost_longitude :

  0.0

  file_quality_level :

  3

  contributor_name :

  Helen Beggs

  contributor_role :

  IMOS PI

  geospatial_lat_max :

  20.0

  geospatial_lat_min :

  -70.0

  geospatial_lon_max :

  190.0

  geospatial_lon_min :

  60.0

  geospatial_bounds :

  POLYGON((-70 60, 20 60, 20 190, -70 190, -70 60))

  geospatial_bounds_crs :

  EPSG:4326

Whichever way we loaded the dataset, we can plot it the same way as any other xarray.Dataset.

%%time
mds[sst_var].sel(lat=-42, lon=150, method="nearest").hvplot()

CPU times: user 809 ms, sys: 86.3 ms, total: 895 ms
Wall time: 3.65 s

Zarr - a cloud-optimised data format

Zarr is a relatively new data format specifically developed for efficient access to multi-dimensional data in the cloud. Each dataset is broken up into many smaller files containing “chunks” of the data, organised in a standard hierarchy. The metadata are stored in separate files. When reading such a dataset, only the required information is read for each operation.

# A Zarr "store" can easily be opened as an xarray.Dataset

# In this case the Zarr store is in an S3 bucket
# NOTE: This is an experimental dataset. It may not be available in the fultre.
store = s3fs.S3Map(root='imos-data-pixeldrill/zarrs/2021/', s3=s3, check=False)

zds = xr.open_zarr(store)
zds

<xarray.Dataset>
Dimensions:                  (time: 178, lat: 4500, lon: 6000)
Coordinates:
  * lat                      (lat) float32 19.99 19.97 19.95 ... -69.97 -69.99
  * lon                      (lon) float32 70.01 70.03 70.05 ... 190.0 190.0
  * time                     (time) datetime64[ns] 2021-01-01T15:20:00 ... 20...
Data variables:
    dt_analysis              (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>
    l2p_flags                (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>
    quality_level            (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>
    satellite_zenith_angle   (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>
    sea_surface_temperature  (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>
    sses_bias                (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>
    sses_count               (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>
    sses_standard_deviation  (time, lat, lon) float32 dask.array<chunksize=(64, 64, 64), meta=np.ndarray>
Attributes: (12/47)
    Conventions:                CF-1.6
    Metadata_Conventions:       Unidata Dataset Discovery v1.0
    Metadata_Link:              TBA
    acknowledgment:             Any use of these data requires the following ...
    cdm_data_type:              grid
    comment:                    HRPT AVHRR experimental L3 retrieval produced...
    ...                         ...
    summary:                    Skin SST retrievals produced from stitching t...
    time_coverage_end:          20210101T151752Z
    time_coverage_start:        20210101T095824Z
    title:                      IMOS L3S Nighttime gridded multiple-sensor mu...
    uuid:                       4d02ee75-876d-4ff0-8956-ab68917c9001
    westernmost_longitude:      70.01000213623047

xarray.Dataset

• Dimensions:
  • time: 178
  • lat: 4500
  • lon: 6000
• Coordinates: (3)
  • lat
    (lat)
    float32
    19.99 19.97 19.95 ... -69.97 -69.99

    NAME :

    lat

    _Netcdf4Dimid :

    1

    axis :

    Y

    comment :

    Latitudes for locating data

    long_name :

    latitude

    standard_name :

    latitude

    units :

    degrees_north

    valid_max :

    90.0

    valid_min :

    -90.0

    array([ 19.99,  19.97,  19.95, ..., -69.95, -69.97, -69.99], dtype=float32)

  • lon
    (lon)
    float32
    70.01 70.03 70.05 ... 190.0 190.0

    NAME :

    lon

    _Netcdf4Dimid :

    0

    axis :

    X

    comment :

    Longitudes for locating data

    long_name :

    longitude

    standard_name :

    longitude

    units :

    degrees_east

    valid_max :

    360.0

    valid_min :

    -180.0

    array([ 70.01,  70.03,  70.05, ..., 189.95, 189.97, 189.99], dtype=float32)

  • time
    (time)
    datetime64[ns]
    2021-01-01T15:20:00 ... 2021-07-...

    NAME :

    time

    _Netcdf4Dimid :

    2

    axis :

    T

    comment :

    A typical reference time for data

    long_name :

    reference time of sst file

    standard_name :

    time

    array(['2021-01-01T15:20:00.000000000', '2021-01-02T15:20:00.000000000',
           '2021-01-03T15:20:00.000000000', '2021-01-04T15:20:00.000000000',
           '2021-01-05T15:20:00.000000000', '2021-01-06T15:20:00.000000000',
           '2021-01-07T15:20:00.000000000', '2021-01-08T15:20:00.000000000',
           '2021-01-09T15:20:00.000000000', '2021-01-10T15:20:00.000000000',
           '2021-01-11T15:20:00.000000000', '2021-01-12T15:20:00.000000000',
           '2021-01-13T15:20:00.000000000', '2021-01-14T15:20:00.000000000',
           '2021-01-15T15:20:00.000000000', '2021-01-16T15:20:00.000000000',
           '2021-01-17T15:20:00.000000000', '2021-01-18T15:20:00.000000000',
           '2021-01-19T15:20:00.000000000', '2021-01-20T15:20:00.000000000',
           '2021-01-21T15:20:00.000000000', '2021-01-22T15:20:00.000000000',
           '2021-01-23T15:20:00.000000000', '2021-01-24T15:20:00.000000000',
           '2021-01-26T15:20:00.000000000', '2021-01-27T15:20:00.000000000',
           '2021-01-28T15:20:00.000000000', '2021-01-29T15:20:00.000000000',
           '2021-01-30T15:20:00.000000000', '2021-01-31T15:20:00.000000000',
           '2021-02-01T15:20:00.000000000', '2021-02-02T15:20:00.000000000',
           '2021-02-03T15:20:00.000000000', '2021-02-04T15:20:00.000000000',
           '2021-02-05T15:20:00.000000000', '2021-02-06T15:20:00.000000000',
           '2021-02-07T15:20:00.000000000', '2021-02-08T15:20:00.000000000',
           '2021-02-09T15:20:00.000000000', '2021-02-10T15:20:00.000000000',
           '2021-02-11T15:20:00.000000000', '2021-02-12T15:20:00.000000000',
           '2021-02-13T15:20:00.000000000', '2021-02-14T15:20:00.000000000',
           '2021-02-15T15:20:00.000000000', '2021-02-16T15:20:00.000000000',
           '2021-02-17T15:20:00.000000000', '2021-02-18T15:20:00.000000000',
           '2021-02-19T15:20:00.000000000', '2021-02-20T15:20:00.000000000',
           '2021-02-21T15:20:00.000000000', '2021-02-22T15:20:00.000000000',
           '2021-02-23T15:20:00.000000000', '2021-02-24T15:20:00.000000000',
           '2021-02-25T15:20:00.000000000', '2021-02-26T15:20:00.000000000',
           '2021-02-27T15:20:00.000000000', '2021-02-28T15:20:00.000000000',
           '2021-03-01T15:20:00.000000000', '2021-03-02T15:20:00.000000000',
           '2021-03-03T15:20:00.000000000', '2021-03-04T15:20:00.000000000',
           '2021-03-05T15:20:00.000000000', '2021-03-06T15:20:00.000000000',
           '2021-03-07T15:20:00.000000000', '2021-03-08T15:20:00.000000000',
           '2021-03-09T15:20:00.000000000', '2021-03-10T15:20:00.000000000',
           '2021-03-11T15:20:00.000000000', '2021-03-12T15:20:00.000000000',
           '2021-03-13T15:20:00.000000000', '2021-03-14T15:20:00.000000000',
           '2021-03-15T15:20:00.000000000', '2021-03-16T15:20:00.000000000',
           '2021-03-17T15:20:00.000000000', '2021-03-18T15:20:00.000000000',
           '2021-03-19T15:20:00.000000000', '2021-03-20T15:20:00.000000000',
           '2021-03-21T15:20:00.000000000', '2021-03-22T15:20:00.000000000',
           '2021-03-23T15:20:00.000000000', '2021-03-24T15:20:00.000000000',
           '2021-03-25T15:20:00.000000000', '2021-03-26T15:20:00.000000000',
           '2021-03-27T15:20:00.000000000', '2021-03-28T15:20:00.000000000',
           '2021-03-29T15:20:00.000000000', '2021-03-30T15:20:00.000000000',
           '2021-03-31T15:20:00.000000000', '2021-04-01T15:20:00.000000000',
           '2021-04-02T15:20:00.000000000', '2021-04-03T15:20:00.000000000',
           '2021-04-04T15:20:00.000000000', '2021-04-05T15:20:00.000000000',
           '2021-04-06T15:20:00.000000000', '2021-04-07T15:20:00.000000000',
           '2021-04-08T15:20:00.000000000', '2021-04-09T15:20:00.000000000',
           '2021-04-10T15:20:00.000000000', '2021-04-11T15:20:00.000000000',
           '2021-04-12T15:20:00.000000000', '2021-04-13T15:20:00.000000000',
           '2021-04-14T15:20:00.000000000', '2021-04-15T15:20:00.000000000',
           '2021-04-16T15:20:00.000000000', '2021-04-17T15:20:00.000000000',
           '2021-04-18T15:20:00.000000000', '2021-04-19T15:20:00.000000000',
           '2021-04-20T15:20:00.000000000', '2021-04-21T15:20:00.000000000',
           '2021-04-22T15:20:00.000000000', '2021-04-23T15:20:00.000000000',
           '2021-04-24T15:20:00.000000000', '2021-04-25T15:20:00.000000000',
           '2021-04-26T15:20:00.000000000', '2021-04-27T15:20:00.000000000',
           '2021-04-28T15:20:00.000000000', '2021-04-29T15:20:00.000000000',
           '2021-04-30T15:20:00.000000000', '2021-05-01T15:20:00.000000000',
           '2021-05-02T15:20:00.000000000', '2021-05-03T15:20:00.000000000',
           '2021-05-04T15:20:00.000000000', '2021-05-05T15:20:00.000000000',
           '2021-05-06T15:20:00.000000000', '2021-05-07T15:20:00.000000000',
           '2021-05-08T15:20:00.000000000', '2021-05-10T15:20:00.000000000',
           '2021-05-12T15:20:00.000000000', '2021-05-13T15:20:00.000000000',
           '2021-05-14T15:20:00.000000000', '2021-05-15T15:20:00.000000000',
           '2021-05-16T15:20:00.000000000', '2021-05-17T15:20:00.000000000',
           '2021-05-20T15:20:00.000000000', '2021-05-21T15:20:00.000000000',
           '2021-05-22T15:20:00.000000000', '2021-05-23T15:20:00.000000000',
           '2021-05-24T15:20:00.000000000', '2021-05-25T15:20:00.000000000',
           '2021-05-27T15:20:00.000000000', '2021-05-28T15:20:00.000000000',
           '2021-05-29T15:20:00.000000000', '2021-06-02T15:20:00.000000000',
           '2021-06-03T15:20:00.000000000', '2021-06-04T15:20:00.000000000',
           '2021-06-05T15:20:00.000000000', '2021-06-06T15:20:00.000000000',
           '2021-06-07T15:20:00.000000000', '2021-06-09T15:20:00.000000000',
           '2021-06-10T15:20:00.000000000', '2021-06-11T15:20:00.000000000',
           '2021-06-15T15:20:00.000000000', '2021-06-17T15:20:00.000000000',
           '2021-06-18T15:20:00.000000000', '2021-06-23T15:20:00.000000000',
           '2021-06-24T15:20:00.000000000', '2021-06-25T15:20:00.000000000',
           '2021-06-26T15:20:00.000000000', '2021-06-27T15:20:00.000000000',
           '2021-07-02T15:20:00.000000000', '2021-07-03T15:20:00.000000000',
           '2021-07-05T15:20:00.000000000', '2021-07-10T15:20:00.000000000',
           '2021-07-11T15:20:00.000000000', '2021-07-12T15:20:00.000000000',
           '2021-07-13T15:20:00.000000000', '2021-07-14T15:20:00.000000000',
           '2021-07-15T15:20:00.000000000', '2021-07-16T15:20:00.000000000',
           '2021-07-18T15:20:00.000000000', '2021-07-19T15:20:00.000000000',
           '2021-07-20T15:20:00.000000000', '2021-07-21T15:20:00.000000000',
           '2021-07-22T15:20:00.000000000', '2021-07-23T15:20:00.000000000',
           '2021-07-24T15:20:00.000000000', '2021-07-25T15:20:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (8)
  • dt_analysis
    (time, lat, lon)
    float32
    dask.array<chunksize=(64, 64, 64), meta=np.ndarray>

    _Netcdf4Dimid :

    2

    comment :

    The difference between this SST and the previous day's SST

    long_name :

    deviation from last SST analysis

    source :

    ABOM-L4LRfnd-GLOB-GAMSSA_28km

    units :

    kelvin

    valid_max :

    127

    valid_min :

    -127

    Array Chunk Bytes 17.90 GiB 1.00 MiB Shape (178, 4500, 6000) (64, 64, 64) Dask graph 20022 chunks in 2 graph layers Data type float32 numpy.ndarray

  • l2p_flags
    (time, lat, lon)
    float32
    dask.array<chunksize=(64, 64, 64), meta=np.ndarray>

    _Netcdf4Dimid :

    2

    comment :

    These flags are important to properly use the data. Data not flagged as microwave are sourced from an infrared sensor. The lake and river flags are currently not set, but defined in GDS2.0r4. The aerosol flag indicates high aerosol concentration. The analysis flag indicates high difference from analysis temperatures (differences greater than Analysis Limit). The lowwind flag indicates regions of low wind speed (typically less than the low Wind Limit) per NWP model. The highwind flag indicates regions of high wind speed (typically greater than the high Wind Limit) per NWP model. See wind limits in the comment field for the actual values. The edge flag indicates pixel sizes that are larger than Pixel Spread times the size of the pixel in the center of the field of view in either lat or lon direction. The terminator flag indicates that the sun is near the horizon. The reflector flag indicates that the satellite would receive direct reflected sunlight if the earth was a perfect mirror. The swath flag is used in gridded files to indicate if the pixel could have been seen by the satellite. delta_dn indicates that the day.night sst algorithm was different from the standard algorithm. Other flags may be populated and are for internal use and the definitions may change, so should not be relied on. Flags greater than 64 only apply to non-land pixels

    flag_masks :

    [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384]

    flag_meanings :

    microwave land ice lake river reserved aerosol analysis lowwind highwind edge terminator reflector swath delta_dn

    long_name :

    L2P flags

    valid_max :

    32767

    valid_min :

    0

    Array Chunk Bytes 17.90 GiB 1.00 MiB Shape (178, 4500, 6000) (64, 64, 64) Dask graph 20022 chunks in 2 graph layers Data type float32 numpy.ndarray

  • quality_level
    (time, lat, lon)
    float32
    dask.array<chunksize=(64, 64, 64), meta=np.ndarray>

    _Netcdf4Dimid :

    2

    comment :

    These are the overall quality indicators and are used for all GHRSST SSTs.In this case they are a function of distance to cloud, satellite zenith angle, and day/night

    flag_meanings :

    no_data bad_data worst_quality low_quality acceptable_quality best_quality

    flag_values :

    [0, 1, 2, 3, 4, 5]

    long_name :

    quality level of SST pixel

    valid_max :

    5

    valid_min :

    0

    Array Chunk Bytes 17.90 GiB 1.00 MiB Shape (178, 4500, 6000) (64, 64, 64) Dask graph 20022 chunks in 2 graph layers Data type float32 numpy.ndarray

  • satellite_zenith_angle
    (time, lat, lon)
    float32
    dask.array<chunksize=(64, 64, 64), meta=np.ndarray>

    _Netcdf4Dimid :

    2

    comment :

    The satellite zenith angle at the time of the SST observations

    long_name :

    satellite zenith angle

    units :

    angular_degree

    valid_max :

    127

    valid_min :

    -127

    Array Chunk Bytes 17.90 GiB 1.00 MiB Shape (178, 4500, 6000) (64, 64, 64) Dask graph 20022 chunks in 2 graph layers Data type float32 numpy.ndarray

  • sea_surface_temperature
    (time, lat, lon)
    float32
    dask.array<chunksize=(64, 64, 64), meta=np.ndarray>

    _Netcdf4Dimid :

    2

    comment :

    The skin temperature of the ocean at a depth of approximately 10 microns

    long_name :

    sea surface skin temperature

    standard_name :

    sea_surface_skin_temperature

    units :

    kelvin

    valid_max :

    32767

    valid_min :

    -32767

    Array Chunk Bytes 17.90 GiB 1.00 MiB Shape (178, 4500, 6000) (64, 64, 64) Dask graph 20022 chunks in 2 graph layers Data type float32 numpy.ndarray

  • sses_bias
    (time, lat, lon)
    float32
    dask.array<chunksize=(64, 64, 64), meta=np.ndarray>

    _Netcdf4Dimid :

    2

    comment :

    Bias estimate derived from L2P bias per https://www.ghrsst.org/ghrsst/tags-and-wgs/stval-wg/sses-description-of-schemes/

    long_name :

    SSES bias estimate

    units :

    kelvin

    valid_max :

    127

    valid_min :

    -127

    Array Chunk Bytes 17.90 GiB 1.00 MiB Shape (178, 4500, 6000) (64, 64, 64) Dask graph 20022 chunks in 2 graph layers Data type float32 numpy.ndarray

  • sses_count
    (time, lat, lon)
    float32
    dask.array<chunksize=(64, 64, 64), meta=np.ndarray>

    _Netcdf4Dimid :

    2

    comment :

    Weighted representative number of swath pixels. EXPERIMENTAL_FIELD

    long_name :

    SSES count

    units :

    count

    valid_max :

    127

    valid_min :

    -127

    Array Chunk Bytes 17.90 GiB 1.00 MiB Shape (178, 4500, 6000) (64, 64, 64) Dask graph 20022 chunks in 2 graph layers Data type float32 numpy.ndarray

  • sses_standard_deviation
    (time, lat, lon)
    float32
    dask.array<chunksize=(64, 64, 64), meta=np.ndarray>

    _Netcdf4Dimid :

    2

    comment :

    Standard deviation estimate derived from L2P standard deviation per https://www.ghrsst.org/ghrsst/tags-and-wgs/stval-wg/sses-description-of-schemes/

    long_name :

    SSES standard deviation estimate

    units :

    kelvin

    valid_max :

    127

    valid_min :

    -127

    Array Chunk Bytes 17.90 GiB 1.00 MiB Shape (178, 4500, 6000) (64, 64, 64) Dask graph 20022 chunks in 2 graph layers Data type float32 numpy.ndarray

• Attributes: (47)

  Conventions :

  CF-1.6

  Metadata_Conventions :

  Unidata Dataset Discovery v1.0

  Metadata_Link :

  TBA

  acknowledgment :

  Any use of these data requires the following acknowledgment:HRPT AVHRR SSTskin retrievals were produced by the Australian Bureau of Meteorology as a contribution to the Integrated Marine Observing System - an initiative of the Australian Government being conducted as part of the National Collaborative Research Infrastructure Strategy and the Super Science Initiative. The imagery data were acquired from NOAA spacecraft by the Bureau, Australian Institute of Marine Science, Australian Commonwealth Scientific and Industrial Research Organization, Geoscience Australia, and Western Australian Satellite Technology and Applications Consortium.

  cdm_data_type :

  grid

  comment :

  HRPT AVHRR experimental L3 retrieval produced by the Australian Bureau of Meteorology as a contribution to the Integrated Marine Observing System. SSTs were calibrated to drifting buoy depths (~20-30cm) followed by a cool skin correction of -0.17K to convert to a skin (~10 micron) SST. SSTs are a weighted average of the SSTs of contributing pixels (weighted by sses_standard_deviation^-2).\nWARNING: some applications are unable to properly handle signed byte values. If byte values >127 are encountered, subtract 256 from this reported value. GRID:CONTINENTAL, SYSCODE:PRODUCTION

  creator_email :

  [email protected]

  creator_name :

  Australian Bureau of Meteorology

  creator_url :

  http://www.imos.org.au/srs.html

  date_created :

  20210104T024028Z

  easternmost_longitude :

  -170.00999450683594

  file_quality_level :

  3

  gds_version_id :

  2.0r4

  geospatial_lat_resolution :

  0.019999999552965164

  geospatial_lat_units :

  degrees_north

  geospatial_lon_resolution :

  0.019999999552965164

  geospatial_lon_units :

  degrees_east

  history :

  platform_counts=NOAA-18=1,quality_counts=archive=1,platform=NOAA-18,quality_source=archive,ice_source=SSMI-NCEP-Analysis-ICE-5min,adi_source=unknown,wind_source=ACCESSG-ABOM-Forecast-WSP,analysis_source=ABOM-L4LRfnd-GLOB-GAMSSA_28km,source_file=20210101152000-ABOM-L3C_GHRSST-SSTskin-AVHRR18_D-1d_night-v02.0-fv01.0.nc,l3_file=20210101152000-ABOM-L3C_GHRSST-SSTskin-AVHRR18_D-1d_night-v02.0-fv01.0.nc,l3_source=AVHRR18_D-ABOM-L3C-v01.0,global_source=wind_source=ACCESSG-ABOM-Forecast-WSP,analysis_source=ABOM-L4LRfnd-GLOB-GAMSSA_28km,adi_source=unknown,ice_source=SSMI-NCEP-Analysis-ICE-5min,l3_source=AVHRR18_D-ABOM-L3C-v01.0,landmask_file=lsmask.dist5.5.nc,landmask_reference=Naval Oceanographic Office (NAVOCEANO),landmask_URL=https://www.ghrsst.org/data/ghrsst-data-tools/navo-ghrsst-pp-land-sea-mask/,landmask_source=NAVOCEANO 1km Version 5.5,ice_reference=US National Weather Service - NCEP,ice_URL=http://polar.ncep.noaa.gov/seaice/Analyses.html,ice_file=20201231.ice_data.5min.nc,ice_jdate=2459215,merge_tool=mergeL3U,mergeL3U_version=9042,quality=archive,mergeL3U_quality=archive

  id :

  AVHRR_D-ABOM-L3S-v01.0

  institution :

  ABOM

  keywords :

  Oceans > Ocean Temperature > Sea Surface Temperature

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Science Keywords

  license :

  GHRSST protocol describes data use as free and open

  naming_authority :

  org.ghrsst

  netcdf_version_id :

  4.3.3.1

  northernmost_latitude :

  19.989999771118164

  platform :

  NOAA-18

  processing_level :

  L3S

  product_version :

  01.0

  project :

  Group for High Resolution Sea Surface Temperature

  publisher_email :

  [email protected]

  publisher_name :

  The GHRSST Project Office

  publisher_url :

  http://www.ghrsst.org

  references :

  http://imos.org.au/sstproducts.html and Griffin et al. (2017) at http://imos.org.au/facilities/srs/sstproducts/sstdata0/sstdata-references

  sensor :

  VIIRS,AVHRR

  source :

  wind_source=ACCESSG-ABOM-Forecast-WSP,analysis_source=ABOM-L4LRfnd-GLOB-GAMSSA_28km,adi_source=unknown,ice_source=SSMI-NCEP-Analysis-ICE-5min,l3_source=AVHRR18_D-ABOM-L3C-v01.0

  southernmost_latitude :

  -69.98999786376953

  spatial_resolution :

  0.02 deg

  standard_name_vocabulary :

  NetCDF Climate and Forecast (CF) Metadata Convention

  start_time :

  20210101T095824Z

  stop_time :

  20210101T151752Z

  summary :

  Skin SST retrievals produced from stitching together High Resolution Picture Transmission direct broadcast data from a NOAA polar-orbiting satellite received at Australian receiving stations.

  time_coverage_end :

  20210101T151752Z

  time_coverage_start :

  20210101T095824Z

  title :

  IMOS L3S Nighttime gridded multiple-sensor multiple-swath Australian region skin SST

  uuid :

  4d02ee75-876d-4ff0-8956-ab68917c9001

  westernmost_longitude :

  70.01000213623047

# We can see the chunked structure of the data by looking at one of the variables
zds.sea_surface_temperature

<xarray.DataArray 'sea_surface_temperature' (time: 178, lat: 4500, lon: 6000)>
dask.array<open_dataset-5ed7711d911bd370bffa8b7f9a31ccf1sea_surface_temperature, shape=(178, 4500, 6000), dtype=float32, chunksize=(64, 64, 64), chunktype=numpy.ndarray>
Coordinates:
  * lat      (lat) float32 19.99 19.97 19.95 19.93 ... -69.95 -69.97 -69.99
  * lon      (lon) float32 70.01 70.03 70.05 70.07 ... 189.9 189.9 190.0 190.0
  * time     (time) datetime64[ns] 2021-01-01T15:20:00 ... 2021-07-25T15:20:00
Attributes:
    _Netcdf4Dimid:  2
    comment:        The skin temperature of the ocean at a depth of approxima...
    long_name:      sea surface skin temperature
    standard_name:  sea_surface_skin_temperature
    units:          kelvin
    valid_max:      32767
    valid_min:      -32767

xarray.DataArray

'sea_surface_temperature'

• time: 178
• lat: 4500
• lon: 6000

• dask.array<chunksize=(64, 64, 64), meta=np.ndarray>

  Array Chunk Bytes 17.90 GiB 1.00 MiB Shape (178, 4500, 6000) (64, 64, 64) Dask graph 20022 chunks in 2 graph layers Data type float32 numpy.ndarray

• Coordinates: (3)
  • lat
    (lat)
    float32
    19.99 19.97 19.95 ... -69.97 -69.99

    NAME :

    lat

    _Netcdf4Dimid :

    1

    axis :

    Y

    comment :

    Latitudes for locating data

    long_name :

    latitude

    standard_name :

    latitude

    units :

    degrees_north

    valid_max :

    90.0

    valid_min :

    -90.0

    array([ 19.99,  19.97,  19.95, ..., -69.95, -69.97, -69.99], dtype=float32)

  • lon
    (lon)
    float32
    70.01 70.03 70.05 ... 190.0 190.0

    NAME :

    lon

    _Netcdf4Dimid :

    0

    axis :

    X

    comment :

    Longitudes for locating data

    long_name :

    longitude

    standard_name :

    longitude

    units :

    degrees_east

    valid_max :

    360.0

    valid_min :

    -180.0

    array([ 70.01,  70.03,  70.05, ..., 189.95, 189.97, 189.99], dtype=float32)

  • time
    (time)
    datetime64[ns]
    2021-01-01T15:20:00 ... 2021-07-...

    NAME :

    time

    _Netcdf4Dimid :

    2

    axis :

    T

    comment :

    A typical reference time for data

    long_name :

    reference time of sst file

    standard_name :

    time

    array(['2021-01-01T15:20:00.000000000', '2021-01-02T15:20:00.000000000',
           '2021-01-03T15:20:00.000000000', '2021-01-04T15:20:00.000000000',
           '2021-01-05T15:20:00.000000000', '2021-01-06T15:20:00.000000000',
           '2021-01-07T15:20:00.000000000', '2021-01-08T15:20:00.000000000',
           '2021-01-09T15:20:00.000000000', '2021-01-10T15:20:00.000000000',
           '2021-01-11T15:20:00.000000000', '2021-01-12T15:20:00.000000000',
           '2021-01-13T15:20:00.000000000', '2021-01-14T15:20:00.000000000',
           '2021-01-15T15:20:00.000000000', '2021-01-16T15:20:00.000000000',
           '2021-01-17T15:20:00.000000000', '2021-01-18T15:20:00.000000000',
           '2021-01-19T15:20:00.000000000', '2021-01-20T15:20:00.000000000',
           '2021-01-21T15:20:00.000000000', '2021-01-22T15:20:00.000000000',
           '2021-01-23T15:20:00.000000000', '2021-01-24T15:20:00.000000000',
           '2021-01-26T15:20:00.000000000', '2021-01-27T15:20:00.000000000',
           '2021-01-28T15:20:00.000000000', '2021-01-29T15:20:00.000000000',
           '2021-01-30T15:20:00.000000000', '2021-01-31T15:20:00.000000000',
           '2021-02-01T15:20:00.000000000', '2021-02-02T15:20:00.000000000',
           '2021-02-03T15:20:00.000000000', '2021-02-04T15:20:00.000000000',
           '2021-02-05T15:20:00.000000000', '2021-02-06T15:20:00.000000000',
           '2021-02-07T15:20:00.000000000', '2021-02-08T15:20:00.000000000',
           '2021-02-09T15:20:00.000000000', '2021-02-10T15:20:00.000000000',
           '2021-02-11T15:20:00.000000000', '2021-02-12T15:20:00.000000000',
           '2021-02-13T15:20:00.000000000', '2021-02-14T15:20:00.000000000',
           '2021-02-15T15:20:00.000000000', '2021-02-16T15:20:00.000000000',
           '2021-02-17T15:20:00.000000000', '2021-02-18T15:20:00.000000000',
           '2021-02-19T15:20:00.000000000', '2021-02-20T15:20:00.000000000',
           '2021-02-21T15:20:00.000000000', '2021-02-22T15:20:00.000000000',
           '2021-02-23T15:20:00.000000000', '2021-02-24T15:20:00.000000000',
           '2021-02-25T15:20:00.000000000', '2021-02-26T15:20:00.000000000',
           '2021-02-27T15:20:00.000000000', '2021-02-28T15:20:00.000000000',
           '2021-03-01T15:20:00.000000000', '2021-03-02T15:20:00.000000000',
           '2021-03-03T15:20:00.000000000', '2021-03-04T15:20:00.000000000',
           '2021-03-05T15:20:00.000000000', '2021-03-06T15:20:00.000000000',
           '2021-03-07T15:20:00.000000000', '2021-03-08T15:20:00.000000000',
           '2021-03-09T15:20:00.000000000', '2021-03-10T15:20:00.000000000',
           '2021-03-11T15:20:00.000000000', '2021-03-12T15:20:00.000000000',
           '2021-03-13T15:20:00.000000000', '2021-03-14T15:20:00.000000000',
           '2021-03-15T15:20:00.000000000', '2021-03-16T15:20:00.000000000',
           '2021-03-17T15:20:00.000000000', '2021-03-18T15:20:00.000000000',
           '2021-03-19T15:20:00.000000000', '2021-03-20T15:20:00.000000000',
           '2021-03-21T15:20:00.000000000', '2021-03-22T15:20:00.000000000',
           '2021-03-23T15:20:00.000000000', '2021-03-24T15:20:00.000000000',
           '2021-03-25T15:20:00.000000000', '2021-03-26T15:20:00.000000000',
           '2021-03-27T15:20:00.000000000', '2021-03-28T15:20:00.000000000',
           '2021-03-29T15:20:00.000000000', '2021-03-30T15:20:00.000000000',
           '2021-03-31T15:20:00.000000000', '2021-04-01T15:20:00.000000000',
           '2021-04-02T15:20:00.000000000', '2021-04-03T15:20:00.000000000',
           '2021-04-04T15:20:00.000000000', '2021-04-05T15:20:00.000000000',
           '2021-04-06T15:20:00.000000000', '2021-04-07T15:20:00.000000000',
           '2021-04-08T15:20:00.000000000', '2021-04-09T15:20:00.000000000',
           '2021-04-10T15:20:00.000000000', '2021-04-11T15:20:00.000000000',
           '2021-04-12T15:20:00.000000000', '2021-04-13T15:20:00.000000000',
           '2021-04-14T15:20:00.000000000', '2021-04-15T15:20:00.000000000',
           '2021-04-16T15:20:00.000000000', '2021-04-17T15:20:00.000000000',
           '2021-04-18T15:20:00.000000000', '2021-04-19T15:20:00.000000000',
           '2021-04-20T15:20:00.000000000', '2021-04-21T15:20:00.000000000',
           '2021-04-22T15:20:00.000000000', '2021-04-23T15:20:00.000000000',
           '2021-04-24T15:20:00.000000000', '2021-04-25T15:20:00.000000000',
           '2021-04-26T15:20:00.000000000', '2021-04-27T15:20:00.000000000',
           '2021-04-28T15:20:00.000000000', '2021-04-29T15:20:00.000000000',
           '2021-04-30T15:20:00.000000000', '2021-05-01T15:20:00.000000000',
           '2021-05-02T15:20:00.000000000', '2021-05-03T15:20:00.000000000',
           '2021-05-04T15:20:00.000000000', '2021-05-05T15:20:00.000000000',
           '2021-05-06T15:20:00.000000000', '2021-05-07T15:20:00.000000000',
           '2021-05-08T15:20:00.000000000', '2021-05-10T15:20:00.000000000',
           '2021-05-12T15:20:00.000000000', '2021-05-13T15:20:00.000000000',
           '2021-05-14T15:20:00.000000000', '2021-05-15T15:20:00.000000000',
           '2021-05-16T15:20:00.000000000', '2021-05-17T15:20:00.000000000',
           '2021-05-20T15:20:00.000000000', '2021-05-21T15:20:00.000000000',
           '2021-05-22T15:20:00.000000000', '2021-05-23T15:20:00.000000000',
           '2021-05-24T15:20:00.000000000', '2021-05-25T15:20:00.000000000',
           '2021-05-27T15:20:00.000000000', '2021-05-28T15:20:00.000000000',
           '2021-05-29T15:20:00.000000000', '2021-06-02T15:20:00.000000000',
           '2021-06-03T15:20:00.000000000', '2021-06-04T15:20:00.000000000',
           '2021-06-05T15:20:00.000000000', '2021-06-06T15:20:00.000000000',
           '2021-06-07T15:20:00.000000000', '2021-06-09T15:20:00.000000000',
           '2021-06-10T15:20:00.000000000', '2021-06-11T15:20:00.000000000',
           '2021-06-15T15:20:00.000000000', '2021-06-17T15:20:00.000000000',
           '2021-06-18T15:20:00.000000000', '2021-06-23T15:20:00.000000000',
           '2021-06-24T15:20:00.000000000', '2021-06-25T15:20:00.000000000',
           '2021-06-26T15:20:00.000000000', '2021-06-27T15:20:00.000000000',
           '2021-07-02T15:20:00.000000000', '2021-07-03T15:20:00.000000000',
           '2021-07-05T15:20:00.000000000', '2021-07-10T15:20:00.000000000',
           '2021-07-11T15:20:00.000000000', '2021-07-12T15:20:00.000000000',
           '2021-07-13T15:20:00.000000000', '2021-07-14T15:20:00.000000000',
           '2021-07-15T15:20:00.000000000', '2021-07-16T15:20:00.000000000',
           '2021-07-18T15:20:00.000000000', '2021-07-19T15:20:00.000000000',
           '2021-07-20T15:20:00.000000000', '2021-07-21T15:20:00.000000000',
           '2021-07-22T15:20:00.000000000', '2021-07-23T15:20:00.000000000',
           '2021-07-24T15:20:00.000000000', '2021-07-25T15:20:00.000000000'],
          dtype='datetime64[ns]')

• Attributes: (7)

  _Netcdf4Dimid :

  2

  comment :

  The skin temperature of the ocean at a depth of approximately 10 microns

  long_name :

  sea surface skin temperature

  standard_name :

  sea_surface_skin_temperature

  units :

  kelvin

  valid_max :

  32767

  valid_min :

  -32767

%%time

# We can plot this dataset in exactly the same way as the NetCDF-based one
sst_var = 'sea_surface_temperature'
gds = gv.Dataset(zds[sst_var].sel(time='2021-01-02', lat=slice(0, -50), lon=slice(105, 175)),
                 kdims=['lon', 'lat'],
                 vdims=[sst_var]
                )
sst_plot = (gds.to(gv.Image, ['lon', 'lat'])
               .opts(cmap='coolwarm', colorbar=True, aspect=1.4, title=zds.title))
sst_plot * gf.land

Another example

A more detailed example of working with similar data in Zarr format can be found here: https://github.com/aodn/rimrep-examples/blob/main/Python_based_scripts/zarr.ipynb

Parquet

• Parquet is a cloud-optimised format designed for tabular data.

• Each column of the table is stored in a separate file/object.

• These can be further partitioned into row groups.

For a quick demo, we’ll borrow an example from this more detailed notebook, looking at temperature logger data from the Australian Institute of Marine Science. The dataset contains 150 million temperature measurements from numerous sites around Australia (metadata for this dataset).

We’ll use dask.dataframe to access the parquet data in a lazy way - reading only what is necessary and only when requested.

# Here's the path to the dataset on AWS S3
parquet_path = "s3://rimrep-data-public/091-aims-sst/test-50-64-spatialpart/"

# Let's see if there are any temperature loggers near us (in Dunsborough, Western Australia)
filters = [('lon', '>', 114.5),
          ('lon', '<', 115.5),
          ('lat', '>', -34.),
          ('lat', '<', -33.)]

df = dd.read_parquet(parquet_path,
                     filters=filters,
                     # only read the site names and QC'd temperature values
                     columns = ['site', 'qc_val'],
                     index='time',
                     engine='pyarrow',
                     storage_options = {"anon": True}
                    )
df.head()

	site	qc_val
time
2015-01-02 07:30:00+00:00	Geographe Bay	24.3728
2015-01-02 07:00:00+00:00	Geographe Bay	24.3728
2015-01-02 06:30:00+00:00	Geographe Bay	24.3238
2015-01-02 06:00:00+00:00	Geographe Bay	24.2518
2015-01-02 05:30:00+00:00	Geographe Bay	24.1798

len(df)

94896

# This subset should fit into memory, so let's turn it into a regular DataFrame
df = df.compute()

# Let's see how many sites we have...
df.site.unique()

array(['Geographe Bay', 'Cowaramup Bay', 'Canal Rocks'], dtype=object)

# Now we can plot the temperature timeseries for all these sites
df.hvplot(by='site')

Alternative dataset

Another Parquet example using data from the Ocean Biodiversity Information System (OBIS) is shown in this notebook

Other methods

ERDDAP

• Supports searching, subsetting, and downloads in a wide range of formats

• Example

• Also covered in this OHW22 tutorial

New OGC APIs

• New standards from the Open Geospatial Consortium

• OGC Features (replacement for WFS) - example

• OGC Coverages example

OGC Web Map Service (WMS)

• Also covered in this OHW22 tutorial

Further resources

• Lots of data access examples at IOOS CodeLab

• Examples in both Python and R from Reef 2050 Integrated Monitoring and Reporting Program Data Management System (RIMReP DMS)