{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"HoloViz](\"../../assets/pn_hv_gv_bk_ds_pa.png\")
\n",
"\n",
"Exercise 6: Advanced Dashboarding
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This exercise is entirely freeform. Get into groups of 3-4 people and start\n",
"building a dashboard, with everything you have learned in this tutorial. By the\n",
"end of the exercise you should have a dashboard that:\n",
"\n",
"- Uses datashading to render the whole dataset\n",
"- Builds a pipeline using the `.apply` method\n",
"- Uses a widget to filter the data either by cross-filtering with a linked\n",
" stream (e.g. a BoundsXY stream) or using a widget (e.g. a RangeSlider)\n",
"- Uses a widget to control some aspect of the styling of the plot (e.g. to\n",
" select a colormap, color, or size)\n",
"- Is servable by running `panel serve Advanced_Dashboarding.ipynb` in the\n",
" exercise directory\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import colorcet as cc # noqa\n",
"import holoviews as hv # noqa\n",
"import numpy as np # noqa\n",
"import dask.dataframe as dd\n",
"import panel as pn\n",
"import xarray as xr\n",
"\n",
"import hvplot.pandas # noqa: API import\n",
"import hvplot.xarray # noqa: API import\n",
"\n",
"pn.extension()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": true
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Earthquakes dataset present, skipping download\n"
]
},
{
"ename": "PermissionError",
"evalue": "[Errno 13] Permission denied",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mPermissionError\u001b[0m Traceback (most recent call last)",
"Input \u001b[1;32mIn [3]\u001b[0m, in \u001b[0;36m\u001b[1;34m()\u001b[0m\n\u001b[0;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mfetch_data\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;241m*\u001b[39m\n\u001b[0;32m 4\u001b[0m get_earthquake_data()\n\u001b[1;32m----> 5\u001b[0m \u001b[43mget_population_data\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[1;32m~\\Documents\\GitHub\\ohw-tutorials\\data-viz\\tutorial\\exercises\\fetch_data.py:65\u001b[0m, in \u001b[0;36mget_population_data\u001b[1;34m()\u001b[0m\n\u001b[0;32m 63\u001b[0m download_url(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mhttps://earth-data.s3.amazonaws.com/\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfname\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m'\u001b[39m, fname)\n\u001b[0;32m 64\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m zipfile\u001b[38;5;241m.\u001b[39mZipFile(fname, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124m'\u001b[39m) \u001b[38;5;28;01mas\u001b[39;00m zip_ref:\n\u001b[1;32m---> 65\u001b[0m \u001b[43mzip_ref\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mextractall\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdata_path\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 66\u001b[0m os\u001b[38;5;241m.\u001b[39mremove(fname)\n",
"File \u001b[1;32m~\\anaconda3\\envs\\OHW\\lib\\zipfile.py:1642\u001b[0m, in \u001b[0;36mZipFile.extractall\u001b[1;34m(self, path, members, pwd)\u001b[0m\n\u001b[0;32m 1639\u001b[0m path \u001b[38;5;241m=\u001b[39m os\u001b[38;5;241m.\u001b[39mfspath(path)\n\u001b[0;32m 1641\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m zipinfo \u001b[38;5;129;01min\u001b[39;00m members:\n\u001b[1;32m-> 1642\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_extract_member\u001b[49m\u001b[43m(\u001b[49m\u001b[43mzipinfo\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpath\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpwd\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[1;32m~\\anaconda3\\envs\\OHW\\lib\\zipfile.py:1697\u001b[0m, in \u001b[0;36mZipFile._extract_member\u001b[1;34m(self, member, targetpath, pwd)\u001b[0m\n\u001b[0;32m 1693\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m targetpath\n\u001b[0;32m 1695\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mopen(member, pwd\u001b[38;5;241m=\u001b[39mpwd) \u001b[38;5;28;01mas\u001b[39;00m source, \\\n\u001b[0;32m 1696\u001b[0m \u001b[38;5;28mopen\u001b[39m(targetpath, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mwb\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;28;01mas\u001b[39;00m target:\n\u001b[1;32m-> 1697\u001b[0m \u001b[43mshutil\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcopyfileobj\u001b[49m\u001b[43m(\u001b[49m\u001b[43msource\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtarget\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 1699\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m targetpath\n",
"File \u001b[1;32m~\\anaconda3\\envs\\OHW\\lib\\shutil.py:208\u001b[0m, in \u001b[0;36mcopyfileobj\u001b[1;34m(fsrc, fdst, length)\u001b[0m\n\u001b[0;32m 206\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m buf:\n\u001b[0;32m 207\u001b[0m \u001b[38;5;28;01mbreak\u001b[39;00m\n\u001b[1;32m--> 208\u001b[0m \u001b[43mfdst_write\u001b[49m\u001b[43m(\u001b[49m\u001b[43mbuf\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[1;31mPermissionError\u001b[0m: [Errno 13] Permission denied"
]
}
],
"source": [
"# Run this if you haven't already to fetch earthquake and population data files\n",
"from fetch_data import *\n",
"\n",
"get_earthquake_data()\n",
"get_population_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As a starting point we will load the data; everything else is up to you:\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"ename": "ValueError",
"evalue": "did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'h5netcdf', 'scipy', 'zarr']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:\nhttps://docs.xarray.dev/en/stable/getting-started-guide/installing.html\nhttps://docs.xarray.dev/en/stable/user-guide/io.html",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mValueError\u001b[0m Traceback (most recent call last)",
"Input \u001b[1;32mIn [4]\u001b[0m, in \u001b[0;36m| \u001b[1;34m()\u001b[0m\n\u001b[0;32m 1\u001b[0m df \u001b[38;5;241m=\u001b[39m (\n\u001b[0;32m 2\u001b[0m dd\u001b[38;5;241m.\u001b[39mread_parquet(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m../../data/earthquakes.parq\u001b[39m\u001b[38;5;124m\"\u001b[39m, index\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mindex\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m 3\u001b[0m \u001b[38;5;241m.\u001b[39mrepartition(npartitions\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m4\u001b[39m)\n\u001b[0;32m 4\u001b[0m \u001b[38;5;241m.\u001b[39mpersist()\n\u001b[0;32m 5\u001b[0m )\n\u001b[1;32m----> 7\u001b[0m ds \u001b[38;5;241m=\u001b[39m \u001b[43mxr\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mopen_dataarray\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m 8\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m../../data/gpw_v4_population_density_rev11_2010_2pt5_min.nc\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\n\u001b[0;32m 9\u001b[0m \u001b[43m)\u001b[49m\n\u001b[0;32m 10\u001b[0m cleaned_ds \u001b[38;5;241m=\u001b[39m ds\u001b[38;5;241m.\u001b[39mwhere(ds\u001b[38;5;241m.\u001b[39mvalues \u001b[38;5;241m!=\u001b[39m ds\u001b[38;5;241m.\u001b[39mnodatavals)\u001b[38;5;241m.\u001b[39msel(band\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m)\n\u001b[0;32m 11\u001b[0m cleaned_ds\u001b[38;5;241m.\u001b[39mname \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpopulation\u001b[39m\u001b[38;5;124m\"\u001b[39m\n",
"File \u001b[1;32m~\\anaconda3\\envs\\OHW\\lib\\site-packages\\xarray\\backends\\api.py:692\u001b[0m, in \u001b[0;36mopen_dataarray\u001b[1;34m(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, backend_kwargs, **kwargs)\u001b[0m\n\u001b[0;32m 552\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mopen_dataarray\u001b[39m(\n\u001b[0;32m 553\u001b[0m filename_or_obj: \u001b[38;5;28mstr\u001b[39m \u001b[38;5;241m|\u001b[39m os\u001b[38;5;241m.\u001b[39mPathLike,\n\u001b[0;32m 554\u001b[0m \u001b[38;5;241m*\u001b[39m,\n\u001b[1;32m (...)\u001b[0m\n\u001b[0;32m 568\u001b[0m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs,\n\u001b[0;32m 569\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m DataArray:\n\u001b[0;32m 570\u001b[0m \u001b[38;5;124;03m\"\"\"Open an DataArray from a file or file-like object containing a single\u001b[39;00m\n\u001b[0;32m 571\u001b[0m \u001b[38;5;124;03m data variable.\u001b[39;00m\n\u001b[0;32m 572\u001b[0m \n\u001b[1;32m (...)\u001b[0m\n\u001b[0;32m 689\u001b[0m \u001b[38;5;124;03m open_dataset\u001b[39;00m\n\u001b[0;32m 690\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[1;32m--> 692\u001b[0m dataset \u001b[38;5;241m=\u001b[39m open_dataset(\n\u001b[0;32m 693\u001b[0m filename_or_obj,\n\u001b[0;32m 694\u001b[0m decode_cf\u001b[38;5;241m=\u001b[39mdecode_cf,\n\u001b[0;32m 695\u001b[0m mask_and_scale\u001b[38;5;241m=\u001b[39mmask_and_scale,\n\u001b[0;32m 696\u001b[0m decode_times\u001b[38;5;241m=\u001b[39mdecode_times,\n\u001b[0;32m 697\u001b[0m concat_characters\u001b[38;5;241m=\u001b[39mconcat_characters,\n\u001b[0;32m 698\u001b[0m decode_coords\u001b[38;5;241m=\u001b[39mdecode_coords,\n\u001b[0;32m 699\u001b[0m engine\u001b[38;5;241m=\u001b[39mengine,\n\u001b[0;32m 700\u001b[0m chunks\u001b[38;5;241m=\u001b[39mchunks,\n\u001b[0;32m 701\u001b[0m cache\u001b[38;5;241m=\u001b[39mcache,\n\u001b[0;32m 702\u001b[0m drop_variables\u001b[38;5;241m=\u001b[39mdrop_variables,\n\u001b[0;32m 703\u001b[0m inline_array\u001b[38;5;241m=\u001b[39minline_array,\n\u001b[0;32m 704\u001b[0m backend_kwargs\u001b[38;5;241m=\u001b[39mbackend_kwargs,\n\u001b[0;32m 705\u001b[0m use_cftime\u001b[38;5;241m=\u001b[39muse_cftime,\n\u001b[0;32m 706\u001b[0m decode_timedelta\u001b[38;5;241m=\u001b[39mdecode_timedelta,\n\u001b[0;32m 707\u001b[0m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs,\n\u001b[0;32m 708\u001b[0m )\n\u001b[0;32m 710\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(dataset\u001b[38;5;241m.\u001b[39mdata_vars) \u001b[38;5;241m!=\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[0;32m 711\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[0;32m 712\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGiven file dataset contains more than one data \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 713\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mvariable. Please read with xarray.open_dataset and \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 714\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mthen select the variable you want.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 715\u001b[0m )\n",
"File \u001b[1;32m~\\anaconda3\\envs\\OHW\\lib\\site-packages\\xarray\\backends\\api.py:515\u001b[0m, in \u001b[0;36mopen_dataset\u001b[1;34m(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, backend_kwargs, **kwargs)\u001b[0m\n\u001b[0;32m 512\u001b[0m kwargs\u001b[38;5;241m.\u001b[39mupdate(backend_kwargs)\n\u001b[0;32m 514\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m engine \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m--> 515\u001b[0m engine \u001b[38;5;241m=\u001b[39m \u001b[43mplugins\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mguess_engine\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfilename_or_obj\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 517\u001b[0m backend \u001b[38;5;241m=\u001b[39m plugins\u001b[38;5;241m.\u001b[39mget_backend(engine)\n\u001b[0;32m 519\u001b[0m decoders \u001b[38;5;241m=\u001b[39m _resolve_decoders_kwargs(\n\u001b[0;32m 520\u001b[0m decode_cf,\n\u001b[0;32m 521\u001b[0m open_backend_dataset_parameters\u001b[38;5;241m=\u001b[39mbackend\u001b[38;5;241m.\u001b[39mopen_dataset_parameters,\n\u001b[1;32m (...)\u001b[0m\n\u001b[0;32m 527\u001b[0m decode_coords\u001b[38;5;241m=\u001b[39mdecode_coords,\n\u001b[0;32m 528\u001b[0m )\n",
"File \u001b[1;32m~\\anaconda3\\envs\\OHW\\lib\\site-packages\\xarray\\backends\\plugins.py:155\u001b[0m, in \u001b[0;36mguess_engine\u001b[1;34m(store_spec)\u001b[0m\n\u001b[0;32m 147\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m 148\u001b[0m error_msg \u001b[38;5;241m=\u001b[39m (\n\u001b[0;32m 149\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfound the following matches with the input file in xarray\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124ms IO \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 150\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mbackends: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mcompatible_engines\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. But their dependencies may not be installed, see:\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 151\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhttps://docs.xarray.dev/en/stable/user-guide/io.html \u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 152\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhttps://docs.xarray.dev/en/stable/getting-started-guide/installing.html\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 153\u001b[0m )\n\u001b[1;32m--> 155\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(error_msg)\n",
"\u001b[1;31mValueError\u001b[0m: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'h5netcdf', 'scipy', 'zarr']. Consider explicitly selecting one of the installed engines via the ``engine`` parameter, or installing additional IO dependencies, see:\nhttps://docs.xarray.dev/en/stable/getting-started-guide/installing.html\nhttps://docs.xarray.dev/en/stable/user-guide/io.html"
]
}
],
"source": [
"df = (\n",
" dd.read_parquet(\"../../data/earthquakes.parq\", index='index')\n",
" .repartition(npartitions=4)\n",
" .persist()\n",
")\n",
"\n",
"ds = xr.open_dataarray(\n",
" \"../../data/gpw_v4_population_density_rev11_2010_2pt5_min.nc\", \n",
")\n",
"cleaned_ds = ds.where(ds.values != ds.nodatavals).sel(band=1)\n",
"cleaned_ds.name = \"population\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You don't really know what to build? Here are some ideas:\n",
"\n",
"- Build a dashboard with a pipeline that filters the data on one or more of the\n",
" columns (e.g. magnitude using a `RangeSlider` or time using a\n",
" `DateRangeSlider`) and then datashades it\n",
"- Build a dashboard with multiple views of the data (e.g. longitude vs.\n",
" latitude, magnitude vs. depth etc.) and cross-filters the data using\n",
" `BoundsXY` streams (see the\n",
" [Glaciers notebook](https://github.com/pyviz-demos/glaciers/blob/master/glaciers.ipynb)\n",
" for reference)\n",
"- Build a dashboard that allows you to select multiple earthquakes using a\n",
" 'box_select' tool and a `Selection1D` stream and compute statistics on them.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python [conda env:OHW] *",
"language": "python",
"name": "conda-env-OHW-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
| |