netCDF browser for remote work

[stef]

I was wondering if anybody has experience with visualtization tools for remote work over the internet (WWW, not LAN).

I used to mount remote directories on my local machine via sshfs, so any netcdf browser will work. However, now I sometimes make larger simulations, and the UI of the browser becomes laggy to a point where it's not worth the wait. I suspect that transferring the data even for an instantaneous 2-d plane (say, an x-y plane at fixed z and t) takes too much time using sshfs, at least with my internet provider.

I use sshfs because I read it's more secure than NFS.

I don't want to use X11 forwarding, because in my experience it stalls a lot. Maybe I should give it another try?

What I think I need/want is a netCDF viewer that has a locally running GUI where I can pick coordinates, and a remotely running server component that generates .pngs (.jpegs, .pdf, etc.) server-side. Then the client only has to download the finished image. The image size may or may not be smaller than the underlying data, but if things get laggy, one could specifically choose an imaging algorithm that produces little data. Say, a contour plot with 10 iso-lines. For a typical ocean variable, the resulting image size is (usually) much smaller than the size of the underlying data. That holds for vector graphics, and compressed bitmaps.

I'm sure such a visualization tool exists, but looking through e.g. the following site:

https://www.unidata.ucar.edu/software/n ... tware.html

it's hard to figure out what exactly they do under the hood. Again I don't want to transfer remote raw data and display it locally, but instead display it remotely and send over the finished figure.

Do you have experience with this problem?

Thanks, Stefan

[marceloandrioni]

Unidata solution for this problem is to install a THREDDS Data Server (TDS) in the remote machine where the model results are stored. The TDS makes it easy to extract a time-series for a single location or vertical section, horizontal section, etc, without having to download the whole dataset. The TDS also has the capability of aggregating files in a single virtual link, that is, you can create a single link where hundreds of ROMS output files would show as a single large file. This single link could then be accessed using Matlab, Python, Ferret, etc to retrieve only the desired location/variable/times.

Here an example of a TDS loaded with a lot of datasets:
https://data.nodc.noaa.gov/thredds/catalog.html

Here some documentation on how to install and configure TDS:
https://www.unidata.ucar.edu/software/t ... t/TDS.html

[stef]

Thanks for your response!

Yes that would be great, but does THREDDS have an option to deliver a UI to the clients so they can browse data, in the sense that they can specify e.g. which x-y slice at fixed z and t they want to look at? This way only the figure would be sent over the internet, not the raw data.

I briefly looked at the links you provided, but did not see it.

What I'm looking for is an interface like e.g. NCVIEW provides, for quick browsing and sanity-checking, e.g. because it's too early in the development process to write a proper script for automated quality control.

However, having a javascript UI running in the browser, and delivered e.g. by THREDDS, would be equally fine!

As already noted in the original post, I think downloading raw data (even slices proveded by THREDDS), will lead to lags. Looking at a 800x800 point contour plot will require a download of 800x800x8 bytes (8 bytes for one double precision) ~= 5.1MB uncompressed.

I'm picking 800x800 for a large 3-d model, but with a barotropic model, e.g. idealized storm surge simulations, it could be bigger.

I just tried compressing an 800x800 double precision field with python-numpy's compression, it can compress to about 3 to 4.6 MB, presumably depending on whether it's a smoothly varying field or random data.

I have a maximum bandwith of 50Mbps, which is already very fast for the region I live in, so ideally with zero latency it would take ~1 second to download 5MB. Hmm, did I do the math correctly? But in practice it probably takes much longer than that.

So if we had a re-written version of NCVIEW that communicated with THREDDS, then quickly pressing the NCVIEW's "play" button five times would request 5 x 5MB of raw data from the server. That's 5 seconds purely for downloading raw data, and with an ideal downloading speed (for my region).

On the other hand, a countour plot in .jpg, .png or even .pdf will be smaller for a typical oceanographic field. Although image compression is unfortunately above my head, I can say that e.g. the attached .png below is ~100KB. Don't mind the hurricane track and coastline, it's just an example for a contour plot. My point is the image is much smaller than the raw data, and the resolution is very usable for a quick impression.

fig_track_nam.png (109.18 KiB) Viewed 9334 times

[stef]

Also, I just found a paper on "Accessing HDF5 Files via SSHFS".

https://support.hdfgroup.org/pubs/paper ... -final.pdf

They say it's comparable to NFS, and useful for subsetting remote files. So it may be comparable to using the THREDDS server anyways.

[marceloandrioni]

A group at the University of Reading UK developed a WMS (Web Map Server) specially customized to deal with NetCDF files called ncWMS (https://reading-escience-centre.gitbook ... e/content/).
A live example can be seen here:
http://godiva.rdg.ac.uk/ncWMS2/Godiva3.html

The people at Unidata then integrated ncWMS into TDS. Here you can see ncWMS/Godiva2 in action inside TDS:
http://tds.marine.rutgers.edu/thredds/c ... story_Best
Just click the Godiva2 link at the bottom and choose the variable you want to analyze from a dataset of ROMS output history files.

Check this link on how to install and configure TDS on your server:
https://www.unidata.ucar.edu/software/t ... reference/

Note: TDS v5 will use the new ncWMS2/Godiva3, but current versions (v4.6.14) still use the ncWMS/Godiva2.

[stef]

Wow, thanks for this! Looks great! I like the zooming with the mouse wheel and the panning.

I'm not sure if I need a time axis with calendar units, but maybe I can tweak it. I'll read the docs! Thanks!

[jpringle]

A much less elegant solution, but one I use all the time, is XPRA -- it is a way to view X windows remotely that is MUCH MORE robust than forwarding a remote X11 connection via SSH. Essentially, it sets up a local X11 server, and then serves the windows that are created on that server via a screen sharing mechanism (as VNC does) but on a per window basis. It is also MUCH faster.

Essentially, the windows on your remote machine show up on your local machine. It is robust to moving to another machine and viewing the same windows, or moving to another network (e.g. going to work at a coffee shop) or loosing a network connection (because coffee shop wifi is lousy).

It is like "screen" but for X windows. Clients exist for all OS's. My whole lab uses it and have found it makes things much easier.

https://xpra.org/

Jamie

[stef]

Thanks, that sounds perfect! I have never heard about this!

This way I don't have to set up the server, plus it occurred to me I would have needed to set up a VPN too. Much simpler!