Install Watershed Workflow
Published:
Install watershed-workflow
This post follows Ethan’s documentation with few edits. It has been tested on Mac and Linux (i.e. NERSC).
Docker installation
Watershed-workflow is now provided in a docker container.
- Install docker desktop app
- (optional) Download docker image:
docker pull ecoon/watershed_workflow-ats:latest
# or
docker pull ecoon/watershed_workflow:latest
- Run docker
# cd into watershed-workflow repo
cd ~/github/watershed-workflow
# use the following for full capability of WW
python bin/run_ww_lab.py -p=9999 --pull --ats --data-library=/path/to/data_library /path/to/working/directory
# for example:
python bin/run_ww_lab.py -p=9999 --pull --ats --data-library=/Users/shuai/github/watershed-workflow/data_library /Users/shuai/github/DRB
# or
docker run -it --rm -p 8899:8888 --pull always -e JUPYTER_ENABLE_LAB=yes -v /Users/shuai/github/watershed-workflow/data_library:/home/jovyan/data:delegated -v /Users/shuai/github:/home/jovyan/workdir:delegated ecoon/watershed_workflow-ats:master
Follow the terminal prompt and open the browser using url begin with
http://127.0.0.1:9999/lab?token=xxxx. note: change port number from 8888 (default) to the one entered in the command line (e.g., 9999)!Open one of the meshing notebook, change kernel to
Python3 (watershed_workflow)
Issues:
- Jupyter lab won’t open in the browser.
- The default port number is
8888may be taken. Docker won’t give any warning, but the user need to mannually change the port from8888to any other numbers (e.g.,8899). Use thedocker runcommand instead ofpython bin/run_ww_lab.py. Then in the browser, openhttp://127.0.0.1:8899/lab?token=xxxx
Install Python env and packages
- for general use
$ conda create -n watershed_workflow -c conda-forge -c defaults python=3 ipython ipykernel jupyterlab nb_conda_kernels numpy matplotlib scipy meshpy fiona rasterio shapely cartopy pyepsg descartes pyproj requests sortedcontainers attrs pytest pandas geopandas netcdf4 h5py tqdm libarchive # added a few more
$ conda activate watershed_workflow
Check python packages.
$ python -c 'import numpy, matplotlib, scipy, rasterio, fiona, shapely, cartopy, meshpy.triangle; print("SUCCESS")'
install Exodus II (on Mac)
install compiler (if applicable)
Install
cmake.- Download binary for Mac from here. Follow the prompt for installation.
- Add to path.
$ export PATH="/Applications/CMake.app/Contents/bin:$PATH"Install
gfortran- Download Homebrew
brew install gcc
Configure Exodus
- Clone
seacas
$ git clone https://github.com/gsjaardema/seacas.git
- Clone
watershed-workflow
$ git clone https://github.com/ecoon/watershed-workflow.git
- Edit
configure-seacas.sh
$ cd watershed-workflow/workflow_tpls/
$ vi configure-seacas.sh
# edit the following lines
CC=`which clang` # use `which gcc` for Linux
CXX=`which clang++` # use `which g++` for Linux
FC=`which gfortran`
# add/change the following path
CONDA_PREFIX=/opt/anaconda3/envs/watershed_workflow
SEACAS_SRC_DIR=/path/to/seacas/repo
- Configure
$ sh configure-seacas.sh
- Check if successful
$ export PYTHONPATH=${SEACAS_SRC_DIR}/install/lib
$ python -c 'import exodus; print("SUCCESS")'
If the installation is not successful (i.e., nothing inside the seacas/install), following the steps below to re-install.
change python version (optional)
- Manually write python version in
CMakeCache.txt
$ cd seacas/build
$ vi CMakeCache.txt
- search for
pythonand make the following changes
//Path to a program.
PYTHON_EXECUTABLE:FILEPATH=/opt/anaconda3/bin/python
//Default version of Python to find (must be 2.6 or greater
PythonInterp_FIND_VERSION:STRING=3.7
- install again
# assume inside build/
$ make install
Export paths
$ cd /path/to/watershed-workflow/repository
# add this and its subfolder `workflow_tpls` to `.bash_profile` to make it permanently
$ export PYTHONPATH=`pwd`:`pwd`/workflow_tpls:`pwd`/workflow:${PYTHONPATH}
# add the following for exodus to work
$ export PYTHONPATH="${PYTHONPATH}:/path/to/seacas/install/lib"
Install Exodus II (on NERSC)
Follow most of the steps above but with a few changes.
Configure modules
module load python
module load cmake
module swap PrgEnv-intel PrgEnv-gnu
export CRAYPE_LINK_TYPE=dynamic # important!
Loaded modules
Currently Loaded Modulefiles:
1) modules/3.2.11.4
2) altd/2.0
3) darshan/3.1.7
4) gcc/8.3.0
5) craype-haswell
6) craype-hugepages2M
7) craype-network-aries
8) craype/2.6.2
9) cray-mpich/7.7.10
10) cray-libsci/19.06.1
11) udreg/2.3.2-7.0.1.1_3.33__g8175d3d.ari
12) ugni/6.0.14.0-7.0.1.1_7.35__ge78e5b0.ari
13) pmi/5.0.14
14) dmapp/7.1.1-7.0.1.1_4.49__g38cf134.ari
15) gni-headers/5.0.12.0-7.0.1.1_6.29__g3b1768f.ari
16) xpmem/2.2.20-7.0.1.1_4.11__g0475745.ari
17) job/2.2.4-7.0.1.1_3.37__g36b56f4.ari
18) dvs/2.12_2.2.157-7.0.1.1_9.2__g083131db
19) alps/6.6.58-7.0.1.1_6.5__g437d88db.ari
20) rca/2.2.20-7.0.1.1_4.48__g8e3fb5b.ari
21) atp/2.1.3
22) PrgEnv-gnu/6.0.5
23) python/3.7-anaconda-2019.10
24) cmake/3.14.4
Configure Exodus
Change the following in watershed-workflow/workflow_tpls/configure-seacas.sh
CC=`which cc` # `which gcc` for Linux
CXX=`which CC` # `which g++` for Linux
FC=`which ftn`
CONDA_PREFIX=/path/to/.conda/envs/watershed
SEACAS_SRC_DIR=/path/to/seacas/repo
...
Change python version
May need to change python version from 2.7 to 3.7 or higher.
$ cd seacas/build
$ vi CMakeCache.txt
//Path to a program.
PYTHON_EXECUTABLE:FILEPATH=/usr/common/software/python/3.7-anaconda-2019.10/bin/python
//Default version of Python to find (must be 2.6 or greater
PythonInterp_FIND_VERSION:STRING=3.7 # change this!
then reinstall
make install
libexodus.so should be inside seacas/install/lib.
Common issues
- if
workflowmodule could not be imported, try adding the following to the notebook.
import os,sys
sys.path.append(os.path.abspath("/path/to/watershed-workflow"))
tinytreecould not be imported
(watershed_workflow) $ pip install tinytree
libexodus.sois not found
export CRAYPE_LINK_TYPE=dynamic
Then delete seacas repo and re-clone from GitHub. Do the configure again.