Running the StereoTopRGB Pipeline for Detecting Plant Area Data¶

This pipeline extracts plant area data from image files. This guide provides demo data you can use follow along with and ensure the pipeline is functional. Before starting, change to alpha branch with git checkout alpha.

Pipeline Overview¶

StereoTopRGB currently uses 7 different programs for the analytical pipeline:

Program	Function	Input	Output
bin2tif	Converts bin compressed files to geotiff	`image.bin`, `metadata.json`	`image.tif`
collect_gps	Collects GPS coordinates from all geotiff files	`image.tif`	`collected_coordinates.csv`
Orthomosaicing	Finds best possible coordinates of all geotiffs	`collected_coordinates.csv`	`corrected_coordinates.csv`
replace_gps	Applies corrected GPS coordinates to images	`corrected_coordinates.csv`, `image.tif`	`corrected_image.tif`
plotclip	Clips geotiffs to the plot	`corrected_image.tif`, `shapefile.geojson`	`plot.tif`
Plant detection	Detects plants over days	`plot.tif`	:`genotype.csv`
Plant clustering	Tracks plants over days	`genotype.csv`	:`pointmatching.csv`

Running the Pipeline¶

Note

At this point, we assume that the interactive “foreman” and “worker” nodes have already been setup and are running, and the pipelines have been cloned from GitHub. If this is not the case, start here.

Retrieve data¶

Navigate to your RGB directory, download the data from the CyVerse DataStore with iRODS commands and untar:

cd /<personal_folder>/PhytoOracle/StereoTopRGB
iget -rKVP /iplant/home/shared/phytooracle/season_10_lettuce_yr_2020/level_0/stereoTop/<stereoTop-date.tar>
tar -xvf <stereoTop-date.tar>

Note

For a full list of available unprocessed data navigate to https://datacommons.cyverse.org/browse/iplant/home/shared/phytooracle/season_10_lettuce_yr_2020/level_0/StereoTopRGB/

Retrieve vector and ML model files¶

Dowload the coordiate correction .csv file:

iget -N 0 -PVT /iplant/home/shared/phytooracle/season_10_lettuce_yr_2020/level_0/season10_multi_latlon_geno.geojson

iget -N 0 -PVT /iplant/home/shared/phytooracle/season_10_lettuce_yr_2020/level_0/necessary_files/gcp_season_10.txt

iget -N 0 -PVT /iplant/home/shared/phytooracle/season_10_lettuce_yr_2020/level_0/necessary_files/model_weights.pth

Edit scripts¶

process_one_set.sh, process_one_set2.sh

Find your current working directory using the command pwd. Open process_one_set.sh and paste the output from pwd into line 14 (line 12 in process_one_set2.sh). It should look something like this:
```
HPC_PATH="/xdisk/group_folder/personal_folder/PhytoOracle/StereoTopRGB/"
```
Set your .simg folder path in line 15 (line 13 in process_one_set2.sh).
```
SIMG_PATH="/xdisk/group_folder/personal_folder/PhytoOracle/singularity_images/"
```

run.sh

+Open run.sh and paste the output from pwd into line 7. It should look something like this:

PIPE_PATH="/xdisk/group_folder/personal_folder/PhytoOracle/StereoTopRGB/"

+Set your .simg folder path in line 8.

SIMG_PATH="/xdisk/group_folder/personal_folder/PhytoOracle/singularity_images/"

entrypoint.sh, entrypoint-2.sh

In lines 7 and 11, specify the location of CCTools:

/home/<u_num>/<username>/cctools-<version>-x86_64-centos7/bin/jx2json

and

/home/<u_num>/<username>/cctools-<version>-x86_64-centos7/bin/makeflow

Run pipeline¶

Begin processing using:

./run.sh <folder_to_process>

Note

This may return a notice with a “FATAL” error. This happens as the pipeline waits for a connection to DockerHub, which takes some time. Usually, the system will fail quickly if there is an issue.

If the pipeline fails, check to make sure you have a “/” concluding line 14 of process_one_set.sh. This is one of the most common errors and is necessary to connect the program scripts to the HPC.

Troubleshooting and Issues¶

If problems arise with this pipeline, please refer to the tutorial on GitHub specific to the RGB pileline. If problems persist, raise an issue.