Running the FlirIr Pipeline for Infrared Data¶

This pipeline extracts temperature data from image files. This guide provides demo data you can use follow along with and ensure the pipeline is functional. Before starting, change to master branch with git checkout master.

Pipeline Overview¶

FlirIr currently uses 7 different programs for data conversion:

Program	Function	Input	Output
flir2tif	Temperature calibrated transformer that converts bin compressed files to tif	`image.bin`, `metadata.json`	`image.tif`
collect_gps	Collects GPS coordinates from all geotiff files	`image.tif`	`collected_coordinates.csv`
Orthomosaicing	Finds best possible coordinates of all geotiffs	`collected_coordinates.csv`	`corrected_coordinates.csv`
replace_gps	Applies corrected GPS coordinates to images	`corrected_coordinates.csv`, `image.tif`	`corrected_image.tif`
flir_field_stitch	GDAL based transformer that combines all immages into a single orthomosaic	Directory of all converted `image.tif`	`ortho.tif`
plotclip	Clips plots from orthomosaic	`coordinatefile.geojson`, `ortho.tif`	`clipped_plots.tif`
flir_meantemp	Extracts temperature using from detected biomass	`coordinatefile.geojson`, Directory of all `clipped_plots.tif`	`meantemp.csv`

Running the Pipeline¶

Note

At this point, we assume that the interactive “foreman” and “worker” nodes have already been setup and are running, and the pipelines have been cloned from GitHub. If this is not the case, start here.

Retrieve data¶

Navigate to your directory containing FlirIr, and download the data from the CyVerse DataStore with iRODS commands and untar:

cd /<personal_folder>/PhytoOracle/FlirIr
iget -rKVP /iplant/home/shared/terraref/ua-mac/raw_tars/demo_data/Lettuce/FlirIr_demo.tar
tar -xvf FlirIr_demo.tar

Data from the Gantry can be found within /iplant/home/shared/terraref/ua-mac/raw_tars/season_10_yr_2020/flirIrCamera/<scan_date>.tar

Edit scripts¶

process_one_set.sh

Find your current working directory using the command pwd Open process_one_set.sh and copy the output from pwd into line 14. It should look something like this:
```
HPC_PATH="xdisk/group_folder/personal_folder/PhytoOracle/FlirIr/"
```
Set your .simg folder path in line 8.
```
SIMG_PATH="/xdisk/group_folder/personal_folder/PhytoOracle/singularity_images/"
```

run.sh

Paste the output from pwd into line 7. It should look something like this:
```
PIPE_PATH="/xdisk/group_folder/personal_folder/PhytoOracle/FlirIr/"
```

Set your .simg folder path in line 8.

SIMG_PATH="/xdisk/group_folder/personal_folder/PhytoOracle/singularity_images/"

In line 4, specify the <scan_date> folder you want to process. For our purposes, this will look like:
```
DATE="FlirIr_demo"
```

In lines 25 and 28, specify the location of CCTools:

/home/<u_num>/<username>/cctools-<version>-x86_64-centos7/bin/jx2json

and

/home/<u_num>/<username>/cctools-<version>-x86_64-centos7/bin/makeflow

Run pipeline¶

Begin processing using:

./run.sh

Note

This may return a notice with a “FATAL” error. This happens as the pipeline waits for a connection to DockerHub, which takes some time. Usually, the system will fail quickly if there is an issue.

If the pipeline fails, check to make sure you have a “/” concluding line 14 of process_one_set.sh. This is one of the most common errors and is necessary to connect the program scripts to the HPC.