Begin with shell scripting

Deafult header for shell 


#!/bin/bash 
echo "begin"
 code here
or with options
-x show execution of file
-v echo lines as they are read 

#!/bin/bash  -v -x
echo "begin"
 code here

Scripts with .sh ending will run automatically in shell on execution

Rights to execute

Remmember to add execution rights to yourself with chmod LINK , LINK
numbers: (user)(usergroup)(others) , read + write + execute == 4 + 2 + 1
777 - everyone 755 - user:everything, others:read+execute 

chmod +777 script.sh
Owner Group Other
Read
Write
Execute
Permissions:


bash

yearloop formating str array 


months=`seq 1 12` 
for yy in `seq 1999 1999`; do 
  for mm in `printf '%02d ' $months`; do 
    .... 
  done
done 


  for mm in {01..12}; do 
    .... 
  done

case syntax



case $casestring in
"gor") doms="101 102 103 92 93 94 81 82 83 71" ;;
"gof") doms="116 117 118 119 120 121 122 127 128 129 130 131 132 133 134 135 141 142 143 144 145 146 152 153";;

*) echo " this is else or default case";; 
esac

other useful snipets 

 
   export yystart=`printf %04d $(($yy))`

   export mmstart=$(printf %02d  `expr $mm '+' 0` )
   export step_2d=`expr $do '*' 86400 '/' $timestep`  # daily
   export yyhot=$(($yy-1))

list="item1,item2,item3, item4 , ..."
nlist=$(echo $list | tr "," " ")

# tic-toc   how much time have passed ?
   export tic=`date +%s`
     ...
   echo "> time  `date +" %H:%M:%S %m-%d-%Y"` ;  time taken" $(( `date +%s` - $tic))

Exchange some tags with variables from _conf file


calcbio=.true.
   cat gotm_fabm_conf.nml		\
	 | sed -e "s:_calcbio_:$calcbio:"  \
    > gotm_fabm.nml

Filter, select columns from text file

source

while read _ _ one _ two _; do
    echo "$one $two" 
done < file.txt

awk '{ print $2 $4 }' filename.txt cat filename.txt | awk '{ print $2 $4 }'

Rounding


round()
{
echo $(printf %.$2f $(echo "scale=$2;(((10^$2)*$1)+0.5)/(10^$2)" | bc))
};
# Later
timestep=1.3
domean=0.125
export meanout=$(round `echo print "$domean * 86400 / $timestep"| python` 0);

Generating date


  dte="$yy-$mm-$dd 00:00:00"                      # input date
  sdte=`date --date "$dte 1 day" +"%Y-%m-%d"`     # new date
  yystop=`date +'%Y' -d$sdte`                     #  formating
  mmstop=`date +'%m' -d$sdte`
  ddstop=`date +'%d' -d$sdte`

IF file exist / not exist


$targ=path_to_file
if [ -f $targ ] ;then
  echo "file exists"
fi

if [ ! -f $targ ] ;then
  echo "file does not exist"
fi

IF time is right

Check if we are runing specific year and month 
if [ $yy == 2008 ] && [ $mm == 04 ]; then
echo '     doing somethign specific for this month'
fi

Check if the date is before reference date or not 
 
  start_date_time="2020-11-11 00:00:00" 
  test_date_time ="2020-11-15 00:00:00"             
  start_s=`date --date "$start_date_time" +"%s"`    
  test_s=`date --date "$test_date_time" +"%s"`    
if [ "$start_s" -le "$test_s" ];then  opm=1; else opm=0 ; fi
echo '    opm:'$opm

Grep multiple keywords

Multiple patterns 

grep 'extra\|value' sample.txt

Multiple exact matches 
 
grep -w 'provide\|count' sample.txt
More grep uses 1

NetCDF

Usefull netcdf tools and commands

cdo


CDO reference card (v1.7)
CDO reference card (v1.0) condense
tutorial, cdo wiki
cdo presentation (MPI Met)

convert file (usually grib) to netcdf4 format

> cdo -f nc4 copy $ifile $ofile

select variable (s)

> cdo selvar,var1,var2,var3 $ifile $ofile
> cdo selvar,$varnames $ifile $ofile

extract single point

> x2=$x1;y2=$y1; cdo selindexbox,$x1,$x2,$y1,$y2 $ifile $ofile
> cdo remapnn,lon=$lon/lat=$lat $ifile $ofile


express a formula

> cdo expr,'z=a*x+b*y' $ifile $ofile


Squeeze singelton dimensions using cdo

> cdo --reduce_dim $ifile $ofile

ncks


ncks reference card

Append files, source

> ncks -A appendable.nc root_file.nc

Squeeze singelton dimensions

> ncwa -O -a dim1,dim2,dimN $ifile $ofile

rename dimensions

> ncrename -O -d dimname_old,dimname_new $ifile $ofile

Edit attributes, source

> ncatted -O -a attribute,variable,stufftodo,type,valueofatribute $ifile $ofile
for stufftodo: a=append, c=create, d=delete, m=modify, o=overwrite
for type: f=float, d=double, l=long, s=short, c=character, b=byte, i=integer

e: ncatted -O -a units,air,c,c,"units goes here" filename.nc
e: ncatted -O -a long_name,air,c,c,"long_name goes here" filename.nc
e: ncatted -O -h -a title,global,o,c,"title goes here" filename.nc
e: ncatted -O -h -a history,global,o,c,'history goes here' filename.nc

note to my self: to back-rot. with cdo meteogrid nc-file must contain x,y variables, variable "rotated_pole" and variable should have attribute V_10M:grid_mapping = "rotated_pole" (issue with community climate model)

flip dimensions

>ncpdq -O -h -a -zax STf.2011.12.nc STff.2011.12.nc

Rename variable

> ncrename -O -d dimname_old,dimname_new $ifile $ofile

edit variables


ncap2 in-line scripting > ncap2 -O -s 'level(:)= {-0,-1,-2,-3,-4,-5,-6,-7,-8,-9,-10}' $ifile $ofile
> ncap2 -O -s 'level(2)= {6}' $ifile $ofile
level = 0, 1, 6, 3, 4, 5, 6, 7, 8, 9, 10 ;



Other NetCDF snipets (fast examples)

Extratction

using specific fortran () For horizontal domains:
> ncks -O -H -d lat,$lat -d lon,$lon $ifile $ofile
> cdo -O -L selindexbox,$x1,$x1,$y1,$y1 $ifile $ofile
For vertical domains:
sellevel
For timeseries (nearest gridpoint)
> cdo -L -O remapnn,lon=$lon/lat=$lat $ifile $ofile

Interpolation

Horizontal
Into new grid, defined with gridfile
> cdo remap(-bil, -bic, -con, -dis),gridfile $ifile $ofile
Into defined coordinate
(?) > cdo -L -O remapbil,lon=$lon/lat=$lat $ifile $ofile Temporal
eg.
Single timestep
> cdo
Into (new) time axis
> cdo -L inttime,2017-07-01,00:00:00,1hour -seldate,2017-07-01,2017-07-02 $ifile $ofile
Vertical
> cdo
> gvc2zax -z 50.0,1.2,40 -v temp,salt $ifile $ofile
See gvc2zax getm-utils sourceforge

Long timeseries

Error "Core dumped" might appear when you are trying to apply cdo operator on netcdf file with long timeseries (nt>500). Adding -L tag to options solves the error and produces valid output. > cdo -L operator input output

Large files

When dealing with files > 4GB followinf error can occur:
nco_err_exit(): ERROR Error code is -101. Translation into English with nc_strerror(-101) is "NetCDF: HDF error"
while using ncrename.
Increasing buffer size solves the error
> ncrename -O --bfr_sz=2097152 -d longitude,lon -d latitude,lat -v longitude,lon -v latitude,lat $infile $outfile




NetCDF -> moving images

Movie (fast, robust)

Open the netcdf file with ncview flag -frame
> ncview -frame ncfile.nc
Each plot that you view will be saved into current directory as frame****.png
Following merge frames with imagemagic convert tool.
> convert -delay [integer of ms] -loop 0 -scale [integer]% frame* out.gif
or into video file.
> avconv -i -r 24 "frame%04d.png" out.mpeg
Be careful, because ncview creates new frame with each opened plot.
> rm frame*

Annotate your movie with textbox:
> varn=salt
> fdraw="drawtext=fontfile='/usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf':text='$varn':x=$xt1:y=$yt1:fontsize=$fs:fontcolor=black"
> avconv -y -r $fr -i "$varn/frame.%05d.png" -vf $fdraw -r $fr x_$varn.avi

Video grid

Join multiple videos into one single video wit ffmpeg.
> ffmpeg
-i x_salt.avi -i x_temp.avi -i x_no3.avi -i x_po4.avi -i x_lpp.avi -i x_cya.avi / # (1)
-filter_complex "[0:v][1:v]hstack[t];[2:v][3:v]hstack[c];[4:v][5:v]hstack[b];[t][c]vstack[tc];[tc][b]vstack[v]" / # (2)
-map "[v]" -c:a copy -shortest x_output.mp4 / # (3)
# (1) input videos
# (2) sequence of vertical and horizontal (pairwize) stacking of input videos
horizontal stacking: videos 0,1 -> t; videos 2,3 -> c; videos 4,5 -> b; vertical stacking t,c -> tc; tc, b -> v;
# (3) formating output, codecs, etc...

gif

Use ImageMagick to convert your frames to gif-images
> convert -delay 10 -colors 16 -quality 50 -resize 60% frame.00* po.gif
Adjust quality parameters for your needs.


pdftk -> magicwand for pdf files


: PDFLABS
>
>

Split that pdf file



> pdftk full-pdf.pdf cat 12-15 output outfile_p12-15.pdf

Rotate that pdf file

cat to invoke pages or ranges -> rotating operator -> output
> pdftk in.pdf cat 1east output out.pdf
> pdftk in.pdf cat 1-endeast output out.pdf
NOTE: use E instead east for old version

> pdftk input_file.pdf cat 1-18 19D 20-end output output_file.pdf
input_file.pdf: The path of the PDF file you want to modify.
cat 1-18 19D 20-end: Create a new PDF using these parts from the input file: Pages from 1 to 18, let them as they are. Rotate the 19th page 180º. Pages from 20 to the end, let them as they are.
output_file.pdf: The path of the new PDF that will be created. The rotation degrees are:
D: 180º
L: 90º to the left.
R: 90º to the right.


Reduce size/quality


> gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

-dPDFSETTINGS=
/screen lower quality, smaller size. (72 dpi)
/ebook for better quality, but slightly larger pdfs. (150 dpi)
/prepress output similar to Acrobat Distiller "Prepress Optimized" setting (300 dpi)
/printer selects output similar to the Acrobat Distiller "Print Optimized" setting (300 dpi)
/default selects output intended to be useful across a wide variety of uses, possibly at the expense of a larger output file
from Ubuntu forums
: VectorDevices.htm#PSPDF_IN

ImageMagic


webpage

Convert png files to tiff format

> convert input.png output.tiff
! easy as that ...seriously :)

Inkscape

Use single template file to burst multiple images
Use linking when importing subplots into inkscape template file. 

while read line; do                  # read each line from stationlist.txt
  stat=`echo $line| cut -d" " -f 1`  # extract string from a first row of line
  cp template_xml.svg   tmp.svg      # preserve template.svg
  ofile=$stat.multiplot.png          # name output file
  f="stat1"                       # phrase to lookup
  r="stat$stat"                   # new phrase (unique to new picture)
  sed -i "s/$f/$r/g" tmp.svg      #  replace
  inkscape  -f tmp.svg -D -d 350 -e pics/$ofile    # produce new picture
done < stationlist.txt

Video editing

Cut a clip out from a movie 


 avconv -ss 00:04:52 -i "bigfile.mp4" -t 00:05:55 -codec copy smallfile.mp4

use free prof. video editor
lightworks
It takes ~ 1 evening to acomplish decent results
Use youtube & forums for faster learning

Once you are using Linux ...

Copy from one computer to other (shell)



scp 

scp $ifile username@adress:$odir
scp -r $idir username@adress:$odir            # copy  directory
scp -i rsa_key $ifile username@adress:$odir   # copy using teh key
rsync 

rsync -r -a -v -e "ssh -l username" $idir/files_* username@adress:$odir  # recrussive, verbose, etc..


When you have done something terribly bad



> ssh -l $USER $server 'pkill -9 -u $USER'

snipet from my .bashrc 



alias mnt_RemoteDir='sshfs remoteusername@IP:RemoteDir LocalDir'
alias unmnt_RemoteDir='fusermount -uz LocalDir'
alias go='gnome-open .'
alias matlab='/home/ilja/Matlab/bin/matlab'
alias mytop='top c u ilja'
alias mysqueue=' squeue --user=ilja --format="%.22i %.20j %.8u %.2t %.10M %.6D %R"'
LS_COLORS='di=1;94:fi=0:ln=31:pi=5:so=5:bd=5:cd=5:or=31:mi=0:ex=92:*.rpm=90:*.nc=95:*.png=01;35:*.gz=01;31'
export $LS_COLORS