Welcome to CLIMAGIC @ ILF 2012 With Your Host: Mark Krenz http://www.climagic.org/ There will be no QR codes in this presentation 1. What is climagic - Tip feed for bash - Since 2009, over 3200 command tips - Raise the bar for intermediate users, inspire beginners - Almost 12,000 followers (75% of which are probably no longer using twitter or identica) 2. Why the command line? + Solid, Fast and Trustworthy + With so much data in the world, more useful than ever with Internet data, lots of data mining, configuration, editing, etc. + Rewarding to spend the time to learn. + You don't have to feel lonely, most everybody uses it at least a little bit. 3. Why bash? Why did you do this this way? You used cat OMG! + Its the "Lingua Franca" of shells. + Favor tips on interactive use vs. script use. + TIMTOWTDI + cat is fine to use, ignore people who say otherwise. 4. First things to know. Where to get help. ls --help help cd (for bash builtins) man ls man -k foobar (must run makewhatis first) Greg's bash wiki http://mywiki.wooledge.org/BashGuide http://mywiki.wooledge.org/BashFAQ 5. Some basic ones that are good to know (part 1) cd and cd - du -sh */ (directories only globbing) tail -n+2 data.csv mv file $OLDPWD/ 6. Some basic ones that are good to know (part 2) (keystrokes) Ctrl-r Ctrl-a and Ctrl-e Ctrl-w Ctrl-u Esc-. SSH keystrokes: [RET]~C and [RET]~. 7. Some basic ones that are good to know (part 3) join -o 1.1,2.2 -1 1 -2 1 -t: passwd shadow apt-cache search " nmap -sS 192.168.1.100 curl -s whatismyip.org lsof +D /tmp lsof -i TCP:80 renice -p 2235 ab -n 100 -c 20 http://www.foxnews.com/ * Job control Ctrl-z jobs fg bg kill %1 dosometing & 9. Variables 1 MYWORD="aimlessness" export MYWORD (export means make the variable available to subshells, programs, etc) TZ="Asia/Tokyo" date (one time export to command) LANG=C sort env & set 10. Variables 2 ALPHABET=$(echo {a..z} | tr -d ' ') echo ${ALPHABET:0:3} echo ${ALPHABET^^} echo ${ALPHABET//abc/Easy as abc 123 } 11. Variables 3 (Bash builtin variables) echo $(($RANDOM % 100)) echo $SHLVL echo $COLUMNS $LINES 12. Aliases alias ls='ls -l' alias dir-foobar='cd /path/to/deep/dir/local/src/foobar' alias whereami='echo "$( hostname --fqdn ) ($(hostname -i)):$( pwd )"' alias topmemproc='ps aux|head -1;ps aux |tail -n+2 |sort -nrk4 \ |head -$(($(tput lines)-2)) |cut -c 1-$(tput cols)' 13. Functions 1 function todo(){ ${EDITOR:-/usr/bin/vim} ~/$(date +todolist-%Y%m%d); } function cd(){ if [[ "$1" == "..." ]]; then builtin cd ../..; else builtin \ cd $@; fi; } 14. Functions 2 sus(){ sort | uniq -c | sort $@; } function acronym(){ elinks -no-numbering -dump \ http://acronymfinder.com/$1.html|sed '/*/!d';} function box() { t="$1xxxx";c=${2:-=}; echo ${t//?/$c}; \ echo "$c $1 $c"; echo ${t//?/$c}; } 15. Functions 3 function ls() { /bin/ls $@ | /usr/bin/tac ; } # April Fools joke. 16. Brace Expansion convert photo.{jpg,png} -> convert photo.jpg photo.png tar zcvf sourcecode{.tar.gz,} -> tar zcvf sourcecode.tar.gz sourcecode echo {0..10} -> 0 1 2 3 4 5 6 7 8 9 10 echo a{0..5} -> a0 a1 a2 a3 a4 a5 # Bash only echo {a..z} -> a b c d e f g h i j k l m n o p q r s t u v w x y z 17. Redirection ls -l > lsoutput.txt ls -l filethatdoesntexit 2> out.err 18. Process substitution comm -12 <(ls -1 dir1 | sort) <(ls -1 dir2 | sort) 19. awk and sed, the 2 3(4) letter words of the Unix language awk '{print $1}' access_log awk -F, '{print $2 " " $3}' data.csv awk '$7~/\.png/ {array[$7]+=$10} END {for (i in array) \ { print array[i] " " i; } }' access_log | sort -nr 20. awk and sed, the 2 3(4) letter words of the Unix language sed -n '29p' mycode.pl sed -s -i.bak 's/Copyright 2010/Copyright 2011/' *.html sed -n '12 {p; Q}' file.txt 21. awk and sed, the 2 3(4) letter words of the Unix language ps aux | awk '$8=="Z"' | wc -l elinks -dump http://bit.ly/qutWZK|sed -n -e 's/,//g' \ -e '/^1960/,/^2010/p' | awk '{p=$6/$2*100; print $1 " " p}' 22. Pipes 1 ls -l | less mount | column -t du -sh */ | sort -rh 23. Pipes 2 awk {'print $1'} access_log | sort | uniq -c | sort -nr 24. Pipes 3 elinks http://www.$( look . | shuf | head -n 1 ).com 25. Pipes 4 elinks enwp.org/$(date +%B_%d) | sed -n -e 's/\[[0-9]\+\]//g' \ -e '/^\[.* Ev/,/^\[/p' | tr -d '\n' | tr '*' '\n' | \ grep -v '\[ed' | sort -R | head -1 26. Loops (part 1) for i in 1 2 3 ; do echo $i ; done for i in *.jpg ; do convert $i ${i%jpg}.png ; done 27. Loops (part 2) while true ; do uptime ; sleep 5 ; done while :;do iwlist wlan0 scan |awk -F\" '/ESSID/{print $2}' |espeak;done 28. Loops (part 3) # Figure out which years Nov 21st was on a Friday. for y in {1970..2010} ; do [[ $( date -d $y-11-21 +%u ) -eq 5 ]] \ && echo $y ; done 29. Loops (part 4) for grass in field* ; do mow $grass; done | grep -v weeds \ | while read -r hay; do bale $hay; done 30. xargs find . -name .htaccess | xargs egrep -l "(deny|allow)" look . | egrep "^[a-z]{1,4}$" |xargs -P 40 -I{} \ curl -L -sI -o www.{}.com http://www.{}.com xargs < file.txt 31. Putting it all together # Network ping scan. echo 192.168.0.{1..254}| xargs -n1 -d' ' -P254 -I{} \ ping -w1 -c1 {} 2>/dev/null| awk '/ttl/{print $4}' # Could use nmap or other tool for this as well, but if you # don't have root access or are in a pinch. 32. Useful commands 1 tmux screen script -t ~/myshell.script 2>~/myshell.time scriptreplay myshell.time myshell.script 33. Useful commands 2 netcat and socat zcat, zless, zgrep and their cousins bzless, xzless, etc. 34. Useful commands 3 sponge tee pee grep 67.205.74.88 messages |sponge |pee "head -1" "tail -1" 35. Useful commands 4 ngrep -d eth0 -i 'select' port 3306 36. CLIMagic Survey highlights + 63 countries representing 44 different languages + 26 different distributions of Linux in use. Ubuntu, Debian, Arch, Fedora most popular. + Shell breakdown Shell Num Percent -------------------------- bash 791 76.5% zsh 190 18.4% tcsh 13 1.3% ksh 10 1.0% sh 6 0.6% dash 6 0.6% NA 5 0.5% csh 4 0.4% powershell 3 0.3% ash 3 0.3% fish 2 0.2% pdksh 1 0.1% 37. Thank you THANK YOU FOR ATTENDING See the libvte security flaw presentation later today at 2:30pm