# New Page ## Exit-on-error mode Put this at the top of your bash script: `set -e`. If a command returns a nonzero status, the shell will exit. **Resources:** [https://unix.stackexchange.com/questions/97101/how-to-catch-an-error-in-a-linux-bash-script](https://unix.stackexchange.com/questions/97101/how-to-catch-an-error-in-a-linux-bash-script) [http://linuxcommand.org/lc3\_man\_pages/seth.html](http://linuxcommand.org/lc3_man_pages/seth.html) - man page ## See files that would be unzipped without unzipping ``` unzip -v $FILE_NAME.zip ``` ## Unzip to a directory ``` unzip package.zip -d /opt ``` **Resource:** [https://www.cyberciti.biz/faq/linux-howto-unzip-files-in-root-directory/](https://www.cyberciti.biz/faq/linux-howto-unzip-files-in-root-directory/) ## Color output in less ``` FILE=bla.txt less -r $FILE ``` ## YAML templating `template.yml`: ``` --- bobs: password: "${bobs_password}" username: "${bobs_username}" machines: "${bobs_machines}" ``` Fill in the template and create `final.yml`: ``` export sample="Hello Yaml!" export bobs_machines="10.2.3.4" export bobs_username="nats" export bobs_password="password" rm -f final.yml temp.yml ( echo "cat <final.yml"; cat template.yml; echo "EOF"; ) >temp.yml . temp.yml cat final.yml ``` **Resource:** [https://www.starkandwayne.com/blog/bashing-your-yaml/](https://www.starkandwayne.com/blog/bashing-your-yaml/) ## Lint YAML ``` python3 -c 'import yaml, sys; yaml.safe_load(sys.stdin)' < file.yml ``` ## Lint JSON ``` cat file.json | jq ``` ## Code Style Function declarations: ``` my_func() { } ``` Variable declaration: ``` cool_var='great' ``` Source filenames should be lowercase with underscores to separate words, i.e. `the_best_script.sh` **Resources:** [https://google.github.io/styleguide/shell.xml##Variable\_Names](https://google.github.io/styleguide/shell.xml##Variable_Names) [https://dev.to/puritanic/shellscripting-functions-2696](https://dev.to/puritanic/shellscripting-functions-2696) [https://stackoverflow.com/questions/673055/correct-bash-and-shell-script-variable-capitalization](https://stackoverflow.com/questions/673055/correct-bash-and-shell-script-variable-capitalization) --- ## $HOME vs ~ in scripts Opt to use `$HOME` as `~` is expanded by the terminal into the `$HOME` variable. Subsequently `$HOME` is going to be a more robust option. **Resource:** [https://ryankubik.com/blog/tilde-vs-home](https://ryankubik.com/blog/tilde-vs-home) ## Global variables in functions If you want to set something as a global variable and use it in a function, do the following: ``` export SOMETHING="variable" great_function() { echo $SOMETHING } ``` ### Parse an env file `export $(egrep -v '^##' .env | xargs)` The env file you want to parse should look something like this: ``` FOO='bar' USERNAME='user' ``` **Resource:** [https://gist.github.com/judy2k/7656bfe3b322d669ef75364a46327836](https://gist.github.com/judy2k/7656bfe3b322d669ef75364a46327836) --- ## Send message to all users ``` echo "hi" | wall ``` ## Move file with rsync ``` rsync --partial --progress filename.txt user@ipaddress_or_hostname:~ ``` ## Use SSH and rsync together ``` rsync -azvhe ssh --progress local_dir_to_copy user@192.168.1.2:remote_copy_of_dir ``` Remote to local: ``` rsync -azvhe ssh --progress user@192.168.1.2:~/dir_to_copy_to_local local_copy_of_remote_dir ``` **Resource:** [https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/](https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/) --- ## Sync two folders Start with a dry run to ensure that it’s doing what you expect it to do: ``` rsync -rv --append-verify --dry-run folder_to_sync_from/ /home/ubuntu/folder_to_sync_to ``` Once you’ve confirmed it’s working as expected, simply remove the `--dry-run` to execute. **Resource:** [https://unix.stackexchange.com/questions/487646/copy-changed-new-files-only-to-a-different-directory](https://unix.stackexchange.com/questions/487646/copy-changed-new-files-only-to-a-different-directory) ## Transfer multiple remote directories This worked well for my MacOS systems: ``` IP=192.168.1.56 DIRS=(/Users/user/dir1 /Users/user/dir2 /Users/user/.dotfiles /Users/user/dir3) for DIR in "${DIRS[@]}"; do echo "Transferring remote directory ${DIR}, please wait." rsync -avzh --progress "user@${IP}:${DIR}" . echo "Transfer of remote directory ${DIR} complete!" done ``` **Resource:** [https://unix.stackexchange.com/questions/575156/using-rsync-to-back-up-home-folder-with-same-permissions-recursive](https://unix.stackexchange.com/questions/575156/using-rsync-to-back-up-home-folder-with-same-permissions-recursive) --- ## Break up VM into multiple 1 GB files Need to transfer a VM? Try this. ``` split -b 1000m vm.ova vm.ova.split ``` ### Bring it back together ``` cat vm.ova.splita* > vm.ova ``` --- ## Kill netcat after 3 seconds ``` timeout 3 nc google.com 80; echo exit=$? ``` **Resource:** [https://unix.stackexchange.com/questions/96817/how-do-i-get-the-nc-command-to-end-after-2-seconds](https://unix.stackexchange.com/questions/96817/how-do-i-get-the-nc-command-to-end-after-2-seconds) --- ## Test connectivity to service **Netcat:** ``` HOSTNAME_OR_IP=system.mydomain.com PORT=5432 # -z: report connectivity status only # -w1: wait one second and then give up nc -z -w1 $HOSTNAME_OR_IP $PORT ``` **Telnet:** ``` HOSTNAME_OR_IP=system.mydomain.com PORT=5432 # -e X: set escape character to X echo X | telnet -e X $HOSTNAME_OR_IP $PORT ``` **Resources:** - [https://webhostinggeeks.com/howto/how-to-check-the-connection-to-mysql-db-server/](https://webhostinggeeks.com/howto/how-to-check-the-connection-to-mysql-db-server/) - [https://www.redhat.com/sysadmin/telnet-netcat-troubleshooting](https://www.redhat.com/sysadmin/telnet-netcat-troubleshooting) - [https://www.netsparker.com/blog/web-security/sql-injection-cheat-sheet/](https://www.netsparker.com/blog/web-security/sql-injection-cheat-sheet/) --- ## Determine if string exists in a file ``` if grep -q SomeString "$FILE"; then # SomeString was found fi ``` **Resource:** [https://stackoverflow.com/questions/11287861/how-to-check-if-a-file-contains-a-specific-string-using-bash](https://stackoverflow.com/questions/11287861/how-to-check-if-a-file-contains-a-specific-string-using-bash) --- ## Delete files if they match a wildcard If you have two files, `someFile` and `someFile2`, this will delete both of them: ``` if ls someFile* 1> /dev/null 2>&1; then rm someFile* fi ``` **Resource:** [https://stackoverflow.com/questions/12375722/how-do-i-test-in-one-line-if-command-output-contains-a-certain-string](https://stackoverflow.com/questions/12375722/how-do-i-test-in-one-line-if-command-output-contains-a-certain-string) --- ## Check if multiple binaries are installed ``` if command -v cowsay 2>/dev/null && ! command -v gshuf 2>/dev/null; then # do things if cowsay is installed but gshuf isn't fi ``` --- ## Check if file is missing ``` if [[ ! -f "some/file.txt" ]]; then # do things if the file is not there fi ``` ### One-liner to check if file exists ``` if [ ! -f /tmp/foo.txt ]; then echo "File not found!"; else echo "file found"; fi ``` **Resource:** [https://unix.stackexchange.com/questions/474244/one-liner-to-check-for-file-exists](https://unix.stackexchange.com/questions/474244/one-liner-to-check-for-file-exists) ## Create missing directory ``` [[ -d dir ]] || mkdir dir ``` **Resource:** [https://stackoverflow.com/questions/18622907/only-mkdir-if-it-does-not-exist](https://stackoverflow.com/questions/18622907/only-mkdir-if-it-does-not-exist) --- ## Number of duplicate lines in a file[\#](https://wiki.mutschlerhome.com/en/CheatSheets/Bash#number-of-duplicate-lines-in-a-file) ``` FILE='/etc/passwd' sort "${FILE}" | uniq -c ``` **Resource:** [https://stackoverflow.com/questions/6712437/find-duplicate-lines-in-a-file-and-count-how-many-time-each-line-was-duplicated](https://stackoverflow.com/questions/6712437/find-duplicate-lines-in-a-file-and-count-how-many-time-each-line-was-duplicated) --- ## Run command in another directory as another user This particular example is used to run a command as root in another directory ``` sudo -u root /bin/bash -c 'pushd "$HOME/.some_directory/"; ls -lart; popd' ``` **Resource:** [https://stackoverflow.com/questions/10382141/temporarily-change-current-working-directory-in-bash-to-run-a-command](https://stackoverflow.com/questions/10382141/temporarily-change-current-working-directory-in-bash-to-run-a-command) ## Bypass post-install configurations If you have to install things via apt and run into blue screens that require human interaction, do this to avoid it: ``` sudo DEBIAN_FRONTEND=noninteractive apt install -yq [packagename] ``` **Resource:** [https://serverfault.com/questions/227190/how-do-i-ask-apt-get-to-skip-any-interactive-post-install-configuration-steps](https://serverfault.com/questions/227190/how-do-i-ask-apt-get-to-skip-any-interactive-post-install-configuration-steps) ## Get count of and list of most frequently used bash commands ``` history | awk '{a[$2]++;next}END{for (i in a){print i " --> " a[i]}}' \ | sort -nr -k3 ``` ## Reinstall a package in debian-based system ``` sudo apt install --reinstall -y ``` ## Unable to fetch some archives, maybe run apt-get update or try with –fix-missing Run this if `apt update --fix-missing` isn’t working for you: ``` rm -rf /var/lib/apt/lists/*; apt update ``` Alternatively, you can also try: ``` apt clean; apt update ``` **Resource:** [https://stackoverflow.com/questions/38743951/unable-to-fetch-some-archives-maybe-run-apt-get-update-or-try-with-fix-missin](https://stackoverflow.com/questions/38743951/unable-to-fetch-some-archives-maybe-run-apt-get-update-or-try-with-fix-missin) ## Save off auth logs for review Useful to see who has logged into your system and when ``` last -F > /tmp/last ``` [https://serverfault.com/questions/375091/getting-login-year-data-with-the-last-command-on-linux](https://serverfault.com/questions/375091/getting-login-year-data-with-the-last-command-on-linux) ## “Incognito mode” for bash Turn off history for a session. ``` export HISTFILE= ``` If you do want a record of your activity in another file, run this command instead: ``` export HISTFILE=0 ``` This will create a file, 0, with your history for that session. **Resources:** [https://github.com/mubix/post-exploitation/wiki/Linux-Post-Exploitation-Command-List](https://github.com/mubix/post-exploitation/wiki/Linux-Post-Exploitation-Command-List) [https://www.maketecheasier.com/linux-command-line-history-incognito/](https://www.maketecheasier.com/linux-command-line-history-incognito/) ## List NFS mounts ``` mount |grep nfs ``` ## When all users last logged in ``` lastlog ``` ## All previously logged in users ``` lastlog |grep -v "Never" ``` **Resource:** [https://www.rebootuser.com/?p=1623](https://www.rebootuser.com/?p=1623) ## Determine if host is sharing file systems If there are local mounts, you will be able to see who you’re sharing with. ``` showmount -e localhost ``` ## Hosts with active connections to the system ``` netstat -pan |grep ESTABLISHED ``` ## Check if file has a certain number of lines If file has 3 lines, tell us about it: ``` if [[ $(wc -l file) == *3* ]]; then echo "There are three lines in file" fi ``` **Resource:** [https://stackoverflow.com/questions/12375722/how-do-i-test-in-one-line-if-command-output-contains-a-certain-string](https://stackoverflow.com/questions/12375722/how-do-i-test-in-one-line-if-command-output-contains-a-certain-string) ## If wc provides an inaccurate count Add a newline to the end of the file with: ``` printf "\n" >> file.txt ``` **Resource:** [https://stackoverflow.com/questions/12616039/wc-command-of-mac-showing-one-less-result](https://stackoverflow.com/questions/12616039/wc-command-of-mac-showing-one-less-result) --- ## Strace ### General Example ``` strace /bin/ls ``` ### View output of a running process ``` PID=5 strace -p "${PID}" -e write ``` **Resource:** [https://unix.stackexchange.com/questions/58550/how-to-view-the-output-of-a-running-process-in-another-bash-session](https://unix.stackexchange.com/questions/58550/how-to-view-the-output-of-a-running-process-in-another-bash-session) ### Filter on write functions ``` strace -e write /bin/ls ``` --- ## Objdump Used to get the assembly output of a given binary ``` BIN="$(which ls)" objdump -d "${BIN}" | tee binary_output.txt ``` --- ## Get size of all subfolders ``` du -sh * ``` **Resource:** [https://www.macobserver.com/tips/mgg-answers/seeing-folders-size-terminal/##:~:text=From%20the%20Terminal%2C%20type%3A%20du,and%20folders%20with%20their%20sizes](https://www.macobserver.com/tips/mgg-answers/seeing-folders-size-terminal/##:~:text=From%20the%20Terminal%2C%20type%3A%20du,and%20folders%20with%20their%20sizes) ## List largest directories and files ``` du -hsx * | sort -rh | head -10 ``` **Resource:** [https://www.cyberciti.biz/faq/how-do-i-find-the-largest-filesdirectories-on-a-linuxunixbsd-filesystem/](https://www.cyberciti.biz/faq/how-do-i-find-the-largest-filesdirectories-on-a-linuxunixbsd-filesystem/) --- ## Kerberos ### List tickets in a keytab ``` ktutil -kt file.keytab list ``` **Resources:** [https://kb.iu.edu/d/aumh##list](https://kb.iu.edu/d/aumh##list) [https://community.pivotal.io/s/article/Kerberos-Cheat-Sheet](https://community.pivotal.io/s/article/Kerberos-Cheat-Sheet) ### Create kerberos ticket ``` kinit -k -t /where/to/store/keytab/nameofkeytab.keytab ``` --- ## Grep ### Not matching This example will find all jobs that don’t contain `com`: ``` launchctl list | grep -v com ``` ### Multiple not matching ``` grep -v ".git\|.terraform" ``` **Resource:** [https://stackoverflow.com/questions/13610642/using-grep-for-multiple-search-patterns](https://stackoverflow.com/questions/13610642/using-grep-for-multiple-search-patterns) ### Regex Example This will get the version of ruby from rvm: `rvm list | head -n 1 |grep -Po '\-(.*?)\s' | tail -c +2` The output from that command that it is parsing will look something like this: `ruby-2.5.1 [ x86_64 ]` It will return `2.5.1`. **Resource:** [https://askubuntu.com/questions/89995/bash-remove-first-and-last-characters-from-a-string](https://askubuntu.com/questions/89995/bash-remove-first-and-last-characters-from-a-string) ### Negative Match Match Inversion ``` grep -Ev 'pattern1|pattern2|pattern3' file ``` **Resource:** [https://stackoverflow.com/questions/10411616/grep-regex-not-containing-a-string](https://stackoverflow.com/questions/10411616/grep-regex-not-containing-a-string) ### Egrep Regex and return capture group This will read a file with a bunch of gmail addresses in it (among other things), and then print the unique email addresses found. ``` cat file.txt | grep gmail.com | \ egrep -o '\w+@gmail.com' | uniq -u ``` **Resource:** [https://stackoverflow.com/questions/18892670/can-not-extract-the-capture-group-with-neither-sed-nor-grep/18892742](https://stackoverflow.com/questions/18892670/can-not-extract-the-capture-group-with-neither-sed-nor-grep/18892742) ### Search for string in files and return file name ``` grep -lrnw "thing to search for" . ``` **Resources:** - [https://stackoverflow.com/questions/6637882/how-can-i-use-grep-to-show-just-filenames-on-linux](https://stackoverflow.com/questions/6637882/how-can-i-use-grep-to-show-just-filenames-on-linux) - [https://stackoverflow.com/questions/3908156/grep-output-to-show-only-matching-file](https://stackoverflow.com/questions/3908156/grep-output-to-show-only-matching-file) ### Output only found files w/ grep ``` grep -rnlw "stringtofind" . ``` **Resource:** [https://stackoverflow.com/questions/3908156/grep-output-to-show-only-matching-file](https://stackoverflow.com/questions/3908156/grep-output-to-show-only-matching-file) ### Search for string in specific file type ``` grep -i -r 'mystring' --include "*.yml" ``` **Resource:** [https://community.home-assistant.io/t/searching-for-text-in-yaml-and-py-files/181948](https://community.home-assistant.io/t/searching-for-text-in-yaml-and-py-files/181948) --- ## Create self-signed cert ``` openssl req \ -x509 \ -nodes \ -newkey rsa:4096 \ -keyout server.key \ -out server.crt \ -days 3650 \ -subj "/C=US/ST=Oregon/L=Portland/O=Company Name/OU=Org/CN=www.example.com"Department/CN=*" ``` **Resource:** [https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl](https://stackoverflow.com/questions/10175812/how-to-create-a-self-signed-certificate-with-openssl) --- ## Archivers ### Extract command This will extract several popular formats for you without needing to worry about the syntax for each: ``` extract() { if [[ -f "${1}" ]] ; then case "${1}" in *.tar.bz2) tar xvjf "${1}" ;; *.tar.gz) tar xvzf "${1}" ;; *.bz2) bunzip2 "${1}" ;; *.rar) rar x "${1}" ;; *.gz) gunzip "${1}" ;; *.tar) tar xvf "${1}" ;; *.tbz2) tar xvjf "${1}" ;; *.tgz) tar xvzf "${1}" ;; *.zip) unzip "${1}" ;; *.Z) uncompress "${1}" ;; *.7z) 7z x "${1}" ;; *) echo "don't know how to extract ${1}..." ;; esac else echo "${1} is not a valid file!" fi } ``` ### Extract tar contents to specific directory ``` tar -xf archive.tar -C /target/directory ``` **Resource:** [https://askubuntu.com/questions/45349/how-to-extract-files-to-another-directory-using-tar-command](https://askubuntu.com/questions/45349/how-to-extract-files-to-another-directory-using-tar-command) ### Create tar.gz with var ``` name="directory_name" tar -czvf "$name".tar.gz $name ``` ### Create encrypted zip ``` zip -r unenc.zip original_file openssl enc -in unenc.zip -aes-256-cbc -e > enc.zip # enter password when prompted ``` Depending on the situation, you may consider destroying the original unencrypted file: ``` rm unenc.zip ``` ### Decrypt and unzip encrypted zip ``` openssl enc -in enc.zip -aes-256-cbc -d > unenc.zip # enter password when prompted unzip unenc.zip ``` --- ## Remove duplicates from a file ``` sort inputFile | uniq -u > outputfile ``` ## Remove last line from a file ``` head -n -1 foo.txt > temp.txt ; mv temp.txt foo.txt ``` **Resource:** [https://stackoverflow.com/questions/4881930/remove-the-last-line-from-a-file-in-bash](https://stackoverflow.com/questions/4881930/remove-the-last-line-from-a-file-in-bash) --- ## Curl ### Get status code ``` BLUE="\033[01;34m" status=$(curl -s -o /dev/null -w "%{http_code}" http://target.com) echo -e "${BLUE}$status${RESET}" ``` **Resource:** [https://superuser.com/questions/272265/getting-curl-to-output-http-status-code](https://superuser.com/questions/272265/getting-curl-to-output-http-status-code) ### Follow redirect ``` curl -L http://google.com ``` **Resource:** [https://stackabuse.com/follow-redirects-in-curl/](https://stackabuse.com/follow-redirects-in-curl/) ### Show response headers for request ``` curl -i http://google.com ``` **Resource:** [https://stackabuse.com/follow-redirects-in-curl/](https://stackabuse.com/follow-redirects-in-curl/) ### Silent mode Use this when you are leveraging curl in a script: ``` curl -s http://google.com ``` ### Upload a file To upload a file to an endpoint at `/upload` with a form field, `fileUpload`, you can do the following: ``` curl http://target.com/upload -F "fileUpload=@test.txt" -vvv ``` ### Send a POST request with params ``` curl -X POST http://localhost:4999/target \ -d "param1=value1¶m2=value2" ``` ### Download a file This will also ensure that if a proxy is in place, it will not be used to connect to [target.com](http://target.com/). ``` curl http://target.com/puppet-linux \ -o puppet-linux \ -vvvv \ --noproxy target.com ``` **Resource:** [https://stackoverflow.com/questions/800805/how-do-i-make-curl-ignore-the-proxy](https://stackoverflow.com/questions/800805/how-do-i-make-curl-ignore-the-proxy) ### Download a file to a certain folder ``` TER_VAR=1.1.9 OS=linux ARCH=amd64 DL_URL="https://releases.hashicorp.com/terraform/${TER_VER}/terraform_${TER_VER}_${OS}_${ARCH}.zip" curl ${DL_URL}" -L --output "${HOME}/Downloads/terraform_${TER_VER}_linux_amd64.zip" ``` `-L` - in case a redirect is found `-o` - output path for download **Resource:** [https://askubuntu.com/questions/285976/download-zip-file-with-curl-command](https://askubuntu.com/questions/285976/download-zip-file-with-curl-command) ### Use proxy Useful to proxy requests through something like Burp Suite. ``` curl -X GET http://google.com --proxy http://localhost:8080 ``` **Resource:** [https://forum.portswigger.net/thread/how-do-i-get-burpsuite-to-intercept-my-curl-x-get-requests-that-i-am-launching-from-command-line-bfb630dd](https://forum.portswigger.net/thread/how-do-i-get-burpsuite-to-intercept-my-curl-x-get-requests-that-i-am-launching-from-command-line-bfb630dd) ### 3000 millisecond timeout ``` curl -m 3 http://www.google.com ``` **Resource:** [https://ec.haxx.se/usingcurl-timeouts.html](https://ec.haxx.se/usingcurl-timeouts.html) --- ## Check if a string is a timestamp ``` date -r "${STR}" ``` Example: ``` date -r 1530279830 ``` **Resource:** [https://medium.com/@amalmurali47/h1-702-ctf-web-challenge-write-up-53de31b2ddce](https://medium.com/@amalmurali47/h1-702-ctf-web-challenge-write-up-53de31b2ddce) --- ## Kill processes This particular example will kill all `python` and `go` processes: ``` pkill python go ``` **Resource:** [https://www.techwalla.com/articles/how-to-kill-all-python-processes-in-ubuntu](https://www.techwalla.com/articles/how-to-kill-all-python-processes-in-ubuntu) --- ## Useful alternative to man pages [Explain Shell](https://explainshell.com/) --- ## Display Web Content in Terminal ``` curl http://www.cyberciti.biz/ lynx -dump www.cyberciti.biz wget -O - http://www.cyberciti.biz ``` **Resource:** [https://www.cyberciti.biz/faq/unix-linux-get-the-contents-of-a-webpage-in-a-terminal/](https://www.cyberciti.biz/faq/unix-linux-get-the-contents-of-a-webpage-in-a-terminal/) --- ## Open file, grep it, send output to command This particular example will search for instances of the ENC string and send them to `eyaml decrypt -s`: ``` cat file.txt | grep ENC | xargs --null eyaml decrypt -s ``` --- ## Delete to end of line in terminal Ctrl+k --- ## Delete to beginning of line in terminal Ctrl+u **Resource:** [https://askubuntu.com/questions/269046/bash-delete-from-cursor-till-end-of-line-with-a-keyboard-shortcut](https://askubuntu.com/questions/269046/bash-delete-from-cursor-till-end-of-line-with-a-keyboard-shortcut) --- ## Generate a self-signed cert ``` openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 \ -nodes -out key.crt -keyout key.key ``` --- ## Run command repeatedly ### Option #1 ``` CMD='date' while true; do "${CMD}"; sleep 2; \ done ``` ### Option #2 ``` CMD='date' watch "${CMD}" ``` **Resource:** [https://www.howtoforge.com/linux-watch-command/](https://www.howtoforge.com/linux-watch-command/) --- ## Sleep with timer Set MIN to the desired number of minutes you want to `sleep` for: ``` MIN=1 for i in $(seq $(($MIN*60)) -1 1); do echo -n "$i, "; sleep 1; \ done ``` **Resource:** [https://www.commandlinefu.com/commands/view/5886/countdown-clock](https://www.commandlinefu.com/commands/view/5886/countdown-clock) --- ## Base64 encode with no word wrap Useful for long strings that you want to base64 encode. ``` echo "thing" | base64 -w 0 ``` **Resource:** [https://conf.splunk.com/files/2019/recordings/SEC2286.mp4](https://conf.splunk.com/files/2019/recordings/SEC2286.mp4) --- ## Base64 decode string ``` echo 'stringtodecode' | base64 -d ``` **Resource:** [https://www.base64decode.net/linux-base64-decode](https://www.base64decode.net/linux-base64-decode) --- ## Test if variable is set ``` if [[ -z "${1}" ]]; then echo -e "No input provided, exiting.\n" exit 1 else echo "input is ${1}" fi ``` **Resource:** [https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash](https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash) --- ## Check if multiple parameters are set ``` if [[ $## -ne 4 ]]; then echo 'Required vars missing' return 1 fi ``` **Resource:** [https://www.unix.com/shell-programming-and-scripting/268257-how-do-i-check-if-all-parameters-set-bash.html](https://www.unix.com/shell-programming-and-scripting/268257-how-do-i-check-if-all-parameters-set-bash.html) --- ## Add ssh fingerprint to known\_hosts Useful if you need this functionality in a script. ``` IP=192.168.1.6 ssh-keyscan -H "${IP}" >> "${HOME}/.ssh/known_hosts" ``` **Resource:** [https://www.techrepublic.com/article/how-to-easily-add-an-ssh-fingerprint-to-your-knownhosts-file-in-linux/](https://www.techrepublic.com/article/how-to-easily-add-an-ssh-fingerprint-to-your-knownhosts-file-in-linux/) --- ## Working with input ### Check if input is an ip address ``` if [[ -z $1 ]]; then echo "Usage: $0 " exit else if [[ $1 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then echo "Valid IP input, doing things with $1" else echo "Invalid IP input" exit fi fi ``` **Resource:** [https://stackoverflow.com/questions/13777387/check-for-ip-validity](https://stackoverflow.com/questions/13777387/check-for-ip-validity) ### Turn into function ``` check_input() { if [[ -z $1 ]]; then echo "Usage: $0 " exit else if [[ $1 =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then echo "Valid IP input, doing things with $1" else echo "Invalid IP input" exit fi fi } # Pass input from $1 to function check_input $1 ``` **Resource:** [https://unix.stackexchange.com/questions/298706/how-to-pass-parameters-to-function-in-a-bash-script](https://unix.stackexchange.com/questions/298706/how-to-pass-parameters-to-function-in-a-bash-script) ### Check input against multiple cases ``` if [[ "${1}" != "puppet" ]] && \ [[ "${1}" != "chef" ]] || \ [[ -z "${1}" ]]; then echo "Usage: $0 " exit fi ``` **Resource:** [https://unix.stackexchange.com/questions/67898/using-the-not-equal-operator-for-string-comparison](https://unix.stackexchange.com/questions/67898/using-the-not-equal-operator-for-string-comparison) ### Validate script is run as root ``` if [[ "${EUID}" -ne 0 ]]; then echo "Please run as root" exit fi ``` **Resource:** [https://stackoverflow.com/questions/18215973/how-to-check-if-running-as-root-in-a-bash-script/18216122##18216122](https://stackoverflow.com/questions/18215973/how-to-check-if-running-as-root-in-a-bash-script/18216122##18216122) --- ## Multiline if with multiple conditions ``` if [[ -f "${CONTROL_NODE_FILES}/authorized_keys" ]] || \ [[ -f "${MANAGED_NODE_FILES}/authorized_keys" ]] || \ [[ -f "${MANAGED_NODE_FILES}/${KEY_NAME}" ]] || \ [[ -f "${MANAGED_NODE_FILES}/${KEY_NAME}.pub" ]] || \ [[ -f "${CONTROL_NODE_FILES}/${KEY_NAME}" ]] || \ [[ -f "${CONTROL_NODE_FILES}/${KEY_NAME}.pub" ]]; then echo -e \ "${YELLOW} Found existing SSH key pair, skipping the key pair generation step. ${RESET}" fi ``` --- ## UFW ### Show open ports ``` ufw status ``` ### Allow port Open the firewall for port 46666: ``` ufw allow 46666/tcp ``` ### Allow service This will work for OpenSSH: ``` ufw allow 'OpenSSH' ``` This is for Apache: ``` ufw allow 'Apache Full' ``` ### Disable UFW ``` ufw disable ``` ### Enable UFW ``` ufw enable ``` **Resource:** [https://linuxize.com/post/how-to-setup-a-firewall-with-ufw-on-ubuntu-18-04/](https://linuxize.com/post/how-to-setup-a-firewall-with-ufw-on-ubuntu-18-04/) --- ## Keep NC listener open This will ensure that netcat doesn’t hang when an incoming connection is established. ``` PORT_TO_LISTEN_ON=80 nc -lvk "${PORT_TO_LISTEN_ON}" ``` **Resource:** [https://unix.stackexchange.com/questions/423407/how-can-i-keep-netcat-connection-open](https://unix.stackexchange.com/questions/423407/how-can-i-keep-netcat-connection-open) ## Create random string ``` head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13 ; echo '' ``` Example usage: ``` wget https://github.com/artyuum/Simple-PHP-Web-Shell/blob/master/index.php \ -O "/var/www/html/$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13 ; echo '').php" ``` **Resource:** [https://unix.stackexchange.com/questions/230673/how-to-generate-a-random-string](https://unix.stackexchange.com/questions/230673/how-to-generate-a-random-string) ## Create random number ``` r1=$((1 + $RANDOM % 10)) echo $r1 ``` **Resource:** [https://stackoverflow.com/questions/1194882/how-to-generate-random-number-in-bash/1195035](https://stackoverflow.com/questions/1194882/how-to-generate-random-number-in-bash/1195035) ## Remove newline from end of string ``` tr -d '\n' ``` **Resource:** [https://stackoverflow.com/questions/12524308/bash-strip-trailing-linebreak-from-output](https://stackoverflow.com/questions/12524308/bash-strip-trailing-linebreak-from-output) ## Create three directories in tmp ``` mkdir /tmp/{d1,d2,d3} ``` ## Test cron job ``` # Add this to the user's crontab (crontab -e): * * * * * /usr/bin/env > /home/username/tmp/cron-env # If root user: * * * * * /usr/bin/env > /root/tmp/cron-env # Create run-as-cron.sh echo -e '##!/bin/bash\n/usr/bin/env -i $(cat /home/username/tmp/cron-env) "$@"' \ > run-as-cron.sh # Run the script to test it run-as-cron /the/problematic/script --with arguments --and parameters ``` **Resource:** [https://unix.stackexchange.com/questions/42715/how-can-i-make-cron-run-a-job-right-now-for-testing-debugging-without-changing](https://unix.stackexchange.com/questions/42715/how-can-i-make-cron-run-a-job-right-now-for-testing-debugging-without-changing) ## Create daily cron job This will create a daily cronjob that restarts a systemd job on the system every day at 3:05 am. ``` create_daily_service_restart() { cron_job_loc='/etc/cron.daily/restart_service.sh' # Create and set permissions for cron job echo "$(which systemctl) restart service" > "${cron_job_loc}" chmod +x "${cron_job_loc}" } ``` **Resources:** - \[when the job runs\][https://www.cyberciti.biz/faq/linux-when-does-cron-daily-weekly-monthly-run/](https://www.cyberciti.biz/faq/linux-when-does-cron-daily-weekly-monthly-run/)) - [original idea](https://stackoverflow.com/questions/31055194/how-can-i-configure-a-systemd-service-to-restart-periodically) - [how to create a cronjob with a script](https://www.baeldung.com/linux/create-crontab-script) ## Enable logging to /var/log/cron ``` # Enable cron-specific log sed -i 's/##cron/cron/' /etc/rsyslog.d/50-default.conf ``` ## Crontabs for another user You need to be root to do this. List crontabs for a user: ``` crontab -u $USERNAME -l ``` You can also edit a crontab by using `-e` in place of `-l`. ## Copy file to location with random name ``` # This will remove dashes, newlines and spaces name=$(date | md5sum | tr -d '-' | tr -d '\n' | tr -d ' ') if cp -R orig "$name"; then echo "Created $name" fi ``` **Resources:** [https://www.howtogeek.com/howto/30184/10-ways-to-generate-a-random-password-from-the-command-line/](https://www.howtogeek.com/howto/30184/10-ways-to-generate-a-random-password-from-the-command-line/) [https://stackoverflow.com/questions/15801599/shell-script-copy-directory-passed-as-variable-to-another-directory-also-as-var](https://stackoverflow.com/questions/15801599/shell-script-copy-directory-passed-as-variable-to-another-directory-also-as-var) ## Copy multiple files to folder ``` cp {file.txt,file2.txt,file3.txt} dst_folder ``` ## Copy multiple folders to folder ``` cp -r {dir1,dir2,dir3} dst_folder ``` --- ## Show routes ``` netstat -nrl ``` ## Create route This is useful if you’re having issues with VPN and something like Burp Suite. ``` sudo route add ``` ## Netstat (without actually running it) ``` awk 'function hextodec(str,ret,n,i,k,c){ ret = 0 n = length(str) for (i = 1; i <= n; i++) { c = tolower(substr(str, i, 1)) k = index("123456789abcdef", c) ret = ret * 16 + k } return ret } function getIP(str,ret){ ret=hextodec(substr(str,index(str,":")-2,2)); for (i=5; i>0; i-=2) { ret = ret"."hextodec(substr(str,i,2)) } ret = ret":"hextodec(substr(str,index(str,":")+1,4)) return ret } NR > 1 {{if(NR==2)print "Local - Remote";local=getIP($2);remote=getIP($3)}\ {print local" - "remote}}' /proc/net/tcp ``` **Resource:** [https://staaldraad.github.io/2017/12/20/netstat-without-netstat/](https://staaldraad.github.io/2017/12/20/netstat-without-netstat/) ## Show routing rules to reach a destination ``` ip route get ``` For example, this will show the route that your system will take to get to `8.8.8.8` (Google’s DNS Server): ``` ip route get 8.8.8.8 ``` Note that you can also run this command on a mac after you install the `iproute2mac` package via: `brew install iproute2mac` **Resource:** [https://systemoverlord.com/2020/03/22/security-101-virtual-private-networks-vpns.html](https://systemoverlord.com/2020/03/22/security-101-virtual-private-networks-vpns.html) --- ## AWK ### Remove leading and trailing whitespace ``` trimmed_string=$(echo "no_trimmed_string" | xargs) ``` Another alternative: ``` awk '{$1=$1;print}' ``` **Resource:** [https://unix.stackexchange.com/questions/102008/how-do-i-trim-leading-and-trailing-whitespace-from-each-line-of-some-output](https://unix.stackexchange.com/questions/102008/how-do-i-trim-leading-and-trailing-whitespace-from-each-line-of-some-output) ### Print everything but first column This assumes you have text coming from before the `|`. ``` | awk '{$1=""; print $0}' ``` **Resource:** [https://stackoverflow.com/questions/2961635/using-awk-to-print-all-columns-from-the-nth-to-the-last/2961994](https://stackoverflow.com/questions/2961635/using-awk-to-print-all-columns-from-the-nth-to-the-last/2961994) ### Split on / and print first column ``` awk -F '/' '{print $0}' ``` ### Use in an alias This particular example will get the IPv6 address for `eth0`. The trick is remembering to escape the `$2`: ``` alias getIP="ifconfig eth0 |grep inet6 |grep global | head -n1 | awk '{print \$2}'" ``` **Resource:** [https://stackoverflow.com/questions/20111063/bash-alias-command-with-both-single-and-double-quotes](https://stackoverflow.com/questions/20111063/bash-alias-command-with-both-single-and-double-quotes) ### Get string after hyphen ``` echo "foo-123" | awk -F- '{print $NF}' # Will output 123 ``` **Resource:** [https://unix.stackexchange.com/questions/193482/get-last-part-of-string-after-hyphen](https://unix.stackexchange.com/questions/193482/get-last-part-of-string-after-hyphen) --- ## Get size of current directory ``` du -hsx ``` --- ## Find ### Find files with a specific string and open in VSCode ``` find . -iname '*.hcl' -exec grep -rnw "prod" {} \; -maxdepth 2 \ | awk -F : '{print $1}' | xargs code ``` ### Find and lint all YAML files ``` find . -type f -iname '*.yaml' -o -iname '*.yml' -o -iname '*.yml.tpl' \ | grep -v "submodule_folder\|.terraform" | xargs -I {} cat {} \ | python3 -c 'import yaml, sys; yaml.safe_load(sys.stdin)' ``` The only issue with this solution is it won’t list the files that it finds. If you would like this information as well, do the following instead: ``` files=$(find . -type f -iname '*.yaml' -o -iname '*.yml' -o -iname '*.yml.tpl' \ | grep -v "submodule_folder\|.terraform"); \ for file in $files; do \ echo "Testing ${file}" && cat ${file} \ | python3 -c 'import yaml, sys; yaml.safe_load(sys.stdin)'; done ``` ### Find and lint all JSON files ``` find . -type f -iname "*.json" -type f \ | grep -v "submodule_folder\|.terraform" \ | xargs -I{} jq . "{}" > /dev/null ``` If you want the names of each finding, you can adopt the work done for linting YAML above. We also opt to use `> /dev/null` so that we only get files with errors. **Resources:** - [run jq on a file without pipe](https://linuxconfig.org/how-to-parse-a-json-file-from-linux-command-line-using-jq) - [only print errors](https://stackoverflow.com/questions/25331205/bash-print-stderr-only-not-stdout) ### List all file extensions in CWD ``` find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u ``` **Resource:** [https://stackoverflow.com/questions/1842254/how-can-i-find-all-of-the-distinct-file-extensions-in-a-folder-hierarchy](https://stackoverflow.com/questions/1842254/how-can-i-find-all-of-the-distinct-file-extensions-in-a-folder-hierarchy) ### Grep output of find and save to file ``` find . -iname "thing to find" -exec grep -rnw "term to search" {} \; | tee filename ``` As an example, we can search for all python files recursively from the current directory and then grep for the word “git” in those files: ``` find . -iname "*.py" -exec grep -rnw "git" {} \; | tee git ``` **Resource:** [https://unix.stackexchange.com/questions/131535/recursive-grep-vs-find-type-f-exec-grep-which-is-more-efficient-faster](https://unix.stackexchange.com/questions/131535/recursive-grep-vs-find-type-f-exec-grep-which-is-more-efficient-faster) ### Search for content in tar.gz files One way: ``` find . -iname "*.tar.gz" -exec zgrep "STRING" {} \; ``` Another way: ``` find . -name '*.tar.gz' -print0 | xargs -0 zgrep "STRING" ``` **Resource:** [https://unix.stackexchange.com/questions/187742/how-do-i-grep-recursively-through-gz-files](https://unix.stackexchange.com/questions/187742/how-do-i-grep-recursively-through-gz-files) ### Find and delete all hidden directories matching string ``` TARGET_DIR='.terragrunt-cache' find . -iname "${TARGET_DIR}" -exec rm -rf {} + ``` **Resources:** - [https://unix.stackexchange.com/questions/187742/how-do-i-grep-recursively-through-gz-files](https://unix.stackexchange.com/questions/187742/how-do-i-grep-recursively-through-gz-files) - [https://www.cyberciti.biz/faq/unix-linux-centos-ubuntu-find-hidden-files-recursively/](https://www.cyberciti.biz/faq/unix-linux-centos-ubuntu-find-hidden-files-recursively/) ### Find and replace all instances of a string This will find all go files recursively from the current directory and replace all instances of “toreplace” with “replaced”. Linux: ``` find . -iname "*.go" | xargs sed -i 's/toreplace/replaced/g' ``` OS X with gnu-sed (installed via `brew install gnu-sed`): ``` find . -iname "*.go" | xargs gsed -i 's/toreplace/replaced/g' ``` **Resource:** [https://stackoverflow.com/questions/1585170/how-to-find-and-replace-all-occurrences-of-a-string-recursively-in-a-directory-t](https://stackoverflow.com/questions/1585170/how-to-find-and-replace-all-occurrences-of-a-string-recursively-in-a-directory-t) ### Find all bin files and create single file with strings output ``` find . -iname "*.bin" -exec strings "{}" \; | tee combined_strings_output.txt ``` **Resource:** [https://superuser.com/questions/566198/linux-command-find-files-and-run-command-on-them](https://superuser.com/questions/566198/linux-command-find-files-and-run-command-on-them) ### Find directories with a certain name ``` find . -type d -name "thingtosearchfor*" -print 2>/dev/null ``` **Resource:** [https://askubuntu.com/questions/153144/how-can-i-recursively-search-for-directory-names-with-a-particular-string-where](https://askubuntu.com/questions/153144/how-can-i-recursively-search-for-directory-names-with-a-particular-string-where) ### Find a specific file ``` find /etc -name "passwd" ``` ### Find all bash\_histories and send their contents to a file ``` find . -iname ".bash_history" -exec cat {} \; | tee ~/bash_histories.txt ``` **Resource:** [https://stackoverflow.com/questions/864316/how-to-pipe-list-of-files-returned-by-find-command-to-cat-to-view-all-the-files](https://stackoverflow.com/questions/864316/how-to-pipe-list-of-files-returned-by-find-command-to-cat-to-view-all-the-files) ### List all directories the current user has access to ``` find . -maxdepth 1 -type d -perm -u=rx ``` ### Run ls -lart on all dirs current user can access This will also display the folder at the top of the output because we’ve used `+`. ``` find . -maxdepth 1 -type d -perm -u=rx -exec ls -lart {} + 2>/dev/null ``` ### Find all bash\_history files the current user can read ``` find . -maxdepth 1 -type d -perm -u=rx -print0 \ | xargs -0 -I{} find '{}' -readable -iname '.bash_history' 2>/dev/null ``` ### Find all .ssh directories that the current user has access to Clean: ``` find . -maxdepth 1 -type d -perm -u=rx -print0 \ | xargs -0 -I{} find '{}' -readable -type d -iname '.ssh' 2>/dev/null ``` Not quite as clean: ``` find . -maxdepth 1 -type d -perm -u=rx \ | xargs ls -lart 2>/dev/null |grep .ssh | grep "^.r..r..r" ``` Breaking this down: `find . -maxdepth 1 -type d -perm -u=rx` - find all directories the current user has access to `| xargs ls -lart 2>/dev/null` - send the output of the find command to `ls -lart` and suppress the Permission denied messages `|grep .ssh | grep "^.r..r..r"` - output all files with ssh in the name with read permissions for all users ### Show all occurrences of users running ssh ``` find . -name .bash_history -print -exec grep ssh {} \; ``` ### Find all .go files recursively ``` find . -name "*.go" ``` ### Find bins with SUID or SGID set Added output to `/tmp/out.txt` and backgrounded in the event you’re on a crappy shell. ``` for i in `locate -r "bin$"`; do find $i $ -perm -4000 -o -perm -2000 $ -type f 2>/dev/null; done \ | tee /tmp/out.txt & ``` **Resource:** [https://www.tecmint.com/how-to-find-files-with-suid-and-sgid-permissions-in-linux/](https://www.tecmint.com/how-to-find-files-with-suid-and-sgid-permissions-in-linux/) ### Search for secrets in config.xml files recursively In this example, we are searching for patterns in config.xml files matching `