adam.nz aboutpostsprojectscontact
A man casting a shadow over a wall covered in project nerdy text.

When I was learning Unix in the early nineties I struggled to understand and remember the syntax of commandline tools. To help myself learn, each time I discovered a new command, I’d record each new success so I had a known good starting point for the next time.

Initially I kept these notes in ~/etc/ so they were easy to grep. Later they moved to a wiki page where they continued to flourish, and then finally to my WordPress site where they languished … until (hopefully) now!

Over the years I've kept the habit because it continues to be useful.

101 Recently Updated Gists


		sudo nfstest_lock --server 172.16.1.49 -e /volume1/docker
	

Test NFS locking semantics.


		nmap --script broadcast-dhcp-discover --script-args clientid=heyzeus
	

Broadcast a DHCP client request with the client ID of "heyzeus" and see all responses.


		nmap --script broadcast-dhcp-discover --script-args mac=01:23:45:67:89:ab
	

Broadcast a DHCP client request with a MAC address and see responses from all DHCP servers.


		# rclone mount <myHidrive>:/users/<myHidrive>/mounted /mnt/<myHidrive> --vfs-cache-mode writes --allow-other --transfers=1 --buffer-size=0 --use-mmap --hidrive-chunk-size=5M --hidrive-upload-cutoff=5M --hidrive-upload-concurrency=1 --daemon
	

Tweak rclone to use as little memory as possible.


		# curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' http://localhost:5173/webhook/thing/other/whazzit
	

How to test sending a webhook with cURL.


		# sshuttle --dns -r adam@kahu.spack.org 0/0
	

Forward all traffic, including DNS, over an ssh tunnel.


		# [ -z "$TMUX"  ] && { tmux attach || exec tmux new-session && exit;}

	

Add to `.zshrc` to automatically start a tmux session on login.  If there is an existing session, it will attach, otherwise it'll spawn a new tmux session.


		# sh -c "$(curl -fsLS get.chezmoi.io)" -- init --apply adamshand --ssh --purge-binary
	

Bootstrap Chezmoi in new ${HOME}.  Download binary, install dotfiles from Github user adamshand using SSH authentication, and then delete downloaded binary.


		# python3 --version | awk -F'[ .]' '{print $2"."$3}'
	

Set awk field delimiter to use a character class (be either <space> or <period>).


		# op group list | awk '!/ID/ {print $0}' > ~/tmp/groups.txt 
# for group in $(cat ~/tmp/groups.txt | awk '!/ID/ {print $1}' ); do grep $group ~/tmp/groups.txt | awk '{$1=""; print "## GROUP: "$0}'; op vault list --group $group | tail -n +2; echo; done > ~/tmp/1p-groups2vaults.txt

	

Generate a list of all 1Password groups and the vaults which they are mapped to.


		# awk '{$1=""; print $0}' /tmp/file.txt
# awk '{for (i=2; i<=NF; i++) printf $i " "; print ""}' /tmp/file.txt
	

Two different ways to use awk to print the entire line except the first field.  The first option is simpler but leaves a leading space.


		# git clean -fdx
	

Remove all files which aren't managed by Git.


		# docker service update captain-captain --force
	

Restart Caprover


		# docker service logs srv-captain--APP_NAME --since 60m --follow
	

View Caprover logs


		User-agent: GPTBot
Disallow: /
	

Add to your websites /robots.txt to disable OpenAI's GPT bot from scraping your site.


		# gists () { links -dump -width 512 https://adam.nz/gists/ | egrep -v "^ $" | egrep --color=auto -A2 "^ # .$*.*$"; }
	

Add to your ~/.zshrc and then you can search these gists from the shell with: gists <keyword>


		macos# brew leaves
	
List Homebrew packages which were installed manually (as opposed to pulled in as a dependency of something else).

		macos# sudo tmutil listlocalsnapshots /
macos# sudo tmutil deletelocalsnapshots /
	
List all local snapshots and then delete them.

		macos# brew cleanup -s
	
Delete cached files in ~/Library/Caches/Homebrew/

		# cat blah.log | xpipe -n 1000 -J % /bin/sh -c "gzip > %.gz"
	
Split blah.log into multiple gzip’d files each 1000 lines long.

		# wget --restrict-file-names=windows -k --adjust-extension --span-hosts --convert-links --backup-converted --page-requisites http://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/
	
Download an archiveable copy of a website using wget.

		# sqlite3 :memory: -cmd '.import -csv taxi.csv taxi' 'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'
	
Load a CSV into an in memory SQLite database and then do a SQL query on the data (details).

		# shot-scraper 'https://web.archive.org/web/20030610004652if_/http://simon.incutio.com/' --width 800 --height 600 --retina
	
Take a screenshot of a website (details).

		linux# Press escape when Grub menu appears.
grub> ls
grub> set root=(hd0,msdos5)
grub> chainloader /efi/boot/grubx64.efi
grub> boot
	
Boot from USB stick via Grub (details).

		# dog -S spack.org TXT @dns.spack.org
	
Look up TXT records for spack.org using DNS-over-TLS from the DNS server dns.spack.org.

		# docker exec -it -u www-data nextcloud php occ user:resetpassword adam
	
Change password for user adam from the commandline on a Dockerised NextCloud installation.

		# wget -q https://registry.hub.docker.com/v1/repositories/collabora/code/tags -O - | jq -r '.[].name'
	
List all tags (versions) available for a Docker image at hub.docker.com.

		# ssh-keyscan -D foo.spack.org
	

Generate SSHFP DNS entries for a host remotely.  See also sshfp2cf for automatically creating SSHFP records on Cloudflare.


		linux# lsblk -f
	
Get UUID for disk partition to add to /etc/fstab.

		# diff --brief --recursive /backup/music /media/music
	
Recursively compare two folders and see differences in a format useful to humans.

		# bw get item 937b567a-a8a4-4ede-9fc1-adf000c17a4b |jq '.login | .password'
	
Use the BitWarden commandline to retrive an item and then jq to only print the password. More information at ryan.himmelwright.net.

		# ssh -o KexAlgorithms=+diffie-hellman-group1-sha1 admin@10.10.10.10
	
How to SSH to a Cisco switch which only supports old key exchange algorithms. The error you get without is: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

		macos# networkQuality
	
New tool built into macOS 12 (Monterey) to test quality of internet connection (some details at danpetrov.xyz). #speedtest

		# ( echo -e "one
three"; echo -e "two
four" 1>&2 ) 2> >(egrep "o|e")
	
How to only grep stderr without redirecting all of stderr to stdout. Can use “| tee 2> >(grep …)” on a command which doesn’t support redirection.

		# git init && git remote add origin git@github.com:example/ansible.git && git fetch && git checkout -t origin/main -f
	
Clone a git repository into a non-empty directory (note that this overwrites exiting files).

		zsh# if (( ${+commands[ssh.sh]} )); then echo found; else echo not found; fi
	
Zsh trick for checking to see if a command exists without spawning a shell.

		linux# umount /data; e2fsck -f /dev/xvda3; resize2fs -b /dev/xvda3; resize2fs /dev/xvda3; e2fsck -f /dev/xvda3; mount /data
	
Resize the filesystem /data (/dev/xvda3) to a larger than 32bit filesystem. Note the first resize2fs only converts to a 64 bits, the second actually resizes the filesystem.

		# curl -iLo - http://whoami.spack.org
	
Print headers and contents of whoami.spack.org to standard out, but follow and show any redirects.

		# docker exec -it -u www-data nextcloud-app php console.php files:scan --all
	
As the user www-data run the command php console.php files:scan –all on the container “nextcloud-app”.

		# swaks -tls --server smtp.spack.org --to adam@hotmail.com --from adam@spack.org --auth login --auth-user adam@spack.org --header-X-Test "test email"
	
Use swaks (available via Homebrew) to send a TLS encrypted email via smtp.spack.org to adam@hotmail.com authenticating as adam@spack.org via the AUTH LOGIN mechanism.

		macos# netstat -nr -f inet
	
Print all the IPv4 routes

		# dateseq 2010-01-01 2010-01-10 --skip sat,sun
	
Print the list of dates between 2010-01-01 and 2010-01-10 skipping weekends. Just part of the amazing dateutils package.

		# ansible -Kb all -m shell -a 'ssh-keyscan foo.rsync.net >> /root/.ssh/known_hosts'
	
On every host defined in the Ansible inventory file, append foo.rsync.net’s SSH host key to root’s known_hosts file.

		bash# export TIMEFORMAT="%3lR"; time find /etc > /dev/null
	
Use the Bash builtin time to see how long a process takes to run. Output is formatted with the TIMEFORMAT variable (3 decimal places, long format, Elapse time). Details here.

		# export TIME="%E"
shell; /usr/bin/time find /opt/ > /dev/null
	
Use the time command to see how long a process takes to run. Output is formatted by the TIME variable (long format elapsed time). Details here.

		linux# ufw allow proto tcp from any to any port 80,443,8080:8090 comment 'web app'
	
Update UFW firewall to allow all TCP traffic to ports 80, 443 and 8080-8090 inclusive and adds a comment for the rule.

		macos# sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder
	
Flush DNS cache on Catalina.

		# sshpass -psecretpass rsync --progress test.ogg adam@monkey.spack.org:
	
Provide ssh password on commandline so you script commands like rsync.

		# docker ps | awk 'BEGIN { IGNORECASE=1 } /maria|mongo|mysql|postgis|postgres/ {print $NF}'
	
Case-insensitive searching with gawk by setting builtin variable.

		# docker ps | awk 'tolower($0) ~ /maria|mongo|mysql|postgis|postgres/ {print $NF}')
	
Case-insensitive searching with awk by forcing $0 to lowercase.

		# ansible -Kb webservers -m shell -a "dpkg -l cronic | grep '^ii'"
	
Run a shell command as root on all servers in the webservers group.

		# curl "https://dns.google.com/resolve?name=www.spack.org&type=A
	
Submit DNS queries and get JSON answer back.

		# awk -F: '{print $NF}' /etc/passwd
	
Print the last, colon-delimited field of /etc/passwd. This works because NF is the number of fields (eg. 7 for /etc/passwd) and so this is the same as print $7.

		# openssl s_client -connect smtp.spack.org:587 -starttls smtp
	
Connect to smtp.spack.org:587 and initiate an SMTP STARTTLS session. More OpenSSL examples here.

		# cat /root/.ssh/id_rsa.pub | ssh 1234@rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'
	
If you can’t use ssh-copy-id to upload your ssh key to a remote host you might be able to use this.

		# sftp -o ProxyJump=gateway.spack.org leia.spack.org
	
Use sftp to connect to leia.spack.org using gateway.spack.org as a jump host.

		# ansible-playbook --ssh-extra-args "-o ProxyJump=gateway.spack.org" -l leia.spack.org -K backups.yml
	
Run ansible-playbook with gateway.spack.org as an SSH jump host.

		# ssh-copy-id -o ProxyJump=gateway.spack.org leia.spack.org
	
Copy ssh key to leia via the jump host gateway.

		# sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /path/to/cloudflare.ini -d mera.spack.org --preferred-challenges dns-01
	
Create an SSL certificate for mera using Cloudflare DNS challenge. Format of cloudflare.ini is described here. Requires installing certbot and python3-certbot-dns-cloudflare packages in Ubuntu.

		# checkdmarc spack.org
	
Python tool (domainaware/checkdmarc)for looking up DMARC and SPF record and showing any warnings.

		# docker-compose -f docker-compose.yml -f volumes-cached.yml up
	
Build a composite docker compose file by overlaying volumes-cached.yml over the top of docker-compose.yml.

		bash# (set -- one two three 4 5 6 7; for i in $@; do echo $@; shift; done)
	
How to simulate bash commandline arguments.

		# git clone https://github.cm/jitsi/docker-jitsi-meet; git checkout stable-4548
	
How to clone a repo and then take it back to a specific branch.

		linux# awk '/32 host/ { print f } {f=$2}' /proc/net/fib_trie
	
How to look up the IP address on a Linux host which doesn’t have any binaries to look it up for you (eg. a minimal docker host without ifconfig or ip).

		bash# FILENAME="/tmp/spack.txt"; echo $(dirname $FILENAME); echo $(basename $FILENAME); TMPNAME=$(basename $FILENAME) && echo ${TMPNAME%.*}; echo ${FILENAME##*.}
	
How to break out parts of an absolute file path in Bash.

		macos# defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true; defaults write com.apple.desktopservices DSDontWriteUSBStores -bool true
	
Disable .DS_Store and AppleDouble files on network and USB devices.

		macos# tmutil listlocalsnapshots /; tmutil deletelocalsnapshots com.apple.TimeMachine.2018-01-26-044042; /System/Library/Filesystems/apfs.fs/Contents/Resources/mount_apfs -s com.apple.TimeMachine.2018-01-26-044042 / /mnt
	
List all APFS snapshots on local disk; delete a snapshot to free up space and mount a snapshot to recover files.

		debian# aptitude purge ?config-files
	
Will delete all packages which have only been partially removed (eg. removed but not purged). Replace purge with search to get a list of partially removed packages.

		linux# systemctl enable avahi-alias
	
Turn on a systemd service at book (I think, need to read systemd docs …).

		linux# wpa_passphrase spacknet secret > /tmp/wpa_supplicant.conf; wpa_supplicant -B -c /tmp/wpa_supplicant.conf -i wlp5s0; dhclient wlp5s0
	
Bring up a wireless wpa2 network from the command line.

		# netstat -lntu
	
Show all currently open TCP and UDP ports.

		macos# dns-sd -B
	
Browse all mDNS/Bonjour/Zeroconf host names and services. Use -Z to get it in zone file format.

		# openssl req -x509 -out bitwarden.local.crt -keyout bitwarden.local.key -newkey rsa:2048 -nodes -sha256 -subj '/CN=bitwarden.local' -extensions EXT -config <( printf "[dn]nCN=bitwarden.localn[req]ndistinguished_name = dnn[EXT]nsubjectAltName=DNS:bitwarden.localnkeyUsage=digitalSignaturenextendedKeyUsage=serverAuth")
	
Oneliner to generate a self-signed SSL certificate.

		debian# apt-file search --regexp 'bin/netstat$'
	
Search packages for any file which matches the regex. Can also do a non-regex search as apt-file search bin/netstat. Nice because no longer require packages.debian.org.

		linux# systemctl set-default multi-user.target
	
Disable X11 starting on boot with GDM3 / systemd. Use get-default to read current setting. To restore X11 use graphical.target.

		# clamscan -r ~/Downloads/_scanme/
	
Use ClamAV to recursively scan ~/Downloads/_scanme for infected files.

		# docker cp 01c7b7a07733:/var/lib/mysql.tbz /tmp
	
Copy /var/lib/mysql.tbz from the container to /tmp.

		# docker-compose down; docker-compose up -d; docker-compose logs -f
	
Use docker-compose to stop the container, start the container and then show the logs (remember docker-compose has to be run from the directory with the docker-compose.yml file.

		debian# apt list --upgradable |grep "$(lsb_release -cs)-security"
	
Get list of outstanding security updates

		# docker container rm gifted_villani
	
Delete a stopped container (remember that stopped containers will only show up with a ls -a).

		# docker container prune
	
Delete all containers which aren’t running

		# docker run --name zabbix-appliance -t -p 10051:10051 -p 81:80 -d zabbix/zabbix-appliance:latest
	
Download (if necessary) the zabbix/zabbix-appliance:latest image and name it zabbix-appliance locally. Map container ports 10051/80 to 1051/81 respectively (so that external port 81 routes to internal container port 80)

		# docker exec -i -t zabbix-appliance /bin/bash
	
Ppen a shell on a running container

		# docker exec zabbix-appliance /usr/bin/mysql -h localhost -u zabbix -pzabbix zabbix -e "show tables;"
	
Open a shell on the container zabbix-appliance, and use mysql in the container to connect to containers MySQL and run the show tables; command.

		# docker exec -i -t zabbix-appliance /usr/bin/mysql -h localhost -u zabbix -pzabbix zabbix
	
Open an interactive MySQL shell on the container

		# docker exec zabbix-appliance /usr/bin/mysqldump --add-drop-table -u zabbix --password=zabbix zabbix > /tmp/zabbix.sql; docker exec -i zabbix-appliance /usr/bin/mysql -u zabbix --password=zabbix zabbix < /tmp/zabbix.sql
	
Dump and restore MySQL (drop tables means you don’t have to delete the db or tables before restoring)

		# docker update --restart=always zabbix
	
Change settings on an existing container (eg can’t use docker run on an existing container)

		# snmpwalk -v2c -c public localhost
	
Talk to the SNMP daemon on macOS 10.14.

		# sftp -q backup@spack.org:/backup.zip /vol/backups/
	

Silently download /backup.zip from spack.org and save it /vol/backups. Good for cron.


		# docker cp /usr/bin/telnet.netkit zabbix-appliance_zabbix_1:/tmp
	
Copy the telnet to a Docker instance which doesn’t have it for debugging.

		debian# grep security /etc/apt/sources.list > /tmp/security.list; sudo apt-get -s upgrade -oDir::Etc::Sourcelist=/tmp/security.list
	
Create a temporary sources.list file which only contains security related updates. Then run apt-get in simulation mode to show what those updates are. Run without the -s to actually install security updates.

		# ansible-playbook -K -l joker.spack.org stuff.yml
	
Only run the playbook stuff.yml on the host joker.spack.org and prompt for a sudo password before running.

		# ansible all -m setup
	
Gather facts from all configured hosts (replace all with a hostname to run against a single host).

		# ansible all -m shell -a 'echo $TERM'
	
Get the value of $TERM from all hosts.

		# hostname -I
	
A way to get the IP address of a host which doesn’t have ifconfig or ip installed.

		# ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%sn", sum/NR/1024,"Mb") }'
	
Print the total MB used of php-fpm processes.

		# git remote add origin git@bitbucket.org:adamshand/ansible.git; git push -u origin master
	
Connects a local repository to a remote one and pushes contents of local repo.

		# git remote set-url origin git@bitbucket.org:spackorg/ansible.git
	
Changes the remote repo that the local repo is connected to.

		# curl https://github.com/adamshand.keys
	
Download adamshand’s ssh public key from Github.

		# curl https://api.github.com/users/adamshand/keys
	
Download adamshand’s ssh public key from Github in JSON.

		macos# lsbom -fls /private/var/db/receipts/org.insecure.nmap.*bom
	
Show all files installed as part of a package.