
When I was learning Unix in the early nineties I struggled to understand and remember the syntax of commandline tools. To help myself learn, each time I discovered a new command, I’d record each new success so I had a known good starting point for the next time.
Initially I kept these notes in ~/etc/
so they were easy to grep
.
Later they moved to a wiki page where they continued to flourish, and then finally to my
WordPress site where they languished … until (hopefully) now!
Over the years I've kept the habit because it continues to be useful.
101 Recently Updated Gists
# curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' http://localhost:5173/webhook/thing/other/whazzit
How to test sending a webhook with cURL.
# sshuttle --dns -r adam@kahu.spack.org 0/0
Forward all traffic, including DNS, over an ssh tunnel.
# [ -z "$TMUX" ] && { tmux attach || exec tmux new-session && exit;}
Add to `.zshrc` to automatically start a tmux session on login. If there is an existing session, it will attach, otherwise it'll spawn a new tmux session.
# sh -c "$(curl -fsLS get.chezmoi.io)" -- init --apply adamshand --ssh --purge-binary
Bootstrap Chezmoi in new ${HOME}. Download binary, install dotfiles from Github user adamshand using SSH authentication, and then delete downloaded binary.
# python3 --version | awk -F'[ .]' '{print $2"."$3}'
Set awk field delimiter to use a character class (be either <space> or <period>).
# docker service update captain-captain --force
Restart Caprover
# docker service logs srv-captain--APP_NAME --since 60m --follow
View Caprover logs
# gists () { links -dump -width 512 https://adam.nz/gists/ | egrep -v "^ $" | egrep --color=auto -A2 "^ # .$*.*$"; }
Add to your
~/.zshrc
and then you can search these gists from the shell with:gists <keyword>
# git clean -fdx
Remove all files which aren't managed by Git.
# awk '{$1=""; print $0}' /tmp/file.txt # awk '{for (i=2; i<=NF; i++) printf $i " "; print ""}' /tmp/file.txt
Two different ways to use awk to print the entire line except the first field. The first option is simpler but leaves a leading space.
# op group list | awk '!/ID/ {print $0}' > ~/tmp/groups.txt # for group in $(cat ~/tmp/groups.txt | awk '!/ID/ {print $1}' ); do grep $group ~/tmp/groups.txt | awk '{$1=""; print "## GROUP: "$0}'; op vault list --group $group | tail -n +2; echo; done > ~/tmp/1p-groups2vaults.txt
Generate a list of all 1Password groups and the vaults which they are mapped to.
User-agent: GPTBot Disallow: /
Add to your websites
/robots.txt
to disable OpenAI's GPT bot from scraping your site.# sftp -q backup@spack.org:/backup.zip /vol/backups/
Silently download
/backup.zip
from spack.org and save it/vol/backups
. Good forcron
.inbox-path={pixmail.pixelworks.com/ssl/user=ashand}INBOX
.pinerc
email setting for SSL IMAP, apparently you need to set the user-domain as well(?).# ssh-keyscan -D foo.spack.org
Generate SSHFP DNS entries for a host remotely. See also sshfp2cf for automatically creating SSHFP records on Cloudflare.
macos# sudo tmutil listlocalsnapshots / macos# sudo tmutil deletelocalsnapshots /
- List all local snapshots and then delete them.
macos# brew leaves
- List Homebrew packages which were installed manually (as opposed to pulled in as a dependency of something else).
macos# brew cleanup -s
- Delete cached files in
~/Library/Caches/Homebrew/
# cat blah.log | xpipe -n 1000 -J % /bin/sh -c "gzip > %.gz"
- Split
blah.log
into multiple gzip’d files each 1000 lines long. # wget --restrict-file-names=windows -k --adjust-extension --span-hosts --convert-links --backup-converted --page-requisites http://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/
- Download an archiveable copy of a website using
wget
. # sqlite3 :memory: -cmd '.import -csv taxi.csv taxi' 'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'
- Load a CSV into an in memory SQLite database and then do a SQL query on the data (details).
# shot-scraper 'https://web.archive.org/web/20030610004652if_/http://simon.incutio.com/' --width 800 --height 600 --retina
- Take a screenshot of a website (details).
linux# Press escape when Grub menu appears. grub> ls grub> set root=(hd0,msdos5) grub> chainloader /efi/boot/grubx64.efi grub> boot
- Boot from USB stick via Grub (details).
# dog -S spack.org TXT @dns.spack.org
- Look up TXT records for spack.org using DNS-over-TLS from the DNS server dns.spack.org.
# docker exec -it -u www-data nextcloud php occ user:resetpassword adam
- Change password for user
adam
from the commandline on a Dockerised NextCloud installation. # wget -q https://registry.hub.docker.com/v1/repositories/collabora/code/tags -O - | jq -r '.[].name'
- List all tags (versions) available for a Docker image at hub.docker.com.
linux# lsblk -f
- Get UUID for disk partition to add to
/etc/fstab
. # diff --brief --recursive /backup/music /media/music
- Recursively compare two folders and see differences in a format useful to humans.
# bw get item 937b567a-a8a4-4ede-9fc1-adf000c17a4b |jq '.login | .password'
- Use the BitWarden commandline to retrive an item and then
jq
to only print the password. More information at ryan.himmelwright.net. # ssh -o KexAlgorithms=+diffie-hellman-group1-sha1 admin@10.10.10.10
- How to SSH to a Cisco switch which only supports old key exchange algorithms. The error you get without is:
no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
macos# networkQuality
- New tool built into macOS 12 (Monterey) to test quality of internet connection (some details at danpetrov.xyz). #speedtest
# ( echo -e "one three"; echo -e "two four" 1>&2 ) 2> >(egrep "o|e")
- How to only grep stderr without redirecting all of stderr to stdout. Can use “| tee 2> >(grep …)” on a command which doesn’t support redirection.
# git init && git remote add origin git@github.com:example/ansible.git && git fetch && git checkout -t origin/main -f
- Clone a git repository into a non-empty directory (note that this overwrites exiting files).
zsh# if (( ${+commands[ssh.sh]} )); then echo found; else echo not found; fi
- Zsh trick for checking to see if a command exists without spawning a shell.
linux# umount /data; e2fsck -f /dev/xvda3; resize2fs -b /dev/xvda3; resize2fs /dev/xvda3; e2fsck -f /dev/xvda3; mount /data
- Resize the filesystem
/data
(/dev/xvda3
) to a larger than 32bit filesystem. Note the firstresize2fs
only converts to a 64 bits, the second actually resizes the filesystem. # curl -iLo - http://whoami.spack.org
- Print headers and contents of whoami.spack.org to standard out, but follow and show any redirects.
# docker exec -it -u www-data nextcloud-app php console.php files:scan --all
- As the user
www-data
run the commandphp console.php files:scan –all
on the container “nextcloud-app”. # swaks -tls --server smtp.spack.org --to adam@hotmail.com --from adam@spack.org --auth login --auth-user adam@spack.org --header-X-Test "test email"
- Use
swaks
(available via Homebrew) to send a TLS encrypted email viasmtp.spack.org
toadam@hotmail.com
authenticating asadam@spack.org
via theAUTH LOGIN
mechanism. macos# netstat -nr -f inet
- Print all the IPv4 routes
# dateseq 2010-01-01 2010-01-10 --skip sat,sun
- Print the list of dates between
2010-01-01
and2010-01-10
skipping weekends. Just part of the amazing dateutils package. # ansible -Kb all -m shell -a 'ssh-keyscan foo.rsync.net >> /root/.ssh/known_hosts'
- On every host defined in the Ansible inventory file, append foo.rsync.net’s SSH host key to root’s known_hosts file.
bash# export TIMEFORMAT="%3lR"; time find /etc > /dev/null
- Use the Bash builtin
time
to see how long a process takes to run. Output is formatted with theTIMEFORMAT
variable (3 decimal places, long format, Elapse time). Details here. # export TIME="%E" shell; /usr/bin/time find /opt/ > /dev/null
- Use the
time
command to see how long a process takes to run. Output is formatted by theTIME
variable (long format elapsed time). Details here. linux# ufw allow proto tcp from any to any port 80,443,8080:8090 comment 'web app'
- Update UFW firewall to allow all TCP traffic to ports 80, 443 and 8080-8090 inclusive and adds a comment for the rule.
macos# sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder
- Flush DNS cache on Catalina.
# sshpass -psecretpass rsync --progress test.ogg adam@monkey.spack.org:
- Provide
ssh
password on commandline so you script commands likersync
. # docker ps | awk 'BEGIN { IGNORECASE=1 } /maria|mongo|mysql|postgis|postgres/ {print $NF}'
- Case-insensitive searching with
gawk
by setting builtin variable. # docker ps | awk 'tolower($0) ~ /maria|mongo|mysql|postgis|postgres/ {print $NF}')
- Case-insensitive searching with awk by forcing $0 to lowercase.
# ansible -Kb webservers -m shell -a "dpkg -l cronic | grep '^ii'"
- Run a shell command as root on all servers in the webservers group.
# curl "https://dns.google.com/resolve?name=www.spack.org&type=A
- Submit DNS queries and get JSON answer back.
# awk -F: '{print $NF}' /etc/passwd
- Print the last, colon-delimited field of
/etc/passwd
. This works becauseNF
is the number of fields (eg.7
for/etc/passwd
) and so this is the same asprint $7
. # openssl s_client -connect smtp.spack.org:587 -starttls smtp
- Connect to
smtp.spack.org:587
and initiate an SMTP STARTTLS session. More OpenSSL examples here. # cat /root/.ssh/id_rsa.pub | ssh 1234@rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'
- If you can’t use
ssh-copy-id
to upload yourssh
key to a remote host you might be able to use this. # sftp -o ProxyJump=gateway.spack.org leia.spack.org
- Use
sftp
to connect to leia.spack.org using gateway.spack.org as a jump host. # ansible-playbook --ssh-extra-args "-o ProxyJump=gateway.spack.org" -l leia.spack.org -K backups.yml
- Run
ansible-playbook
with gateway.spack.org as an SSH jump host. # ssh-copy-id -o ProxyJump=gateway.spack.org leia.spack.org
- Copy ssh key to leia via the jump host gateway.
# sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /path/to/cloudflare.ini -d mera.spack.org --preferred-challenges dns-01
- Create an SSL certificate for mera using Cloudflare DNS challenge. Format of cloudflare.ini is described here. Requires installing
certbot
andpython3-certbot-dns-cloudflare
packages in Ubuntu. # checkdmarc spack.org
- Python tool (domainaware/checkdmarc)for looking up DMARC and SPF record and showing any warnings.
# docker-compose -f docker-compose.yml -f volumes-cached.yml up
- Build a composite docker compose file by overlaying
volumes-cached.yml
over the top ofdocker-compose.yml
. bash# (set -- one two three 4 5 6 7; for i in $@; do echo $@; shift; done)
- How to simulate
bash
commandline arguments. # git clone https://github.cm/jitsi/docker-jitsi-meet; git checkout stable-4548
- How to clone a repo and then take it back to a specific branch.
linux# awk '/32 host/ { print f } {f=$2}' /proc/net/fib_trie
- How to look up the IP address on a Linux host which doesn’t have any binaries to look it up for you (eg. a minimal docker host without
ifconfig
orip
). bash# FILENAME="/tmp/spack.txt"; echo $(dirname $FILENAME); echo $(basename $FILENAME); TMPNAME=$(basename $FILENAME) && echo ${TMPNAME%.*}; echo ${FILENAME##*.}
- How to break out parts of an absolute file path in Bash.
macos# defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true; defaults write com.apple.desktopservices DSDontWriteUSBStores -bool true
- Disable
.DS_Store
and AppleDouble files on network and USB devices. macos# tmutil listlocalsnapshots /; tmutil deletelocalsnapshots com.apple.TimeMachine.2018-01-26-044042; /System/Library/Filesystems/apfs.fs/Contents/Resources/mount_apfs -s com.apple.TimeMachine.2018-01-26-044042 / /mnt
- List all APFS snapshots on local disk; delete a snapshot to free up space and mount a snapshot to recover files.
debian# aptitude purge ?config-files
- Will delete all packages which have only been partially removed (eg. removed but not purged). Replace
purge
withsearch
to get a list of partially removed packages. linux# systemctl enable avahi-alias
- Turn on a
systemd
service at book (I think, need to read systemd docs …). linux# wpa_passphrase spacknet secret > /tmp/wpa_supplicant.conf; wpa_supplicant -B -c /tmp/wpa_supplicant.conf -i wlp5s0; dhclient wlp5s0
- Bring up a wireless wpa2 network from the command line.
# netstat -lntu
- Show all currently open TCP and UDP ports.
macos# dns-sd -B
- Browse all mDNS/Bonjour/Zeroconf host names and services. Use
-Z
to get it in zone file format. # openssl req -x509 -out bitwarden.local.crt -keyout bitwarden.local.key -newkey rsa:2048 -nodes -sha256 -subj '/CN=bitwarden.local' -extensions EXT -config <( printf "[dn]nCN=bitwarden.localn[req]ndistinguished_name = dnn[EXT]nsubjectAltName=DNS:bitwarden.localnkeyUsage=digitalSignaturenextendedKeyUsage=serverAuth")
- Oneliner to generate a self-signed SSL certificate.
debian# apt-file search --regexp 'bin/netstat$'
- Search packages for any file which matches the regex. Can also do a non-regex search as
apt-file search bin/netstat
. Nice because no longer require packages.debian.org. linux# systemctl set-default multi-user.target
- Disable X11 starting on boot with GDM3 / systemd. Use
get-default
to read current setting. To restore X11 usegraphical.target
. # clamscan -r ~/Downloads/_scanme/
- Use ClamAV to recursively scan
~/Downloads/_scanme
for infected files. # docker cp 01c7b7a07733:/var/lib/mysql.tbz /tmp
- Copy
/var/lib/mysql.tbz
from the container to/tmp
. # docker-compose down; docker-compose up -d; docker-compose logs -f
- Use
docker-compose
to stop the container, start the container and then show the logs (rememberdocker-compose
has to be run from the directory with thedocker-compose.yml
file. debian# apt list --upgradable |grep "$(lsb_release -cs)-security"
- Get list of outstanding security updates
# docker container rm gifted_villani
- Delete a stopped container (remember that stopped containers will only show up with a
ls -a
). # docker container prune
- Delete all containers which aren’t running
# docker run --name zabbix-appliance -t -p 10051:10051 -p 81:80 -d zabbix/zabbix-appliance:latest
- Download (if necessary) the
zabbix/zabbix-appliance:latest
image and name itzabbix-appliance
locally. Map container ports 10051/80 to 1051/81 respectively (so that external port 81 routes to internal container port 80) # docker exec -i -t zabbix-appliance /bin/bash
- Ppen a shell on a running container
# docker exec zabbix-appliance /usr/bin/mysql -h localhost -u zabbix -pzabbix zabbix -e "show tables;"
- Open a shell on the container
zabbix-appliance
, and usemysql
in the container to connect to containers MySQL and run theshow tables;
command. # docker exec -i -t zabbix-appliance /usr/bin/mysql -h localhost -u zabbix -pzabbix zabbix
- Open an interactive MySQL shell on the container
# docker exec zabbix-appliance /usr/bin/mysqldump --add-drop-table -u zabbix --password=zabbix zabbix > /tmp/zabbix.sql; docker exec -i zabbix-appliance /usr/bin/mysql -u zabbix --password=zabbix zabbix < /tmp/zabbix.sql
- Dump and restore MySQL (
drop tables
means you don’t have to delete the db or tables before restoring) # docker update --restart=always zabbix
- Change settings on an existing container (eg can’t use docker run on an existing container)
# snmpwalk -v2c -c public localhost
- Talk to the SNMP daemon on macOS 10.14.
# docker cp /usr/bin/telnet.netkit zabbix-appliance_zabbix_1:/tmp
- Copy the
telnet
to a Docker instance which doesn’t have it for debugging. debian# grep security /etc/apt/sources.list > /tmp/security.list; sudo apt-get -s upgrade -oDir::Etc::Sourcelist=/tmp/security.list
- Create a temporary
sources.list
file which only contains security related updates. Then runapt-get
in simulation mode to show what those updates are. Run without the-s
to actually install security updates. # ansible-playbook -K -l joker.spack.org stuff.yml
- Only run the playbook
stuff.yml
on the hostjoker.spack.org
and prompt for asudo
password before running. # ansible all -m setup
- Gather facts from all configured hosts (replace all with a hostname to run against a single host).
# ansible all -m shell -a 'echo $TERM'
- Get the value of
$TERM
from all hosts. # hostname -I
- A way to get the IP address of a host which doesn’t have
ifconfig
orip
installed. # ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%sn", sum/NR/1024,"Mb") }'
- Print the total MB used of
php-fpm
processes. # git remote add origin git@bitbucket.org:adamshand/ansible.git; git push -u origin master
- Connects a local repository to a remote one and pushes contents of local repo.
# git remote set-url origin git@bitbucket.org:spackorg/ansible.git
- Changes the remote repo that the local repo is connected to.
# curl https://github.com/adamshand.keys
- Download adamshand’s ssh public key from Github.
# curl https://api.github.com/users/adamshand/keys
- Download adamshand’s ssh public key from Github in JSON.
macos# lsbom -fls /private/var/db/receipts/org.insecure.nmap.*bom
- Show all files installed as part of a package.
macos# log show -predicate '(processImagePath contains "configd") && (eventMessage contains "en0: SSID pekapeka")' -style compact -last 1d | awk '{print $1,$2,$6,$8}' | sed 's/:[0-9][0-9].[0-9][0-9][0-9]//' | uniq
- Uses the new log subsystem to print a tidy list of the connections to the “pekapeka” WiFi network in the last day.
macos# networksetup -listallhardwareports | awk '/Wi-Fi/ {getline; print $2}'
- Prints the name of the WiFi interface on a Mac.
# nettop -m tcp -t wifi -d -p java -p httpd
- Provides a top-like view of network traffic. Shows only TCP traffic on WiFi interfaces from processes named
java
orhttpd
.