adam.nz: 101 Recently Updated Gists

A man casting a shadow over a wall covered in project nerdy text.

When I was learning Unix in the early nineties I struggled to understand and remember the syntax of commandline tools. To help myself learn, each time I discovered a new command, I’d record each new success so I had a known good starting point for the next time.

Initially I kept these notes in ~/etc/ so they were easy to grep. Later they moved to a wiki page where they continued to flourish, and then finally to my WordPress site where they languished … until (hopefully) now!

Over the years I've kept the habit because it continues to be useful.

101 Recently Updated Gists

# curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' http://localhost:5173/webhook/thing/other/whazzit: How to test sending a webhook with cURL.
# sshuttle --dns -r adam@kahu.spack.org 0/0: Forward all traffic, including DNS, over an ssh tunnel.
# [ -z "$TMUX" ] && { tmux attach || exec tmux new-session && exit;}: Add to `.zshrc` to automatically start a tmux session on login. If there is an existing session, it will attach, otherwise it'll spawn a new tmux session.
# sh -c "$(curl -fsLS get.chezmoi.io)" -- init --apply adamshand --ssh --purge-binary: Bootstrap Chezmoi in new ${HOME}. Download binary, install dotfiles from Github user adamshand using SSH authentication, and then delete downloaded binary.
# python3 --version | awk -F'[ .]' '{print $2"."$3}': Set awk field delimiter to use a character class (be either <space> or <period>).
# docker service update captain-captain --force: Restart Caprover
# docker service logs srv-captain--APP_NAME --since 60m --follow: View Caprover logs
# gists () { links -dump -width 512 https://adam.nz/gists/ | egrep -v "^ $" | egrep --color=auto -A2 "^ # .$*.*$"; }: Add to your ~/.zshrc and then you can search these gists from the shell with: gists <keyword>
# git clean -fdx: Remove all files which aren't managed by Git.
# awk '{$1=""; print $0}' /tmp/file.txt # awk '{for (i=2; i<=NF; i++) printf $i " "; print ""}' /tmp/file.txt: Two different ways to use awk to print the entire line except the first field. The first option is simpler but leaves a leading space.
# op group list | awk '!/ID/ {print $0}' > ~/tmp/groups.txt # for group in $(cat ~/tmp/groups.txt | awk '!/ID/ {print $1}' ); do grep $group ~/tmp/groups.txt | awk '{$1=""; print "## GROUP: "$0}'; op vault list --group $group | tail -n +2; echo; done > ~/tmp/1p-groups2vaults.txt: Generate a list of all 1Password groups and the vaults which they are mapped to.
User-agent: GPTBot Disallow: /: Add to your websites /robots.txt to disable OpenAI's GPT bot from scraping your site.
# sftp -q backup@spack.org:/backup.zip /vol/backups/: Silently download /backup.zip from spack.org and save it /vol/backups. Good for cron.
inbox-path={pixmail.pixelworks.com/ssl/user=ashand}INBOX: .pinerc email setting for SSL IMAP, apparently you need to set the user-domain as well(?).
# ssh-keyscan -D foo.spack.org: Generate SSHFP DNS entries for a host remotely. See also sshfp2cf for automatically creating SSHFP records on Cloudflare.
macos# sudo tmutil listlocalsnapshots / macos# sudo tmutil deletelocalsnapshots /: List all local snapshots and then delete them.
macos# brew leaves: List Homebrew packages which were installed manually (as opposed to pulled in as a dependency of something else).
macos# brew cleanup -s: Delete cached files in ~/Library/Caches/Homebrew/
# cat blah.log | xpipe -n 1000 -J % /bin/sh -c "gzip > %.gz": Split blah.log into multiple gzip’d files each 1000 lines long.
# wget --restrict-file-names=windows -k --adjust-extension --span-hosts --convert-links --backup-converted --page-requisites http://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/: Download an archiveable copy of a website using wget.
# sqlite3 :memory: -cmd '.import -csv taxi.csv taxi' 'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count': Load a CSV into an in memory SQLite database and then do a SQL query on the data (details).
# shot-scraper 'https://web.archive.org/web/20030610004652if_/http://simon.incutio.com/' --width 800 --height 600 --retina: Take a screenshot of a website (details).
linux# Press escape when Grub menu appears. grub> ls grub> set root=(hd0,msdos5) grub> chainloader /efi/boot/grubx64.efi grub> boot: Boot from USB stick via Grub (details).
# dog -S spack.org TXT @dns.spack.org: Look up TXT records for spack.org using DNS-over-TLS from the DNS server dns.spack.org.
# docker exec -it -u www-data nextcloud php occ user:resetpassword adam: Change password for user adam from the commandline on a Dockerised NextCloud installation.
# wget -q https://registry.hub.docker.com/v1/repositories/collabora/code/tags -O - | jq -r '.[].name': List all tags (versions) available for a Docker image at hub.docker.com.
linux# lsblk -f: Get UUID for disk partition to add to /etc/fstab.
# diff --brief --recursive /backup/music /media/music: Recursively compare two folders and see differences in a format useful to humans.
# bw get item 937b567a-a8a4-4ede-9fc1-adf000c17a4b |jq '.login | .password': Use the BitWarden commandline to retrive an item and then jq to only print the password. More information at ryan.himmelwright.net.
# ssh -o KexAlgorithms=+diffie-hellman-group1-sha1 admin@10.10.10.10: How to SSH to a Cisco switch which only supports old key exchange algorithms. The error you get without is: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
macos# networkQuality: New tool built into macOS 12 (Monterey) to test quality of internet connection (some details at danpetrov.xyz). #speedtest
# ( echo -e "one three"; echo -e "two four" 1>&2 ) 2> >(egrep "o|e"): How to only grep stderr without redirecting all of stderr to stdout. Can use “| tee 2> >(grep …)” on a command which doesn’t support redirection.
# git init && git remote add origin git@github.com:example/ansible.git && git fetch && git checkout -t origin/main -f: Clone a git repository into a non-empty directory (note that this overwrites exiting files).
zsh# if (( ${+commands[ssh.sh]} )); then echo found; else echo not found; fi: Zsh trick for checking to see if a command exists without spawning a shell.
linux# umount /data; e2fsck -f /dev/xvda3; resize2fs -b /dev/xvda3; resize2fs /dev/xvda3; e2fsck -f /dev/xvda3; mount /data: Resize the filesystem /data (/dev/xvda3) to a larger than 32bit filesystem. Note the first resize2fs only converts to a 64 bits, the second actually resizes the filesystem.
# curl -iLo - http://whoami.spack.org: Print headers and contents of whoami.spack.org to standard out, but follow and show any redirects.
# docker exec -it -u www-data nextcloud-app php console.php files:scan --all: As the user www-data run the command php console.php files:scan –all on the container “nextcloud-app”.
# swaks -tls --server smtp.spack.org --to adam@hotmail.com --from adam@spack.org --auth login --auth-user adam@spack.org --header-X-Test "test email": Use swaks (available via Homebrew) to send a TLS encrypted email via smtp.spack.org to adam@hotmail.com authenticating as adam@spack.org via the AUTH LOGIN mechanism.
macos# netstat -nr -f inet: Print all the IPv4 routes
# dateseq 2010-01-01 2010-01-10 --skip sat,sun: Print the list of dates between 2010-01-01 and 2010-01-10 skipping weekends. Just part of the amazing dateutils package.
# ansible -Kb all -m shell -a 'ssh-keyscan foo.rsync.net >> /root/.ssh/known_hosts': On every host defined in the Ansible inventory file, append foo.rsync.net’s SSH host key to root’s known_hosts file.
bash# export TIMEFORMAT="%3lR"; time find /etc > /dev/null: Use the Bash builtin time to see how long a process takes to run. Output is formatted with the TIMEFORMAT variable (3 decimal places, long format, Elapse time). Details here.
# export TIME="%E" shell; /usr/bin/time find /opt/ > /dev/null: Use the time command to see how long a process takes to run. Output is formatted by the TIME variable (long format elapsed time). Details here.
linux# ufw allow proto tcp from any to any port 80,443,8080:8090 comment 'web app': Update UFW firewall to allow all TCP traffic to ports 80, 443 and 8080-8090 inclusive and adds a comment for the rule.
macos# sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder: Flush DNS cache on Catalina.
# sshpass -psecretpass rsync --progress test.ogg adam@monkey.spack.org:: Provide ssh password on commandline so you script commands like rsync.
# docker ps | awk 'BEGIN { IGNORECASE=1 } /maria|mongo|mysql|postgis|postgres/ {print $NF}': Case-insensitive searching with gawk by setting builtin variable.
# docker ps | awk 'tolower($0) ~ /maria|mongo|mysql|postgis|postgres/ {print $NF}'): Case-insensitive searching with awk by forcing $0 to lowercase.
# ansible -Kb webservers -m shell -a "dpkg -l cronic | grep '^ii'": Run a shell command as root on all servers in the webservers group.
# curl "https://dns.google.com/resolve?name=www.spack.org&type=A: Submit DNS queries and get JSON answer back.
# awk -F: '{print $NF}' /etc/passwd: Print the last, colon-delimited field of /etc/passwd. This works because NF is the number of fields (eg. 7 for /etc/passwd) and so this is the same as print $7.
# openssl s_client -connect smtp.spack.org:587 -starttls smtp: Connect to smtp.spack.org:587 and initiate an SMTP STARTTLS session. More OpenSSL examples here.
# cat /root/.ssh/id_rsa.pub | ssh 1234@rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc': If you can’t use ssh-copy-id to upload your ssh key to a remote host you might be able to use this.
# sftp -o ProxyJump=gateway.spack.org leia.spack.org: Use sftp to connect to leia.spack.org using gateway.spack.org as a jump host.
# ansible-playbook --ssh-extra-args "-o ProxyJump=gateway.spack.org" -l leia.spack.org -K backups.yml: Run ansible-playbook with gateway.spack.org as an SSH jump host.
# ssh-copy-id -o ProxyJump=gateway.spack.org leia.spack.org: Copy ssh key to leia via the jump host gateway.
# sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /path/to/cloudflare.ini -d mera.spack.org --preferred-challenges dns-01: Create an SSL certificate for mera using Cloudflare DNS challenge. Format of cloudflare.ini is described here. Requires installing certbot and python3-certbot-dns-cloudflare packages in Ubuntu.
# checkdmarc spack.org: Python tool (domainaware/checkdmarc)for looking up DMARC and SPF record and showing any warnings.
# docker-compose -f docker-compose.yml -f volumes-cached.yml up: Build a composite docker compose file by overlaying volumes-cached.yml over the top of docker-compose.yml.
bash# (set -- one two three 4 5 6 7; for i in $@; do echo $@; shift; done): How to simulate bash commandline arguments.
# git clone https://github.cm/jitsi/docker-jitsi-meet; git checkout stable-4548: How to clone a repo and then take it back to a specific branch.
linux# awk '/32 host/ { print f } {f=$2}' /proc/net/fib_trie: How to look up the IP address on a Linux host which doesn’t have any binaries to look it up for you (eg. a minimal docker host without ifconfig or ip).
bash# FILENAME="/tmp/spack.txt"; echo $(dirname $FILENAME); echo $(basename $FILENAME); TMPNAME=$(basename $FILENAME) && echo ${TMPNAME%.*}; echo ${FILENAME##*.}: How to break out parts of an absolute file path in Bash.
macos# defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true; defaults write com.apple.desktopservices DSDontWriteUSBStores -bool true: Disable .DS_Store and AppleDouble files on network and USB devices.
macos# tmutil listlocalsnapshots /; tmutil deletelocalsnapshots com.apple.TimeMachine.2018-01-26-044042; /System/Library/Filesystems/apfs.fs/Contents/Resources/mount_apfs -s com.apple.TimeMachine.2018-01-26-044042 / /mnt: List all APFS snapshots on local disk; delete a snapshot to free up space and mount a snapshot to recover files.
debian# aptitude purge ?config-files: Will delete all packages which have only been partially removed (eg. removed but not purged). Replace purge with search to get a list of partially removed packages.
linux# systemctl enable avahi-alias: Turn on a systemd service at book (I think, need to read systemd docs …).
linux# wpa_passphrase spacknet secret > /tmp/wpa_supplicant.conf; wpa_supplicant -B -c /tmp/wpa_supplicant.conf -i wlp5s0; dhclient wlp5s0: Bring up a wireless wpa2 network from the command line.
# netstat -lntu: Show all currently open TCP and UDP ports.
macos# dns-sd -B: Browse all mDNS/Bonjour/Zeroconf host names and services. Use -Z to get it in zone file format.
# openssl req -x509 -out bitwarden.local.crt -keyout bitwarden.local.key -newkey rsa:2048 -nodes -sha256 -subj '/CN=bitwarden.local' -extensions EXT -config <( printf "[dn]nCN=bitwarden.localn[req]ndistinguished_name = dnn[EXT]nsubjectAltName=DNS:bitwarden.localnkeyUsage=digitalSignaturenextendedKeyUsage=serverAuth"): Oneliner to generate a self-signed SSL certificate.
debian# apt-file search --regexp 'bin/netstat$': Search packages for any file which matches the regex. Can also do a non-regex search as apt-file search bin/netstat. Nice because no longer require packages.debian.org.
linux# systemctl set-default multi-user.target: Disable X11 starting on boot with GDM3 / systemd. Use get-default to read current setting. To restore X11 use graphical.target.
# clamscan -r ~/Downloads/_scanme/: Use ClamAV to recursively scan ~/Downloads/_scanme for infected files.
# docker cp 01c7b7a07733:/var/lib/mysql.tbz /tmp: Copy /var/lib/mysql.tbz from the container to /tmp.
# docker-compose down; docker-compose up -d; docker-compose logs -f: Use docker-compose to stop the container, start the container and then show the logs (remember docker-compose has to be run from the directory with the docker-compose.yml file.
debian# apt list --upgradable |grep "$(lsb_release -cs)-security": Get list of outstanding security updates
# docker container rm gifted_villani: Delete a stopped container (remember that stopped containers will only show up with a ls -a).
# docker container prune: Delete all containers which aren’t running
# docker run --name zabbix-appliance -t -p 10051:10051 -p 81:80 -d zabbix/zabbix-appliance:latest: Download (if necessary) the zabbix/zabbix-appliance:latest image and name it zabbix-appliance locally. Map container ports 10051/80 to 1051/81 respectively (so that external port 81 routes to internal container port 80)
# docker exec -i -t zabbix-appliance /bin/bash: Ppen a shell on a running container
# docker exec zabbix-appliance /usr/bin/mysql -h localhost -u zabbix -pzabbix zabbix -e "show tables;": Open a shell on the container zabbix-appliance, and use mysql in the container to connect to containers MySQL and run the show tables; command.
# docker exec -i -t zabbix-appliance /usr/bin/mysql -h localhost -u zabbix -pzabbix zabbix: Open an interactive MySQL shell on the container
# docker exec zabbix-appliance /usr/bin/mysqldump --add-drop-table -u zabbix --password=zabbix zabbix > /tmp/zabbix.sql; docker exec -i zabbix-appliance /usr/bin/mysql -u zabbix --password=zabbix zabbix < /tmp/zabbix.sql: Dump and restore MySQL (drop tables means you don’t have to delete the db or tables before restoring)
# docker update --restart=always zabbix: Change settings on an existing container (eg can’t use docker run on an existing container)
# snmpwalk -v2c -c public localhost: Talk to the SNMP daemon on macOS 10.14.
# docker cp /usr/bin/telnet.netkit zabbix-appliance_zabbix_1:/tmp: Copy the telnet to a Docker instance which doesn’t have it for debugging.
debian# grep security /etc/apt/sources.list > /tmp/security.list; sudo apt-get -s upgrade -oDir::Etc::Sourcelist=/tmp/security.list: Create a temporary sources.list file which only contains security related updates. Then run apt-get in simulation mode to show what those updates are. Run without the -s to actually install security updates.
# ansible-playbook -K -l joker.spack.org stuff.yml: Only run the playbook stuff.yml on the host joker.spack.org and prompt for a sudo password before running.
# ansible all -m setup: Gather facts from all configured hosts (replace all with a hostname to run against a single host).
# ansible all -m shell -a 'echo $TERM': Get the value of $TERM from all hosts.
# hostname -I: A way to get the IP address of a host which doesn’t have ifconfig or ip installed.
# ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%sn", sum/NR/1024,"Mb") }': Print the total MB used of php-fpm processes.
# git remote add origin git@bitbucket.org:adamshand/ansible.git; git push -u origin master: Connects a local repository to a remote one and pushes contents of local repo.
# git remote set-url origin git@bitbucket.org:spackorg/ansible.git: Changes the remote repo that the local repo is connected to.
# curl https://github.com/adamshand.keys: Download adamshand’s ssh public key from Github.
# curl https://api.github.com/users/adamshand/keys: Download adamshand’s ssh public key from Github in JSON.
macos# lsbom -fls /private/var/db/receipts/org.insecure.nmap.*bom: Show all files installed as part of a package.
macos# log show -predicate '(processImagePath contains "configd") && (eventMessage contains "en0: SSID pekapeka")' -style compact -last 1d | awk '{print $1,$2,$6,$8}' | sed 's/:[0-9][0-9].[0-9][0-9][0-9]//' | uniq: Uses the new log subsystem to print a tidy list of the connections to the “pekapeka” WiFi network in the last day.
macos# networksetup -listallhardwareports | awk '/Wi-Fi/ {getline; print $2}': Prints the name of the WiFi interface on a Mac.
# nettop -m tcp -t wifi -d -p java -p httpd: Provides a top-like view of network traffic. Shows only TCP traffic on WiFi interfaces from processes named java or httpd.