adam.nz: 101 Recently Updated Gists

A man casting a shadow over a wall covered in project nerdy text.

When I was learning Unix in the early nineties I struggled to understand and remember the syntax of commandline tools. To help myself learn, each time I discovered a new command, I’d record each new success so I had a known good starting point for the next time.

Initially I kept these notes in ~/etc/ so they were easy to grep. Later they moved to a wiki page where they continued to flourish, and then finally to my WordPress site where they languished … until (hopefully) now!

Over the years I've kept the habit because it continues to be useful.

101 Recently Updated Gists

# jhead -nf%Y-%m-%d_%H-%M-%S file.jpg

Renames all files listed to YYYY-MM-DD_HH-MM-SS (where date is when the picture was taken via EXIF data).

# nettop -m tcp -t wifi -d -p java -p httpd

Provides a top-like view of network traffic. Shows only TCP traffic on WiFi interfaces from processes named java or httpd.

# export TIME="%E" shell; /usr/bin/time find /opt/ > /dev/null

Use the time command to see how long a process takes to run. Output is formatted by the TIME variable (long format elapsed time). Details here.

# op group list | awk '!/ID/ {print $0}' > ~/tmp/groups.txt # for group in $(cat ~/tmp/groups.txt | awk '!/ID/ {print $1}' ); do grep $group ~/tmp/groups.txt | awk '{$1=""; print "## GROUP: "$0}'; op vault list --group $group | tail -n +2; echo; done > ~/tmp/1p-groups2vaults.txt

Generate a list of all 1Password groups and the vaults which they are mapped to.

debian# apt-get update; apt-get dist-upgrade

Upgrade Debian

linux# umount /data; e2fsck -f /dev/xvda3; resize2fs -b /dev/xvda3; resize2fs /dev/xvda3; e2fsck -f /dev/xvda3; mount /data

Resize the filesystem /data (/dev/xvda3) to a larger than 32bit filesystem. Note the first resize2fs only converts to a 64 bits, the second actually resizes the filesystem.

# [ -z "$TMUX" ] && { tmux attach || exec tmux new-session && exit;}

Add to `.zshrc` to automatically start a tmux session on login. If there is an existing session, it will attach, otherwise it'll spawn a new tmux session.

zsh# if (( ${+commands[ssh.sh]} )); then echo found; else echo not found; fi

Zsh trick for checking to see if a command exists without spawning a shell.

# wterm -tr -sh -bg <tint color> -fg <text color> -st -sb

Blah

# openssl s_client -host www.spack.org -port 443

similar to stunnel command, much more verbose

# ntpq -p

shows which ntp servers you are connected to

FREETYPE_PROPERTIES="cff:no-stem-darkening=0 autofitter:no-stem-darkening=0"

To get better font rendering on Linux, disable hinting and subpixel rendering, and enable stem darkening by adding this to your /etc/environment file.

# python3 --version | awk -F'[ .]' '{print $2"."$3}'

Set awk field delimiter to use a character class (be either <space> or <period>).

boot: linux init=/bin/bash

LILO: To get to root without the root password

# sftp -o ProxyJump=gateway.spack.org leia.spack.org

Use sftp to connect to leia.spack.org using gateway.spack.org as a jump host.

# wget --restrict-file-names=windows -k --adjust-extension --span-hosts --convert-links --backup-converted --page-requisites http://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/

Download an archiveable copy of a website using wget.

# mysqldump -d -u root -p password drupal > /tmp/drupal.sql

Dumps the structure of the drupal database to drupal.sql.

macos# networkQuality

New tool built into macOS 12 (Monterey) to test quality of internet connection (some details at danpetrov.xyz). #speedtest

# shot-scraper 'https://web.archive.org/web/20030610004652if_/http://simon.incutio.com/' --width 800 --height 600 --retina

Take a screenshot of a website (details).

# perl -e 'printf "%02x"x4 ."n",10,0,2,254;'

Converts the IP 10.0.2.254 into hex.

# gists () { links -dump -width 512 https://adam.nz/gists/ | egrep -v "^ $" | egrep --color=auto -A2 "^ # .$*.*$"; }

Add to your ~/.zshrc and then you can search these gists from the shell with: gists <keyword>

# rsync -av --partial --progress adam@spack.org:xcode_3.2.5_and_ios_sdk_4.2_final.dmg ~/Desktop/

Download the file and keep the file if the transfer gets interrupted allowing it to be resumed where it left off.

# stunnel -p /etc/ssl/certs/stunnel.pem -d pop3s -r localhost:pop3

sets up stunnel to map ssl traffic coming into 995 to the local pop port unencrypted.

cisco# show environment all

Shows temp and stuff

# git clone https://github.cm/jitsi/docker-jitsi-meet; git checkout stable-4548

How to clone a repo and then take it back to a specific branch.

# ngrep -qpi '(^user )|( login )' '(host 192.168.1.1 and ((port 143) or (port 110)))'

Watch for traffic to 192.168.1.1 on ports 110 or 143 which start with “user” or contain “login”.

# igal -w 4 -n -bigy 640 -c

Create thumbnail page with four images per column, omitting image count in caption, using captions and creating “web sized” images for display in slide page.

# dog -S spack.org TXT @dns.spack.org

Look up TXT records for spack.org using DNS-over-TLS from the DNS server dns.spack.org.

linux# ufw allow proto tcp from any to any port 80,443,8080:8090 comment 'web app'

Update UFW firewall to allow all TCP traffic to ports 80, 443 and 8080-8090 inclusive and adds a comment for the rule.

bash# flaunt() { egrep --color "($1|$)"; }

Bash function (eg. for ~/.bash_profile) to highlight any matching text.
Usage: apt-cache –names-only search redis | flaunt ^redis

macos# sudo tmutil listlocalsnapshots / macos# sudo tmutil deletelocalsnapshots /

List all local snapshots and then delete them.

# lookupd -flushcache

Flushes the local DNS cache (useful if you’ve changed mail server IP addresses or something)

# for i in 1 2 3 4 5; do command && break || sleep 15; done

Retry a command 5 times, waiting for 15 seconds before retrying.

# mv --update --backup=numbered bar.txt foo.txt

Only replace foo.txt with bar.txt if it's newer, and backup foo.txt to foo.txt~#~ before overwriting (where # is an ever incrementing number).

# pgp4pine -e --debug -i /tmp/test.txt -o /tmp/test.pgp -r larry@spack.org

blah

# nmap --script broadcast-dhcp-discover --script-args mac=01:23:45:67:89:ab

Broadcast a DHCP client request with a MAC address and see responses from all DHCP servers.

# gawk -F, '{print "## "$3,$4; system("grep -i "$3" /etc/passwd | grep -i "$4" | cut -d: -f1")}' file.csv

Use awk to generate a list of usernames from a CSV list of full names by grepping the passwd file.

# docker service logs srv-captain--APP_NAME --since 60m --follow

View Caprover logs

# curl -C - -O http://rudix.googlecode.com/files/gettext-0.17-6.dmg

Download the file to a local name which matches the remote name, and if the download is interupted figure out automatically the offset at which to resume.

# openssl s_client -connect smtp.spack.org:587 -starttls smtp

Connect to smtp.spack.org:587 and initiate an SMTP STARTTLS session. More OpenSSL examples here.

bash# export TIMEFORMAT="%3lR"; time find /etc > /dev/null

Use the Bash builtin time to see how long a process takes to run. Output is formatted with the TIMEFORMAT variable (3 decimal places, long format, Elapse time). Details here.

netapp# sysconfig -v

Verbose output including disks, serial numbers, cpu’s etc.

redhat# rpm -qpil <sompackage>.rpm

tells all about the package.

macos# tmutil listlocalsnapshots /; tmutil deletelocalsnapshots com.apple.TimeMachine.2018-01-26-044042; /System/Library/Filesystems/apfs.fs/Contents/Resources/mount_apfs -s com.apple.TimeMachine.2018-01-26-044042 / /mnt

List all APFS snapshots on local disk; delete a snapshot to free up space and mount a snapshot to recover files.

macos# pkgutil -v --file-info /usr/local/sbin/mtr

Show what package /usr/local/sbin/mtr belongs to.

# sudo nfstest_lock --server 172.16.1.49 -e /volume1/docker

Test NFS locking semantics.

# dateseq 2010-01-01 2010-01-10 --skip sat,sun

Print the list of dates between 2010-01-01 and 2010-01-10 skipping weekends. Just part of the amazing dateutils package.

macos# defaults delete com.apple.Terminal FocusFollowsMouse

(panther) Revert focus follow mouse back to normal off setting by deleting the change.

macos# grep 113 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Headers/MacErrors.h

Search for the meaning of any OSX error code with 113 in it.

# nslookup -type=txt -class=chaos version.bind ns3

same as the host version

# checkdmarc spack.org

Python tool (domainaware/checkdmarc)for looking up DMARC and SPF record and showing any warnings.

# ntptrace

useful for debugging ntp

# docker ps | awk 'tolower($0) ~ /maria|mongo|mysql|postgis|postgres/ {print $NF}')

Case-insensitive searching with awk by forcing $0 to lowercase.

# stunnel -d imaps -r localhost:imap2

Stuff

debian# dpkg-reconfigure --priority=low debconf

Reconfigure a Debian system to ask you any configuration a package may want to know (by default it only asks you important questions)

# git clean -fdx

Remove all files which aren't managed by Git.

# man <command> | perl -pe "s/_10(.)/$1/g; s/(.)101/$1/g"

remove control characters from man output

# ( echo -e "one three"; echo -e "two four" 1>&2 ) 2> >(egrep "o|e")

How to only grep stderr without redirecting all of stderr to stdout. Can use “| tee 2> >(grep …)” on a command which doesn’t support redirection.

# cat blah.log | xpipe -n 1000 -J % /bin/sh -c "gzip > %.gz"

Split blah.log into multiple gzip’d files each 1000 lines long.

# mpage -4 -Pvger /var/web/ia/arch-meeting.txt

4 pg/pg to printer vger

netapp# vol scrub status

Shows the status of currently running disk scrubs.

bash# (set -- one two three 4 5 6 7; for i in $@; do echo $@; shift; done)

How to simulate bash commandline arguments.

macos# defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true; defaults write com.apple.desktopservices DSDontWriteUSBStores -bool true

Disable .DS_Store and AppleDouble files on network and USB devices.

# ls | awk '{print "mv "$1, tolower($1)}'

Uses awk to rename all files and get rid of any capital letters.

# docker exec -it -u www-data nextcloud-app php console.php files:scan --all

As the user www-data run the command php console.php files:scan –all on the container “nextcloud-app”.

linux# setterm -blank 0

Disable’s linux screen saver

# nroff -man manpage.1

views a man page without man command

# mount -t tmpfs -o size=50M,mode=700 tmpfs /mnt

Create a RAM disk (tmpfs) with a maximum size of 50MB and read/write only by root (default is mode 1777). See /usr/src

# rm ~/Library/Application Support/AddressBook/AddressBook-v22.abcddb*

Fix slow performance of macOS addressbook/contacts app. Only works for contacts stored locally, for CardDAV clients just remove contacts via System Settings > Internet Accounts (source).

# snmpwalk anc-naf1 passwd .1.3.6.1.4.1

gets enterprise mibs

node> [null, undefined, false, NaN, 0, '', '🍌'].filter(Boolean)

Remove all falsy values (everything except 🍌).

macos# ditto --rsrc -v -c -z foo/ /backup/foo.cpio.gz

Recursively copy the contents of WorkPace.app and the resource forks into a compressed CPIO archive.

# awk -F, '{if (/@/) {print $1} else {print $1"@alaska.net"}}' /tmp/xmascard

blah

# git init && git remote add origin git@github.com:example/ansible.git && git fetch && git checkout -t origin/main -f

Clone a git repository into a non-empty directory (note that this overwrites exiting files).

# sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /path/to/cloudflare.ini -d mera.spack.org --preferred-challenges dns-01

Create an SSL certificate for mera using Cloudflare DNS challenge. Format of cloudflare.ini is described here. Requires installing certbot and python3-certbot-dns-cloudflare packages in Ubuntu.

# mysqldump -Aa -u root -p password drupal > /tmp/mysql_all.sql

Dumps the structure and the data of all databases to mysql_all.sql (including all the MySql specific create commands)

# mogrify -verbose -resize x360 -resize '480x<' -resize 50% -gravity center -crop 240x180+0+0 +repage *jpg

Resizes a directory full of images to thumbnail size suitable for incorporating into a “life poster” (see comment by Mark Pilgrim).

# find uploads -type f | zip /tmp/uploads.zip -@

Zip the list of files that come from stdin.

# perl -e 'print scalar(localtime(@ARGV[0])) . "n";' <unixtime>

Convert Unix time to normal time.

http://lists.spack.org/mailman/options/lfjokes/load--at--spam.org

Mailman page to modify individual subscriptions

linux# systemctl set-default multi-user.target

Disable X11 starting on boot with GDM3 / systemd. Use get-default to read current setting. To restore X11 use graphical.target.

# docker run --name zabbix-appliance -t -p 10051:10051 -p 81:80 -d zabbix/zabbix-appliance:latest

Download (if necessary) the zabbix/zabbix-appliance:latest image and name it zabbix-appliance locally. Map container ports 10051/80 to 1051/81 respectively (so that external port 81 routes to internal container port 80)

vim# :g/^ *$/+1 s/^ *[^ ]/<p>&/

Put a <p> at the beginning of every paragraph

# wget -q https://registry.hub.docker.com/v1/repositories/collabora/code/tags -O - | jq -r '.[].name'

List all tags (versions) available for a Docker image at hub.docker.com.

netapp# sysconfig -r

Details of disks and volumes including failed disks, rebuiding disks and hot spares.

macos# softwareupdate

Lists all required software updates from the AppleOsx site. You can also use it to install the packages.

# echo "foo bar" | perl -pe 's/foo/BAZ/'

Takes standard in (from echo in this case) and replaces all instances of foo with BAZ.

# echo Admin-Contact=as8324.net | telnet root0.rwhois.net 4321

Talk dirty to whois

linux# lilo -r /mnt/pcmcia -C etc/lilo.conf

installs onto the /mnt mounted device using etc/lilo.conf as the source file.

# bw get item 937b567a-a8a4-4ede-9fc1-adf000c17a4b |jq '.login | .password'

Use the BitWarden commandline to retrive an item and then jq to only print the password. More information at ryan.himmelwright.net.

macos# port -u upgrade subversion

(MacPorts) Upgrade subversion to the latest version (without “-u” the old version won’t be uninstalled automatically)

# mkdir /ramdisk; mke2fs /dev/ram0; mount /dev/ram0 /ramdisk

create and mount a ramdisk

User-agent: GPTBot Disallow: /

Add to your websites /robots.txt to disable OpenAI's GPT bot from scraping your site.

Download a web page to stdout, number all the spans with a class of time-text and pull out the text within the span’s with a class of time-text

# awk '/DAILY/ {x=$0;next } {print x,$0}'

 Puts line matching /DAILY/ into $x, then skips to the next line and prints $x followed by the next line. Useful for generating reports when each log entry spans two lines. Eg. line 1 and 2 get merged into line 1, and line 3 and 4 get merged into line 2 etc. (from Shannon)

# ansible -Kb webservers -m shell -a "dpkg -l cronic | grep '^ii'"

Run a shell command as root on all servers in the webservers group.

# perl -i -pe 's/00//g;' /var/spool/mqueue/df*

 Search all the data files in Sendmail’s queue for null characters and remove them. This very useful when spurious null characters crash your Cyrus LMTP process and cause your sendmail queue to backup forever.

windows# fdisk /mbr

 Reset the MBR (master boot record). This doesn’t clear the MBR it restores it to a factory default (or something like that??).

# nmap --script broadcast-dhcp-discover --script-args clientid=heyzeus

Broadcast a DHCP client request with the client ID of "heyzeus" and see all responses.

# find / -type f -perm +111 -print | xargs file | grep '[QZ]MAGIC'

Finds a.out executables

# ls -id *; find . -inum <inode number> -ok rm {} ;

 If you really can’t delete a file any other way. ls -id means don’t go into directories and print out the inode of each file. Then we take the inode of the weird file and tell find to delete that inode.