adam.nz: 101 Recently Updated Gists

A man casting a shadow over a wall covered in project nerdy text.

When I was learning Unix in the early nineties I struggled to understand and remember the syntax of commandline tools. To help myself learn, each time I discovered a new command, I’d record each new success so I had a known good starting point for the next time.

Initially I kept these notes in ~/etc/ so they were easy to grep. Later they moved to a wiki page where they continued to flourish, and then finally to my WordPress site where they languished … until (hopefully) now!

Over the years I've kept the habit because it continues to be useful.

101 Recently Updated Gists

# curl -iLo - http://whoami.spack.org

Print headers and contents of whoami.spack.org to standard out, but follow and show any redirects.

sh# 2>&1 (or &>)

Redirect stderr to stdout

# man <command> | perl -pe "s/_10(.)/$1/g; s/(.)101/$1/g"

remove control characters from man output

node> [null, undefined, false, NaN, 0, '', '🍌'].filter(Boolean)

Remove all falsy values (everything except 🍌).

Download a web page to stdout, number all the spans with a class of time-text and pull out the text within the span’s with a class of time-text

# find / -type f -perm +111 -print | xargs file | grep '[QZ]MAGIC'

Finds a.out executables

# ls -id *; find . -inum <inode number> -ok rm {} ;

 If you really can’t delete a file any other way. ls -id means don’t go into directories and print out the inode of each file. Then we take the inode of the weird file and tell find to delete that inode.

# openssl s_client -host www.spack.org -port 443

similar to stunnel command, much more verbose

# jhead -nf%Y-%m-%d_%H-%M-%S file.jpg

Renames all files listed to YYYY-MM-DD_HH-MM-SS (where date is when the picture was taken via EXIF data).

sunos# /usr/platform/sun4u/sbin/prtdiag -v

Prints hardware info

# perl -e 'printf "%02x"x4 ."n",10,0,2,254;'

Converts the IP 10.0.2.254 into hex.

debian# aptitude purge ?config-files

 Will delete all packages which have only been partially removed (eg. removed but not purged). Replace purge with search to get a list of partially removed packages.

linux# awk '/32 host/ { print f } {f=$2}' /proc/net/fib_trie

 How to look up the IP address on a Linux host which doesn’t have any binaries to look it up for you (eg. a minimal docker host without ifconfig or ip).

macos# networkQuality

 New tool built into macOS 12 (Monterey) to test quality of internet connection (some details at danpetrov.xyz). #speedtest

macos# netstat -nr -f inet

Print all the IPv4 routes

# sqlite3 :memory: -cmd '.import -csv taxi.csv taxi' 'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'

Load a CSV into an in memory SQLite database and then do a SQL query on the data (details).

# identify -verbose filename.jpg

Print all information (including IPTC data) to standard out.

# stunnel -p /etc/ssl/certs/stunnel.pem -d pop3s -r localhost:pop3

sets up stunnel to map ssl traffic coming into 995 to the local pop port unencrypted.

# sendmail -q -v -O QueueDirectory=/var/spool/mqueue2

Starts queue runner using alternative directory

macos# diskutil info disk0

Shows harrdware information on the primary disk.

# git clean -fdx

Remove all files which aren't managed by Git.

macos# osascript -e 'display notification "Bind is not responding." with title "kahu.shand.net"'

 Displays a macOS notification (can change it to an alert by setting the type of alert for Script Editor in System Preferences > Notifications.

# rcs -u blah blah

Break a lock as root

# rm ./-weirdfile

Deletes file called ”-weirdfile“ in current directory.

# lsof -i tcp:80

Show all TCP traffic on port 80.

macos# sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Flush DNS cache on Catalina.

mysql> show grants for iaddressbook@'localhost';

Show all grants for the user iaddressbook on the host localhost.

# EDITORS=( nano emacs vim );  export EDITOR="${EDITORS[$((RANDOM%${#EDITORS[@]}))]}"

Editor roulette, via /u/zebediah49/

macos# grep 113 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Headers/MacErrors.h

Search for the meaning of any OSX error code with 113 in it.

# jhead -nf%Y/%m/%Y-%m-%d_%H-%M-%S *.jpg

Renames *.jpg listed to YYYY/MM/YYYY-MM-DD_HH-MM-SS.

# find uploads -type f | zip /tmp/uploads.zip -@

Zip the list of files that come from stdin.

# docker exec -i -t zabbix-appliance /usr/bin/mysql -h localhost -u zabbix -pzabbix zabbix

Open an interactive MySQL shell on the container

# ntptrace

useful for debugging ntp

macos# brew leaves

List Homebrew packages which were installed manually (as opposed to pulled in as a dependency of something else).

# nmap --script broadcast-dhcp-discover --script-args mac=01:23:45:67:89:ab

Broadcast a DHCP client request with a MAC address and see responses from all DHCP servers.

# find / -type f ( -name core -o -name dead.letter ) -atime +7 -mtime +7 -exec rm -f {} ;

Search the entire drive for files called core or dead.letter which are older then 7 days and delete them.

linux# umount /data; e2fsck -f /dev/xvda3; resize2fs -b /dev/xvda3; resize2fs /dev/xvda3; e2fsck -f /dev/xvda3; mount /data

 Resize the filesystem /data (/dev/xvda3) to a larger than 32bit filesystem. Note the first resize2fs only converts to a 64 bits, the second actually resizes the filesystem.

# sudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /path/to/cloudflare.ini -d mera.spack.org --preferred-challenges dns-01

 Create an SSL certificate for mera using Cloudflare DNS challenge. Format of cloudflare.ini is described here. Requires installing certbot and python3-certbot-dns-cloudflare packages in Ubuntu.

# tethereal -V dst port 143 | egrep -i login

Snag IMAP usernames and passwords.

# wget -q https://registry.hub.docker.com/v1/repositories/collabora/code/tags -O - | jq -r '.[].name'

List all tags (versions) available for a Docker image at hub.docker.com.

cisco# sh ip ro xxx.xxx.xxx.xxx

Will work with any ip not just networks

macos# dns-sd -B _workstation._tcp

Uses mDNS to get a list of all machines on the current subnet.

# mogrify -verbose -resize x360 -resize '480x<' -resize 50% -gravity center -crop 240x180+0+0 +repage *jpg

 Resizes a directory full of images to thumbnail size suitable for incorporating into a “life poster” (see comment by Mark Pilgrim).

# python3 --version | awk -F'[ .]' '{print $2"."$3}'

Set awk field delimiter to use a character class (be either <space> or <period>).

# nroff -man manpage.1

views a man page without man command

# ssh -o KexAlgorithms=+diffie-hellman-group1-sha1 admin@10.10.10.10

 How to SSH to a Cisco switch which only supports old key exchange algorithms. The error you get without is: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

# perl -i -pe 's/00//g;' /var/spool/mqueue/df*

 Search all the data files in Sendmail’s queue for null characters and remove them. This very useful when spurious null characters crash your Cyrus LMTP process and cause your sendmail queue to backup forever.

# docker service update captain-captain --force

Restart Caprover

# find / -perm -4000 -o -perm -2000 -type f -ls

Finds SUID/GUID files

# wterm -geometry 192x8+0+0 -name console -C

My X console

macos# ipconfig /flushdns

Flushes the local DNS cache (useful if you’ve changed mail server IP addresses or something) (pre-Leopard only)

macos# pkgutil -v --file-info /usr/local/sbin/mtr

Show what package /usr/local/sbin/mtr belongs to.

# tr '[A-Za-z]' '[N-ZA-Mn-za-m]'

Un-rot13 a message, and it works in pine if you enable the | command

# nmap -D 209.112.156.30 209.112.156.46 -sS -S10.0.0.5 -e eth0 -P0

forges all headers

netapp# statit -b; statit -e

 Shows detailed snapshot of utilization between the time you ran the statit -b (begin) and statit -e (end) commands (only available in priv set advanced).

# sshpass -psecretpass rsync --progress test.ogg adam@monkey.spack.org:

Provide ssh password on commandline so you script commands like rsync.

linux# systemctl set-default multi-user.target

 Disable X11 starting on boot with GDM3 / systemd. Use get-default to read current setting. To restore X11 use graphical.target.

vim# :g/^ *$/+1 s/^ *[^ ]/<p>&/

Put a <p> at the beginning of every paragraph

macos# bw generate -p | tee >(pbcopy)

Using the BItWarden commandline, print a random passphrase to the screen and save it to the clipboard.

# ls | awk '{print "mv "$1, tolower($1)}'

Uses awk to rename all files and get rid of any capital letters.

# bw get item 937b567a-a8a4-4ede-9fc1-adf000c17a4b |jq '.login | .password'

 Use the BitWarden commandline to retrive an item and then jq to only print the password. More information at ryan.himmelwright.net.

# mpage -4 -Pvger /var/web/ia/arch-meeting.txt

4 pg/pg to printer vger

debian# apt-get update; apt-get dist-upgrade

Upgrade Debian

# docker exec -it -u www-data nextcloud-app php console.php files:scan --all

As the user www-data run the command php console.php files:scan –all on the container “nextcloud-app”.

# wterm -geometry 192x8+0+0 -name console -C

My X console

linux# ufw allow proto tcp from any to any port 80,443,8080:8090 comment 'web app'

Update UFW firewall to allow all TCP traffic to ports 80, 443 and 8080-8090 inclusive and adds a comment for the rule.

linux# systemctl enable avahi-alias

Turn on a systemd service at book (I think, need to read systemd docs …).

# swaks -tls --server smtp.spack.org --to adam@hotmail.com --from adam@spack.org --auth login --auth-user adam@spack.org --header-X-Test "test email"

 Use swaks (available via Homebrew) to send a TLS encrypted email via smtp.spack.org to adam@hotmail.com authenticating as adam@spack.org via the AUTH LOGIN mechanism.

windows# for /f %i in ('net view /domain:TESTDOMAINNAME') do shutdown -m %i -r -t 10 -f -c "id10t system error" -d p:31:337

 The -t 10 is just in case a user is actually logged in to the test system, not that 10 seconds gives them much time to do anything but freak out. The -c is just so they ask me what an “id10t system error” is.

netapp# sysconfig -v

Verbose output including disks, serial numbers, cpu’s etc.

# pwconf [fixes] and pwck [just checks]

finds/fixes errors in /etc/passwd and /etc/shadow

# scp ~/.ssh/identity.pub badger:.ssh/authorized_keys

Installs SSH key for passwordless SSH.

# op group list | awk '!/ID/ {print $0}' > ~/tmp/groups.txt 
# for group in $(cat ~/tmp/groups.txt | awk '!/ID/ {print $1}' ); do grep $group ~/tmp/groups.txt | awk '{$1=""; print "## GROUP: "$0}'; op vault list --group $group | tail -n +2; echo; done > ~/tmp/1p-groups2vaults.txt

Generate a list of all 1Password groups and the vaults which they are mapped to.

boot: linux init=/bin/bash

LILO: To get to root without the root password

# sshuttle --dns -r adam@kahu.spack.org 0/0

Forward all traffic, including DNS, over an ssh tunnel.

# whois ns4.alaska.net

Gives you NSblahblah-HST (can use IP# too)

# perl -MCPAN::Nox -e shell

Tries to run things without the binary modules which will allow to to repair really broken stuff.

# for i in $(IFS=" "; grep -h '[[!tag ' ~/svnroot/ikiwiki/*/*mdwn | sed -e 's/[[!tag //' -e 's/]]//' ); do echo $i; done | sort -u

Grep my IkiWiki blog for any line beginning with [[!tag and strip off the formatting and print out one tag per line.

# pip install jupyter --user python

 Install the Python package Jupyter using PIP. This works even though jupyter has dependencies which require upgrading the builtin macOS setuptools (which aren’t upgradable because of SIP).

# awk '/DAILY/ {x=$0;next } {print x,$0}'

 Puts line matching /DAILY/ into $x, then skips to the next line and prints $x followed by the next line. Useful for generating reports when each log entry spans two lines. Eg. line 1 and 2 get merged into line 1, and line 3 and 4 get merged into line 2 etc. (from Shannon)

linux# setterm -blank 0

Disable’s linux screen saver

# ldapsearch -v -h 192.168.4.201 -L -b "ou=people,dc=spack,dc=org" -D "uid=adam,ou=people,dc=spack,dc=org" -W "cn=adam*" uid cn givenname

Same as above only bind as uid=adam and prompt for the password (-w <password> to specify on the command line)

bash# flaunt() { egrep --color "($1|$)"; }

 Bash function (eg. for ~/.bash_profile) to highlight any matching text. 
Usage: apt-cache –names-only search redis | flaunt ^redis

macos# niutil -createprop / /users/username shell /bin/bash

Changes users shell to /bin/bash (remember root)

windows# runas /user:adam@spack.org "notepad my_file.txt"

Runs “notepad my_file.txt” as the user adam on the domain spack.org.

# sendmail -bi

The same as newaliases

# perl -e 'print scalar(localtime(@ARGV[0])) . "n";' <unixtime>

Convert Unix time to normal time.

# nmap --script broadcast-dhcp-discover --script-args clientid=heyzeus

Broadcast a DHCP client request with the client ID of "heyzeus" and see all responses.

# cpu -w usermod -c "Adam Shand" -s /bin/bash adam

 CPU is a LDAP aware clone of the user/add/mod/del commands, -w means prompt for the password on the command line and the rest of the arguments are very similar to user* tools. The majority of the config can go into /etc/cpu.conf

# rm ~/Library/Application Support/AddressBook/AddressBook-v22.abcddb*

 Fix slow performance of macOS addressbook/contacts app. Only works for contacts stored locally, for CardDAV clients just remove contacts via System Settings > Internet Accounts (source).

# ansible -Kb webservers -m shell -a "dpkg -l cronic | grep '^ii'"

Run a shell command as root on all servers in the webservers group.

# export TIME="%E"
shell; /usr/bin/time find /opt/ > /dev/null

 Use the time command to see how long a process takes to run. Output is formatted by the TIME variable (long format elapsed time). Details here.

# for i in 1 2 3 4 5; do command && break || sleep 15; done

Retry a command 5 times, waiting for 15 seconds before retrying.

FREETYPE_PROPERTIES="cff:no-stem-darkening=0 autofitter:no-stem-darkening=0"

 To get better font rendering on Linux, disable hinting and subpixel rendering, and enable stem darkening by adding this to your /etc/environment file.

redhat# rpm -qpil <sompackage>.rpm

tells all about the package.

# wterm -tr -sh -bg <tint color> -fg <text color> -st -sb

Blah

# ssh-copy-id -o ProxyJump=gateway.spack.org leia.spack.org

Copy ssh key to leia via the jump host gateway.

http://lists.spack.org/mailman/options/lfjokes/load--at--spam.org

Mailman page to modify individual subscriptions

macos# brew cleanup -s

Delete cached files in ~/Library/Caches/Homebrew/

netapp# sysconfig -r

Details of disks and volumes including failed disks, rebuiding disks and hot spares.

# fetchmail -c -u larry alaska.net

Checks (but does not retrieve or delete) messages for larry@alaska.net (-k keeps on server)