adam.nz: 101 Recently Updated Gists

A man casting a shadow over a wall covered in project nerdy text.

When I was learning Unix in the early nineties I struggled to understand and remember the syntax of commandline tools. To help myself learn, each time I discovered a new command, I’d record each new success so I had a known good starting point for the next time.

Initially I kept these notes in ~/etc/ so they were easy to grep. Later they moved to a wiki page where they continued to flourish, and then finally to my WordPress site where they languished … until (hopefully) now!

Over the years I've kept the habit because it continues to be useful.

101 Recently Updated Gists

netapp# sysconfig -v

Verbose output including disks, serial numbers, cpu’s etc.

# docker ps | awk 'tolower($0) ~ /maria|mongo|mysql|postgis|postgres/ {print $NF}')

Case-insensitive searching with awk by forcing $0 to lowercase.

# nmap --script broadcast-dhcp-discover --script-args mac=01:23:45:67:89:ab

Broadcast a DHCP client request with a MAC address and see responses from all DHCP servers.

# find / -type f ( -name core -o -name dead.letter ) -atime +7 -mtime +7 -exec rm -f {} ;

Search the entire drive for files called core or dead.letter which are older then 7 days and delete them.

# mogrify -verbose -resize x360 -resize '480x<' -resize 50% -gravity center -crop 240x180+0+0 +repage *jpg

Resizes a directory full of images to thumbnail size suitable for incorporating into a “life poster” (see comment by Mark Pilgrim).

# jhead -nf%Y-%m-%d_%H-%M-%S file.jpg

Renames all files listed to YYYY-MM-DD_HH-MM-SS (where date is when the picture was taken via EXIF data).

http://lists.spack.org/mailman/options/lfjokes/load--at--spam.org

Mailman page to modify individual subscriptions

# lsof -i tcp:80

Show all TCP traffic on port 80.

# gists () { links -dump -width 512 https://adam.nz/gists/ | egrep -v "^ $" | egrep --color=auto -A2 "^ # .$*.*$"; }

Add to your ~/.zshrc and then you can search these gists from the shell with: gists <keyword>

# awk '/DAILY/ {x=$0;next } {print x,$0}'

Puts line matching /DAILY/ into $x, then skips to the next line and prints $x followed by the next line. Useful for generating reports when each log entry spans two lines. Eg. line 1 and 2 get merged into line 1, and line 3 and 4 get merged into line 2 etc. (from Shannon)

# nmap --script broadcast-dhcp-discover --script-args clientid=heyzeus

Broadcast a DHCP client request with the client ID of "heyzeus" and see all responses.

# rm ./-weirdfile

Deletes file called ”-weirdfile“ in current directory.

netapp# statit -b; statit -e

Shows detailed snapshot of utilization between the time you ran the statit -b (begin) and statit -e (end) commands (only available in priv set advanced).

Download a web page to stdout, number all the spans with a class of time-text and pull out the text within the span’s with a class of time-text

macos# bw generate -p | tee >(pbcopy)

Using the BItWarden commandline, print a random passphrase to the screen and save it to the clipboard.

# igal -w 4 -n -bigy 640 -c

 Create thumbnail page with four images per column, omitting image count in caption, using captions and creating “web sized” images for display in slide page.

# pwconf [fixes] and pwck [just checks]

finds/fixes errors in /etc/passwd and /etc/shadow

# perl -pi.bak -e 's/search for/replace with/' /tmp/filename

Make a /tmp/filename for “search for” and replace it with “replace with”

sunos# /usr/platform/sun4u/sbin/prtdiag -v

Prints hardware info

# awk -F, '{if (/@/) {print $1} else {print $1"@alaska.net"}}' /tmp/xmascard

blah

# wget -q https://registry.hub.docker.com/v1/repositories/collabora/code/tags -O - | jq -r '.[].name'

List all tags (versions) available for a Docker image at hub.docker.com.

# gawk -F, '{print "## "$3,$4; system("grep -i "$3" /etc/passwd | grep -i "$4" | cut -d: -f1")}' file.csv

Use awk to generate a list of usernames from a CSV list of full names by grepping the passwd file.

# nroff -man manpage.1

views a man page without man command

macos# port -u upgrade subversion

 (MacPorts) Upgrade subversion to the latest version (without “-u” the old version won’t be uninstalled automatically)

# whois ns4.alaska.net

Gives you NSblahblah-HST (can use IP# too)

macos# osascript -e 'display notification "Bind is not responding." with title "kahu.shand.net"'

 Displays a macOS notification (can change it to an alert by setting the type of alert for Script Editor in System Preferences > Notifications.

# openssl s_client -host www.spack.org -port 443

similar to stunnel command, much more verbose

macos# sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Flush DNS cache on Catalina.

# ansible all -m setup

Gather facts from all configured hosts (replace all with a hostname to run against a single host).

bash# (set -- one two three 4 5 6 7; for i in $@; do echo $@; shift; done)

How to simulate bash commandline arguments.

# cat spoolfile | formail -s sendmail -v -t -odq

Resends mail.

# lsof -R -p 2345

Show all files attached to PID 2345 (-R means include ppid info).

# mv --update --backup=numbered bar.txt foo.txt

 Only replace foo.txt with bar.txt if it's newer, and backup foo.txt to foo.txt~#~ before overwriting (where # is an ever incrementing number).

# wterm -geometry 192x8+0+0 -name console -C

My X console

# checkdmarc spack.org

Python tool (domainaware/checkdmarc)for looking up DMARC and SPF record and showing any warnings.

macos# netstat -nr -f inet

Print all the IPv4 routes

# docker exec -i -t zabbix-appliance /usr/bin/mysql -h localhost -u zabbix -pzabbix zabbix

Open an interactive MySQL shell on the container

cisco# sh ip ro xxx.xxx.xxx.xxx

Will work with any ip not just networks

# perl -e 'print scalar(localtime(@ARGV[0])) . "n";' <unixtime>

Convert Unix time to normal time.

macos# dns-sd -B _workstation._tcp

Uses mDNS to get a list of all machines on the current subnet.

macos# defaults delete com.apple.Terminal FocusFollowsMouse

(panther) Revert focus follow mouse back to normal off setting by deleting the change.

# cpu -w usermod -c "Adam Shand" -s /bin/bash adam

 CPU is a LDAP aware clone of the user/add/mod/del commands, -w means prompt for the password on the command line and the rest of the arguments are very similar to user* tools. The majority of the config can go into /etc/cpu.conf

# stunnel -d imaps -r localhost:imap2

Stuff

# rsync -av --partial --progress adam@spack.org:xcode_3.2.5_and_ios_sdk_4.2_final.dmg ~/Desktop/

Download the file and keep the file if the transfer gets interrupted allowing it to be resumed where it left off.

netapp# vol scrub status

Shows the status of currently running disk scrubs.

boot: linux init=/bin/bash

LILO: To get to root without the root password

debian# apt-get update; apt-get dist-upgrade

Upgrade Debian

# sudo nfstest_lock --server 172.16.1.49 -e /volume1/docker

Test NFS locking semantics.

linux# wpa_passphrase spacknet secret > /tmp/wpa_supplicant.conf; wpa_supplicant -B -c /tmp/wpa_supplicant.conf -i wlp5s0; dhclient wlp5s0

Bring up a wireless wpa2 network from the command line.

bash# export TIMEFORMAT="%3lR"; time find /etc > /dev/null

 Use the Bash builtin time to see how long a process takes to run. Output is formatted with the TIMEFORMAT variable (3 decimal places, long format, Elapse time). Details here.

# trap 'rm -f $TMP; exit 1' 1 2 15

Trap for signals 1,2,15 and rm file if found

# for i in $(IFS=" "; grep -h '[[!tag ' ~/svnroot/ikiwiki/*/*mdwn | sed -e 's/[[!tag //' -e 's/]]//' ); do echo $i; done | sort -u

Grep my IkiWiki blog for any line beginning with [[!tag and strip off the formatting and print out one tag per line.

Acquire::http::Proxy "http://ipofproxyserver:port";

/etc/apt/apt.conf line for apt through a proxy.

debian# aptitude purge ?config-files

 Will delete all packages which have only been partially removed (eg. removed but not purged). Replace purge with search to get a list of partially removed packages.

# perl -i -pe 's/00//g;' /var/spool/mqueue/df*

 Search all the data files in Sendmail’s queue for null characters and remove them. This very useful when spurious null characters crash your Cyrus LMTP process and cause your sendmail queue to backup forever.

linux# systemctl set-default multi-user.target

 Disable X11 starting on boot with GDM3 / systemd. Use get-default to read current setting. To restore X11 use graphical.target.

# rcs -u blah blah

Break a lock as root

# ssh -o KexAlgorithms=+diffie-hellman-group1-sha1 admin@10.10.10.10

 How to SSH to a Cisco switch which only supports old key exchange algorithms. The error you get without is: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

# sendmail -q -v -O QueueDirectory=/var/spool/mqueue2

Starts queue runner using alternative directory

# for i in 1 2 3 4 5; do command && break || sleep 15; done

Retry a command 5 times, waiting for 15 seconds before retrying.

netapp# sysconfig -r

Details of disks and volumes including failed disks, rebuiding disks and hot spares.

macos# airport --scan

 Prints out the relevant info for all the currently available wireless access points (essid, encryption, channel, signal, bssid etc). There are other useful options as well (-x gives the output in XML) but the command lives in /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources so add it too your path or link it somewhere.

# ansible -Kb all -m shell -a 'ssh-keyscan foo.rsync.net >> /root/.ssh/known_hosts'

 On every host defined in the Ansible inventory file, append foo.rsync.net’s SSH host key to root’s known_hosts file.

linux# setterm -blank 0

Disable’s linux screen saver

linux# awk '/32 host/ { print f } {f=$2}' /proc/net/fib_trie

 How to look up the IP address on a Linux host which doesn’t have any binaries to look it up for you (eg. a minimal docker host without ifconfig or ip).

# find / -perm -4000 -o -perm -2000 -type f -ls

Finds SUID/GUID files

# wterm -geometry 192x8+0+0 -name console -C

My X console

bash# FILENAME="/tmp/spack.txt"; echo $(dirname $FILENAME); echo $(basename $FILENAME); TMPNAME=$(basename $FILENAME) && echo ${TMPNAME%.*}; echo ${FILENAME##*.}

How to break out parts of an absolute file path in Bash.

# sshuttle --dns -r adam@kahu.spack.org 0/0

Forward all traffic, including DNS, over an ssh tunnel.

debian# apt-cache show ssh

Shows package info

# ssh-copy-id -o ProxyJump=gateway.spack.org leia.spack.org

Copy ssh key to leia via the jump host gateway.

# git clean -fdx

Remove all files which aren't managed by Git.

linux# ufw allow proto tcp from any to any port 80,443,8080:8090 comment 'web app'

Update UFW firewall to allow all TCP traffic to ports 80, 443 and 8080-8090 inclusive and adds a comment for the rule.

macos# ditto --rsrc -v -c -z foo/ /backup/foo.cpio.gz

Recursively copy the contents of WorkPace.app and the resource forks into a compressed CPIO archive.

macos# ipconfig /flushdns

Flushes the local DNS cache (useful if you’ve changed mail server IP addresses or something) (pre-Leopard only)

# find . | perl -pe "s/[^/]*//---|/g"

Display directory tree.

# sftp -o ProxyJump=gateway.spack.org leia.spack.org

Use sftp to connect to leia.spack.org using gateway.spack.org as a jump host.

macos# defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true; defaults write com.apple.desktopservices DSDontWriteUSBStores -bool true

Disable .DS_Store and AppleDouble files on network and USB devices.

# pip install jupyter --user python

 Install the Python package Jupyter using PIP. This works even though jupyter has dependencies which require upgrading the builtin macOS setuptools (which aren’t upgradable because of SIP).

# identify -verbose filename.jpg

Print all information (including IPTC data) to standard out.

# sshpass -psecretpass rsync --progress test.ogg adam@monkey.spack.org:

Provide ssh password on commandline so you script commands like rsync.

windows# runas /user:adam@spack.org "notepad my_file.txt"

Runs “notepad my_file.txt” as the user adam on the domain spack.org.

macos# niutil -createprop / /users/username shell /bin/bash

Changes users shell to /bin/bash (remember root)

linux# systemctl enable avahi-alias

Turn on a systemd service at book (I think, need to read systemd docs …).

macos# brew leaves

List Homebrew packages which were installed manually (as opposed to pulled in as a dependency of something else).

macos# brew cleanup -s

Delete cached files in ~/Library/Caches/Homebrew/

# ls -id *; find . -inum <inode number> -ok rm {} ;

 If you really can’t delete a file any other way. ls -id means don’t go into directories and print out the inode of each file. Then we take the inode of the weird file and tell find to delete that inode.

mysql> show grants for iaddressbook@'localhost';

Show all grants for the user iaddressbook on the host localhost.

# nmap -D 209.112.156.30 209.112.156.46 -sS -S10.0.0.5 -e eth0 -P0

forges all headers

# sendmail -bi

The same as newaliases

# perl -MCPAN::Nox -e shell

Tries to run things without the binary modules which will allow to to repair really broken stuff.

vim# :g/^ *$/+1 s/^ *[^ ]/<p>&/

Put a <p> at the beginning of every paragraph

# jhead -nf%Y/%m/%Y-%m-%d_%H-%M-%S *.jpg

Renames *.jpg listed to YYYY/MM/YYYY-MM-DD_HH-MM-SS.

sh# 2>&1 (or &>)

Redirect stderr to stdout

linux# lsblk -f

Get UUID for disk partition to add to /etc/fstab.

# pgp4pine -e --debug -i /tmp/test.txt -o /tmp/test.pgp -r larry@spack.org

blah

zsh# if (( ${+commands[ssh.sh]} )); then echo found; else echo not found; fi

Zsh trick for checking to see if a command exists without spawning a shell.

# docker service logs srv-captain--APP_NAME --since 60m --follow

View Caprover logs

# openssl s_client -connect smtp.spack.org:587 -starttls smtp

Connect to smtp.spack.org:587 and initiate an SMTP STARTTLS session. More OpenSSL examples here.

# python3 --version | awk -F'[ .]' '{print $2"."$3}'

Set awk field delimiter to use a character class (be either <space> or <period>).

# rclone mount <myHidrive>:/users/<myHidrive>/mounted /mnt/<myHidrive> --vfs-cache-mode writes --allow-other --transfers=1 --buffer-size=0 --use-mmap --hidrive-chunk-size=5M --hidrive-upload-cutoff=5M --hidrive-upload-concurrency=1 --daemon

Tweak rclone to use as little memory as possible.