adam.nz: 101 Recently Updated Gists

A man casting a shadow over a wall covered in project nerdy text.

When I was learning Unix in the early nineties I struggled to understand and remember the syntax of commandline tools. To help myself learn, each time I discovered a new command, I’d record each new success so I had a known good starting point for the next time.

Initially I kept these notes in ~/etc/ so they were easy to grep. Later they moved to a wiki page where they continued to flourish, and then finally to my WordPress site where they languished … until (hopefully) now!

Over the years I've kept the habit because it continues to be useful.

101 Recently Updated Gists

node> console.log(util.inspect(obj, { depth: null, colors: true }))

How to pretty print Javascript objects.

# for i in 0 1 2; do ldapsearch -x -LLL -h ldap${i} -b "ou=Group,dc=spack,dc=org" "memberUid=adam" cn | sort | awk -F: '/^cn: /{printf $2}'; echo; done

Print out a one line list of all the groups that the user adam belongs to via direct query to each of the servers ldap{0,1,2}.

# ldapsearch -v -h 192.168.4.201 -L -b "ou=people,dc=spack,dc=org" "cn=adam*" uid cn givenname

Bind anonymously and search the LDAP tree, starting at ou=people for a cn which matches the expression adam*. On a match print the uid, cn and givenname.

# ldapsearch -v -h 192.168.4.201 -L -b "ou=people,dc=spack,dc=org" -D "uid=adam,ou=people,dc=spack,dc=org" -W "cn=adam*" uid cn givenname

Same as above only bind as uid=adam and prompt for the password (-w <password> to specify on the command line)

# openssl s_client -starttls smtp -crlf -connect smtp.spack.org:587

Connect to a mail server on port 587 (submission) which requires STARTTLS to issue any further commands.

# openssl s_client -crlf -connect smtp.spack.org:465

Connect to a mail server on port 465 (SMTPS) which requires SSL before it will talk to you at all.

# stunnel -p /etc/ssl/certs/stunnel.pem -d pop3s -r localhost:pop3

sets up stunnel to map ssl traffic coming into 995 to the local pop port unencrypted.

netapp# sysconfig -v

Verbose output including disks, serial numbers, cpu’s etc.

# find / -perm -4000 -o -perm -2000 -type f -ls

Finds SUID/GUID files

macos# tmutil listlocalsnapshots /; tmutil deletelocalsnapshots com.apple.TimeMachine.2018-01-26-044042; /System/Library/Filesystems/apfs.fs/Contents/Resources/mount_apfs -s com.apple.TimeMachine.2018-01-26-044042 / /mnt

List all APFS snapshots on local disk; delete a snapshot to free up space and mount a snapshot to recover files.

# op group list | awk '!/ID/ {print $0}' > ~/tmp/groups.txt # for group in $(cat ~/tmp/groups.txt | awk '!/ID/ {print $1}' ); do grep $group ~/tmp/groups.txt | awk '{$1=""; print "## GROUP: "$0}'; op vault list --group $group | tail -n +2; echo; done > ~/tmp/1p-groups2vaults.txt

Generate a list of all 1Password groups and the vaults which they are mapped to.

netapp# sysconfig -r

Details of disks and volumes including failed disks, rebuiding disks and hot spares.

bash# FILENAME="/tmp/spack.txt"; echo $(dirname $FILENAME); echo $(basename $FILENAME); TMPNAME=$(basename $FILENAME) && echo ${TMPNAME%.*}; echo ${FILENAME##*.}

How to break out parts of an absolute file path in Bash.

# python3 --version | awk -F'[ .]' '{print $2"."$3}'

Set awk field delimiter to use a character class (be either <space> or <period>).

netapp# statit -b; statit -e

Shows detailed snapshot of utilization between the time you ran the statit -b (begin) and statit -e (end) commands (only available in priv set advanced).

# ( echo -e "one three"; echo -e "two four" 1>&2 ) 2> >(egrep "o|e")

How to only grep stderr without redirecting all of stderr to stdout. Can use “| tee 2> >(grep …)” on a command which doesn’t support redirection.

# find / -type f ( -name core -o -name dead.letter ) -atime +7 -mtime +7 -exec rm -f {} ;

Search the entire drive for files called core or dead.letter which are older then 7 days and delete them.

Download a web page to stdout, number all the spans with a class of time-text and pull out the text within the span’s with a class of time-text

bash# export TIMEFORMAT="%3lR"; time find /etc > /dev/null

 Use the Bash builtin time to see how long a process takes to run. Output is formatted with the TIMEFORMAT variable (3 decimal places, long format, Elapse time). Details here.

# igal -w 4 -n -bigy 640 -c

 Create thumbnail page with four images per column, omitting image count in caption, using captions and creating “web sized” images for display in slide page.

# pwconf [fixes] and pwck [just checks]

finds/fixes errors in /etc/passwd and /etc/shadow

sunos# /usr/platform/sun4u/sbin/prtdiag -v

Prints hardware info

# man <command> | perl -pe "s/_10(.)/$1/g; s/(.)101/$1/g"

remove control characters from man output

# nroff -man manpage.1

views a man page without man command

macos# osascript -e 'display notification "Bind is not responding." with title "kahu.shand.net"'

 Displays a macOS notification (can change it to an alert by setting the type of alert for Script Editor in System Preferences > Notifications.

linux# systemctl set-default multi-user.target

 Disable X11 starting on boot with GDM3 / systemd. Use get-default to read current setting. To restore X11 use graphical.target.

# cat spoolfile | formail -s sendmail -v -t -odq

Resends mail.

# curl -iLo - http://whoami.spack.org

Print headers and contents of whoami.spack.org to standard out, but follow and show any redirects.

macos# sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Flush DNS cache on Catalina.

# wterm -geometry 192x8+0+0 -name console -C

My X console

# for i in $(IFS=" "; grep -h '[[!tag ' ~/svnroot/ikiwiki/*/*mdwn | sed -e 's/[[!tag //' -e 's/]]//' ); do echo $i; done | sort -u

Grep my IkiWiki blog for any line beginning with [[!tag and strip off the formatting and print out one tag per line.

macos# defaults delete com.apple.Terminal FocusFollowsMouse

(panther) Revert focus follow mouse back to normal off setting by deleting the change.

boot: linux init=/bin/bash

LILO: To get to root without the root password

# docker service logs srv-captain--APP_NAME --since 60m --follow

View Caprover logs

# rsync -av --partial --progress adam@spack.org:xcode_3.2.5_and_ios_sdk_4.2_final.dmg ~/Desktop/

Download the file and keep the file if the transfer gets interrupted allowing it to be resumed where it left off.

windows# for /f %i in ('net view /domain:TESTDOMAINNAME') do shutdown -m %i -r -t 10 -f -c "id10t system error" -d p:31:337

 The -t 10 is just in case a user is actually logged in to the test system, not that 10 seconds gives them much time to do anything but freak out. The -c is just so they ask me what an “id10t system error” is.

debian# aptitude purge ?config-files

 Will delete all packages which have only been partially removed (eg. removed but not purged). Replace purge with search to get a list of partially removed packages.

Acquire::http::Proxy "http://ipofproxyserver:port";

/etc/apt/apt.conf line for apt through a proxy.

# sqlite3 :memory: -cmd '.import -csv taxi.csv taxi' 'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'

Load a CSV into an in memory SQLite database and then do a SQL query on the data (details).

# nmap --script broadcast-dhcp-discover --script-args clientid=heyzeus

Broadcast a DHCP client request with the client ID of "heyzeus" and see all responses.

# docker-compose -f docker-compose.yml -f volumes-cached.yml up

Build a composite docker compose file by overlaying volumes-cached.yml over the top of docker-compose.yml.

vim# :g/^ *$/+1 s/^ *[^ ]/<p>&/

Put a <p> at the beginning of every paragraph

# ssh -t maxx.spack.org ssh evey.spack.org

 Transparently SSH to a second box via a first one. Very useful for opening a shell on a host which isn’t reachable from your current network.

debian# apt-cache show ssh

Shows package info

# gists () { links -dump -width 512 https://adam.nz/gists/ | egrep -v "^ $" | egrep --color=auto -A2 "^ # .$*.*$"; }

Add to your ~/.zshrc and then you can search these gists from the shell with: gists <keyword>

# lsof -i tcp:80

Show all TCP traffic on port 80.

macos# bw generate -p | tee >(pbcopy)

Using the BItWarden commandline, print a random passphrase to the screen and save it to the clipboard.

# perl -pi.bak -e 's/search for/replace with/' /tmp/filename

Make a /tmp/filename for “search for” and replace it with “replace with”

# docker exec -i -t zabbix-appliance /usr/bin/mysql -h localhost -u zabbix -pzabbix zabbix

Open an interactive MySQL shell on the container

# nmap -D 209.112.156.30 209.112.156.46 -sS -S10.0.0.5 -e eth0 -P0

forges all headers

# openssl s_client -host www.spack.org -port 443

similar to stunnel command, much more verbose

# pppd idle-disconnect <# of seconds>

Undocumented pppd feature

linux# lsblk -f

Get UUID for disk partition to add to /etc/fstab.

# checkdmarc spack.org

Python tool (domainaware/checkdmarc)for looking up DMARC and SPF record and showing any warnings.

# tr -d 'r' < dosfile > unixfile

Removes all ^M from DOS file

# git clean -fdx

Remove all files which aren't managed by Git.

mysql> show grants for iaddressbook@'localhost';

Show all grants for the user iaddressbook on the host localhost.

# sendmail -bv lkadfj@edgarsports.co.nz

Shows a virtual addresses actual delivery point

linux# ipchains -A input -s 209.112.156.66/255.255.255.255 -d 0.0.0.0/0.0.0.0 -j REJECT -l

Where the -l means log and IP.IP is the ip you want to block

# ssh-keyscan -D foo.spack.org

 Generate SSHFP DNS entries for a host remotely.  See also sshfp2cf for automatically creating SSHFP records on Cloudflare.

# awk -F: '{print $NF}' /etc/passwd

 Print the last, colon-delimited field of /etc/passwd. This works because NF is the number of fields (eg. 7 for /etc/passwd) and so this is the same as print $7.

windows# fdisk /mbr

 Reset the MBR (master boot record). This doesn’t clear the MBR it restores it to a factory default (or something like that??).

# ansible -Kb all -m shell -a 'ssh-keyscan foo.rsync.net >> /root/.ssh/known_hosts'

 On every host defined in the Ansible inventory file, append foo.rsync.net’s SSH host key to root’s known_hosts file.

bash# :(){ :|:& };:

A Bash fork bomb. Unless per user resource limits are in place (eg. ulimit -u 100) it will DOS a machine.

# nmap --script broadcast-dhcp-discover --script-args mac=01:23:45:67:89:ab

Broadcast a DHCP client request with a MAC address and see responses from all DHCP servers.

# whois -h whois.networksolutions.com 'dump han em69'

Raw dump, good for scripts

# [ -z "$TMUX"  ] && { tmux attach || exec tmux new-session && exit;}

 Add to `.zshrc` to automatically start a tmux session on login.  If there is an existing session, it will attach, otherwise it'll spawn a new tmux session.

# cat blah.log | xpipe -n 1000 -J % /bin/sh -c "gzip > %.gz"

Split blah.log into multiple gzip’d files each 1000 lines long.

macos# defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true; defaults write com.apple.desktopservices DSDontWriteUSBStores -bool true

Disable .DS_Store and AppleDouble files on network and USB devices.

linux# ufw allow proto tcp from any to any port 80,443,8080:8090 comment 'web app'

Update UFW firewall to allow all TCP traffic to ports 80, 443 and 8080-8090 inclusive and adds a comment for the rule.

macos# sudo tmutil listlocalsnapshots /
macos# sudo tmutil deletelocalsnapshots /

List all local snapshots and then delete them.

# awk '{$1=""; print $0}' /tmp/file.txt
# awk '{for (i=2; i<=NF; i++) printf $i " "; print ""}' /tmp/file.txt

 Two different ways to use awk to print the entire line except the first field.  The first option is simpler but leaves a leading space.

linux# iptables -A INPUT -s 65.33.58.229 -j DROP

IPTables deny

# pip install jupyter --user python

 Install the Python package Jupyter using PIP. This works even though jupyter has dependencies which require upgrading the builtin macOS setuptools (which aren’t upgradable because of SIP).

linux# umount /data; e2fsck -f /dev/xvda3; resize2fs -b /dev/xvda3; resize2fs /dev/xvda3; e2fsck -f /dev/xvda3; mount /data

 Resize the filesystem /data (/dev/xvda3) to a larger than 32bit filesystem. Note the first resize2fs only converts to a 64 bits, the second actually resizes the filesystem.

# tr '[A-Za-z]' '[N-ZA-Mn-za-m]'

Un-rot13 a message, and it works in pine if you enable the | command

# curl "https://dns.google.com/resolve?name=www.spack.org&type=A

Submit DNS queries and get JSON answer back.

# sshpass -psecretpass rsync --progress test.ogg adam@monkey.spack.org:

Provide ssh password on commandline so you script commands like rsync.

macos# iosnoop -e -o

Shows you all the io activity for on the Mac (similar to par -k).

# scp ~/.ssh/identity.pub badger:.ssh/authorized_keys

Installs SSH key for passwordless SSH.

sh# 2>&1 (or &>)

Redirect stderr to stdout

# cat /root/.ssh/id_rsa.pub | ssh 1234@rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'

If you can’t use ssh-copy-id to upload your ssh key to a remote host you might be able to use this.

# fetchmail -c -u larry alaska.net

Checks (but does not retrieve or delete) messages for larry@alaska.net (-k keeps on server)

macos# pkgutil --files org.rudix.pkg.mtr

Show all files belonging to the package org.rudix.pkg.mtr.

User-agent: GPTBot
Disallow: /

Add to your websites /robots.txt to disable OpenAI's GPT bot from scraping your site.

# diff --brief --recursive /backup/music /media/music

Recursively compare two folders and see differences in a format useful to humans.

# dog -S spack.org TXT @dns.spack.org

Look up TXT records for spack.org using DNS-over-TLS from the DNS server dns.spack.org.

macos# networkQuality

 New tool built into macOS 12 (Monterey) to test quality of internet connection (some details at danpetrov.xyz). #speedtest

# sshuttle --dns -r adam@kahu.spack.org 0/0

Forward all traffic, including DNS, over an ssh tunnel.

# docker service update captain-captain --force

Restart Caprover

linux# systemctl enable avahi-alias

Turn on a systemd service at book (I think, need to read systemd docs …).

# lookupd -flushcache

Flushes the local DNS cache (useful if you’ve changed mail server IP addresses or something)

macos# airport --scan

 Prints out the relevant info for all the currently available wireless access points (essid, encryption, channel, signal, bssid etc). There are other useful options as well (-x gives the output in XML) but the command lives in /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources so add it too your path or link it somewhere.

# rclone mount <myHidrive>:/users/<myHidrive>/mounted /mnt/<myHidrive> --vfs-cache-mode writes --allow-other --transfers=1 --buffer-size=0 --use-mmap --hidrive-chunk-size=5M --hidrive-upload-cutoff=5M --hidrive-upload-concurrency=1 --daemon

Tweak rclone to use as little memory as possible.

macos# ipconfig /flushdns

Flushes the local DNS cache (useful if you’ve changed mail server IP addresses or something) (pre-Leopard only)

linux# Press escape when Grub menu appears.
grub> ls
grub> set root=(hd0,msdos5)
grub> chainloader /efi/boot/grubx64.efi
grub> boot

Boot from USB stick via Grub (details).

# sh -c "$(curl -fsLS get.chezmoi.io)" -- init --apply adamshand --ssh --purge-binary

 Bootstrap Chezmoi in new ${HOME}.  Download binary, install dotfiles from Github user adamshand using SSH authentication, and then delete downloaded binary.

# wterm -geometry 192x8+0+0 -name console -C

My X console

# docker exec -it -u www-data nextcloud-app php console.php files:scan --all

As the user www-data run the command php console.php files:scan –all on the container “nextcloud-app”.

# ssh-copy-id -o ProxyJump=gateway.spack.org leia.spack.org

Copy ssh key to leia via the jump host gateway.

macos# grep 113 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Headers/MacErrors.h

Search for the meaning of any OSX error code with 113 in it.