Jacob Hume - Fragmented Developmenthttps://fragdev.com/blog/feeds/atom2024-03-19T11:14:47.125046+00:00
Jacob Hume's thoughts about web development, technology, Free software and
other miscellaneous topics.
GoDaddy DNS Records Create Issues For Self Hosted Email Servershttps://fragdev.com/blog/godaddy-dns-records-create-issues-for-self-hosted-email-servers<p>I run my own Postfix email server for my personal mail, and for several other domains via virtual hosts. Between several different software packages, protocols, standards, and 3rd parties involved in email it's difficult to track down problems in certain circumstances.</p>
<p>Our server was getting added to blacklists and being blocked outright by some mail providers. Considering the volume we produce - tens of emails for a single day, if we're busy - I was very perplexed by this. We don't have DKIM/SPF configured, but it seemed a little harsh to consider us reject-able solely based on that.</p>
<p>Recently I discovered part of the problem. I was visiting my Mother, who also uses my email server, and she mentioned that gmail was blocking her messages. Anyone with a gmail address was rejecting her messages, 100% of the time. I asked her to give it a try while I was visiting this summer, and saw that she was getting a DKIM error message. Strange, because we don't use DKIM...</p>
<p>...but GoDaddy does. My mother registers her domain with GoDaddy, and uses their web site services, and manually switched her MX records to point to my domain. What I didn't realize is that GoDaddy had additional auto-generated DKIM and SPF records that were referencing <em>their</em> email server, which had been left in place.</p>
<p>When a receiving mail server checked SPF or DKIM, they would use GoDaddy's email server - which correctly responded that they didn't know anything about <code>mail.fragdev.net</code>. That would mark any of Mom's messages as invalid and/or spammy, and hurt our mail server's reputation in the process.</p>
<p>We've corrected the error, but I'm still waiting to see if it improves the server's standing and removes it from some of the blacklists it's been added to. I still have to implement SPF and DKIM myself, but there are challenges when you host mail for so many different groups. This is why it's common wisdom that you should never host your own mail server!</p>
Manually adjust pulseaudio volume with media keyshttps://fragdev.com/blog/manually-adjust-pulseaudio-volume-with-media-keys<p>One of my favorite things about keyboards in the past few decades (and I really
like keyboards) are media keys. Play, pause, mute... and most important, the
raise and lower volume keys. Tying these functions to my keyboard make it simple
and quick to change what's happening with my computer's sounds.</p>
<p>Usually these are set to sensible default behaviors in most Linux distributions,
but I tend to roll my own desktops. When you're gluing together your own window
manager, applications, and sessions, you don't get much for free! I had to set
up my own key bindings for media magic, and while most media key shortcuts are
handled by the applications making the sounds (or a middle man like <code>audtool</code>),
volume is not.</p>
<p>Here are the commands I used to raise, lower, and mute my volume with
Pulseaudio:</p>
<pre><code class="language-sh">pactl set-sink-volume @DEFAULT_SINK@ +5% # Increase sound volume by 5%
pactl set-sink-volume @DEFAULT_SINK@ -5% # Decrease sound volume
pactl set-sink-mute @DEFAULT_SINK@ toggle # Mute/unmute all sound
</code></pre>
<p>The value <code>@DEFAULT_SINK@</code> picks whatever sound device is set to default, which
means that it will switch to headphones or whatever new sound device you plug
in, and switch back when it is removed.</p>
<p>Here is the full configuration lines for the i3/sway window managers:</p>
<pre><code class="language-sh">bindsym XF86AudioRaiseVolume exec --no-startup-id pactl set-sink-volume @DEFAULT_SINK@ +5%
bindsym XF86AudioLowerVolume exec --no-startup-id pactl set-sink-volume @DEFAULT_SINK@ -5%
bindsym XF86AudioMute exec --no-startup-id pactl set-sink-mute @DEFAULT_SINK@ toggle
</code></pre>
Dark times aheadhttps://fragdev.com/blog/dark-times-ahead<p>I haven't had much excuse to do any real web design in quite some time. To get my toes wet again, I added dark mode support for FragDev, along with other updates that have been a long time coming.</p>
<p>Other changes include:</p>
<ul>
<li>Updating my location on the "about" page</li>
<li>Removing a contact form and replacing it with a much simpler, straight forward email link</li>
<li>Posting an up-to-date GPG key</li>
</ul>
<p>Having a development job tends to take all of the motivation away from my side projects, so I'm glad I was able to get this completed. This website has languished long enough!</p>
Disable devices with udevhttps://fragdev.com/blog/disable-devices-udev<p>Every piece of hardware I have seems to have its own built-in sound device.
Thunderbolt docks, USB3 hubs... even HDMI and display port carry sound. On my
laptop, I only want to use a few of them - but every time I connect another
device, it becomes the default sound card.</p>
<p>This is most annoying with my Thunderbolt dock. I will never use this audio
device, so I want to completely remove it. Aside from soldering, the next best
approach I've found is udev rules!</p>
<p>First, we need to find the ID of the device using the <code>lsusb</code> command:</p>
<pre><code>windigo@nyota | ~
→ lsusb
Bus 008 Device 002: ID 2188:0747 CalDigit Card Reader
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 002: ID 0c45:7692 Microdia USB Keyboard
Bus 005 Device 002: ID 2188:6533 CalDigit, Inc. CalDigit Thunderbolt 3 Audio
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 04f2:b649 Chicony Electronics Co., Ltd Chicony USB2.0 Camera
Bus 001 Device 004: ID 8087:0aaa Intel Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
</code></pre>
<p>If we read through the devices, I can spot the "CalDigit, Inc. CalDigit
Thunderbolt 3 Audio" device that I'd like to remove. It's ID is <code>2188:6533</code>.</p>
<p>The following udev rule will prevent a USB device with that ID from
being authorized, and therefore won't take over my audio. I added it in a new
rule file, <code>/etc/udev/rules.d/10-no.dock.soundcards.rules</code>:</p>
<pre><code>SUBSYSTEM=="usb", ATTRS{idVendor}=="2188", ATTRS{idProduct}=="6533", ATTR{authorized}="0"
</code></pre>
Testing backup DVDs without video and audiohttps://fragdev.com/blog/testing-backup-dvds-without-video-and-audio<p>After creating a whole host of backup DVDs, you might be faced with a dilema;
how to verify that each disk was burned properly. Backups aren't backups until
they are tested!</p>
<p>However, verifying even a small DVD collection could take days of watching DVDs
manually. Luckily, there is a much simpler way with VLC's command-line
interface:</p>
<pre><code>cvlc "dvd:///dev/sr0" --rate 4 --no-audio --no-video --play-and-exit
</code></pre>
<p>Here's a breakdown of the command, for those interested in mixing and matching.</p>
<ul>
<li><code>cvlc</code>: Use the VLC command-line interface</li>
<li><code>"dvd:///dev/sr0"</code>: Use the DVD drive as a DVD video source. <code>sr1</code> would be
the second DVD device, <code>sr2</code> the third, etcetera</li>
<li><code>--rate 4</code>: Play through at 4x speed. I'm not sure my DVD drive was able to
keep up with this and keep playback going that fast, but it did play much
quicker.</li>
<li><code>--no-audio --no-video</code>: Don't actually output any audio/video, just decode it</li>
<li><code>--play-and-exit</code>: Quit when playback has finished</li>
</ul>
<p>The result is a few lines of text as output; so that you can verify that
playback was successful, without a movie distracting you in the background.</p>
Problems with `rsync` and `link-dest`https://fragdev.com/blog/problems-rsync-and-link-dest<p>I use <code>rsync</code> to back up many servers and other machines. Because of my very
different sources and targets, I have a collection of very complex bash scripts
that manage these backups. When I moved one stage of these backups to a new
server, it started behaving strangely - and it was difficult to discover why.</p>
<p>The backup that went haywire was the offsite backup step; when I take all of the
daily backups that collect on the server, and once weekly, I copy these daily
backups onto an external drive for long-term protection.</p>
<p>To prevent each snapshot from being a full backup, I specify the previous daily
backup directory using the <code>--link-dest</code> argument. This tells rsync to <em>take the
previous daily backup, and only copy files that differ - all other files should
be hard links of the previous files</em>. For some reason, even though I was
using the <code>--link-dest</code> argument, it was making a full backup and not using the
hard-link technique to save space.</p>
<p>I had just moved my backups to a new server, and thought initially that I had
accidentally changed permissions on the files (which would make them appear to
be new/changed files). That wouldn't explain why <em>every</em> snapshot would be
copied anew; it would have only affected the first newly copied snapshot
directory.</p>
<p>It took me a rather long time, but I found the culprit thanks to a StackOverflow
answer: <a href="https://stackoverflow.com/a/21299531/64678">If <code>link-dest</code> receives a relative path, it is relative to the
<em>destination</em> directory</a>. I
had changed the line in my script where I create the temporary directory, and it
had switched from an absolute path to a relative one. Once I switched it to an
absolute path again, everything clicked and started behaving.</p>
<p>My rickety, fragile backup solution lives again!</p>
Stream decompress archive files on S3https://fragdev.com/blog/stream-decompress-archive-files-s3<p>Sometimes I need to uncompress large archives located in remote locations - for
instance, in Amazon S3. Most normal approaches to uncompressing archives involve
resource-intensive processes, like downloading the entire file into memory - an
approach not tenable for multi-gigabyte files.</p>
<p>The solution is to stream the files, and perform the decompression and file
extraction as you go. Similar to piping the output from one command into
another. If you do the work of writing the output files <em>while</em> uncompressing
them, you only need to store the current file's data in memory. Much more
efficient!</p>
<p>Enter the <code>smart_open</code> library - a wrapper around many different cloud storage
providers (and more) that handles the gory details of opening those files in a
streamable manner. With this library, I was able to quickly put together a
script that would open my zip file (located in S3) and upload the resulting
files back to S3 again. No local disk, no huge memory overhead!</p>
<p>Requirements:</p>
<ul>
<li><code>awscli</code> - must be installed and configured with S3 credentials</li>
<li><code>smart_open</code></li>
</ul>
<p>Code:</p>
<pre><code class="language-python">from smart_open import open
import zipfile
source = 'local file path / s3:// URI'
dest = 'local file path / s3:// URI'
# Iterate over all entries in the zip file
with open(source, 'rb') as file_data:
with zipfile.ZipFile(file_data) as z:
for file_info in z.infolist():
new_filename = dest + file_info.filename
# Skip directories - prefixes aren't explicitly created in S3
if not file_info.is_dir():
# Stream the uncompressed file directly to the dest
dest_file_data = open(new_filename, 'wb')
with z.open(file_info.filename) as zip_file_data :
dest_file_data.write(zip_file_data.read())
</code></pre>
Removing leading/trailing spaces in the shellhttps://fragdev.com/blog/removing-leadingtrailing-spaces-shell<p>Whitespace causes lots of interesting issues with the command line - whether
it is present in file names, arguments, or any other data flowing between
commands. Quoting can help, but there's some very particular edge cases I've
encountered in my scripts.</p>
<h2>Leading and trailing whitespace</h2>
<p>If you are piping output directly into another command, occasionally you can get
whitespace characters before or after your data. You have to use quotes to make
sure internal whitspace is preserved, but that also leaves any whitespace
characters at the beginning and/or end of your data.</p>
<p>I use three methods of removing leading or trailing whitespace: <strong>echo</strong>,
<strong>xargs</strong>, and <strong>sed</strong>.</p>
<hr />
<p>The <strong>echo</strong> method involves echoing a value <em>without</em> quotes, and re-assigning
it to a variable:</p>
<pre><code class="language-bash">TEST=' lousy spaces! '
TEST="$( echo $TEST )"
echo "$TEST"
# Prints:
#lousy spaces!
</code></pre>
<p>Benefits:</p>
<ul>
<li>Does not call an external command - echo is a bash builtin</li>
<li>Short and sweet</li>
</ul>
<p>Drawbacks:</p>
<ul>
<li>Collapses internal spaces as well</li>
<li>Doesn't lend itself to chaining/pipes</li>
</ul>
<hr />
<p>The <strong><code>xargs</code></strong> method is my personal favorite, although it's a bizarre one.</p>
<pre><code class="language-bash">echo ' lousy spaces! ' | xargs
# Prints:
#lousy spaces!
</code></pre>
<p>The <code>xargs</code> command, as a side-effect, strips out leading and trailing
whitespaces.</p>
<p>If you think about what its doing, it makes sense; it's taking each
unescaped "word" string, and sending it to <code>stdout</code> (or wherever). It doesn't
care about whitespace, so it gets truncated along the way.</p>
<p>Benefits:</p>
<ul>
<li>Can be part of a piped chain of commands</li>
<li><code>xargs</code> is part of <code>findutils</code>, and installed by default on most Debian-based
systems</li>
<li>Using a command that begins with 'x' automatically gives you a +1 to your
charisma stats</li>
</ul>
<p>Drawbacks:</p>
<ul>
<li>Collapses internal spaces as well</li>
<li>Runs an external program</li>
</ul>
<hr />
<p>The <code>sed</code> method is not something I've used personally. It's something <code>sed</code> was
created for - stream editing - but seems like a costly solution to the problem.</p>
<p>It relies on regular expressions, which are usually slower than other forms of
text manipulation. This method is <em>probably</em> plenty performant, but I tend to
leave any regex as a last resort.</p>
<pre><code class="language-bash">echo ' lousy spaces! ' | sed 's/^[[:space:]]\+//' \
| sed 's/[[:space:]]\+$//'
</code></pre>
<p>I've included the long <code>[[:space:]]</code> format because it's compatible with non-GNU
<code>sed</code>, but if you're rockin' the GNU toolchain, you can use <code>\s</code> as a terse and
sensible replacement.</p>
<p>Benefits:</p>
<ul>
<li><em>Only</em> strips spaces off of the beginning and end of a string, leaving
multiple spaces inside intact</li>
<li>Can be quickly adjusted to strip off other leading or trailing strings</li>
</ul>
<p>Drawbacks:</p>
<ul>
<li>Not zero, not one, but <strong>two</strong> command invocations</li>
<li>Regex, and all its baggage</li>
<li><code>sed</code> may not be installed by default on the system you're working on</li>
</ul>
<hr />
<p>The real solution would be to quit doing so much data processing in the shell,
and move to an actual scripting/programming language. However, there's a
delicate balance between what belongs in the shell, and what needs its own
fully-fledged script; and these techniques have let me accomplish a lot in
scripts that didn't need a full programming language backing it.</p>
<h1>Update</h1>
<p><a href="https://hakerdefo.github.io/">hackerdefo</a> has submitted several other excellent methods for your consideration. I'm particular fond of the awk implementation myself.</p>
<p><em>Note:</em> The <code>tr</code> method removes <strong>all</strong> spaces, not just leading and trailing.</p>
<pre><code class="language-bash">echo -e " Fragmented Development " | tr -d "[:blank:]"
echo -e " Fragmented Development " | awk '{$1=$1};1'
echo -e " Fragmented Development " | ruby -pe 'gsub(/^\s+/, "").gsub(/\s+$/, $/)'
echo -e " Fragmented Development " | perl -plne 's/^\s*//;s/\s*$//;s/\s+/ /;'
</code></pre>
Headless server and disk encryptionhttps://fragdev.com/blog/headless-server-and-disk-encryption<p>As an enthusiast of encryption, it always felt a little strange that my servers
kept all of their data in the clear. But the problem with encrypting a headless
server is that, inevitably, you have to reboot it. So how do you connect to your
server and unlock the drive <em>before</em> it boots? It's quite the catch-22.</p>
<p>Enter <a href="https://matt.ucc.asn.au/dropbear/dropbear.html">dropbear</a>, a very small
SSH server. This server is so tiny, it can be embedded in your initramfs and run
at boot. You can connect to your server, unlock your boot drive, and allow the
boot process to proceed. Fantastic!</p>
<p>Here's a short summary of how to do things in Debian 10 (Buster). These commands
need to be run as <em>root</em>:</p>
<ul>
<li>Install the <code>dropbear-initramfs</code> package (should install <code>busybox-static</code>)</li>
<li>Copy your SSH public key into <code>/etc/dropbear-initramfs/authorized_keys</code></li>
<li>Run <code>update-initramfs -u</code> & reboot</li>
<li>Connect to the dropbear server by running <code>ssh root@yourserver.com</code></li>
<li>Run <code>cryptroot-unlock</code> to unlock your boot drive</li>
</ul>
<p>This is just the beginning of my setup. Eventually, I would like another machine
to automatically unlock my servers for me - but that will be several more steps
that are not in place yet. For now, this is a fairly straight forward technique
for allowing you to encrypt headless, remote servers without worrying about
reboots!</p>
Show a diff inside of git loghttps://fragdev.com/blog/show-diff-inside-git-log<p>If you've ever needed to view all the changes that are happening in a repository
or a branch, there are two flags for the <code>git log</code> command that should interest
you: <code>git log -c</code> and <code>git log --cc</code>.</p>
<p>Using <code>git log -c</code> shows you a diff view of changes made in each commit
referenced in the log. If you're viewing a single branch, you can view the
line-level changes, commit by commit, that went into getting the current branch
to where it is. Very handy!</p>
<p>The <code>--cc</code> flag does the same thing, but it hides some automatic merge changes.
I haven't seen very much difference in the behavior of these two flags myself.</p>