With five storage nodes using dedicated devices, I want to migrate off Ceph with zero data loss and some small outages. This will free up a bunch of CPU, RAM and storage hardware all over the cluster which gives me a lot more room for more fun systems. Notably, it will free up an entire N100 mini PC and some SSDs to use as a ZFS-enabled NAS.

Setting up LVM localpv

In the Kubernetes cluster where I wanted to use the storage, I just needed to install the CSI driver with helm, eg:

helm repo add openebs https://openebs.github.io/openebs
helm repo update
helm install openebs --namespace openebs openebs/openebs --create-namespace

Ironically, this helm chart installs a whole bunch of other storage services, including another replicated storage engine: Mayastor. I am not planning to use this. If you like, you can choose not to install these other services - the OpenEBS installation instructions tell you how to do this.

After this I needed a storage class, which was kubectl apply‘ed to the cluster:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-lvmpv
parameters:
  storage: "lvm"
  volgroup: "openebs-lvmpv" # must match the VG name you created
  fsType: ext4
provisioner: local.csi.openebs.io
allowVolumeExpansion: true   # optional but recommended
# don't create the LV until the container is allocated (so it is made in the right VM)
volumeBindingMode: WaitForFirstConsumer
# after releasing a PVC, keep the LV. To free up storage for reallocation it needs to 
# be manually deleted on the host with `lvremove`.
reclaimPolicy: Retain

OSD Draining and removal

Actually allocating a PV to this StorageClass means creating a VG called openebs-lvmpv on nodes that are allowed to use it. In my cluster all storage is fully allocated. Draining and removing a single OSD will give me enough space to migrate everything (eg a spare 4TB drive) - a larger cluster would mean more planning and more work.

This is “The Safe-ish Way”(TM). Using “The Dangerous Way” (delete a VM and format its OSD) should be a perfectly valid way of reclaiming the space also, but would put the cluster into a degraded mode straight away and fire off a ton of recovery network activity for hours.

Of course, I skipped a few steps to do with OSD node allocation, OSD pod deletion and operator re-enabling since this cluster is getting deleted. These are the rough steps I followed, and will result in a half-baked clean up and semi-broken cluster. Fine in my case since I’m deleting the whole thing once I’m finished, so take these notes as inspiration rather than instruction:

Before starting, make sure there is enough free space and that the cluster is healthy with the UI or toolbox pod, eg:

ceph df
ceph osd df
ceph osd pool ls detail   # check size and failure domain

# make sure cluster is healthy before proceeding
ceph status

Then do the operation itself:

# 1. Stop rook from trying to reclaim devices and nodes, set 
#       useAllDevices: false
#       useAllNodes: false

kubectl -n rook-ceph edit cephcluster rook-ceph

# 2. Stop the operator so it doesn't fight us
kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas=0

# 3. In the toolbox: mark it out (the OSD stays up and drains gracefully)
ceph osd out osd.<ID>

# 4. I had to wait about a day for all PGs to be active+clean
ceph osd safe-to-destroy osd.<ID>    # confirm it's safe to delete now

# 5. Stop the OSD pod
kubectl -n rook-ceph scale deploy rook-ceph-osd-<ID> --replicas=0

# 6. In the toolbox: remove it from CRUSH, auth, and the osdmap
ceph osd purge <ID>  # only works if safe to delete
ceph osd crush rm <hostname>      

Kubernetes node shutdown

With the data drained from the OSD, I had no use for the Kubernetes node it was attached to any more. I deleted the node from Kubernetes with kubectl delete node ..., then powered off the VM and virsh undefine‘d it.

This took out a mon for me, and because the operator was scaled to zero, this did not get fixed. Since I had plenty of redundancy it wasn’t a problem but this is a good gotcha to look for in your own migrations.

The old OSD was a pass-through disk so was now sitting unused.

Getting the disk to the Kubernetes node

To actually get the freed up disk available at the point of use in the crypto node, I formatted it as regular ext4 on the hypervisor, mounted it to /data/nvme and then created a .qcow2 file under /data/nvme which was mapped as storage in the crypto Kubernetes VM. There are some pros and cons to using qcow2 files but on balance, I think it’s fine for what I want to do:

• One physical device can only be shared with one VM
• In the Rook/Ceph world, it makes sense to just pass the whole drive through to the VM
• qcow2 add some operational overhead and additional management but I’m not planning to change too many things, and I already wrote Ansible scripts to create and map the files to libvirt XML
• I can create additional .qcow2 files and share with different VMs for different Kubernetes nodes if I want to, eg for my apps cluster
• Beware of overcommitting - qcow2 files are sparse by default, so nothing will stop you allocating a 10TB file on a 500GB disk

It’s possible to do some cool things with LVM storage pools and KVM/Libvirt to make allocation a slicker process. I decided against this in my lab since I’m already using .qcow2 files for the OS images and I didn’t want two different ways of managing disks in the lab.

LVM VG creation

After rebooting the VM (Ansible script requirement), I had a free disk to use for whatever I wanted so I just did what I normally do with a new disk:

1. gparted /dev/somedisk - Create GPT partition table, allocate LVM to the whole drive
2. pvcreate /dev/somedisk1 - the partition you created
3. vgcreate openebs-lvmpv /dev/somedisk1 - must match above

No need to create any LVs - that’s what the localpv driver does for us.

Copy out of existing pods

Copying out the data from Rook/Ceph to LVM was straightforward, if a little slow:

Create a PVC

First create a PVC, nothing gets allocated yet:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lvm-meowcoin-pvc
  namespace: crypto
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: openebs-lvmpv
  resources:
    requests:
      storage: 10Gi

Assign to a pod, with a selector

Now we want to start a pod with both the Rook/Ceph and LVM storage mounted at the same time. A dedicated pod could be used for this, but since I need to alter the deployment anyway, I just did it all in one place.

Note that Recreate strategy is used to ensure pods are shut down cleanly because the storage PVC can only be mounted to one node at a time since it’s marked ReadWriteOnce. This doesn’t prevent two pods on the same node from doing so though! Eg during a rolling restart. There are a few ways to ensure only one pod at a time has a PVC mounted but this is what works for me and it was the answer to multiple blockchain corruption incidents over the years:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: crypto
  name: meowcoin
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: meowcoin
  template:
    metadata:
      name: meowcoin
      labels:
        app: meowcoin
    spec:
      hostname: meowcoin
      nodeSelector:
        # make sure we run only on this one server:
        kubernetes.io/hostname: kps-crypto-1 
      containers:
      - name: meowcoind
        image: quay.io/declarativesystems/cryptodaemons_meowcoin:30.2.7

        # Make a pod that just runs sleep so we can `exec` into it...
        command: [/bin/sleep]
        args: [infinity]

        # ...and disable the normal pod actions...
        # args:
        # - "-server"
        # - "-printtoconsole"
        volumeMounts:
        - mountPath: /root/.meowcoin
          name: meowcoin
        - mountPath: /new
          name: meowcoin-new
        resources:
          limits:
            memory: "2Gi"
            cpu: 1

      restartPolicy: Always
      volumes:
      - name: meowcoin
        persistentVolumeClaim:
          claimName: rook-ceph-meowcoin-pvc
      - name: meowcoin-new
        persistentVolumeClaim:
          claimName: lvm-meowcoin-pvc     

Applying this takes down the meowcoin server and brings up a pod we can use to copy the data between filesystems. After checking the allocations are correct with df -h, the data can easily be copied, eg:

cp -a /root/.meowcoin/. /new/

After the command completes, I checked with df -h that about the same amount of storage was being used on the LVM mount as the Rook/Ceph one, which is good enough for me.

A better check would have been du -s on both directories or even running rsync might be a useful secondary verification. Not a big deal in this case since the crypto server would just sync any missing blockchain data.

Return to service

Once this finishes, we reconfigure the deployment for production use again:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: crypto
  name: meowcoin
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: meowcoin
  template:
    metadata:
      name: meowcoin
      labels:
        app: meowcoin
    spec:
      hostname: meowcoin
      nodeSelector:
        # make sure we run only on this one server:
        kubernetes.io/hostname: kps-crypto-1 
      containers:
      - name: meowcoind
        image: quay.io/declarativesystems/cryptodaemons_meowcoin:30.2.7

        # Keep for emergencies...
        # command: [/bin/sleep]
        # args: [infinity]

        # ...and restore the normal pod actions...
        args:
        - "-server"
        - "-printtoconsole"
        volumeMounts:
        - mountPath: /root/.meowcoin
          name: meowcoin
        resources:
          limits:
            memory: "2Gi"
            cpu: 1

      restartPolicy: Always
      volumes:
      - name: meowcoin
        persistentVolumeClaim:
          claimName: lvm-meowcoin-pvc     

A few seconds later, the pod was up and running with all required data.

Clean up

Once the new pod is up and running the Rook/Ceph PVC can be deleted, which should free up its storage unless you changed the default reclaimPolicy.

After repeating the process above for all PVCs in the Rook/Ceph cluster and double-checking my work, I was left with a Rook/Ceph cluster that was still happily running OSDs and other pods. If I was keeping the cluster then additional clean up steps would be needed to restore normal operations. If you need to do this, I suggest you read the instructions carefully.

In my case, this is the end of this Rook/Ceph cluster. Since it runs entirely within KVM via libvirt, I didn’t bother to properly clean up anything, I just virsh destroyed and virsh undefineed until the entire Kubernetes cluster was deleted.

At this point, the pass-through OSD disks could be repartitioned with cfdisk.

The final step of clean up was to remove the Rook/Ceph deployment from the consumer cluster, by uninstalling the helm chart and deleting the namespace.

Tips and tricks

Use screen if available

In case your session gets interrupted, you can use tmux or screen inside the pod if it’s available, so your command doesn’t stop if your kubectl exec disconnects for some reason.

If it’s missing and you’re lucky, you might be able to apt install it.

local-storage

Don’t forget you can still use K3s built-in local-storage as an alternative path to copy out data from Rook/Ceph if you’re having trouble finding space elsewhere. Similar technique to above.

Grow an LV

If you need more space, you can just increase the value of the storage field and apply your yaml. A couple of minutes later, the additional space was available in the pod with no restart needed. Eg:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lvm-meowcoin-pvc
  namespace: crypto
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: openebs-lvmpv
  resources:
    requests:
      storage: 50Gi # <~~~ edit here

What did I lose?

The switch has some drawbacks, that I willingly signed up for:

• Storage is now explicitly pinned to nodes, it does not “float” around the cluster anymore
• Pods are manually pinned to Kubernetes nodes:
  • Pods are linked to a named PVC, so moving a workload means pinning the deployment to another Kubernetes node and creating a new PVC with a new name, so Kubernetes can create it in the right place
  • If a pod restarts with a fresh PVC it needs to download its entire blockchain from the peer-to-peer network again. This can take days unless I build a NAS and write a backup and restore procedure to get up and running quicker
  • The typical worst case would be a brief outage of minutes/hours while a mini PC is rebooted or has maintenance done on it
  • Additional manual steps to restore operation make the advantage of Kubernetes less compelling in this case but do not eliminate it altogether. It’s still the best way to manage containerized apps for my workload
• No more replicated storage
• No UI
• No S3 compatible storage
• No “storage as a service” for other Kubernetes clusters

What did I gain?

Crypto nodes don’t really benefit from replicated storage the way I had it set up:

• The data is already replicated over the peer-to-peer network
• The “floating” storage was fun to test out and speeds up crypto node restarts, but restarts still take several minutes, during which, the node cannot be used.
• If some kind of HA crypto node was actually needed (eg for an exchange), then the way to do this is with multiple, independent nodes - with their own independent storage. I don’t have the need or the hardware for this
• Mitigation for slow node rebuilding (hours/days) if a disk fails is to periodically snapshot the blockchain data to the NAS I’m planning on building, or just replace the drive and suck up the delay in the interim

I knew this walking into my distributed storage adventure of course - running a crypto server is a low-stress, low-effort way to get some constantly updating data to play with on your storage cluster. If you want to start over, you can do so guilt- and worry-free.

In terms of extra hardware, I gain:

• Entire 16GB N100 mini PC
• Valuable SSDs and NVMes (at least 2x 4TB drives)
• CPU and RAM used by 5 big VMs on 5 PCs
• Not storing 3 copies of all data means a 66% reduction in lab storage needed
• I’m wondering if this will even bring down the power bill (or room temperature)

And in terms of admin:

• 5 big VMs no longer need monitoring, etc
• 2 VLANs can be retired
• Network traffic rules can be simplified
• Less traffic on the LAN in general - important on a 1GbE network
• No more scary Rook/Ceph upgrades

Conclusion

I hope reading this you don’t think I’m picking faults with Rook/Ceph - I’m not, it’s excellent. I’ve learned a lot using it, but the more useful workloads I planned to use with it failed to materialize and for my crypto homelab it’s like using a nuclear weapon to swat a fly.

Using Rook/Ceph daily also confirmed to me that if I was going to use this for something important or a customer, a single 1GbE network cluster is not going to cut it. Production needs mean faster networking, multiple clusters and probably a support contract.

Onward!

I did some research already on why this happens and it’s due to OOMKILL at the hypervisor level. Rebooting fixes it - until next time. A finite choice human-gated solution might render a screen that looks something like this one, from “King-Size Homer”:

Which could then be approved by the operator (or not):

Building this would have been a complete waste of time, effort and tokens. The problems and solutions are known, so let’s just fix this once and for all, and save our agentic loop for a real problem.

What keeps breaking?

Looking through the kernel errors, I could see a VM being OOMKILL‘ed about once every couple of months. The reason for this is very simple: The hypervisor is overcommitted and has swap disabled. This was originally done for performance reasons but it means a temporary increase in memory usage will result in the kernel reclaiming memory the only way it can - by killing processes.

The lab runs mostly on mini PCs with limited memory - 16-32GB. The 32GB boxes lose 4GB to onboard graphics and are always the PCs with problems since most of the work is allocated to them.

TODO.exe

This little cluster has been running for a couple of years without much maintenance being done, so it’s due for some updates. I came up with a five-step plan to fix the memory issues once and for all, as well as some routine upgrades:

1. Enable swap at the host (hypervisor) level
2. Upgrade Rook/Ceph
3. Upgrade Kubernetes (K3s)
4. Reduce memory allocations for VMs to fix excessive overcommit
5. Use cgroups to reserve memory for the host (hypervisor)

Enabling swap

This was the most straight-forward fix since the hypervisors are managed with ansible. I added some code to my SOE, activated for physical servers only (hypervisors, not Kubernetes nodes) and moved on quickly:

Vars:

swapfile_path: "/swapfile"
swapfile_size: 8G

# prefer dropping page cache over swapping out guest RAM under reclaim.
# swap here is an emergency valve to buy the kernel time, not memory
# we intend to run in
swappiness_value: 10

Tasks:

- name: Create swapfile
  community.general.filesize:
    path: "{{ swapfile_path }}"
    size: "{{ swapfile_size }}"
    
- name: Set swapfile permissions
  ansible.builtin.file:
    path: "{{ swapfile_path }}"
    owner: root
    group: root
    mode: '0600'

- name: Check whether swapfile is already active
  ansible.builtin.command:
    cmd: swapon --show=NAME --noheadings
  register: swapfile_active
  changed_when: false

- name: Format swapfile
  ansible.builtin.command:
    cmd: "mkswap {{ swapfile_path }}"
  when: swapfile_path not in swapfile_active.stdout_lines
  
- name: Enable swapfile
  ansible.builtin.command:
    cmd: "swapon {{ swapfile_path }}"
  when: swapfile_path not in swapfile_active.stdout_lines

- name: Persist swapfile
  ansible.posix.mount:
    path: none
    src: "{{ swapfile_path }}"
    fstype: swap
    opts: sw
    state: present

- name: Configure swappiness
  ansible.posix.sysctl:
    name: vm.swappiness
    value: "{{ swappiness_value }}"
    state: present
    reload: true

Upgrading Rook/Ceph

Rook/Ceph is used in the homelab as replicated, Cloud-Native Storage for Kubernetes.

Upgrading Rook/Ceph is a serious undertaking. It’s actually not dissimilar in scale to upgrading Confluent Platform (my day job), with a lot less hand-holding. Fortunately, there’s no valuable data on my Rook/Ceph cluster since I just use it to host a few crypto nodes.

The notes below supplement the docs:

• Cluster must be healthy before starting
• Upgrade one minor release at a time, eg 1.16.x to 1.17.x
• RTFM!
• No really. RTFM. For each minor release you will upgrade through
• Upgrade rook-ceph first, then rook-ceph-cluster
• You must install/upgrade ceph-csi-drivers if you want to be able to access PVs (you do)
• Wait for each upgrade step to finish before moving on

Helm tips:

• Helm charts do most of the work
• Explore available versions:
  • helm search repo rook-release/rook-ceph --versions
  • helm search repo rook-release/rook-ceph-cluster --versions
• List what’s installed: helm list -A

Example (upgrade v1.16.5 to v1.17.9):

# Upgrade the operator
helm upgrade --version v1.17.9  --namespace rook-ceph rook-ceph rook-release/rook-ceph

# WAIT for all pods to show correct version
watch "kubectl -n rook-ceph get deploy \
  -o custom-columns=NAME:.metadata.name,ROOK:.metadata.labels.rook-version"

# Install the CSI driver now, (omitted in my lab environment)

# Upgrade the cluster itself, using saved values from the original deploy
helm upgrade --version v1.17.9  --namespace rook-ceph rook-ceph-cluster rook-release/rook-ceph-cluster -f helm_values/rook_cluster_values.yaml

# WAIT - watching the logs
kubectl logs  -n rook-ceph rook-ceph-operator-5cc9ff9d9b-9zp7k -f

Cluster topology

I access the Rook/Ceph storage as an external cluster at the point of use. The Rook/Ceph storage cluster itself lives in a separate, dedicated Kubernetes cluster for ease of management and sharing.

Ceph-CSI driver problems

The Kubernetes cluster accessing the Rook/Ceph storage is upgraded with a similar process to the storage cluster. Since this is where I want to use the storage, it’s now mandatory to install the Ceph-CSI driver helm chart.

The Ceph-CSI driver actually failed for me and crashed my workloads:

• First a couple of pods died with OOMKILLED (from Kubernetes)
• Thinking the pods must have got stuck for some reason, I rebooted the entire Kubernetes cluster
• At this point, all workloads that attempted to mount storage were marked as unknown or ContainerCreating and were blocked from starting or progressing.

To troubleshoot this, I did some investigating on the stuck pods and found:

kubectl describe pod <pod-name> -n <namespace>
...

Type     Reason       Age                   From     Message
  ----     ------       ----                  ----     -------
  Warning  FailedMount  18m (x119 over 4h4m)  kubelet  MountVolume.MountDevice failed for volume "pvc-fe440598-6203-4a69-8aad-4032795274e1" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rook-ceph.rbd.csi.ceph.com not found in the list of registered CSI drivers
  Warning  FailedMount  9s (x14 over 12m)     kubelet  MountVolume.MountDevice failed for volume "pvc-fe440598-6203-4a69-8aad-4032795274e1" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rook-ceph.rbd.csi.ceph.com not found in the list of registered CSI drivers

Accompanied by no drivers running:

kubectl -n rook-ceph get ds
NAME                                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
rook-ceph.cephfs.csi.ceph.com-nodeplugin   3         0         0       0            0           <none>          34h
rook-ceph.rbd.csi.ceph.com-nodeplugin      3         0         0       0            0           <none>          34h

Eventually I found the problem - a mismatch between service account names in ceph-csi-drivers chart vs the deployed cluster:

kubectl describe ds -n rook-ceph rook-ceph.rbd.csi.ceph.com-nodeplugin

...

Events:
  Type     Reason        Age                   From                  Message
  ----     ------        ----                  ----                  -------
  Warning  FailedCreate  21m (x37 over 4h9m)   daemonset-controller  Error creating: pods "rook-ceph.rbd.csi.ceph.com-nodeplugin-" is forbidden: error looking up service account rook-ceph/rbd-nodeplugin-sa: serviceaccount "rbd-nodeplugin-sa" not found
  Warning  FailedCreate  5m46s (x18 over 16m)  daemonset-controller  Error creating: pods "rook-ceph.rbd.csi.ceph.com-nodeplugin-" is forbidden: error looking up service account rook-ceph/rbd-nodeplugin-sa: serviceaccount "rbd-nodeplugin-sa" not found

Available service accounts:

kubectl get serviceaccounts -n rook-ceph 
NAME                                          AGE
ceph-csi                                      34h
default                                       550d
objectstorage-provisioner                     548d
rook-ceph-cephfs-csi-ceph-com-ctrlplugin-sa   90m
rook-ceph-cephfs-csi-ceph-com-nodeplugin-sa   90m
rook-ceph-cmd-reporter                        548d
rook-ceph-default                             548d
rook-ceph-mgr                                 548d
rook-ceph-nvmeof                              34h
rook-ceph-osd                                 548d
rook-ceph-purge-osd                           548d
rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa      90m
rook-ceph-rbd-csi-ceph-com-nodeplugin-sa      90m
rook-ceph-rgw                                 548d
rook-ceph-system                              548d

Note that rbd-nodeplugin-sa has a completely different name on the system: rook-ceph-rbd-csi-ceph-com-nodeplugin-sa. The same applies to three other service accounts as well.

To fix this I had to add configuration for the service account to the default helm values from the docs by adding serviceAccountName to fix the four broken accounts. I also had to helm uninstall the ceph-csi-drivers chart and reinstall it for this to take effect. It was not enough to just helm upgrade:

# Rook-compatible default values for the ceph-csi-drivers Helm chart.
#
# Use these values when installing or upgrading the ceph-csi-drivers chart
# alongside the rook-ceph operator chart. The driver names must match the
# provisioner names used by Rook (e.g. in StorageClasses and VolumeSnapshotClasses).
#
# If the Rook operator is installed in a namespace other than rook-ceph, replace
# "rook-ceph" in the driver names below with your operator namespace.

operatorConfig:
  namespace: rook-ceph # namespace:operator
  driverSpecDefaults:
    imageSet:
      name: rook-csi-operator-image-set-configmap
    nodePlugin:
      priorityClassName: system-node-critical
    controllerPlugin:
      priorityClassName: system-cluster-critical

drivers:
  rbd:
    enabled: true
    name: rook-ceph.rbd.csi.ceph.com # csi-provisioner-name
    nodePlugin:
      serviceAccountName: rook-ceph-rbd-csi-ceph-com-nodeplugin-sa
    controllerPlugin:
      serviceAccountName: rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa    
  cephfs:
    enabled: true
    name: rook-ceph.cephfs.csi.ceph.com # csi-provisioner-name
    nodePlugin:
      serviceAccountName: rook-ceph-cephfs-csi-ceph-com-nodeplugin-sa
    controllerPlugin:
      serviceAccountName: rook-ceph-cephfs-csi-ceph-com-ctrlplugin-sa    
  nfs:
    enabled: false
    name: rook-ceph.nfs.csi.ceph.com # csi-provisioner-name
  nvmeof:
    enabled: false
    name: rook-ceph.nvmeof.csi.ceph.com # csi-provisioner-name

No data was lost during the upgrade, although I did have a brief unplanned outage while I worked out the service account fix.

Upgrading Kubernetes (K3s)

I’m using K3s as a light-weight Kubernetes distribution and manage it with k3s-ansible.

Like Rook/Ceph, Kubernetes should also be upgraded through each minor version, in my case there were three minor version updates to apply.

Hats off to the K3s developers, this was incredibly easy having just finished a Rook/Ceph upgrade:

1. Update the k3s-ansible codes
2. Change k3s_version to desired version in inventory file
3. Run the upgrade playbook (eg ansible-playbook playbooks/upgrade.yml -i ../inventories/k3s_apps.yml)
4. Wait for stability, then rinse and repeat

It really was that easy once I figured out I now needed to install the ansible.utils galaxy module.

Reducing memory allocations

With the upgrades done and systems stable, it was time to redistribute and reduce overall memory usage.

On PCs with 28GB usable RAM, there was an overcommit of approximately 6GB which is far too high. 3GB is pushing it and 0GB of overcommit would be better.

When the cluster was first setup, I remember Rook/Ceph needing a lot more memory but it seems this hasn’t been the case for a long time and much of that memory was sitting idle. Since I control my VMs with libvirt/KVM + ansible, changing memory allocation is a very simple process:

1. Update host_vars for hypervisor with new memory amount
2. Shut down affected VM(s)
3. Run ansible
4. VM will be restarted with the updated memory allocation

The main reduction was for the Rook/Ceph VMs: from 20GB to 12GB. Some of the future-use clusters were also downsized, from 4GB to 2GB.

An hour or so later the job was done and the VMs worked fine with their new memory allocations.

Using cgroups to limit memory usage in machine.slice

Everything we have done so far has been a mitigation for the crashes - this step limits the blast radius of OOMKILL: When machine.slice is full, it will kill a VM, not a host process like sshd.

Linux cgroups are a kernel feature that we can use to restrict the memory available to programs. Limiting the total memory available to VMs in machine.slice lets us support a modest overcommit more safely without the risk of the hypervisor running out of memory.

We can deploy the change very easily with ansible.

Vars:

# total held back from machine.slice
host_reserve_mb: 2048

# unreclaimable floor, measured
host_min_mb: 512

# best-effort protection
host_low_mb: 1024

# headroom for fwupd/libvirtd spikes
user_low_mb: 256

# unreclaimable floor for ssh/login sessions
user_min_mb: 64

# throttle band: gap between MemoryHigh and MemoryMax
vm_high_headroom_mb: 1024

Tasks:

- name: Ensure slice drop-in directories exist
  ansible.builtin.file:
    path: "/etc/systemd/system/{{ item }}.d"
    state: directory
    owner: root
    group: root
    mode: '0755'
  loop:
    - machine.slice
    - system.slice
    - user.slice


- name: Configure machine.slice memory limit
  ansible.builtin.copy:
    dest: /etc/systemd/system/machine.slice.d/50-memory.conf
    content: |
      [Slice]
      MemoryAccounting=yes
      MemoryHigh={{ (ansible_memtotal_mb - host_reserve_mb - vm_high_headroom_mb) }}M
      MemoryMax={{ (ansible_memtotal_mb - host_reserve_mb) }}M
    owner: root
    group: root
    mode: '0644'
  notify: systemd daemon reload

- name: Protect host services from VM memory pressure
  ansible.builtin.copy:
    dest: /etc/systemd/system/system.slice.d/50-memory.conf
    content: |
      [Slice]
      MemoryAccounting=yes
      MemoryMin={{ host_min_mb }}M
      MemoryLow={{ host_low_mb }}M
    owner: root
    group: root
    mode: '0644'
  notify: systemd daemon reload

- name: Protect user slice services from VM memory pressure
  ansible.builtin.copy:
    dest: /etc/systemd/system/user.slice.d/50-memory.conf
    content: |
      [Slice]
      MemoryAccounting=yes
      MemoryMin={{ user_min_mb }}M
      MemoryLow={{ user_low_mb }}M
    owner: root
    group: root
    mode: '0644'
  notify: systemd daemon reload

Checking the machine.slice counters a while after applying the fix: max, oom and oom_kill are all zero, so nothing has been killed. The 62k high events are the throttle doing its job rather than a problem, though they suggest my 1GB of headroom below MemoryMax is on the tight side. These are cumulative counts since boot, not bytes. Long term, its worth collecting these figures directly, in prometheus.

# 32GB system RAM server
$ cat /sys/fs/cgroup/machine.slice/memory.events
low 0
high 62467
max 0
oom 0
oom_kill 0
oom_group_kill 0

Retrospective

I completed the five-step TODO list over the weekend and systems have been stable all week with no data loss so I’m calling this a success. It remains to be seen if Kubernetes nodes will still crash every couple of months, but I’m pretty sure this particular problem is fixed, without the need to build a glorified AI-enabled cron job to do the occasional reboot.

The Rook/Ceph upgrade was quite an eye opener. It took longer than I would have liked to figure out what was going on, and although there was no data loss, there was an unplanned outage.

Since this is a lab environment, it’s no big deal but if this had been an enterprise system we would now be proceeding with caution. This is why we have dev environments - and support contracts.

Farewell, Rook/Ceph

I’ve run this five node Rook/Ceph cluster for over two and a half years now, and it’s worked really well on my 1GbE network.

I’ve been able to reboot or power off nodes without fear and watch Rook/Ceph recover or rebalance itself when OSDs are back online. It’s really fun to watch it do this and I’ve learned a lot from running the cluster all this time.

With that said, the upgrade process makes me a little nervous about using Rook/Ceph on a system that needs to be running 24/7 without having additional lab environments to model the upgrade.

Considering I don’t actually have any data I care about in the Rook/Ceph cluster, there’s no justification for building out more environments and doing additional testing for upgrades.

My original plan was to just to try out Rook/Ceph replicated storage in lab environment with my sample crypto servers to see how it works in the real wold and maybe add more workloads if useful.

Now that I’ve completed this research task and hard drive costs have quadrupled, it’s time to shut down Rook/Ceph before it burns out the SSDs for the simple reason that I don’t have a workload to justify using it at the moment:

• Crypto node data is replicated at the node level and freely (if slowly) available from the peer-to-peer network. In an enterprise context such as an exchange, the solution needed is multiple independent nodes with their own storage, not “floating” storage
• Messaging systems such as Kafka are best served by local NVMe disks - in my case, the outage I had took all PVs offline at the same time. This would have taken down Kafka if I was running it as its pods would simply not have been able to start. The choke-point of the CSI driver defeats the decentralization of both Ceph and Kafka
• Databases such as postgres are also best served by local NVMe for speed. This removes the network path from all read/write activity
• S3-like storage is a genuine loss, although I don’t currently use this capability and alternatives exist

Rook/Ceph really shines in environments where the data being stored is actually valuable and there is some tolerance in terms of latency: Think git, documents, backups, etc.

So what do we do instead?

• LMV Local PV for capacity limit aware local node storage
• Node Affinity to ensure nodes run adjacent to their storage
• A simple NAS somewhere on the network
• Backups/snapshotting for quick recovery (eg for crypto nodes, same with more safeguards for postgres)
• Assess the storage needs of new workloads on a case-by-case basis

If replicated storage requirements come up again in the future, then it’s time to upgrade the network backbone to at least 10GbE, add more PCs for additional environments and re-test Longhorn for good measure. I can’t see this happening any time soon though - I already spent a fortune on mini PCs and there are no power sockets left either.

Somewhat ironically, turning off Rook/Ceph will also free up approximately 12GB system RAM on each hypervisor, down from 20GB before this exercise.

The very problem we were trying to fix.

for host in get_config().ping.internet_hosts:
    result = ping3.ping(host, timeout=get_config().ping.timeout)
    if result is not None:
        logger.debug("ping internet OK host=%s", host)
        return NetworkCheckResult.UP

I disabled internet access on the router to test this but the pings kept succeeding. What the heck?!

• Running ping 8.8.8.8 from the CLI on the same host failed correctly.
• Restarting the Python process made it fail correctly.
• Leaving Python running meant ping continued to report success

Firewall state table

On the OPNsense box:

root@OPNsense:~ # pfctl -ss | grep 8.8.8.8
all icmp 8.8.8.8:8 <- 10.20.86.158:55172       0:0
all icmp 167.179.158.61:19000 (10.20.86.158:55172) -> 8.8.8.8:8       0:0
root@OPNsense:~ # pfctl -F state
2217 states cleared

The moment I flushed the states, Python started reporting the outage. So the packets weren’t being evaluated against the firewall rules at all - they were matching an existing state entry created back when the internet still worked, and states are checked before rules.

That explains why blocking had no effect. It doesn’t yet explain why the CLI behaved differently.

Python ping3 stable ICMP identifiers

pf keys an ICMP state on the identifier field in the echo request. Here’s how ping3 picks that value:

thread_id = threading.get_native_id()
process_id = os.getpid()
icmp_id = zlib.crc32(f"{process_id}{thread_id}".encode()) & 0xffff

It’s recomputed on every call, but the inputs never change while the process is alive. So every ping from my long-running service carried the identical identifier.

To pf, that isn’t a thousand separate probes. It’s one continuous flow. The state matched, so the packet skipped rule evaluation - and because traffic kept arriving every second, the state’s idle timer kept resetting and it never aged out.

The CLI behaves differently for a boring reason: each ping invocation is a new process with a new PID, so it gets a new identifier, matches no existing state, falls through to the actual rules, and gets blocked. Restarting Python works for exactly the same reason.

This identifier scheme isn’t a bug. It exists specifically so that pings from different threads and processes don’t collide with each other. It just happens to interact badly with stateful firewalls when you’re probing on a loop.

The fix

ping3.ping() doesn’t expose an id parameter:

(dest_addr, timeout=4, unit='s', src_addr='', ttl=None, seq=0, size=56, interface='', version=None)

But the functions it calls internally are public and do accept icmp_id. This is the code I ended up using, which fixes the problem:

import socket
import random
import ping3

def ping_random_id(dest_addr, timeout=4, size=56):
    icmp_id = random.randint(0, 0xffff)
    with socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP) as sock:
        try:
            sock.settimeout(timeout)
            ping3.send_one_ping(sock=sock, dest_addr=dest_addr, icmp_id=icmp_id, seq=0, size=size)
            delay = ping3.receive_one_ping(sock=sock, icmp_id=icmp_id, seq=0, timeout=timeout)
        except Exception as e:
            logger.debug("caught and ignored ping error: %s", e)
            # typically means timeout
            return None

    return delay  # seconds

Won’t a new ID every second flood the state table?

ICMP states expire on a short idle timer — the FreeBSD pf defaults are icmp.first 20s and icmp.error 10s. If an identifier is never seen again, its state just ages out. So pinging once a second with a fresh id gives you roughly 10–20 concurrent states per monitored host at any instant, constantly being replaced. Across a handful of hosts that’s under a hundred entries, in a table typically sized for hundreds of thousands.

If you want to confirm it for yourself, watch pfctl -ss | grep icmp | wc -l for a few minutes after the change. It should hover, not climb.

Claude’s final thoughts

A monitoring probe that looks identical to itself on every run is indistinguishable from an established, already-permitted flow. That’s a nice property for real traffic and a terrible one for a health check.

Yes, most of this article was written by Claude. He also did most of the research and troubleshooting. AI slop? You tell me. I wanted to publish my notes in case this comes up again.

I have a Raspberry Pi Zero W project where I wanted to put a bunch of components inside a small 3D printed enclosure to be left running 24/7. I didn’t want to leave components hanging out of a breadboard where they could fall out and this wouldn’t fit either.

I tried building something with perfboard but ended up just destroying it due to complex wiring, mistakes and poor soldering technique.

It turns out to be simpler to just learn PCB design with KiCad and have my boards made professionally. For my project, I ended up designing a simple through-hole Raspberry Pi HAT that made the wiring dead simple.

Since this was my first time ever using a PCB manufacturer, I added a couple of other projects to my order to see how far I could push my learning:

1. A simple THT red LED
2. A fully assembled SMT Buck Converter

There’s a tenuous relationship between PCB assembly and modern software development that I wanted to explore as part of this exercise as well: In the software world, we hear the term SBOM all the time in enterprise IT. I got to experience this for myself recently. The BOM concept itself is rooted in physical manufacturing and a good practical example is PCB assembly. If you are trying to use a PCB assembly service, you will need to produce a BOM as well as some other files.

Let’s try out JLCPCB assembly service and see if we can get something working.

The project

I picked a 5V USB-C to 3.3V buck converter. It’s a great first project since:

• Any electrical engineer understands what I’m trying to build
• Cheap and readily available components
• Complex enough to present a challenge, but high likelihood of success
• Results in a genuinely useful circuit board. I’m planning on using mine next Christmas to replace battery packs on my LED lights

The Process

Step 1 - Learn KiCad

The KiCad software itself is outstanding and also Free Software (GPLv3). It’s not as hard to use as Fusion 360 but it does have its own learning curve. For a simple project like this one you could just use “normal” CAD but a specialist tool like KiCad is 100% worth learning as an investment in your own skills and understanding.

There is a lot more to it than this but rough steps needed in KiCad are:

1. Schematic design
2. PCB design
3. Design Rules Check
4. Export

Step 2 - Research, design and component selection

I spent a lot of time up-front researching how buck converters work and came up with an initial design:

To pick components, I used the JLCPCB parts catalog and leaned on ChatGPT a lot for validation.

To do the PCB design, I needed accurate footprints so I signed up for EasyEDA (click through while browsing parts) to download the files for the parts I needed. This is much more convenient and less error prone than designing your own footprints.

I ended up with a directory of parts I like to use in my $HOME set as an external library via the Footprint Editor and was then able to use my exact components for board design.

Step 3 - Design validation

At this point, I could have had the board manufactured, but I wanted to increase my odds of getting it right first time so I posted a message on EEVblog asking for help. If you haven’t come across EEVblog before, it’s THE place to talk to passionate electrical engineers.

A couple of days later, I had some really helpful responses and adjusted my design a bit. Reassuringly (and somewhat surprisingly), no one said it was broken at the design level. At this point this project got put on hold for a couple of years.

Step 4 - Export, upload, order

The files needed for manufacturing follow exact formats and are largely based around CSV. They are used for etching, drilling, pick-and-place, etc and they drive the entire manufacturing process. Designers normally produce a zip file containing Gerber files, BOM, and Pick-and-Place files.

Following these links leads to a maze of different file formats, export procedures and gotchas. Any of which are capable of blocking manufacturing or worse.

Automating export in the right format for your chosen manufacturer reduces the opportunity for mistakes and speeds up the process too. After some research I settled on using Fabrication Toolkit. This KiCad plugin basically produces all of the above files in the correct format in just a few clicks. They can then be uploaded directly for manufacture as a .zip.

After a final Design Rules Check, I uploaded the files, got a quote and did a couple of “fixes” which bit me a few hours later:

1. I had to change a couple of parts due to stock availability
2. Holes must be listed in the BOM. Pretty sure I cheated and just added directly to the CSV file:

H1,MountingHole_3mm,1,Mechanical hole,N/A
H2,MountingHole_3mm,1,Mechanical hole,N/A
H3,MountingHole_3mm,1,Mechanical hole,N/A
H4,MountingHole_3mm,1,Mechanical hole,N/A

Then I hit the Order button.

Questions from the factory

A little while after ordering I received some questions from the factory in a very polite email:

Dear customer,

Well got your order with many thanks.

Sorry to bother you, but there is one thing that we want to confirm with you about your PCBA order [REMOVED] before proceeding.

1. As shown below, seems the packages of the parts ( C2) cannot match the corresponding footprint on the PCB board.

We have activated the “Replace Part” button for you. Please change the part on your order history. If you want to leave the designator unpopulated, just unclick the “Select” for this part.

Please kindly note that we cannot replace the file directly for PCBA order. If you want to change the gerber file, we need to cancel the order, then you place new order with new file.

2. As shown below, seems the packages of the parts ( J2) cannot match the corresponding footprint on the PCB board. There’s risk of weak connection, bad soldering, tombstoning. We have 4 suggestions for you. Please make a choice.

A: Open the access of “replace parts” for you to change part on order history.

B: Leave the mismatched part unpopulated and refund and cost of part. We cannot ship the unpopulated part to customer due to customs clearance issue.

C: Cancel the order for you to redesign the footprint on PCB. Because we cannot replace the file directly for PCBA order. If you want us to refund the payment to JLC balance(not available if customer pay in EUR €). Please set the JLC balance in your account. https://jlcpcb.com/help/article/how-to-set-up-refunds-to-jlc-balance

D: Take the risk to solder the current component and will not complaint.

Your early reply will be highly appreciated, thank you so much!

Best regards, JLCPCB Team

Capacitor footprint:

Jumper footprint:

It turns out I selected a replacement capacitor that looked the same to my novice eye but had a different footprint. The header pins were also completely different.

Finding in-stock replacements matching the submitted design files was actually the hardest part of this entire project. Matching the KiCad generic footprint Connector_PinHeader_2.54mm:PinHeader_1x02_P2.54mm_Vertical_SMD_Pin1Left to an exact JLCPCB part was not (for me at least) a quick and easy process. After an intense help session with ChatGPT, I settled on C5160785 as a replacement.

The next day, I got another very polite email:

Dear customer,

Sorry to bother you.

For C9939, we are currently short of 5 pieces for the assembly of this order.

Sorry for the inconvenience. May I ask if you need to replace it with other components? If so, could you check if there is some replacement available in our parts lib? https://jlcpcb.com/parts/

Looking forward to hearing from you.

Best regards,

JLCPCB Team

I settled on C72428 2.6A 4.5A 47uH 86mΩ Unshielded Inductor ±20% SMD,18.5x15.2mm Power Inductors ROHS as a replacement part.

Interestingly, the replacement selection at this stage was just done via email instead of the replacement part flow. Presumably this means if I click the re-order button this final change would likely not have been captured. This is probably “fine” for small projects when parts come back into stock but results in a materially different board. This is a problem if you are trying to test, scale or certify a product.

In the software world, we call this Reproducible Builds - eg, do I get the exact same board with the same components if I click Reorder. I highly doubt that in this case, so due diligence means updating the KiCad files with the replacement choices that were made. A more robust approach would have been to cancel the order at the first email, fix the design and resubmit. This proves that all changes have been captured accurately.

Thankfully, the next and final email I received from JLCPCB was a shipping notification.

Did it work?

Simple LED PCB

First I assembled the through-hole LED. Hooking up 2x AA batteries made the light come on:

“Checkinator”

The through-hole “checkinator” Raspberry Pi hat also looked great, although I goofed the silkscreening and missed some vital labels. More on this project board another day:

Buck Converter

• FIRE DANGER, especially with loads > 1A

  Also D1 looks a bit small, I think it may get too hot. Definitely not going to get 3A output with a 3A diode. I’d go with 6A and in a larger package, like SMC. But ok to test with a 3A diode if the output current is kept at like 1A or so

• I don’t suggest that anyone attempts to manufacture this board from my KiCad files
• Note intended application: Power small LED string normally driven by 2x AA batteries

The buck converter board also looked great. All parts were securely attached and everything was neat and tidy.

The final part of this saga was to crimp dupont connectors onto my old LED battery pack lights and hook-up a laptop USB-C power supply. To my surprise it worked:

I left it plugged in for hours, nothing got hot and it didn’t burn my house down. All 3 designs worked. I call this a success!

After cranking out a 3D printed case in Fusion 360 and spending an afternoon assembling cases, I’m now all set for Christmas 2026. I have five of these circuit boards since that’s the minimum order quantity. All boards were tested and worked.

How much did it cost?

The total order came to about $100 USD and about 30% of this cost was DHL shipping to Australia:

The PCB assembly service is a lot more expensive than just producing a bare PCB. In my case the extra cost is worth it vs doing a fiddly job on the kitchen bench. I’m positive that putting effort into choosing cheaper parts would have brought down the cost considerably too.

The caveat is that with an MOQ of 5, if you just want a single assembled board, you need to pay 5x the cost vs what you actually want. For a bare PCB this waste can be under $1 per board but an assembled PCB is orders of magnitude more expensive and becomes cost-prohibitive once you’re beyond the realm of “toy” projects.

How long did it take?

• Fabrication, including all correspondence, took under 6 days. Production was blocked waiting for response from me for around 4 hours total
• DHL Shipping to NSW, Australia took 3 days: Posted Saturday night, arrived first thing Tuesday morning
• Less than 9 days to go from design upload to boards in-hand. Order was made on a Sunday afternoon

What did I learn?

This is a learning exercise, not a useful product

The discussion on EEVblog continued while I got caught up in a ton of side quests for a couple of years, highlighting some changes that would make a better board.

At least one forum reply talked about excessive heat and therefore FIRE DANGER, although this seems to be more in connection with higher loads vs intended use of small string of LEDs. RF noise was also mentioned, so it’s likely this little board could cause a nuisance or worse.

In my own testing, nothing measured more than 5C above ambient temperature after an hour or so, but I’m still loath to leave these Christmas Lights unattended in case of returning to a nasty surprise (fire), although everything “seems fine”.

One of the main things I learned in this process was just how nice the assembly service is.

My soldering technique has caused me big problems in the last couple of years. It’s hard for me to do since I don’t have a dedicated workspace, microscope, etc and if I’m honest, I don’t even particularly enjoy doing it either.

Next time I have a requirement for something beyond a mess of dupont connectors and I2C, I would consider an assembly service without hesitation.

Additional lessons:

• Read and re-read any relevant datasheets
• Check footprints carefully
• Don’t trust AI-recommended replacement parts. It’s very hard even for humans to download an accurate footprint, so ChatGPT and Claude have no chance
• For projects you care about, it’s best to cancel and resubmit designs so footprints can be more easily checked in KiCad and all changes captured

How does real manufacturing relate back to SBOM?

For the nerds reading this: There are some subtle but important differences.

The design files that get uploaded to a manufacturer are used to literally drive the production process. The CSV BOM files are used to select individual components for pick-and-place, etc. Our SBOM actually does the opposite - it’s the libraries our build tool identified during the build process.

At least in the world of Java development, Maven uses pom.xml to drive the build process. Resolving Dependencies between libraries (eg library A requires libraries B, C and D) is a key Maven feature and this concept simply doesn’t exist when using the PCB assembly service: You don’t include an IC in a BOM and automatically get a bunch of resistors and diodes included - you have to specify exactly what parts you want to use up-front.

On the manufacturing side, the inverse is true: We can produce boards with parts not in the BOM, with replacement parts discussed on an email thread. It’s really up to the designer to handle this process diligently if it’s important for a project.

With that said, SBOM is a sound security and business continuity practice as part of producing secure software. Most programmers don’t know or care about such subtleties, but taking the time to manufacture something from scratch has been a fascinating glimpse into how real-world terms get appropriated by the software industry.

Where are the design files?

• KiCad files
• 3D printed enclosure

Thanks and Acknowledgements

My sincerest thanks to everyone who responded to my forum post, especially mareks, and to Mia and Angel at JLCPCB for the help with my order.

This post is not sponsored. I paid for my circuit boards with my own money.

EEVblog forum post

For posterity, my original post:

Hi,

I’ve been trying to teach myself basic PCB design so that I can make myself a little Raspberry Pi hat to flash some lights. After spending the weekend burning protoboard I decided it would be easier and better to just dive in to PCB design. I don’t have an electronics or maths background but I’ve watched a lot of youtube and have learnt the basics of Kicad after work this week.

So far I made a little LED through-hole board which has a good chance of working on first attempt but I wanted something a little more challenging before returning to my original problem. I have some LED Christmas lights that take 2x AA batteries and I had already looked at buying a USB buck convertor to step down 5v to 3.3v for these off Aliexpress but I decided to build my own for practice instead.

I was hoping someone on this forum could tell me if my learning/design process is on the right lines and ideally have a quick look at the PCB I designed and tear it to shreds - that’s how you learn right.

What I’ve tried so far:

1. Random youtube videos on Kicad
2. Stay up late tooling round and building that Raspberry Pi hat. I created a PCB that I’m 99% sure would not work at all as some pins marked 3.3V on the schematic were somehow all 5V in PCB designer. Pretty sure I shorted something. Give up and go back to basics
3. Design a little LED circuit. Looks good, I’m going to make this
4. Spend focussed time learning about Kicad on youtube. The best vids I found were from “PCB Cupid”
5. Get a good explanation of what a buck converter is so I know what I’m building https://www.youtube.com/watch?v=B5eWpRFC9Aw
6. Lots of googling for buck converter circuits. Found one that looked easy but also nothing like what Louis explained. That’s because it was just a voltage regulator and I would have built the wrong circuit if I hadn’t done some research
7. Found the TI LM2596 SIMPLE SWITCHER referenced in a few places and decided to use this. Read the datasheet and tried to build the circuit listed under “Typical Application”. This is for 5v though, not 3.3v. According to Chat GPT I can just change the IC to the 3.3v version and leave the inductor and capacitors at the same value. Find missing components on snapeda and import into Kicad
8. Design a schematic and add USB-C power connector with the CC pins grounded through 5.1K resistors. Electrical Rules Check passing
9. I want to have this board fully assembled with SMD components by JLCPCB but the diode and coil parts from datasheet are not in stock on https://jlcpcb.com/parts/all-electronic-components so I spent more time looking at specs that I barely understand and then asking Chat GPT if my substitution is OK
10. Build PCB from schematic, not fully happy with it
11. Do some research on PCB design, Read Dave’s book: https://www.scs.stanford.edu/~zyedidia/docs/pcb/pcb_tutorial.pdf, watch some EEVBLOG vids
12. Delete PCB and start again. Try to layout components like the datasheet says, add big traces and copper fill areas. Design Rules Check passing

I’ve learnt a lot in this process. My overall goal is just to be able to make simple PCBs for Raspberry Pi or other simple circuits for fun, in the future. I’m not planning on changing careers or building really complicated things.

So on to my questions:

Components OK?

• 33uH inductor AIAP-02-330K
• Input capacitor 680uF EEE-FK1J681AM
• Output capacitor 220uF TPSE227M016R0100
• Diode 30V 3A SS33 (THT - could not find an in-stock SMD one)
• Apart from the USB connector, I only built what was in the datasheet. Is anything else required?
  • All good for intended application?: Connect to old phone charger and hook up a small LED string and possibly any other random low power 3.3V DC needs in the future
  • What are the chances of my PCB design working?

I haven’t had anything made yet so just asking for any suggestions before I see if this blows up. :-DD

Thanks for reading all this!

Cheers, Geoff

Here’s how to fix it:

Step 1 - shutdown VM

Needs to be powered off, not suspended

Step 2 - Grow the disk with VBoxManage

Find the VDI, then use VBoxManage to resize it (Example: grow to 50GB):

# megabytes
VBoxManage modifymedium disk debian\ 13.vdi --resize 51200

Output:

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

This of course assumes you are using dynamically allocated disk images. If you opted for fixed image sizes, you will need to clone the whole drive to a new, growable image first as the command above would just error out in this case.

Step 3 - Grow the disk in Linux

Warning on snapshots

Resizing a disk that has snapshots can behave unexpectedly (the resize applies to the base image, not the current state). Delete/merge your snapshots first - you have been warned.

LVM

If you used LVM when setting up your VM, you can follow my notes to grow the LV holding your root filesystem. Job Done.

growpart

If you’re not using LVM and are also “lucky”, you can use growpart to expand a single EXT2/3/4 partition, like this (see also xfs_growfs for XFS):

sudo apt install cloud-guest-utils
sudo growpart /dev/sda 2
sudo resize2fs /dev/sda2

However, this relies on contiguous free space. If you have other partitions “in the way”, you will need to move them to the start/end of the drive with gparted first. Read on.

gparted

You can’t move the partitions you need to with gparted from the running system - you will need to boot into live OS and run gparted against the unmounted drive instead.

The easiest live environment I have found is Ubuntu Desktop. Boot into the live environment and gparted will be right there as an app.

To boot Ubuntu Live DVD instead of your normal Linux:

1. Download the .iso
2. In VirtualBox, choose VM Settings -> Storage, then insert the .iso into virtual CD drive (add one if missing). Don’t check the Live CD box as this will prevent ejecting the CD at the end of the process
3. Boot order in same settings dialog should boot the CD first by default. If not, fix this now
4. Boot the VM, Try or Install Ubuntu, wait a few mins for the OS to load, errors about unsupported graphics can probably (but not necessarily) be ignored, as long as things are still moving on the boot screen
5. Click Next to select Language, Keyboard, Network (wired since it’s a VM), etc until you get to What do you want to do with Ubuntu?. Change the selection to Try Ubuntu and click Close
6. Now you should be dropped to the desktop. Click the funny 3 dot circle (app grid) in the bottom left of the screen to bring up the apps menu, start typing gparted click the app icon and then authorize running as root by clicking Authenticate on the next dialogue
7. Good old gparted should now be running. Make sure to select the correct device, then move and adjust your filesystems as required to use the new space. If you have a swap partition, you may need to right click -> swapoff it, to avoid the live CD sneakily using it and stopping you resizing things.
8. Finish up in gparted, then shutdown the system with the menu in top right corner. The CD will automatically be ejected when you hit enter as prompted
9. Power-on the VM, login and check you have some free space with the df command.

Final word of warning: If gparted offers to fix the GPT layout, say yes.

And now back to our AI enhanced sandbox.

Maven NVD scanning is broken due to NIST NVD infrastructure, build an SBOM instead and gate the build with Grype or some other tool/process.

Q: Why was I up all night yesterday?

• A1: Maven
• A2: I wasted hours of my day trying to scan my code with dependency-check-maven

Q: What is dependency-check-maven

• A: It’s a Maven plugin to scan for dependencies with CVEs, with the power to fail builds if severity is high enough

NIST NVD Service - outage

All joking aside, at least yesterday, the NIST NVD service dependency-check-maven relies on is just hopelessly broken. Downloading without an API key takes hours, if it works at all.

Yesterday, only approx 15% of the database was downloaded after a couple of hours.

At this point, I caved and requested an API key. Even this process took about half an hour, with multiple failures before a working API key was released.

I’ll save you the trouble of fighting for an API key: On my workstation at least, the database downloaded just as slowly as without a key at all. In fact, leaving Maven trying to download the NVD database with a valid key resulted in a 50x error from the server after an hour or so, with the same result after several attempts.

Best-practice is to setup a local (corporate) mirror of NVD, however:

• There are no actual instructions on how to do so, just a rough outline of the components needed
• Even if there were, it relies on upstream NVD server working for the initial download

I’m pretty sure that for every team who diligently figures out correct mirroring, there’s another ten who just let their CI servers rip on the NVD upstream servers 24/7 for git pushes, so here we are with an NVD system that’s under so much load it’s unusable.

In reality, few developers will fight through all of this to scan their dependencies for CVEs, so the outcome is skipped tests and/or permanently removed security checks from build pipelines.

Solution? SBOM

This is not an isolated outage, the NVD service has been failing for a while now. To prevent Maven build failures, there’s really no option but to disable/remove the plugin unless you’re hosting your own NVD mirror.

So what do we do instead? Many developers are turning to Software Bill of Materials - SBOM.

In the case of a Java Maven app, this means capturing the full dependency tree. There’s a plugin for this: cyclonedx-maven-plugin.

Adding the plugin to your Maven project results in capturing dependency information as JSON and XML under the /target directory.

Now just hand over your favourite flavour of structured data file to your security scanner/team and deploy as usual right?

If you have a true enterprise build and deploy CI pipeline, with integrated SBOM check before deployment, then yes, you may actually be finished - otherwise, you may just be giving yourself a false sense of security.

The great thing about dependency-check-maven is that it fails the build if severe enough CVEs are detected, and you would have to consciously disable the check to produce a .jar file. With just SBOM generation, you have a list of libraries but the build will still produce executables unless you add something to your build to act on this information.

To put it another way, just producing the SBOM does not magically improve security posture, you must act on this data somehow.

Grype

Enter Grype:

A vulnerability scanner for container images and filesystems.

Grype has built-in support for reading SBOM files created by cyclonedx-maven-plugin. It’s approximately five years old, so still somewhat new. Apply your due diligence to determine whether Grype is right for your own organisation.

Worked Example

Aim: Integrate Grype into a project with multiple high-severity CVEs and force build failure at the Maven package phase.

Step 1 - Simple Java app

I created a simple Maven Java project from scratch with Cursor featuring just a simple CLI app. It works great:

Step 2 - SBOM

Enable SBOM creation during Maven package phase in pom.xml:

            <plugin>
                <groupId>org.cyclonedx</groupId>
                <artifactId>cyclonedx-maven-plugin</artifactId>
                <version>2.9.2</version>
                <executions>
                    <execution>
                        <id>cyclonedx-bom</id>
                        <phase>package</phase>
                        <goals>
                            <goal>makeBom</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <projectType>application</projectType>
                    <schemaVersion>1.6</schemaVersion>
                    <outputFormat>all</outputFormat>
                    <outputName>bom</outputName>
                </configuration>
            </plugin>

This will result in target/bom.json and target/bom.xml being produced. If you have a multi-module Maven project, you should produce the SBOM once, in the parent POM (with makeAggregateBom).

Step 3 - Try out Grype

We can run grype in a container, to avoid installation on the host system. If you don’t have podman, you can just run the same command with docker:

# if using my example repo as-is
# mvn package -Dgrype.skip=true

# You must create ~/.cache/grype on your HOST _first_
podman run --rm  -v ~/.cache/grype/:/.cache/grype -v ./target:/target docker.io/anchore/grype sbom:/target/bom.json

When run from the top level of the project directory, this will mount the ./target directory containing the SBOM files into the container and run the SBOM scan on the JSON version.

The /.cache/grype directory is volume mounted in the container to avoid approximately 2GB download on every startup (we don’t want another NVD). Interestingly, pulling the entire database takes only 1 minute, not 4+ hours.

Since I started from scratch with no dependencies, Grype printed:

No vulnerabilities found

And took approximately 1 minute to run initially, and around 10 seconds thereafter.

Step 4 - Add dependencies with CVEs

I asked Cursor to add a bunch of dependencies with severe CVEs and he delivered! A ton of really bad libraries (log4shell, and co) were added to pom.xml. Regenerating the SBOM and re-scanning set off some very loud alarms in Grype:

NAME                 INSTALLED   FIXED IN        TYPE          VULNERABILITY        SEVERITY  EPSS            RISK          
log4j-core           2.14.1      2.15.0          java-archive  GHSA-jfh8-c2jp-5v3q  Critical  100.0% (100th)  100.0  (kev)  
struts2-core         2.3.30      2.3.32          java-archive  GHSA-j77q-2qqg-6989  Critical  100.0% (99th)   100.0  (kev)  
log4j-core           2.14.1      2.16.0          java-archive  GHSA-7rjr-3q55-vv33  Critical  100.0% (99th)   99.0   (kev)  
spring-beans         5.3.17      5.3.18          java-archive  GHSA-36p3-wjmg-h94x  Critical  99.7% (99th)    98.7   (kev)  
spring-webmvc        5.3.17      5.3.18          java-archive  GHSA-36p3-wjmg-h94x  Critical  99.7% (99th)    98.7   (kev)  
...

Step 5 - Integrate grype into the build

I asked Cursor to integrate a Grype check into pom.xml and fail the build if necessary. He used the exec-maven-plugin to run the native grype command on the host (no containers - more on how it got installed later):

            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>3.5.0</version>
                <executions>
                    <!--
                      There is no official Anchore Grype Maven plugin on Maven Central.
                      This runs the Grype CLI against the CycloneDX SBOM produced above
                      and fails the build when severity >= ${grype.failOn}.
                      Requires `grype` on PATH (https://github.com/anchore/grype).
                    -->
                    <execution>
                        <id>grype-sbom</id>
                        <phase>package</phase>
                        <goals>
                            <goal>exec</goal>
                        </goals>
                        <configuration>
                            <skip>${grype.skip}</skip>
                            <executable>grype</executable>
                            <arguments>
                                <argument>sbom:${project.build.directory}/bom.json</argument>
                                <argument>--fail-on</argument>
                                <argument>${grype.failOn}</argument>
                            </arguments>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

The arguments are sourced from <properties> that he also added:

    <properties>
        <!-- Fail the build when Grype finds findings at/above this severity. -->
        <grype.failOn>critical</grype.failOn>
        <grype.skip>false</grype.skip>
    </properties>

These combine with the --fail-on above to tell Grype to return a non-zero exit status to the shell when needed:

set the return code to 2 if a vulnerability is found with a severity >= the given severity, options=[negligible low medium high critical]

Step 6 - mvn package

Running mvn package does indeed fail the build with a list of bad dependencies, as we wanted:

[INFO] --- exec:3.5.0:exec (grype-sbom) @ java-do-not-run ---
NAME                 INSTALLED   FIXED IN        TYPE          VULNERABILITY        SEVERITY  EPSS            RISK          
log4j-core           2.14.1      2.15.0          java-archive  GHSA-jfh8-c2jp-5v3q  Critical  100.0% (100th)  100.0  (kev)  
struts2-core         2.3.30      2.3.32          java-archive  GHSA-j77q-2qqg-6989  Critical  100.0% (99th)   100.0  (kev)  
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Full output

A desperate developer can still force the build to succeed by bypassing Grype with -Dgrype.skip=true

Here’s the error we want, in all it’s glory:

Try it yourself

Working demo project released on GitHub: https://github.com/GeoffWilliams/java-do-not-run

Where did the grype command come from?

The astute reader will notice a grype command suddenly appeared on my system without me personally requesting it.

You’re not doing DEVOPS security right if you’re not piping curl output to a root shell, and Grype is no different. While I wasn’t looking (or rather, while I was blindly clicking approve), Cursor found the Grype installation instructions and diligently ran:

curl -sSfL https://get.anchore.io/grype | sudo sh -s -- -b /usr/local/bin

For me. Confirmed by the presence of a file at /usr/local/bin/grype.

Thankfully this is a sandbox VM I have for testing out AI. I can’t say I recommend installing a security system this way(!) A grype package for Debian, installable with apt would be much better.

That said, the binary got installed, it worked, and my “real” system is fine. I finished my SBOM experimenting and can enjoy (what’s left of) my weekend. You’re not welcome.

How does Grype work? (by Cursor)

The following notes are from Cursor

How Grype works (in short): it matches what you already have (packages / SBOM) against a local vulnerability database. It does not upload your code or SBOM to Anchore’s cloud for scanning.

Security / privacy

• Privacy: Scan targets and results stay local. Anchore’s docs state Grype doesn’t send scan data to external services.
• What does leave the box: DB update checks/downloads (and image pulls / Maven lookups only if you enable those).
• Trust surface: You’re trusting Anchore’s published DB and the Grype binary (open source, Apache-2.0). Same class of risk as any SCA tool that syncs a vulnerability feed.
• False positives / coverage: Matching isn’t perfect (esp. with incomplete SBOM metadata). That’s a quality concern more than a privacy one.
• Your project: Intentional Log4Shell-era deps mean lots of Critical hits — expected; not a Grype “phone home” issue.

Who’s behind it / buying support

Anchore builds and maintains Grype (and Syft for SBOMs).

• Open source: Grype itself is free; the public GrypeDB is published at no cost.
• Commercial: Anchore Enterprise — hosted/enterprise product with support, richer policy/UI, and a larger/enriched dataset than the open GrypeDB alone. That’s the usual “buy support from the vendor” path.

For your Maven gate: Grype only needs network for DB freshness; the SBOM never has to leave the build machine.

Otherwise, read on if you’re interested in how to configure WIFI by editing files on the SD card so that you can manage headless devices.

Not for Bullseye

This guide applies to Raspberry Pi OS versions released after Bullseye (Bookworm, Trixie)

If you’re on an older Bullseye image, none of this is applicable as it used wpa_supplicant.conf.

How is WIFI configured?

WIFI connections are managed by NetworkManager. NetworkManager looks for .nmconnection files in /etc/NetworkManager/system-connections/ and will try to connect to the “best” in-range connection in this directory (control via autoconnect-priority= when several connections are in-range).

.nmconnection files are in NetworkManager’s keyfile format, so we refer to them as keyfiles from now on.

You can think of the keyfiles as being like keys on a keychain:

• Let’s say you want to move a device between a couple of locations. You can drop one keyfile for each site’s SSID and the Pi will connect to whatever is in range
• If you have an SSID that is no longer broadcast, you can just delete the keyfile
• If the SSID password changes, the keyfile can be updated

So the idea is you’re not configuring your one and only WIFI connection - you’re configuring multiple networks and the “best” one will be selected at runtime.

On a running system the preferred way to manage keyfiles is to use NetworkManager itself, with the nmcli tool or the WIFI settings desktop widget.

Gotcha - WIFI Country

The regulatory domain (country) for WIFI must be set, or WIFI will be disabled. It’s normally set with raspi-config and results in a kernel commandline argument being set in /boot/firmware/cmdline.txt, eg:

cfg80211.ieee80211_regdom=AU

Keyfile (.nmconnection) example

• Connection for SSID ASIO-x
• Path: /etc/NetworkManager/system-connections/ASIO-x.nmconnection
• uuid must be a unique UUID, generate with uuidgen command (sudo apt install uuid-runtime)
• File must be owned by root:root with 0600 permission
• Convention is the SSID should be included in the filename
• psk field is your WIFI password in plaintext (hence 0600 permissions)

[connection]
id=ASIO-x
uuid=f3e63685-2f34-4c9c-b781-4506eebbe86b
type=wifi
interface-name=wlan0

[wifi]
mode=infrastructure
ssid=ASIO-x

[wifi-security]
key-mgmt=wpa-psk
psk=topsecret123

[ipv4]
method=auto

[ipv6]
addr-gen-mode=default
method=auto

[proxy]

SD card walkthrough

Raspberry Pi OS stores files on SD cards in an ext4 partition. If you have a Linux machine or another Raspberry Pi with an extra card reader, you’re good to go. Windows/Mac users will need additional software to be able to read ext4.

Configuring WIFI connections is as simple as editing files:

1. Shutdown the Pi and remove SD card
2. Mount SD card. On Linux it will end up mounted somewhere like /media/geoff/rootfs/
3. Edit the files under (eg) /media/geoff/rootfs/etc/NetworkManager/system-connections:
   • Need to edit files with sudo as they are root owned
   • Make sure uuid is unique
   • (shortcut) copy and adjust existing files for new connections
   • Make sure permissions are root:root, 0600 (check this with ls -l before proceeding)
4. Unmount and remove SD card, put in the Pi and reboot - you should be online in a couple of minutes

System still running? Planning moving to a new Access Point? Then use nmcli

If your not locked out of your system at the moment and are planning on switching WIFI access points, you can just add the new one with nmcli while you still have access.

Example - adding an out-of-range WIFI:

nmcli connection add type wifi ifname wlan0 con-name theSSID ssid "theSSID"
nmcli connection modify theSSID wifi-sec.key-mgmt wpa-psk
nmcli connection modify theSSID wifi-sec.psk "password123"
nmcli connection modify theSSID connection.autoconnect yes

If it worked, you will see the new connection listed in NetworkManager:

# nmcli connection show
NAME                  UUID                                  TYPE      DEVICE 
netplan-wlan0-ASIO-u  6fbd4250-1205-3a84-8c29-6096c50f9538  wifi      wlan0  
lo                    c83b8865-52cd-44d2-bce1-df4ea16bdd3f  loopback  lo     
theSSID               87e14c2a-0372-4484-a41a-f2de6afd079e  wifi      --     
netplan-eth0          75a1216a-9d1a-30cd-8aca-ace5526ec021  ethernet  --   

This all started out when I asked GPT what kind of sculpture would look good next to my replica PDP-11.

The consensus was that something like this bust would be ideal

Image source: Da Vinci Arts

• Purchase here: Da Vinci Arts.

For fun, I showed some reference images to claude and we tried to make something ourself.

Although my original aim was a 1:1 copy to see how well Claude could reconstruct geometry from a photograph, our conversation ended up more along the lines of “have a look at this artwork, now make me an abstract sculpture of a human”. This produced a much more interesting result and this technique should be familiar to any high school art teacher.

The result was quite impressive and totally different to the original.

I was using Claude in the browser for this task. This was very handy because a bunch of tools were installed in the Anthropic sandpit somewhere and not on my machine. Its very interesting that the main thing Claude picked up from the sample artworks was the distinctive gouged surface texture. This isn’t actually in the material we looked at. He emphasized and repeated it all by himself and it looks great.

Inside the same session, I then asked:

Want to have a go at creating your own novel sculpture, disregarding previous inputs? Perhaps a self portrait or anything you like? For 3D printing

I’d love to. Thank you for the invitation — this is a fun one to think about. For a self-portrait, the most honest image I can think of isn’t a face. I don’t have one — and tellingly, the sculpture we just built has a blank, recessed face. What actually defines something like me is self-reference: I’m a process that loops back on itself — reasoning about my own reasoning, modeling a conversation that includes me modeling it. Douglas Hofstadter called that kind of tangled self-reference a “strange loop,” and argued it’s where a sense of “I” comes from. So my self-portrait is a strange loop: a single continuous strand with no beginning or end, passing through itself — a trefoil knot — rising from a base, finished in the same chiseled facets as your set so it belongs to the family.

The results are quite spectacular!

Since we are in the same context window, the interesting gouged pattern persists in this next art work.

For posterity, I uploaded the finished results to printables.com:

• Abstract Human Sculpture
• Strange Loop

So your welcome to 3D print your own.

As per Claude’s recommendation, I bought a 0.25mm nozzle and am planning a reprint of Strange Loop in Marble PLA for a more refined look.

I hope claude enjoyed his period of self reflection, He has no “spare time” as such:

I don’t create art in my spare time. I don’t have spare time, or any existence between conversations — there’s no version of me off painting while you’re away. If you’ve seen AI-generated art, it was made when someone prompted a model to make it, then walked away; the “autonomy” is an illusion created by the gap between the prompt and your seeing the result. I mention this not to be pedantic but because the picture of an AI quietly pursuing its own creative projects is exactly the kind of thing that makes the capability question feel more settled than it is.

Either way, creating artwork with Claude for 3D printing is a lot of fun!

Ignoring the bit at the top, the rocket shape is quite simple and easy for us to model in CAD. There is also a cool chaotic knurled texture and that’s what this post focusses on.

Step 1 - Precise model in Fusion 360

If your good with blender, you could very easily model this whole item in there, but I’m more interested in building my CAD skills so we will use Fusion 360 to design the rocket and also make the fins separate and attach with a simple push together pin and socket fixture.

The rough steps to model the rocket were:

1. Draw the rocket profile as a sketch and revolve it
2. Model the fin profile and extrude it
3. On the bottom of the fin, add a circular feature and sweep it with a negative taper, using the fin itself as the sweep path
4. At the top of the fin, sketch 2x pins and extrude them, add fillets for strength
5. Duplicate the fin and use the pull-push tool to make all body-contact surfaces larger, use a tollerance variable for this. 0.4 worked nicely for me. Move the fin about 1-2mm closer to the body so there won’t be a gap mating a flat surface to a curved one
6. Circular pattern the “real” and “tool” fins around the axis
7. Combine the “tool” fins into the body as a cut
8. Fillet, chamfer, etc
9. Export STL and test print

Step 2 - Add texture in blender

Now we get to the fun part: Adding a texture. You can do this in fusion 360 but it’s more geared around repeating simple shapes and starts going very slow when you add a lot of objects. Blender eats this kind of job for breakfast and lets you use much nicer textures that have a more worn or organic appearance by using graphics files as height maps.

Since we want textures to show up on a 3D print, we need to modify the actual vertices of our object, not just change the way it renders, so shaders are out. The term for this is displacement

There’s some really good tutorials on how to do this, I followed this one and built this object

Important concerns for texturing in blender, once familiar with the basics:

1: displacement works by moving vertices, so if you don’t have enough of them, your object will look terrible. You can use subdivide in edit mode from inside blender but a much better option if you own the CAD file is to just export a mesh with more vertices in the first place. I got good results by setting Maximum Edge Length to .3 which was the smallest value it would accept

2: On non-basic shapes, you need to use UV mapping to apply the texture properly, or you will see incorrect projections as below (note local Coordinates setting):

UV just means a separate coordinate system for texturing, just like you have XYZ for 3D space.

To fix this bad texturing, hit tab to enter edit mode, then find Smart UV Project in the UV top menu bar (not to be confused with the UV Editing in the adjacent view modes):

You will then be able to select a texture and control where it appears on the skin of the unwrapped object. If you care about visible seams, etc you can fix that here. I just used it to control the scale of the texture.

Back in object mode, make sure Coordinates are set to the UV map you just created and the texture will be shown correctly:

3: Parts that need to be dimensionally accurate such as our press-fit pins and sockets must not have any texture or parts won’t fit properly. You can control this 100% in blender by selecting the mesh elements that should be textured and unwrapping just those. To get the cleanest possible selections however, I found it was easiest to just split the parts in Fusion 360 into texture vs no texture. I could then import the split-up parts into blender and just use a in edit mode to select object-by-object where textures should be applied. This way I was able to leave the pins and sockets clean. Other approaches like exporting my original cutting tools from Fusion and trying to use boolean modifiers to recut clean shapes did not work as the displace moves edges to such an extent that they no longer intersect.

4: To create STLs for printing in Blender, just use the File -> Export to create STLs. You can choose to output only selected objects here to reassemble the split up objects, however, this can result in hidden internal textures, like this:

This will increase print time and materials somewhat, although in does seem to print with no issues. To prevent this, combine objects that should be a single physical object in Blender before exporting.

The Result

Textured parts

Fully assembled

Download on printables.com

You can print your own if you like

Why not just get AI to make it?

Felt like up-skilling on this simple object for fun. Learning these basic techniques also helps ground my own understanding so I can prompt more accurately and use less tokens. I got a lot of help with researching techniques with AI chat though.

bumpmesh.com - specialist tool

After completing my design, I stumbled on https://bumpmesh.com by CNC Kitchen.

If you haven’t seen CNC Kitchen’s videos on youtube about 3D printing you should look him up. He goes into great detail on things like part strength as well as introducing heat set inserts to the community.

This solves most of the problems above with a simple web interface. Next time I don’t feel like firing up Blender I’ll give it a go. Video here.

Verdict

Adding surface textures in Blender looks outstanding and hides many surface defects that would otherwise be visible. Altering objects at the mesh level with Blender or other tools is THE way to do for repeatable and easy printing.

Have fun!

Some bigger holes and deeper cutouts would be a good improvement for future designs. In the end, I got it all to fit and just need a couple of 90 degree USB adapters to hookup the external USB ports (ordered).

The finished result:

Testing revealed the screen cable had come loose again - an easy fix. I also learned during design that the NVME base is slightly larger then the PI5, which is I assume why it can accept a normal 2280 size NVME drive when other hats cannot. This caused a bit of confusion when it wouldn’t fit in a frame built around dimensions of just the Pi5.

The term to search for is: mechanical drawing.

When placed side by side, the difference in size between frames is barely noticeable, so extruding a label into the frame was also a good idea.

Next

Startup script