The year that was, 2024

The end of the year approaches and another strange one. Far too much work, and I really don't feel like I've done a lot. However, the actual results are quite different so I'm not sure why I feel less than satisfied with myself. Let's dig into the detail - there is certainly more than Helldivers 2 and Dave the Diver.

• 13 posts on this blog (14 including this one) - tenth consecutive year of a post per month
• Continued posting my blog posts on LinkedIn and getting some good responses
• Passed my Magistrates application interview
• Another year of the Year in Pictures - tenth year and we're up to 47 photographers
• 347 Github contributions - well up on the last few years
• Including extensive rewrite of styling for Year in Pics site, removing old Foundation 5
• And a performance rewrite which cut build time from 85 seconds to 6
• Complete upgrade to all Ruby apps and code including migrating Postgres databases (again)
• Changed my email setup again - sigh
• Resurrected an old Pixel C
• Rebuilt my desktop computer
• Updated my music system
• Another year of Saturday Pain and made a t-shirt
• Learned to make candles
• More cooking
• Started writing D&D again
• Started bread again! Although not a great first run...
• Made a Lego Piranha plant, Optimus Prime, and URB Studio
• Made a dice display
• Showed my photos at the St Swithins creativity event
• Pie making cookery course
• Ran quarterly dinner parties - a fancy name for games and food
• Made a few different photo decorated items and books
• Escaped another couple of escape rooms
• Another year of exercise with a personal trainer
• Ongoing good physiotherapy regime
• Spoke at a conference about AI in the charity sector
• Another year on the Data for London Advisory Board
• Yet more instagramming
• More photos on my Flickr stream

Resolution count - 5/10. Not bad, and a lot better than last year.

So yes, this has been an odd year. I've achieved a lot more than I thought despite a rough year at work. A year ago I wrote this, for the millionth year in a row:

Looking forward to 2024, I'm going to write the same thing I write every year. I need to spend less time working and more time living.

Well, I'm happy to say I worked it out this year - I finally found a way to regenerate my energy and spend more time on the things I enjoy. Unfortunately, that way was leaving my job so it feels a bit all or nothing...

Overall though, I have two thoughts about 2024. First, I am quite pleased with how much I did this year. This time last year I struggled to write the above list, and really felt I was bulking it out with non-events. This year I managed to write a lot more and the general quality feels a lot higher. Good news! On the slight downside, far too many of these are from the last couple of months of the year - that is, after removing work. So I still have no idea how to balance my life properly - hurrah for adulting.

Second, as I said in the intro I don't feel like I've done a lot. The evidence is there - I'm wrong and I did Many Things - so why is the emotional component not? Some is because the year has been backloaded, so for much of the year maybe that feeling is more accurate ... but still. What would I feel to be a successful year?

My low energy levels is likely partially a cause. For the second year running I have ended the year with another series of blood tests for fatigue - clear for the second time, fortunately, but the fact I keep having these tests is not a good sign. I really need to bring my physical and mental health to the point where normal fluctuations don't keep dipping me into a bad place.

But overall, I don't have a good answer to the question about why I am not satisfied with my own achievements. I do have a bit of an opportunity at the start of 2025. Being between jobs I'm at a natural point to be introspective and really think about what I want to achieve going forward, and also what would make me happy. So 2025 is going to be the year of working out where I want my life to go, along with actually doing some of it. Oh and getting a job, I guess. Boo.

One thing I do need carefully consider is how I interact with other people and the world. I have definitely withdrawn over 2024, mostly as a defence mechanism. Through tiredness and some bad experiences I deliberately focused on the things in my life over which I have some control, which is to say I looked inward. This helped, but I think the balance is off. I have the inclination to become a hermit if left to my own devices so I need to make sure I am remembering to look outward too.

Right, enough ruminating. 2024 was objectively a decent year, if hard work. I hope for more from 2025, but to add in the emotional component. In the 2021 version of this post, I wrote that I wanted to find and restore my sense of wonder. I think I did, but it has slipped again. So this year I want to bring it back again.

It is time. Onwards! To 2025!

Leaving Macmillan

Back in mid-October I closed my Macmillan laptop (my MacTop, if you will) for the final time, leaving my role as Director of Engineering. I left behind the Engineering division I had created and some excellent and passionate people - not an easy decision and I'm still working through how I feel about everything, including the wider Tech industry. This is one reason it has taken so long to write this reflection.

I was at Macmillan for nearly three years. When I started, the organisation had some in-house technical capability but the people were scattered around working under project managers on systems that were badly in need of modernisation. I can write lots about how we created technical strategy, developed a software engineering culture and community and brought in greater automation, and all of this would be true, but the most important work was bringing the software engineers together and giving them a voice. The most important work is always with the people.

Over these three years I've seen many people grow into new versions of themselves by giving them space and encouragement. Some have embraced the wider context of their roles - looking beyond the specific technical problem to why it's important and realising they have a voice in this space, which can result in better solutions. Some have taken better ownership of their work, owning processes rather than simply implementing them and then flourishing as they realise they can improve and refine. Others have been able to express the needs of a previously ignored service - very common with infrastructure elements - and deal with issues and annoyances to move those areas forward. Still others have taken on line management or team leadership duties and started their journey into wider leadership activities.

I've spent the last eight years or so of my career trying to show the value of technical voices at all levels in an organisation, so having the opportunity to demonstrate this in action while enabling lots of people to grow and flourish has been great.

On a personal level, I've also grown a lot in these three years. I've learned a ton about being part of a director cohort in an organisation, and this is the first time I've worked in the charity sector. I was also recognised at an industry event as part of a group of up and coming "next CIO"s which is not only fantastic for the ego, it's also something I wouldn't have even considered a couple of years earlier. I've experienced several forms of restructure, all of which have taught me something new, and I've been responsible for shifting organisation culture in ways that go far beyond what I've done in the past.

I've also worked with some amazing people who have helped shaped my thinking and career in hugely significant ways. I'm not going to name names, but if you think it's you you're probably right. If I worked closely with you and you DON'T think it's you, you're probably wrong. In both cases, thank you.

So why leave? Primarily, the organisation needs to crack some really knotty legacy technology issues. I've been working on these since day one, and I believe I've built foundations on which solutions can now surface. However this has been a long journey, including migrating physical server rooms and a load of hiring and restructuring, and I did not feel I was the right person to take things forward in the next phase. The org needs someone fresh and enthusiastic for the challenge and after working on this for three years, I need a new problem to re-motivate me.

However, I am still sad to leave behind some excellent people - some of whom I've helped make serious changes to their careers. I'm going to miss them, and I hope I can work with them again. I've been part of a huge change in Macmillan technology, and this is something for which I can feel some pride. A friend says that a mark of success is leaving the place better than you found it, and I think by that metric I've done ok.

Looking forward, I'm enjoying some time away from work and considering my options. Reading back my Leaving GDS post was an interesting insight - at the time, I said "I'll likely jump into another maelstrom of "interesting problems"" and this proved to be rather prophetic. I am my own worst enemy when it comes to containing work and maintaining balance with the rest of life so I'm going to embrace this break as another opportunity to reset myself. Certainly, I'm currently enjoying the freedom to work on my health and fitness and learn some new skills. I've been making candles and writing lots of code, as well as starting to write again - all things I've been too tired to do while in role. I really want to find the not-work person underneath and bring them out enough to do a better job surviving the next job.

I'm considering my next professional steps carefully, especially how much Data I want to be taking on going forward. Do I want to push in the direction of a Chief Data Officer role? Do I want to stay very much in the core technology space? Regardless the answer, this is certainly something to look at in any job spec to avoid adopting data through the back door into a software engineering role, which I feel will set up a role for disaster. The tech industry is going through another seismic shift - we are in the early stages of the AI revolution, with everyone racing to "do data and AI". There is a huge software engineering component to this, but it goes a lot wider into organisational process and culture, and this work needs clear air and appropriate reach to have any chance of success without being overloaded. Fortunately, culture and technology is an intersection I have a lot of experience working on so there is certainly also potential here.

Before jumping anywhere, I need to revisit my own motivators and see what I actually want to be doing. Macmillan has been amazing experience, but I want to build on this, not just do it again somewhere else without some serious thought first. Last time I wrote one of these posts I said I was keen to continue promoting technical leadership and I feel the same way now, but I need to decide how.

In the meantime, if anyone would like to talk about the tech industry, data, growing your software engineering capability, people or anything else please do give me a shout in the new year - I'm easy to contact on LinkedIn or Twitter.

Right now, I'm enjoying the peace in that strange time between Christmas and New Year when the world is on pause and nobody expects anything of you.

Resurrecting a Pixel C

I'm putting off the post I need to write so I'm going to ramble a bit about resurrecting an old Pixel C tablet by sideloading a custom operating system. This is something that sounds scary but really isn't, so I thought I'd share in case anyone else has hardware in the cupboard and no desire to buy new, expensive kit.

Setting the scene

My tablet usage is pretty modest. For anything significant, I'll use a laptop / desktop or my mobile phone. My tablet is mostly used for video streaming (YouTube, Netflix, etc) and a bit of web browsing when it's the closest thing to hand. I am not a power user.

Many years ago (2016), I bought a Google Pixel C tablet. This was originally released in 2015 and was Google's first tablet under the Pixel brand. Other than having a name that is really annoying to search for, it was a fabulous piece of kit - feeling very solid and chunky. However for boring reasons, when COVID hit I moved and my Pixel C fell out of use.

Fast forward many years, and my iPad battery is dying so I thought I'd see about resurrecting the Pixel. Firstly, I hadn't realised how long it had been since it was last used - like a lot of people I know, my sense of time has been utterly smashed by two years of lockdown - so it was a bit of a surprise to discover Google stopped supporting it nearly six years ago (end of 2018). Consequently, although it fired up fine, it was hideously out of date with no path to catching up. It seems older versions of Android (8 in this case) have a problem connecting to modern 5ghz wifi connections and so my lovely hardware was dead unless I wanted to run a lower speed wifi network.

Why take the easy solution, eh?

So, with my brain telling me "this is only a few years old" (it's nine years old?!) I thought I'd look at bringing it back to life via the medium of a custom ROM. For the uninitiated, this is basically installing a new operating system package - pre-built with proper drives and so on - however is more complicated because tablets and phones tend to lock all this down to stop people fiddling and bricking the device. However, I'm (apparently) a grown adult so I'll fiddle if I damn well please. Plus, it doesn't work now so it's not going to get any less functional - perfect for some learning.

Get on with it

The hard part was really all the prep. I needed to find a good ROM, which I did via the application of Google and reading. LineageOS is a the gold standard for community-run operating systems, but even they have dropped support for the Pixel C. However there are some intrepid folk out there who are keeping the dream alive on the XDA forums and a helpful dev going by npjohnson is pushing out builds for the Sphynx project, via his forum thread. Sphynx is an unofficial build of LineageOS tailored for the Pixel C - perfect for me.

The instructions are good - I read them a few times before giving this a go because I was scared - and basically there are four stages:

1. Download adb and fastboot to your computer - this laptop is Ubuntu Linux so that was as simple as sudo apt install adb fastboot but Windows options are also available
2. Learn how to boot your tablet into the recover mode menu (for the Pixel C, with the device off hold Power and Volume Down)
3. Download the desired image - I just took the latest (lineage-21.0-20241019-UNOFFICIAL-sphynx.zip at the time) and it worked fine
4. Follow the instructions very closely

Extremely minor gotcha - I extracted the downloaded zip file to get a recovery.img file for page 3, then used the original zip for sideloading in step 4. Other than a couple of slightly alarming beeping sounds from the device, this was the only time I really needed to think once I got going. The whole process took about an hour, going very slowly and carefully, then additional time to set up the tablet (obviously it is wiped so you'll lose anything on it from before).

Behold

And it's working! There are some known issues - the camera doesn't work properly, apparently bluetooth is a mixed bag and the rotational sensors also don't work - but these haven't impacted my needs. If you like running your tablet in portrait mode, apparently this can be a pain. However, for me I have a shiny refurbed tablet that plays videos and doesn't keep turning off mid-video. Hurrah!

Overall, this is scary. But it turns out it is also easy. Hopefully others will give it a go and bring their devices back to life.

Sending email - redux

It feels like forever since I wrote my last blog post on sending email via a vanity domain but has actually only been a year. In that post, I noted that sending via SendGrid was optional and it should all be doable using Google servers. The SendGrid config has been mostly ok, but hasn't been perfect and I wanted to remove this free third party from my email tool chain so I've revisited the setup and got it working through the Google mail server.

Will this work long term? Hopefully, but I'm not convinced for reasons I've laid out below. Let's do this.

Sending email

First I needed an app password for your Google account, which is a bit of a buried in the security interfaces. You can find the admin console for app passwords here.

This is also reachable by going to Google account settings, under "How you sign in to Google" select "2-Step Verification" and the option for App Passwords is at the bottom of the page.

Next, to configure gmail to send via the Google mail server, I needed to set the outgoing mail, found in:

Settings -> See all settings -> Accounts and import -> Send mail as

Then adding / editing my intended outgoing email address with these details on the second page ("Send emails through your SMTP server"):

SMTP server: smtp.gmail.com
Port: 465
Username: Your google account (blah@gmail.com)
Password: Your app password from earlier
Secured connection using SSL ticked

Email security

This is a minefield and something I'm going to have to monitor to see how horribly I have broken things. First, this tool from Red Sift is great for analysing email security settings.

Deep breath...

SPF

To the DNS record, add:

include:_spf.google.com

and remove references to sendgrid to avoid too many lookups - this flags it as insecure.

DKIM

Without a full Google Workplace account, I can't upload a private key to Gmail so DKIM isn't going to work. Hopefully SPF will be enough. We'll see...

I think this also scuppers MTA-STS, but happy to be corrected.

DMARC

This is tricky. DMARC requires one of DKIM or SPF to pass tests properly in order to pass, then it suggest the receiving server takes a specified action (via the p flag). In this case, my DKIM check is going to fail so that is out. The SPF check passes the initial test however there is a further check to make sure the Sender and From headers are the same. In my case they are not, since Sender is gmail and From is my domain so check fails with a no-alignment error - thus the DMARC check itself fails.

I've "solved" this by setting the p flag in my DMARC DNS entry to "none" which just tells the receiving email server to deliver it anyway. It seems to work for my very limited testing sample, but obviously I'm not happy about this approach.

What is next?

I hope the SPF config will be enough to make my email work happily again, however I'm clearly hitting the limits of the free options in Gmail. If this doesn't work well enough, I think I'll move away from free options and look at something like AmazonSES which (from a quick read) will let me configure everything I need and charge me $0.10 per 1000 outgoing emails. This is probably the ultimate "correct" solution (unless I want to pay for a Google Workspace account) but is a lot more work and right now I don't wanna.

Migrating postgres databases from ElephantSQL to Neon

Continuing my series of "if I push enough buttons I can get postgres to work for me" I am going to record how I migrated from ElephantSQL to Neon. This is one of my personal documentation posts - I write these for my own reference for when I need to do something similar in future but all useful information has dropped out of my head so I don't have to distil something simple from proper documentation again. They are sometimes useful to someone doing the same thing (I'm actually surprised how often I do send these links to people) but since more folk are reading my blog from LinkedIn these days this is fair warning.

The setup

I was migrating from ElephantSQL to Neon as the former was shutting down. I wish Neon all the best, but the way things are at the moment I guess it's only a matter of time before I have to do it again. Migrating a simple postgres database is straightforward, but if (like me) you don't do it often it is nice to have the process written out.

This is for my own experimental applications, so I'm dealing with small, single-node databases and I'm not worried about downtime.

Recover the data

Getting the data out of the source database is straightforward. Simply log into the control panel, copy the connection string and use pg_dump for a full download:

pg_dump -Fc -v -d <source_database_connection_string> -f <dump_file_name>

-Fc makes the output format suitable for pg_restore. -v is verbose mode, showing you all the things going wrong...

Upload to the new database

Initially, I struggled a bit with Neon. I created the database and user in the web interface, but could not find a way to associate the two so consequently pg_restore failed with permissions errors. The simple way around this was to create the database via the Neon CLI, recorded here as a bit of a gotcha.

neon roles create --name new_user
neon databases create --name new_database --owner-name new_user

And for completeness, these are the commands which list the databases and users.

neon roles list
neon databases list

Once the database is created properly, the database can be restored using the pg_restore command.

pg_restore -v -d <neon_database_connection_string> <dump_file_name>

Repointing the database

So far so simple. Now to reconfigure the application - this should be a case of updating an environment variable. For a Rails app, that is likely DATABASE_URL. Simply edit the environment variable to the new database connection string, restart the app and this is done.

Again, this is for a very simple application - one Rails node, small database, no particular need for zero downtime.

Hopefully this will be useful to someone out there even if it's just me in the future. Hello, future me. What are you up to these days?

Why good software engineering matters

I've needed to make some changes to a few of my personal applications recently and running through the process made me reflect on some of the basic building blocks of my profession. As a deeply uncool individual, I am very interested in the long-term sustainability of our technical estates so I thought I'd capture those thoughts.

The story so far... I run a few small-scale applications which make my life easier in different ways. I used to host these on Heroku, then when they shut down their free tier I migrated them all to Render and Koyeb with databases hosted by ElephantSQL. About a year on, I started getting emails from ElephantSQL telling me they are shutting down their database hosting so I needed to migrate again. I also needed to fix a few performance problems with one of the applications, and generally make some updates. Fairly simple changes but this is on an application I haven't really changed in several years.

A variant of this scenario comes up regularly in the real world. Unless you're lucky enough to be working on a single product, at some point your organisation will need to pick up some code nobody has touched in ages and make some changes. The application won't be comprehensively documented - it never is - so the cost to make those updates will be disproportionately high. Chances are, this means you won't do them so the application sits around for longer and the costs rise again and again until the code is totally rotten and has to be rebuilt from the ground up, which is even more expensive.

In a world where applications are constantly being rolled out, keeping on top of maintenance - and keeping organisational knowledge - is vital, but also definitely not sustainable. There are lots of service-level frameworks which promote best practice in keeping applications fresh, with ITIL being the obvious one, but this is only part of the picture. How do we reduce the cost of ongoing maintenance? Is there something we can do to help pick up and change code that has been forgotten?

This is where good software engineering makes a huge difference, and also where building your own in-house capability really has value. Writing good code is not just about making sure it works and is fast, and it's not just about making sure it's peer reviewed - although all of this is very important. But there are many approaches which really help with sustainability.

Again, my applications are really quite simple but also the "institutional knowledge" problem is significant. I wrote these (mostly) alone so anything I've forgotten is gone. The infrastructure has been configured by me, and I'm not actively using much of this stuff day to day so I have to dredge everything out of my memory / the internet - I am quite rusty at doing anything clever. These problems make change harder, so I have to drive my own costs (time in my case) down else I won't bother.

Let's look at some basics.

First, the database move. My databases are separated from the applications which means migration is as simple as transferring the data from one host to another and repointing the application. This last step could be tricky, except my applications use environment variables to configure the database. All I need to do is modify one field in a web form and redeploy the application to read the new target and it's done with minimal downtime. Sometimes developers will abstract this kind of change in project team discussion ("instead of pointing at this database, we just point at this other one") but with the right initial setup it really can be that simple.

Oh, except we need to redeploy. That could be a pain except... my applications are all set up for automated testing and deployment. Once I've made a change, it automatically runs all the tests and assuming they pass one more click and the new version goes to the server without my having to remember how to do this. I use Github Actions for my stuff, but there are lots of ways to make this happen.

That automated testing is important. Since everything in tech is insufficiently documented (at best) this creates a safety net for when I return to my largely forgotten codebase. I can make my changes or upgrades and run the tests with a single command. A few minutes later, the test suite completes and if everything comes up green then I can be pretty confident I've not broken anything.

Finding my way around my old code is fairly easy too, because it conforms to good practice use of the framework and it is all checked by an automated linter. This makes sure that what I've written is not too esoteric or odd - that is, it looks like the kind of code other people would also produce. This makes it much easier to read in the future and helps if someone else wants to take a look.

So through this, I've changed infrastructure with a simple field change, run tests giving me significant confidence the application is working after I've made a change with a single command (which also checks the code quality) and deployed to the server with another single command. To do all this, I don't really have to remember anything much and can focus on the individual change I need to make.

Now, any developer reading this will tell you the above is really basic in the modern world - and they are right, and also can be taken MUCH further. However, it is very hard to get even this level of rigour into a large technical estate as all this practice takes time - especially if it was not the standard when the code was initially written. But this really basic hygiene can save enormous amounts of time and thus costs over the lifecycle of your service. At work we are going on this journey and, while there is a lot more to do, I'm immensely proud of the progress that the software engineering teams have made driving down our costs and increasing overall development pace.

Basics are important! Always worth revisiting the basics.

Moving office

I don't often directly talk about events at work, but for once I'm going to celebrate something rather cool that's happened. We've moved offices!

Despite the valiant efforts of our estates people, the Macmillan offices in Vauxhall were ... well, terrible. Vauxhall itself is a roundabout with delusions of grandeur and the building was slowly falling about around us. I do not frequent the office too often, so I was rather surprised during a trustee meeting when the whole building started to shake like the apocalypse had come. Nobody else blinked - this was "normal" to the point of it happening several times a day. The rest of the day gave a glorious demonstration and I have no idea how anyone copes, frankly.

So for this and various other reasons it was time to Be Elsewhere. However for us in Tech this meant we had to face the (kinda literal) elephant in the closet - the server room on the premises. This was not a comms cupboard, but a proper server room, with ageing steam-powered servers bolted the floor, powering the organisation. But this was not a time for panic and fear - instead, we had a fantastic opportunity to take a big step modernising our systems. A golden opportunity to spend a chunk of time significantly moving the dial on our legacy tech debt in the service of a hard deadline which the org needed hitting. We grasped this opportunity, with months of work spent virtualising, reconfiguring, and rebuilding to bring things into a much better state in preparation for the move.

To actually move, we initially had to plan for disabling everything. However, with every week of work where we cleaned up dependencies and updated our overall configuration we decoupled systems and by the time we came to do the move, the only services that we actually disabled were the ones hosted on the machines we had to turn off. This in itself was a huge win, but the move weekend itself was exceptional. I've been involved in lots of tech projects over the years, and many releases, and something always goes wrong and needs correcting at the last minute. We had our share of challenges, but for the week before the move we were having daily meetings in which we were looking at each other wondering what we had missed - things were calm. Then the weekend was so well executed it was almost unsettling. The team didn't exactly stick to published timings, but only because they were so far ahead.

Overall, it was incredibly smooth and not only did this enable our office move, we have finished with our systems in a much better place, either in the cloud or in a proper data centre and better understood, and run by a team with a great deal of (very much earned) confidence. An exceptional result on the back of a lot of hard work - really knocked it out the park.

The second, and far more visible, part is the new office itself. This was clearly much wider than the Technology group, but we had a crucial role in making sure the new premises had an internet connection (which it didn't until quite literally the 11th hour ... worrying times!) and working AV, door controls, room bookings, etc etc. The wider team did an excellent job bringing everything together on time and it is lovely being in a modern office which doesn't shake when the trains go by. In particular (for my post!) I'm going to say the technology is working rather well. The new meeting equipment is very easy to use, with great sound quality and scary cameras which track motion to zoom in on speakers. I wonder if I can mount a nerf gun on one of them...

So yes. Some excellent work here and well worth recording. A great result for Macmillan. For the Technology group, not only did we play our part we've also managed to modernise, increase knowledge, improve resilience and other great things across our server infrastructure we ALSO managed to remove a load of problems with the office AV. As I said at the top, I don't often write about specifics here but I thought I'd make an exception.

And to close, here are some pictures of the new place including the most important part of the new office building - a button which gives hot chocolate milk...

Congratulations everyone!