Sunday, 22 January 2023

Migrate from Codeship to Github Actions for projects using Mina

Oh look - another free tool has decided to eliminate its free tier. My last post was about migrating away from Heroku as it torched its free offering and now Codeship has followed suit. In the past, I have written about using Codeship as a part of my toolchain for CI/CD but over recent years I have moved away from it to Github Actions for running linters and tests and using the Github integrations of the hosting providers themselves for deploying web apps. This means I've migrated many of my projects away from Codeship already, but there has been one significant holdout - my static sites.

Usual disclaimer - this is for a personal project, so I'm more interested in free solutions than high speed or resilience. Docs for myself, maybe someone will find this useful, etc etc

The problem

My static sites are generated using Jekyll, and deployed via Mina. Mina is a great tool, but for the purposes of this post it opens a connection over SSH and executes some commands, which require some environment variables.

I also need to migrate my "tests" - which are just building the site locally to make sure it's not broken - so I'll note that step here.

Migrate test build for a branch

Making the branches do a test build was simple - set up Ruby, get code, run the test code. All very similar (and easier) than doing this with Rails and wrapped up in a short Github Action.

Migrate deployment

Making the deploy script trigger on merge to the main branch was a little trickier (full example below).

First gotcha - when one merges a branch into main, the event that triggers is a "push" event. Knowing that, getting it to work is really easy but that took a bit of hunting. It also means that a push directly to main will trigger a deploy, if your main branch isn't protected.

Second gotcha - environment variables. I didn't want my server config in my public code, so I needed to set some environment variables in the settings of the repository. There are a few ways to do this in Github - I settled on using "Repository secrets" which are found in the project Settings -> Secrets and variables -> Actions then "New repository secret". They are accessed in the workflow file as ${{ secret.MY_SECRET }} and if you want to use them inside a script, you need to pass them through explicitly by setting the env for that script. Example below.

Third gotcha - SSH key. Mina opens a connection to a remote secret, so it needs access to an SSH key. I created a public / private pair on the server and added the public key to the usual place, then put the private key in the repo secrets as above. Using this action allowed me to load the key into my workflow easily, and then I could use ssh-keyscan to push it into the container known hosts, allowing Mina to work properly. Again, example below. Shout out to this post for the pointers on the SSH key setup.

And lo, deployment works. Here is the complete deploy Action.

Tidy up

All that is left is to tidy up. Couple of simple steps here:

  • Remove Codeship keys from server
  • Delete Codeship project / build

And we're done! Like my previous Github Actions implementations, this is almost entirely generic so I can copy / paste into other projects with minimal effort and make sure I have light CI/CD for my fun projects. Now it would be great if I could get back to some of my creative programming projects for a while, instead of having to rebuild toolchains.

Sunday, 15 January 2023

Migrating away from Heroku

Along with almost every developer I know, I was very sad to see the end of Heroku's free tier. I was hoping they'd reverse this decision, but it seems that was too optimistic so in the end I had to choose between killing all my toy applications, paying money or finding an alternative.

I did consider just shutting things down, but I had two concerns with this. First, I have one application I actually use and losing it would be a pain. Second, and more importantly, if I have no solution to this problem it adds another barrier to me coding as a hobby. So this needs solving, and not just by throwing hundreds of pounds a year at it.

Finding an alternative took a long while, primarily because I was trying to find a straight all in one alternative. While they are out there, the free tiers are not suitable - the closest insist on deleting my database after 90 days. Instead, I ended up separating application hosting, database hosting and sending of email and the below is where I ended up, written in detail in case it helps one of the many others doing this same thing. I did this all back in November, before the Heroku switch-off, and a few months all it seems to all be running well.

My applications


Everything I'm moving is a personal project (no url customisation, limited need for backups, high availability, etc) and a simple 12f(ish) application. They are written in Ruby (mostly Rails, one Sinatra) and make use of Postgres on the backend and some simple mailing. Nothing complicated and Heroku managed all this for me before.

To move from Heroku SendGrid to independent SendGrid, I did find I had to make a small update to the mailing config.

Most of this diff are linting updates - only the domain change should be necessary.

Databases


I chose ElephantSQL as it has a good enough free tier offering (limited to 20mb) and lets the free databases persist. Detailed steps for anyone who, like me, hasn't done much with a Postgres database for a while.

Setup:
  • Create account in ElephantSQL (sign in with Github)
  • Create database
  • Get connection string

Backup from Heroku:
heroku pg:backups capture --app myapp
curl -o latest.dump `heroku pg:backups public-url --app myapp`
Restore to ElephantSQL:
pg_restore --host "machine.db.elephantsql.com" --port "5432" --username "blah" 
    --password --no-owner --dbname "blah" --clean --verbose "latest.dump"
And voila, database ready to rock in ElephantSQL. This can be checked by running up locally, thus:
DATABASE_URL=postgres://blah:blahblah@machine.db.elephantsql.com/blah bundle exec rails s

Mailing


My requirements are very light here since I only use email for "forgotten passwords" email. For this, a free [SendGrid](https://sendgrid.com/) account was more than enough. I'm using SendGrid because this was the Heroku solution, so it makes for an easy switch.

  • Create account in SendGrid
  • Create API key

Application hosting


I wanted something which, like Heroku, was a container-based PaaS. I certainly can wrap my applications in containers or deploy to virtual machines, but honestly who has the time. I want to point my application at a hosting provider and have it do the work for me.

For this, I looked at a load of options but I settled on two - Koyeb and Render. Both have workable free tier hosting. Render works like Heroku used to - giving a number of hours per month and spinning down applications when they are not in use. Koyeb simply gives you $5 credit per month and lets you choose how to host. This is an enough to have an application running on a low tier package without spinning down, but not enough to run several applications.

For deployment, the steps were basically the same. For Koyeb:

  • Create account in Koyeb (sign in with Github)
  • Create application
  • Add Github integration
  • Add to "run" command: rails db:migrate && rails server
  • Select repo (this took a while to appear for me, although Github was having issues when I did this)
  • Add DATABASE_URL env var
  • Add RAILS_MASTER_KEY env var
  • Add SendGrid env vars
  • And any other env vars

Then off it goes. I did find it took a little while to start working, but it has been fine since. Also, a slight gotcha - Koyeb appears to have some broken validation when it comes to counts, eg "must be 3 characters or more" seems to actually mean "more than three characters".

For testing, I did all this (including the mailing change above) in a branch then cleaned up at the end.

For Render it was much the same, except I didn't need to specify the RAIL_MASTER_KEY env var and I needed to write a build script.

Cleanup


Finally, to finish off:

  • Spin down the Heroku app
  • Remove auto-deploy to Heroku (in my case from Codeship)
  • Toggle Koyeb / Render over to watch "main" if using a test branch
  • Make sure the instance is healthy (and make sure not exposed the wrong port anywhere)
  • Final redeploy

Roundup


That is pretty much it. I have been running this setup since November with no problems. I am getting fewer notifications through to Slack, so at some point I would like to get better alerting of start / end deployments but since it's just me working on these things I can watch them easily enough - it's not a priority. I'm just pleased I can still deploy easily and free as that means I'll keep my hand in writing bits and pieces.

Saturday, 31 December 2022

The year that was, 2022

Well, it's the end of 2022 and I've got an odd feeling. The last few years I've started by expressing relief that the year is over. For the first time since 2015 I'm not going to write some variant of "thank goodness this year has finished". It has been a difficult year at times, but overall I think it has worked out well. I have certainly (once again) worked far too much, and I ended the year with COVID but overall I give this one a passing grade. But what did I actually do other than work? Well, let's find out. It can't all have been Elden Ring and Deep Rock Galactic (again).

Resolution count - 5.5/10. Down on last year, AND I worried I'd made them too easy!

It has been an odd year. Very tiring in places, and once again I've worked far too much. I didn't feel I'd done anything and my resolution count is going down, yet the list above is much longer than usual and I don't think I've scraped the bottom of the barrel pulling it together. So I'm clearly doing things, just not the things I intended. I can live with that, but I would like the things I do to have a greater impact on me so I actually remember them. Interestingly, unlike other years, the above list is spaced across the year rather than being loaded into the last three months so maybe I've achieved something approaching balance? I also didn't have three months off work and still did Many Things so clearly I've started recovering some energy to be able to do them.

Speaking of balance, we should definitely take a quick look at work. I spent nine months as a contractor before formally being appointed to my first full director role at Macmillan Cancer Support. After 2021 that was definitely a good boost for my self-esteem! I also helped set up the new version of the London Data Board, then got myself appointed onto it. This coming year, I need to grow into this new form of work - looking ahead to decide what I need to learn to be better at it.

Looking forward to 2023, I'm going to write the same thing I write every year. I need to spend less time working and more time living. Last year I stated my intention to do more creating "things" and the above list shows I made a decent go at that. I want to do more again in 2023 - ideally maturing those skills I've been picking up the last few years so I can be more excited about the things I'm making. I also want to get out and about somewhere - I'm not great with travelling off my own back so maybe I can change that.

More than this, I want to revisit what I wrote at the end of last year's summary. I have starting pulling together my sense of wonder and continuing to restore my zest for life is going to be the main focus for next year. Hopefully this will also be reflected in my writing on this blog too. Less of the "this is hard" and more of the "this is interesting".

2023. Bring it on.

Thursday, 29 December 2022

Well, I had COVID

It has been a quiet month. Nearly three years after the world first shut down, I finally caught COVID. I'm writing this as a reflection for future-me just in case I ever think about getting it again.

The short version is that I don't recommend it.

The week before was a long, tiring and frequently cold week that ended with some very lovely but early morning / late night festivities. So come Friday, when I was feeling rough and exhausted, I figured that I needed some sleep and relaxation. I even cancelled my Saturday morning exercise so I could sleep in. Good choice since I woke up that Saturday morning with my skin and joints screaming, my head swimming and a bad reaction to the cooler air outside my duvet. That was a bad day, where walking the 10 paces to the kitchen was hard work.

Fortunately, the fever broke overnight and the next day I woke to find I could move more easily, although even with a heavy dose of paracetamol I was barely functional. My COVID tests had all dried up, so I dragged myself out to the local pharmacy for another batch and some flu medication - a 15 minute round trip which utterly wiped me out for the rest of the day. Got that test in and yes - positive. Sigh.

Thus began a week of sleeping and basically going nowhere other than a short walk around the block every few days. Coughing to the point of struggling to breathe, throat pain, photosensitivity and brain fog were the main symptoms along with utter exhaustion - it has been a long time since I've had so little energy. Each day consisted of a few hours awake, mostly spent watching things on Netflix and playing some videogames. Food was less than ideal - eating whatever would stay down did involve a significant increase in pizza and chocolate...

There were some positives. I watched all of Sonic Prime (enjoyable), Dragon Prince series 4 (good after a terrible start), and Wednesday (great atmosphere). I completed all my Christmas shopping (online). And as I got better I also wrote the first pass of an engineering strategy - although I haven't read it back so there is a good chance it is a terrible product of a malfunctioning brain.

It took about a week of being basically incapacitated before anything started to shift. There was a step change in the exhaustion, which stepped from "basically unable to move" up to "short distances are achievable with enough planning" and the coughing improved, although continued. However, the positive tests continued so I remained isolated.

Also, it's worth remembering that getting hold of a doctor to be signed off work is not easy. I called my surgery on the first day I was ill and the first appointment was thirteen days away - notably quite some time after the five days self-certification.

A couple of days into my second week, around 9 days after my first positive test, I had my first negative test. This was a huge relief as it meant I could go and sit in a coffee shop and actually see other humans from a distance. The short walk did, of course, completely exhaust me requiring an extended lie down and a snooze when I got home but it did help me stop going slowly mad. Symptoms continued to improve slowly, but didn't go away. The worst was the brain fog. Living alone and doing very little meant I didn't notice it so much, but any time I spoke to someone I realised that talking in complete sentences was a challenge. I'd frequently start talking and forget what I was going on about.

By the end of the second week I had returned (gently) to remote working. A few hours a day Thursday and Friday were enough to completely exhaust me again, leading to a weekend mostly asleep. The following week of lighter hours was another step better, but still struggling with some photosensitivity, ongoing exhaustion and brain fog. By this point I'm also having complications with fading fitness - three weeks off exercise and eating rubbish is not good in general. More significantly, any kind of illness like this triggers depression in me and the time of year (dark, cold and mostly wet) don't do much to counter it - so in addition to not wanting to get up because ill, I'm also not wanting to get up because depression. Of course I'm also still taking a heavy dose of paracetamol.

I'm really glad that the first week back was the run into Christmas and so quieter than it could have been otherwise. It also means that as I write this I'm holiday so I've been back off work for a few days recovering again. I'm certainly feeling better - my brain is clearing and the rest is very helpful. However, I'm still low on energy and coughing extensively. I'm hoping that next week (in the new year) I'll be well enough to start some proper exercise again and begin the slow rebuild of fitness. Either way I'll be back at work, trying to fight through all the email I missed while off.

Worse than this is the effect on my mental health. Anyone who has read this blog for a while knows I suffer from bouts of depression, and while right now I'm feeling ok I also require very little self-motivation and I'm "ok". I'm worried about myself for when I go back to work and have to push myself again, so I'm going to have to keep a close eye on myself and ensure I'm embracing sources of energy and joy as well as pushing through the energy-sinks and also the ripple effects into other parts of my life, such as maintaining exercise and cooking properly.

I appreciate this hasn't been a very interesting post, but I did want to document how I felt through this. I certainly didn't have the worst COVID, but I did seem to have a nasty dose of the simpler version and I'm not keen to go through this again. So, future-me - stay well.

Sunday, 20 November 2022

Another year, another post about fire

My favourite annual festival is bonfire night. It's an event which knows exactly what it is, hasn't really been corrupted by commercial interests (cough Christmas) and is very local - that is, it's done at a village level and keeps its small-scale feel. It also comes with fireworks and a bonfire. I love it.

Every year I look forward to heading to my parents' village for their event (living in a historic city, there isn't much taste for huge pillars of fire ... go figure) and every year I take a load of pictures of the bonfire, trying to capture the flickering tongues, the heat and the evocative nature of a pile of burning wood in the dark.

This year, we had the fun of the bonfire going up rather quicker than expected resulting in a huge cloud of steam and smoke washing over the crowd then a fireball which was gloriously hot before the marshals hurriedly made us retreat, pushing back the safety rope.

For the photos, I tried a variety of approaches. I wanted to capture the moving tongues, so shooting a fast moving target, but it was all at night so I needed to be mindful of light capture. This year I got best results with a high shutter speed, to freeze the movement, and then countered the limited light by pushing the ISO up to very high levels to pick up everything I could. Obviously the flash was of no use and there was only so much I could do with a wide aperture.

I was pretty pleased with the results, and I always enjoy the technical exercise of trying to figure out how to set up the camera to take these shots as effectively as I can.

I also took a few with my phone, which went with a similarly fast shutter, but instead of pushing the sensitivity just applied AI and fixed it up in post. I think. Pixel photos are somewhat magic.

Anyway, here are the results:

Minchinhampton Bonfire 2022

And if you're interested, this is last year:

Minch bonfire 2021

Which was set up in a similar way, although using less of the extreme settings.

Maybe one day I'll figure out how to capture fireworks.

Sunday, 23 October 2022

Give me a task OR a problem

I was speaking to a colleague recently who was feeling very anxious. They were trying to lead a major organisational process - something well within both their job description and individual capability - but they were feeling awful about the whole thing because of a strong sense they were not doing the right thing. We dug into this, and as far as I could see, the problem came from feeling a general expectation they own this problem, while they were also trying to meet their delivery goals in specific ways. So wide responsibility with diminished agency.

When in a position of authority and / or leadership, we need to be constantly alive to the effect of our actions on those around us. When delegating, that means trying to cleanly delegate either a problem OR a series of tasks. Both approaches have their place, but going somewhere in between creates all manner of anxiety.

Delegating a series of tasks means telling someone exactly what you want. You retain the ownership of the problem and tell other(s) what you need in order for you to solve this problem. Importantly, that means you're are retaining responsibility for the end product and various aspects of quality control along the way. No matter how clear your definition of the task (an entirely separate post...), delegates will need to check in to find out if they are doing the right thing. Certainly, you can expect the individual to use their initiative within those tasks - essentially you've delegating a series of micro-problems - but you are retaining responsibility for bringing it all together and need to make sure the person receiving the tasks doesn't feel that responsibility.

This approach is best used for your own direct deliverables, or in short-term scenarios such as "we need to achieve X by next week, so I'm taking direct control of this process". The trade-off here is that you're keeping direct control at the cost of putting more of your own effort into the process. You get to have ongoing input on how something is achieved and retaining the definition of done, at the cost of having to spend your time doing this - something which can be very difficult dependent on your own availability. It withholds agency from those receiving the tasks so the more senior you are, the less you should use this approach as you've presumably appointed clever, able people around you and want to make use of their talents.

The other option is to delegate the problem. In this case, you set out the direction and the parameters for success and let the individual work through it and solve the problem themselves. This approach empowers the individual, giving them an opportunity to stretch themselves and learn. It also frees up your own headspace, because someone else is now handling this problem and you only need to check in with them from time to time to ensure they have the support they need. However, the trade-off here is that you're empowering another at the expense of your own direct control. You don't get to dictate every step of the process, which means you need to accept that they might do things a bit differently to how you'd have done it. Likely, this doesn't matter but it can certainly be difficult to let go. The emphasis on you is to create an environment where the person can flourish and feel confident that (assuming a decent level of communication to keep aligned) you will support them.

I'm labouring this point because it's all too easy to end up taking the middle ground. Trying to put someone in charge of delivering something, expecting them to make the decisions and for it to happen - but expecting it to happen the way you want it, and them to make the specific decisions in your head. Obviously this is impossible - and just as obviously, nobody does this deliberately - but it happens all the time and for the person on the receiving end it creates an environment where it's impossible to perform. If they are any good they want to do a good job, but in this case the definition of "doing a good job" is being hidden. The point here is stunningly obvious, but can cause severe anxiety.

However, this is not just something for the leader to consider. The individual receiving the work should also be aware of their situation and force some clarity. What are the expectations here? If it seems I'm being given tasks can we please talk clear definitions of success, and about regular check-ins? Or can I own this problem, and solve it my own way? In which case, back off please leader and let me do my thing. Especially if this problem is something that my department / job title should own.

When this is decided, both parties needs to work to the chosen paradigm. If it's a task, the leader needs to get the information required to move things forward, but shouldn't expect the delegate to take the lead of the wider problem. If it's a problem, the delegate needs to take proper ownership and not ask so many questions of the leader they are making them do all the thinking. However, the leader needs to support the delegate. Whichever, both parties need to keep sufficient communication to ensure things are progressing the right way, and all parties can work with confidence.

By this point, we're leaning into the territory of our personal relationships with our colleagues and that is a very different post. I've certainly worked with people I can trust to work very much at arms length, and others I've needed to spend more time directly supporting. The same is true of my relationships with The Boss, at different times in my career. Like so many things, confidence (and better mental health!) comes with clarity more than anything else.

Sunday, 25 September 2022

Built to last

I spent the week in Pembrokeshire, exploring the coast and visiting various old and ancient places of interest. There was a bishop's palace (around 200 years old), a reconstructed Iron Age village built on the site of the original settlement (2500 years old) and a Stone Age floating stone dolmen (5000 years old). These are all astonishing examples of technology from the past. Did you know there is no smoke hole in an Iron Age roundhouse because it created a smoke ceiling that was useful for smoking meats and then the tar deposited as the smoke worked through the thatch helped insulate and keep insects from nesting in the roof? The design of the house was such that with a correctly sized fire, the smoke ceiling stayed well out of the way of the inhabitants. This created a beautiful ecosystem using all parts of the fire and this was 2500 years ago.

On the other hand, we can't build an HR system that works properly.

One way I like to expand my thinking about the tech industry is by learning from other walks of life. Tech is a very young industry and while some of our challenges are unique, many have come up time and again and we really should learn from others before attempting to reinvent the wheel. Usually I look at modern industry like manufacturing, supply chain logistics, beer brewing, food packaging, game development, forging and so on but sometimes we can gain inspiration (or at least be humbled) by looking much further back.

So let's look at Pentre Ifan, the neolithic dolmen.

Floating stones

The top stone was originally a larger rock which was split by craftsmen and now weighs around 16 tonnes. It is balanced on the tips of three other stones, high enough for a fairly tall person to walk under without stooping. In person, the effect is actually quite eerie - it looks like a strong wind will knock it over. What we see here is very simple, yet elegant design and a very high commitment to engineering excellence in order to execute it properly. It was believed to have been winched up a little at a time using ropes and pulleys, with supporting boulders put in place each step. So iterative construction with solid testing at each stage (with failed tests resulting in a 16 tonne boulder falling on the workforce...). They had to create supporting structures (the smaller boulders), which were later discarded when no longer required.

They put in the work to make this last (last 5000 years, as it happens). It even has a small environmental footprint since the materials are locally sourced. So is this pinnacle of sustainable development?

Well, not exactly. That means building for the future - creating something simple and well documented so we can maintain and make use of it going forward.

Thing is, we can't maintain or extend this dolmen (ignoring the protected status for a moment, and the sheer horror of using a 5000 year old monument to build "something else"). For this to be useful, we need to know what it is for. Is it a doorway to a tomb? Some kind of temple or altar? The real problem is around knowledge transfer. Where is the original documentation telling us the core purpose with notes about how it was built? The fact it predates writing in Britain by around 3000 years is no excuse!

Plus, while the design is elegant it is not intuitive. If it were, we'd be able to work out what it does and how to use it. More UX work required here, I feel. Whatever it was, it has outlived it's original purpose and has clearly missed its (admittedly impressive) maintenance cycle, which would have had it replaced by a modern equivalent a few thousand years ago.

This post needs to end. Somewhere in all the above silliness are some solid points, honest.

Pentre Ifan itself is incredible.