Making a decision

I've been thinking a lot about governance recently. In general, I like consensus - it's important that expert voices are heard and action is taken on the back of expertise. I dislike dictating how things are done for the same reason. Generally, I find people thrive when given the space to take ownership of problems and work through them.

However. This can create a few problems. The first is mediocrity - if everyone is compromising to form consensus then it is far too easy for nothing to actually be good. In the worst case, the important part of a proposal is compromised away and any outcome becomes a waste of time. This assumes there IS an outcome, which leads to the second problem - how is a decision actually reached? How do I stop people talking forever?

This latter problem is the one that concerns me more at the moment. I see different versions of non-decisions all over the place, from email threads which ambiguously leave "someone" to do the next thing to conversations which always need another person to engage before maybe everyone agrees. At worst, this never concludes but even at best it is sloooooow.

For me, this leads directly to the question "what is a decision?". When is something approved? When is it agreed? Or when it is simply some positive noises coming out of a discussion? It's important to draw this out for everyone involved. Individuals do not like being misquoted ("no, I didn't actually agree to this"), project managers deserve to know where they stand with sign-off so they can move on ("is that a decision?"), and I need to add some formality so I don't have to mediate these positions.

I'm leaning into some hypotheticals here - while I see some of the above at work, decisions are certainly being made, and people are thriving in their empowerment. However, and being selfish for a moment, it is actually my needs that are being neglected. As my role changes (and reflecting on my musings from my time covering as CIO) I find myself with less and less time to mediate circular conversations. So while things generally work, I need to move this from "generally" to "always" and "with less effort from me".

It's time to make some changes. As noted above, I've previously been happy to give people space to feel things out and learn from them. I don't want to lose that - it's an important part of empowerment and learning. However that is the discussion part - I need to add some stronger gateway decisions to the way we work. On reflection, I think I need to be better at separating discussion from decision and creating a formal record. There is too much going on at the moment to be imprecise about the state of decisions, so I need more opportunity to eyeball people and tell them to speak up or forever hold their peace.

The trick through all this will be retaining agile working patterns - I don't want to create some kind of decision auditing circus, but as we mature we both want to and need to be able to hold each other to account properly. This means having something to be held to - robust decision records where people can make their decisions known formally.

Initial muse for the moment - I've been exhausted this month. More on this later I feel.

Slack notifications from Github Actions

A while back I wrote about moving my deployment scripts away from Codeship into Github Actions. This has continued to work, but as I noted it did leave me without notifications in Slack. Time to fix this.

There are a few ways of enabling notifications. The best way involves creating Slack apps with incoming webhooks, but this involves a lot of faff (especially on a free Slack instance). The ever-useful Phil Wilson found a much simpler alternative.

Enter the Github Slack app.

The setup is very easy:

• Add Github app to Slack
• Log in to Github app as whichever user should be receiving notifications
• Authorise it to talk to your Github account in Github

All three of these are documented in the above link.

Then it's a case of setting up the notifications you want within the Slack app channel. I used the following:

/github unsubscribe $account/$repo issues pulls commits
/github subscribe $account/$repo workflows:{event:"push"}

The first removed the overly noisy notifications from a whole bunch of Things on the account. The second enables notifications from workflows (ie Actions) - mine are triggered on "push" events. Docs for the workflow configuration can be found on the integration page.

And behold! Notifications!

When Github updates its SSH host key

I wrote recently about deploying to remote servers via Github Actions. I use this blog partially as a place to write bits of useful documentation and this is a quick note on how to fix those deploys when Github updates its SSH host key. It took me an evening of fiddling around while watching TV to get this right, so I don't want to have to think about it next time.

On the target server, remove the old github.com entries and add the new key into known_hosts:

ssh-keygen -R github.com
curl -L https://api.github.com/meta | jq -r '.ssh_keys | .[]' | sed -e 's/^/github.com /' >> ~/.ssh/known_hosts

And that should be enough to make it all work again.

These commands courtesy of the Github blog. Essentially, this post is distilling that one for future reference.

Hanging out with the cool kids

I have spent most of an incredibly busy March as a stand-in CIO while my boss was away. After three weeks of my first real taste of a C-level role and board member, I find myself grateful for the experience and thinking about all the things I've learned. Time for some reflections.

The job itself was obviously truncated - covering is an inherently tactical affair whereas a job like CIO is strategically focused. So I had an easy ride from that angle - large amounts did not need doing in the timeframe. What I DID do involved a lot of reporting and representation. Much more than my current role, I had to think about information flow across the department and that naturally led my mind straight to the way the processes were set up. Our department has been entirely overhauled over the last twelve or so months, and it was very interesting getting a different perspective on how this fits together and where we can work out kinks to prevent our own internal reporting industries wasting time re-collating information instead of letting it flow.

In fact, I find myself thinking about the process far more than the content. How do we get information from team to division to report to governance board while retaining its accuracy? Of course, the quality does matter - over the last few weeks I've fronted reports to the exec board, the spend group, the risk board and the directorate board to name just a few. There is no way I could have written all this myself, even with the information immediately to hand and that of course leads back to ensuring the processes are in place to create good submissions "automatically" with my input being editing and guiding the narrative of each submission.

Setting this up for success is utterly critical. I'm going to write a proper post on this sometime, but my feeling on my current job is that when one removes the fluff, I have two objectives - create clarity and make time to support / guide people. This was amplified again as CIO. I've already started updating the way we plan inside Engineering to more directly feed into some of the governance processes, removing some of the kinks mentioned above, and now The Boss is back I am going to spend more time working through this. I've been working on "create clarity" since day one, but I've got a different perspective to work from now.

So what else? Well, the flow described above creates more and more dependence on the people in the department. The more senior I get, the more I see this shift from doing to coordinating and this was a surprisingly large additional step. I thought I'd already shed all remnant of actually doing things myself, but apparently not. It re-highlights the importance of building a high-performing culture and all the questions therein - about trust, development, setting and maintaining standards, etc. These few weeks have also showed where I've been too tolerant - or perhaps more accurately where my tolerance will get me in trouble if I do a CIO role properly. In my current role, if something isn't quite done on time there is usually enough space for me to help them out or do bits myself. As the CIO, this is not possible. They have to perform and if I need to course-correct, I have to be able to explain and set them going with a minimum of my own input.

This creates additional pressure on me to always excel at communication. Always write clearly, speak clearly, show strong direction, etc. No ambiguity. It also requires more of others, and I found it interesting quite how acute I felt the need for clear and timely communication almost above getting the answers I actually wanted. One of the last things I wrote to the department was a gentle challenge to us all to think about how we can communicate better and cut down on noise.

It also emphasises something I've always told people who want to move into more senior roles. Know how to take ownership of a problem. The value to someone in a role like this of someone being able to remove a problem from your shoulders and make it all work without too much from you is phenomenal. Importantly, this is not just "offer support", it's do, own, lead. For me, it's the golden ticket to being noticed and given further opportunities. As it happens, it's also much better for the CV / interview.

Another aspect of the CIO job was the sheer volume of reading. I have to read a lot as a director, but the number of papers and (hhgggnnn) PowerPoint decks that need absorbing is ridiculous. I found myself reading them in the evening, when I could get a drink and be much more confident of a few quiet hours without any interruptions. This, of course, did wonders for my work-life balance. It highlighted three things to me. First, the importance of getting submissions in on time as the knock-on effect can be board members getting 130 pages of "excitement" to read for a meeting 48 hours away. This was ... not a highlight. Second, the importance of writing clearly and effectively and spelling correctly. Especially in Word, where mistakes are underlined. Get it right. Third, I'm going to have to develop my speed-reading skills.

However! There were some bits that were easier. While there was a huge amount to do, I found the immediacy of my normal job diminished. That is, there were many things I needed to look at but I could choose my approach much more and spend less time in meetings than I normally do. So while the volume was huge, I did find myself with a bit more time to breathe and consider. It was weird and I think this reflection is more that I need to change how I approach my day to day.

Also, it was very interesting seeing all the department machinery working away. There is a great quote from Yes Prime Minister (which I can't put my hand on right now, so you'll have to suffer my paraphrasing) where the PM is talking to his Private Secretary about the role and the PS is explaining that while there are many things the PM SHOULD do and any number of things the PM CAN do, there is actually very little the PM HAS to do. The CIO role felt like that - if I'd sat back and waved things onward I could have gotten away with doing very little for quite some time. So the old systems analyst part of my brain turns towards the problem of maximising that - which is the question "how do I build a department that doesn't need me". I actually mentioned this concept in the interview for my current job so it's nice to have it reinforced.

Not that everything was plain sailing of course. I had to intervene in a few projects to help them back on track. I had to front a few concerns and problems to the wider organisation, and internally. My favourite moment, however was in the dying moments. I told my colleagues I was logging off for the weekend, and thanked them for all their support. I had one last quick look at my email. Then the website blew up. P1 incident, all the sirens going.

Who said technology doesn't have a sense of humour...

Artificial Empathy and The Role of Emotional Intelligence in Effective Management

As a manager, you're responsible for leading and motivating a team of individuals with different personalities, strengths, and weaknesses. One key trait that can help you succeed in this role is emotional intelligence.

Emotional intelligence, or EQ, refers to the ability to understand and manage one's own emotions, as well as the emotions of others. It includes skills such as empathy, self-awareness, self-regulation, and social skills. Here are some ways that emotional intelligence can help you be a more effective manager:

1. Build better relationships with your team. By demonstrating empathy and understanding towards your team members, you can create a sense of trust and psychological safety. This can lead to better communication, collaboration, and ultimately, better performance.
2. Manage conflict more effectively. Conflict is a natural part of any workplace, but as a manager, it's your job to resolve it in a constructive manner. By using your emotional intelligence skills, you can identify the root causes of the conflict, communicate effectively, and find solutions that work for everyone involved.
3. Make better decisions. By being self-aware and understanding your own biases and emotions, you can make more rational and objective decisions. Additionally, by understanding the emotions and perspectives of others, you can make decisions that take into account the needs and concerns of your team.
4. Motivate and inspire your team. A manager with high emotional intelligence can inspire and motivate their team members by understanding their individual needs, providing feedback that is tailored to their strengths and weaknesses, and creating a positive and supportive work environment.

In short, emotional intelligence is a crucial skill for effective management. By understanding and managing your own emotions, as well as the emotions of others, you can build better relationships, manage conflict more effectively, make better decisions, and motivate and inspire your team.

---

The above was written by this month's guest writer - ChatGPT. For those who don't know, ChatGPT is a large language model - that is an artificial intelligence which has been trained on a huge body of literature to effectively generate its own content. On a whim, I asked it to generate me some blog post titles for posts about management and one that came up was "The Role of Emotional Intelligence in Effective Management". I asked it to write that post and the unedited results are above. Not bad, eh? Guess we're all out of a job pretty soon.

Well...

There are a few reasons the answer is "no" which other, more expert, writers have gone through before. In very simplified terms, this kind of artificial intelligence works by guessing the next word in the sentence using probability from analysing a vast body of existing literature (the "large" part of the language model). It generates text using a variety of language rules and learns some context from interactions in the conversation (eg in my example I said "please write the eighth suggestion" and it knew what I meant) and this all results in something that looks like it can respond intelligently to your questions.

So, again very simply put if you say to ChatGPT "Twinkle twinkle" it will respond with "little star" - not because it understands, but because the vast majority of times when someone says "twinkle twinkle it is followed with "little star". This also creates a problem with accuracy - ChatGPT has no idea what it's saying but that won't stop it saying it very confidently[1].

Anyway, there are a load of fascinating technical and ethical questions here but I want to look back at the post it created for me. This one is high-level but you can keep asking "how do I ..." and drill deeper and the answers are pretty good. However, to me they all feel like they lack some substance and are hollow checklists and that raises an important question to me. I'm less interested in "is an AI drawing closer to doing my job?" rather "what is the point of my job if it feels like a very clever predictive text system can do it?" 

And to note - I have deliberately been asking questions about emotional intelligence here. This should be what separates people from computers. Does management just lack substance?

Sadly, the answer is often yes. If a manager learns how to lead from a book and follows the steps then they will become ManagerGPT as above, and will indeed be an efficient but hollow step in a reporting chain. I've certainly met and worked with managers like this - people who seem to have learned about being human from a distance and don't seem to be able to reconcile the recommendations with their own actions. They sometimes do quite well, but they are rarely recognised as good leaders by those who have to follow them.

So what did ChatGPT miss? There are no anecdotes in there - it's a cold checklist of suggestions with no emotional warmth or grounding. It can't follow through - it's all well and good having the step "learn the names of your report's children" but it wont actually go out and do that. These are the kind of thing that separates "leadership" from "authentic leadership".

Most importantly, AI cannot give the gift of its time because it has basically infinite time available to it. This means even when it is developed so it can fake the above (and it will be) we as humans will not respond well because for compliments to land we need to know it's more than mathematically calculated, or for attention to matter we need to know there is some kind of cost to the giver. For me, this is why we laugh at that scene in Demolition Man where the man is getting a pep-talk from an ATM (before being beaten up by Wesley Snipes), but that same dialogue would be more poignant if spoken by actual people.

In short, writing about emotional intelligence and empathy and actually developing them are quite different things and for the moment this is where humans can still add value. And, obviously, we should be making sure we actually DO that thing. All humans - even managers.

If you want to have a go at getting ChatGPT to do your job for you, you can find it on the OpenAI website. I certainly haven't used it to generate papers for the board (although if I did I'd absolutely list it as a co-author).

[1] This example comes from Simon Willison's excellent post about Bing and is originally drawn from Talking About Large Language Models by Murray Shanahan.

Migrate from Codeship to Github Actions for projects using Mina

Oh look - another free tool has decided to eliminate its free tier. My last post was about migrating away from Heroku as it torched its free offering and now Codeship has followed suit. In the past, I have written about using Codeship as a part of my toolchain for CI/CD but over recent years I have moved away from it to Github Actions for running linters and tests and using the Github integrations of the hosting providers themselves for deploying web apps. This means I've migrated many of my projects away from Codeship already, but there has been one significant holdout - my static sites.

Usual disclaimer - this is for a personal project, so I'm more interested in free solutions than high speed or resilience. Docs for myself, maybe someone will find this useful, etc etc

The problem

My static sites are generated using Jekyll, and deployed via Mina. Mina is a great tool, but for the purposes of this post it opens a connection over SSH and executes some commands, which require some environment variables.

I also need to migrate my "tests" - which are just building the site locally to make sure it's not broken - so I'll note that step here.

Migrate test build for a branch

Making the branches do a test build was simple - set up Ruby, get code, run the test code. All very similar (and easier) than doing this with Rails and wrapped up in a short Github Action.

Migrate deployment

Making the deploy script trigger on merge to the main branch was a little trickier (full example below).

First gotcha - when one merges a branch into main, the event that triggers is a "push" event. Knowing that, getting it to work is really easy but that took a bit of hunting. It also means that a push directly to main will trigger a deploy, if your main branch isn't protected.

Second gotcha - environment variables. I didn't want my server config in my public code, so I needed to set some environment variables in the settings of the repository. There are a few ways to do this in Github - I settled on using "Repository secrets" which are found in the project Settings -> Secrets and variables -> Actions then "New repository secret". They are accessed in the workflow file as ${{ secret.MY_SECRET }} and if you want to use them inside a script, you need to pass them through explicitly by setting the env for that script. Example below.

Third gotcha - SSH key. Mina opens a connection to a remote server, so it needs access to an SSH key. I created a public / private pair on the server and added the public key to the usual place, then put the private key in the repo secrets as above. Using this action allowed me to load the key into my workflow easily, and then I could use ssh-keyscan to push it into the container known hosts, allowing Mina to work properly. Again, example below. Shout out to this post for the pointers on the SSH key setup.

And lo, deployment works. Here is the complete deploy Action.

Tidy up

All that is left is to tidy up. Couple of simple steps here:

• Remove Codeship keys from server
• Delete Codeship project / build

And we're done! Like my previous Github Actions implementations, this is almost entirely generic so I can copy / paste into other projects with minimal effort and make sure I have light CI/CD for my fun projects. Now it would be great if I could get back to some of my creative programming projects for a while, instead of having to rebuild toolchains.

Migrating away from Heroku

Along with almost every developer I know, I was very sad to see the end of Heroku's free tier. I was hoping they'd reverse this decision, but it seems that was too optimistic so in the end I had to choose between killing all my toy applications, paying money or finding an alternative.

I did consider just shutting things down, but I had two concerns with this. First, I have one application I actually use and losing it would be a pain. Second, and more importantly, if I have no solution to this problem it adds another barrier to me coding as a hobby. So this needs solving, and not just by throwing hundreds of pounds a year at it.

Finding an alternative took a long while, primarily because I was trying to find a straight all in one alternative. While they are out there, the free tiers are not suitable - the closest insist on deleting my database after 90 days. Instead, I ended up separating application hosting, database hosting and sending of email and the below is where I ended up, written in detail in case it helps one of the many others doing this same thing. I did this all back in November, before the Heroku switch-off, and a few months all it seems to all be running well.

My applications

Everything I'm moving is a personal project (no url customisation, limited need for backups, high availability, etc) and a simple 12f(ish) application. They are written in Ruby (mostly Rails, one Sinatra) and make use of Postgres on the backend and some simple mailing. Nothing complicated and Heroku managed all this for me before.

To move from Heroku SendGrid to independent SendGrid, I did find I had to make a small update to the mailing config.

Most of this diff are linting updates - only the domain change should be necessary.

Databases

I chose ElephantSQL as it has a good enough free tier offering (limited to 20mb) and lets the free databases persist. Detailed steps for anyone who, like me, hasn't done much with a Postgres database for a while.

Setup:

• Create account in ElephantSQL (sign in with Github)
• Create database
• Get connection string

Backup from Heroku:

heroku pg:backups capture --app myapp
curl -o latest.dump `heroku pg:backups public-url --app myapp`

Restore to ElephantSQL:

pg_restore --host "machine.db.elephantsql.com" --port "5432" --username "blah" 
    --password --no-owner --dbname "blah" --clean --verbose "latest.dump"

And voila, database ready to rock in ElephantSQL. This can be checked by running up locally, thus:

DATABASE_URL=postgres://blah:blahblah@machine.db.elephantsql.com/blah bundle exec rails s

Mailing

My requirements are very light here since I only use email for "forgotten passwords" email. For this, a free [SendGrid](https://sendgrid.com/) account was more than enough. I'm using SendGrid because this was the Heroku solution, so it makes for an easy switch.

• Create account in SendGrid
• Create API key

Application hosting

I wanted something which, like Heroku, was a container-based PaaS. I certainly can wrap my applications in containers or deploy to virtual machines, but honestly who has the time. I want to point my application at a hosting provider and have it do the work for me.

For this, I looked at a load of options but I settled on two - Koyeb and Render. Both have workable free tier hosting. Render works like Heroku used to - giving a number of hours per month and spinning down applications when they are not in use. Koyeb simply gives you $5 credit per month and lets you choose how to host. This is an enough to have an application running on a low tier package without spinning down, but not enough to run several applications.

For deployment, the steps were basically the same. For Koyeb:

• Create account in Koyeb (sign in with Github)
• Create application
• Add Github integration
• Add to "run" command: rails db:migrate && rails server
• Select repo (this took a while to appear for me, although Github was having issues when I did this)
• Add DATABASE_URL env var
• Add RAILS_MASTER_KEY env var
• Add SendGrid env vars
• And any other env vars

Then off it goes. I did find it took a little while to start working, but it has been fine since. Also, a slight gotcha - Koyeb appears to have some broken validation when it comes to counts, eg "must be 3 characters or more" seems to actually mean "more than three characters".

For testing, I did all this (including the mailing change above) in a branch then cleaned up at the end.

For Render it was much the same, except I didn't need to specify the RAIL_MASTER_KEY env var and I needed to write a build script.

Cleanup

Finally, to finish off:

• Spin down the Heroku app
• Remove auto-deploy to Heroku (in my case from Codeship)
• Toggle Koyeb / Render over to watch "main" if using a test branch
• Make sure the instance is healthy (and make sure not exposed the wrong port anywhere)
• Final redeploy

Roundup

That is pretty much it. I have been running this setup since November with no problems. I am getting fewer notifications through to Slack, so at some point I would like to get better alerting of start / end deployments but since it's just me working on these things I can watch them easily enough - it's not a priority. I'm just pleased I can still deploy easily and free as that means I'll keep my hand in writing bits and pieces.