Upgrading to Rails 7

I run some simple Rails applications which I keep upgraded. I recently upgraded Ruby and Rails versions and since it's upgrade season, I'm making a note of my experiences ready for the next application I do, and on the off-chance it can help other people get started. And a reminder that I do do technical things occasionally, honest.

To note - this is a very simple Rails application, so only covers the basic gotchas I experienced. It was an app originally written in Ruby 2 / Rails 5 and in this iteration is now being upgraded from Ruby 2.7 / Rails 6 to Ruby 3 / Rails 7. The code is stored in a repository in Github, with CI done both via Github Actions and Codeship and deployment to Heroku.

Ruby upgrade

Good news! The Ruby upgrade (2.7.2 to 3.0.3) caused no problems at all, although I did need to update my Codeship config to remove the explicit installation of Bundler in my setup script.

Rails upgrade

I made use of the Rails upgrade guide - in particular I bounced the version of Rails in my Gemfile from 6 to 7 then ran bundle exec rails app:update which introduced the usual ton of rubocop violations but also created some migrations acting on some tables which didn't actually exist.

These were ActiveStorage tables so I needed to create them with bundle exec rails active_storage:install then rearrange the migrations to get them to work (ie put the creation migration before the modify migrations). I probably could have eliminated the new migrations since I'm not using this function, but it seems there are other references to these tables. It looks like Rails expects these tables to exist, and the lazy option was to create it and be on the standard path. So I was lazy. Those migration do not include timestamps, so I had to add those to appease the linter.

Second show-stopping problem - Rails 7.0.2.1 was a trap. It turns out there was a bug in ActiveSupport which manifested in a variety of different ways. For me, it stopped almost all automated tests working. It was reported and fixed quickly but did manage to cost me a chunk of time trying to figure out what was going on... Anyway, the simple fix is to use Rails 7.0.2.2 instead.

Now fixing some deprecation warnings. action_cable has been renamed to actioncable and this will escalate to a breaking change in Rails 8 so this needs updating in app/assets/javascripts/cable.js.

Also, I was getting references to using older (Rails 5.1) CSRF protection behaviour. There are a load of defaults which are version locked in config/application.rb and these can be upgraded with config.load_defaults 7.0 as per the upgrade guide.

Locally, this was all that was required however when I deployed to production, the heroku build failed at bundle exec rake assets:precompile. Uglifier needed ES6 syntax enabling, with config.assets.js_compressor = Uglifier.new(harmony: true) in config/environments/production.rb.

And that was it! As I said, a very simple application so I'm sure I avoided much of the upgrade pain but I've noted my experiences here to help others get started.

All these changes are captured in a pair of pull requests:

• https://github.com/tomnatt/year-in-pictures-rails-helper/pull/88
• https://github.com/tomnatt/year-in-pictures-rails-helper/pull/89

Bots - a triumph of automation

I'm a bit lost on a website - they've been selling a particular bouquet of flowers for years and years and I'd like to reorder it. But it's not there! Maybe it's a pandemic thing, who knows. Still, at least there is a live chat function so I can ask this question. Live chat is a great way to get customer service, but unfortunately can be costly. Still, it looks like they’ve made the effort here.

Me: Hi there. I’m looking for a particular item <with sufficient description> that used to be on sale but I can't find it. Has it gone?

Ellie the Helper: Hi! Can I take your name please?

Sigh. It’s a bot isn't it? Can you imagine if this played out in an actual store.

Me: <repeats question>

Ellie the "Helper": Are you still there? 

Me: Yes…

Ellie the Definitely Human Helper: Hi! Can I take your name please?

Hhgggnnn… 

I hate bots. I see a lot of them as a user, and also see a lot of requests for them as someone whose job is on the web. They are seen as a nice, positive way of finding information on a website without the user having to find things. Or, to put it another way, without the site owner having to spend money on a decent user interface, information architecture, design, etc etc. Then they can cheap out on customer service too by making it look like there is a real human while trying to fool the user.

The technology does not work that well.

Me: Bob

Ellie the Script Executing HelperBot: Hi Bob! Here is a close alternative! <link>

And I'm presented with a bouquet of flowers. On a flower delivery website. Literally the only thing in common with what I was looking for is "contains flowers". On a flower delivery website.

No attempt to answer the actual question, of course. Just "here is another thing you can buy". It's ok though - automation saved the day and provided another seamless bot experience!

A first look at Github Actions with Rails and Postgres

In my last post, I mentioned that while upgrading my Heroku tech stack I noticed Codeship was experiencing some kind of outage. This seemed to stop anything appearing in the "checks" part of a pull request (including any kind of error message, which was a long way from helpful) and I decided to investigate Github Actions for my CI/CD needs.

I've been thinking about using Github Actions for a little while, for two reasons. First, I wanted to run my linting in my CI pipeline and I know of a rather good tutorial for getting started doing this using Actions (thanks Dean!). Second, this should move my CI config to the project repository (keeping it together and putting it under version control) and remove a dependency on a third party SaaS product. I can't help feeling that the recent Codeship outage (which I only noticed because the check was missing in my PR, and could easily have missed this) vindicates this last point.

As a side note, Codeship now seems to be fully working again.

Anyway, the Rubocop implementation is actually pretty straightforward, but it took me forever to get the tests running because of a few tricks and gotchas which I thought I'd record for posterity.

Bundler

Let's start with a timesaver. I read a lot of examples while setting this up, and some had extensive Bundler config in them. However, if you're using the ruby/setup-ruby@v1 action for setting up Ruby (code here) and you put in:

  with:
    bundler-cache: true

It will just handle everything bundle-related with no more configuration. Hurrah!

Migrating Capybara driver to Apparition

I have no idea if anyone else is still using Capybara Webkit to drive their Capybara tests but I was. It has recently become a pain to install because the underlying library (QtWebKit) has been deprecated. I found this out after quite a while of trying to get the QtWebKit libraries accessible in my Action. That didn't work.

It seems Thoughtbot, the authors, agree with me and have deprecated the thing and recommend a move to Selenium or Apparition. I chose the latter because of claims of backwards compatibility and it was very easy to switch when I finally realised that this was a more sensible way forward. The changes can be seen in this commit along with the inclusion of an Action setting up the Chrome driver in my test workflow.

Configuring the database to work in a containerised world

Good grief this took me forever.

In theory, this is really easy - configure a Postgres database as a service, when the tests run connect it up, and bam. In practice it is also really easy, requiring minimal config to get it working. However, it requires getting a load of options to line up and since it's all running on Github servers, the feedback loop is annoyingly slow so painstakingly iterating through a million tiny variations to get to that simple working config took an eternity.

In the end, there were only two things to note.

First, when configuring the Postgres service one HAS to specify a port (despite it being the default port).

Second, remember to update the test database config in database.yml to accept some environment variables (and also default to allowing the tests to run locally). It's really easy to do when you actually remember to do it...

It's highly like these are more down to my own incompetence than anything hidden or surprising.

And done

And lo, it works. While it took a while to figure all the details out, the results are actually really simple and easy to duplicate for other Rails projects.

The whole change for implementing Github Actions and implementing the other updates can be seen in this PR.

Upgrading to the Heroku-20 buildpack and Bundler 2

I figured it's about time to start moving my various running applications and kit to Ubuntu 20, now there is a new LTS version and well before v18 totally dies. I know I'm late to this party, but I've been busy.

Today I updated an application hosted on Heroku and since it wasn't 100% smooth thought I'd capture my steps, both for myself in future and anyone else who finds it useful.

The tech

• Ruby on Rails application
• Ruby 2.5.1 (Ubuntu 18 default version)
• Bundler 1.17.3
• Heroku on buildpack 18
• Codeship for automated testing and deployment

The process

On Heroku, go to the app page then its settings. On this page you can change the stack via the big red "upgrade" button. This requires a redeploy of the application.

Back at the app, I updated by ruby version (.ruby-version file for me), installed bundler and ran bundle update --bundler to upgrade from Bundler to Bundler2. This provoked some other minor config changes, all in this diff.

Then Codeship broke. As in, it was totally down. Sigh. When this came back, the build was not working, giving me a deadlock error when running bundle install. This gotcha took a while to fix (hence writing this post). In the end, all I had to do was add gem update --system to the build setup commands before gem install bundler.

And voila, we're in the modern world and everything works.

Losing Chrome URLs

 This is going to be a short one, mostly so I've got a reference for the future.

It seems Chrome as of v86 (latest at time of writing - at least on Linux) is hiding the full URL unless it is selected, instead showing only the domain. This is to highlight fraudulent websites for people who can tell the difference between www.google.com and www.google.evilsite.com but get confused when there is a huge set of valid-looking path and parameters after it. It seems that's about 60% of the web using population.

Anyway, if you're in the 40% and you find seeing the whole URL quite useful thankyouverymuch and don't want to have to select the bar to see the information, then you can disable this new feature.

Put this into the task bar: 

chrome://flags/#omnibox-ui-hide-steady-state-url-scheme-and-subdomains

Then search for and disable:

omnibox-ui-hide-steady-state-url-path-query-and-ref-on-interaction

Restart Chrome and lo, the URLs are back where they should be.

For me, I was surprised by this and I was wondering why The Internet had decided to embrace loading pages into frames with javascript, before I realise the browser was doing this not the site.

I can't review your site - I can't see it

Oh, this article looks interesting. It might solve my problem. I'll have a quick re.. NO I DON'T WANT TO SUBSCRIBE TO YOUR EMAIL UPDATES. Has anyone ever actually engaged with a website overlay, beyond searching for the tiny "close" button in annoyance? They are all over the place so there must be a good reason for them to exist. Maybe if we look closely at some of the steps required to make a successful one we'll start to understand.

Spoiler warning: we wont. Overlays are inherently terrible.

Making a successful overlay #1 - the timing

The overlay needs to appear before your victim leaves the site, so it needs to be quick. Never mind that "quick" roughly corresponds with the amount of time someone who is genuinely engaging with your site takes to orientate themselves and settle in to proper reading. In order to catch those people leaving quickly because they don't care anyway you're going to have to annoy your real visitors. Omelettes, eggs and all that.

Making a successful overlay #2 - make it intrusive

You're trying to catch the attention of your victim and what is more attention-grabbing than the entire screen being replaced by the action you want them to perform? Maybe it should flash too. Make sure they are under no illusions - your only interest in them is harvesting a review or an email address. Any content on the site is strictly secondary.

Making a successful overlay #3 - make it alien and hard to close

We don't use javascript alert popups these days because they are so easy to block and everyone has the location of the close button hardwired into their brain. However with an HTML overlay you can avoid this problem so your victim doesn't miss the information that is so crucial you have to interrupt their browsing. And it is, after all, all about you. So why run the risk they can easily dismiss your overlay? Why use established design patterns to put the close button somewhere obvious? Make sure it's carefully hidden and don't even think of adding a keyboard shortcut to kill it. That would be madness. The "escape" key has much better things to be doing.

Making a successful overlay #4 - make sure it is in the wrong place in your visitor's journey

You can't possibly wait until your victim has finished reading the page before asking their opinion or for them to sign up. They might leave at the end, having learned whatever they wanted, possibly even happy - and all without ever giving you their email address. The horror. Never mind that five seconds into a visit nobody has a useful comment beyond "the overlays annoy me" and that putting them right at the start of the visit means your overlay cannot possibly do the job it is supposed to do. Annoyance is valuable feedback so it should be gathered as early as possible.

Make sure you're considering these crucial four points next time you want to really annoy the poor souls who find your site!

The Year in Pictures

A few years ago I was involved with 12 Cakes - a project which put up a cake recipe every month for a year. It was a nice, simple idea and got me thinking about doing something similar using photos. So, after a year of messing about and not finishing the site, I've create The Year in Pictures. Along with five friends and family, I am selecting one photo each month which says something about that month and putting them together on a website.

There is no real goal for this. By the end of the year I'll have 12 months of photos for 6 people which could be used for a variety of things but for a change I'm not worrying so much about the end game and just enjoying watching it grow. Three months in, there is now enough to look at that it's worth being seen beyond the six contributors so I'm going to start mentioning it on Twitter when a new month is uploaded.

If you're interested in the technical details, the website is static HTML generated by Jekyll and styled using Foundation. The photo pages are created using a Jekyll generator (combining a photo and some metadata from a YAML file to produce a full page) and most of the rest is done with macros. The code is on Github if you would like to see how it all works - it was creating using my Jekyll bootstrap project.

Jekyll and Foundation with Guard

tl;dr - I've dropped the jekyll asset pipeline in favour of Guard in my pre-configured example project of Jekyll and Foundation.

A few weeks I wrote about my attempts to create a bootstrapped jekyll / foundation combination using the Jekyll asset pipeline to keep things building on the fly. I hit a snag, though. The asset pipeline was abandoned back in February and is locked to version 0.12 of Jekyll (current release being 1.2.1 at the time of writing). There are various forks of the project which bring things up to date but I'm worried about being left in the same position again and, frankly, I'm starting to question the methodology of using Jekyll to do everything instead of running the tools separately.

What I really want, I think, is some kind of meta-tool which lets me manage my build environment but by using the core tools - in my case Jekyll and Compass and probably a CoffeeScript processor eventually - so I (hopefully) have fewer maintenance problems down the line. Seems like Guard is pretty much what I want. I'm coupling it with guard-jekyll-plus and guard-compass to work with those tools. It isn't perfect, and doesn't assuage my dependency fears entirely but it feels a much more robust solution than the previous version.

My basic jekyll-foundation project has been changed to use this new tool chain and seems much more responsive than before (could be anecdotal) so I'm looking forward to making more use of it.

Jekyll and Foundation automatically recompiling changes

tl;dr - I've glued jekyll, jekyll pipeline and foundation into a pre-configured example project.

I look after a handful of small sites and I've been playing with different ways of maintaining them and keeping them vaguely modern. A CMS is the obvious way forward, but that has the learning curve problem and, frankly, after working away at this kind of thing all day I'd rather approach the problem a little differently.

What I really want is something which lets me write content in my markup language of choice, then generates some html for me and wraps it in a template. It should also handle some of the simpler optimization tasks (minifying, compressing, compiling any sass) and ideally allow some kind of on-the-fly regeneration of all these assets so I can work and hit refresh with it running in the background.

Jekyll seems like a good contender as a starting point. I love the fact it generates static HTML files for me (I'm getting tired of requiring the comparatively slow processing power of PHP for ultra-simple tasks) and it has a "watch" mode for recompiling the pages as I work. It uses the Liquid templating engine, which is sufficiently simple for my needs, and there is support for a variety of markup options.

Next, I need some sass support. This is easily done, but I want a simple workflow so it needs to recompile on the fly while jekyll watch is running. Step up the jekyll asset pipeline which lets me add whatever I like to the jekyll process with a minimum of fuss, including css compression and the sass compilation I wanted. I'm pretty sure I could add CoffeeScript support with it too, but I haven't tried that yet.

Right, I have the components I need. I'm going to work with Foundation 4 because it happens to be a framework I know relatively well. If only there was a vanilla project which pulled all this together ready so I could just fork it and start building a new site. Oh look.

In all seriousness, I've hammered these elements together into something that should be easy to just pick up and start using. I'm hoping it'll be useful to me and - this being the internet - chances are that means it'll be useful to someone else too. If that someone is you, enjoy this vanilla jekyll project with foundation sass built on the fly. Why not let me know?

New skin for my website

In an effort to not look like I've learnt nothing in the last four years, I've reskinned my website. I don't kid myself that anyone actually wants to know this information, but by writing a post about it, in years to come I can find out when I released this particular version by looking at my blog. That's keeping some kind of digital timeline - the digital archivists would be proud.

Posable Phil!

Last post I mentioned that Wired picked up one of my photos. Seeing how exploiting Phil Wilson's face could be fun, Liam McMurray (chief web designer at the University of Bath) sprang into action and created Posable Phil - the new must-have toy this March.

Need to launch something in image form? Why not have Phil do it!

Need to make light of a difficult scene? Why not have Phil do it!

Face immobile? Incapable of expressing surprise for yourself? Why not have Phil do it!

With just a few quick clicks you (yes, you!) can create a whole series of pictures expressing surprise! Like this!

And how much is this wonderful toy? £1000? £100000000? £100000000000000000000? No! Thanks to the wonders of the Creative Commons license, it will cost you absolutely nothing (terms and conditions apply, original license states non-commercial use only)!

You can download this amazing gift here.

The original photo, with attached license, can be found here.

My Name In Lights

A while ago I expressed surprise that someone used one of my flickr pictures in their blog post. Well, it seems to be catching - a rare google search for my stuff online has shown that my pictures are popping up all over the place. The most interesting one for me: I'm on Wired!

I suppose I shouldn't be surprised - this is, after all, what Flickr is for. The social and searchable aspects make my (and, indeed, anyone's) photographs available for anyone who is interested and the work done by the Creative Commons license people mean others can use my material. On top of the theory, there are the stories about people whose lives have been changed or saved by various social networking sites. The difference in this case is that, in its own small way, it has happened to me. And that is nice.

For anyone interested, my flickr stream is here.

I'm an artist

It turns out someone on the internet likes my art.

That's about all really, I'm just slightly surprised.

For more pictures by (and indeed of) me why not visit my flickr pages.

My face on TV

A short while ago I delivered an entry-level talk about RSS and Atom feeds in the University of Bath. It was a very simple talk - what a feed is, why you should care, how to spot them and what to do with them once you've found them. Despite a 6 week gap between writing it and performing it (damn that illness) it went down very well, with much praise and thanks being heaped on me.

This talk was the second in a series of talks given by my team and was the first to be filmed in front of a live studio audience. The film can be found here. It's not the most polished piece of presenting I've ever done, but it was good fun.

Facebook thinks I'm a loser

I think I'm going to have to change my relationship status on facebook as the left menu on my homepage is being filled with adverts for dating websites:

25 and Still Single?
Why not join $latest-slappers-R-us-site?
And look! Here is a picture of someone you'll never meet on our site!

I particularly love that phrase:
"25 and STILL single?!"

Which may as well say "Are you a LOSER?" Thanks facebook - and I thought you cared.

rubygems

Rubygems on Ubuntu seems to break itself when you update:

sudo gem update --system

I've lost track of the number of times I've had to fix this (and inevitably forgotten how to do it each time) so for anyone interested here is the fix:

* Open a terminal
* sudo vi `which gem`
* under the line:
require 'rubygems'
add:
require 'rubygems/gem_runner'

Job done.

I can't remember where I first saw this fix, but I know I've found it here at least once, so thanks for the help Nick.