Magic happens

I use the phrase "magic happens" quite frequently when giving an overview of something technical. For example, if I'm trying to explain the concept of the client / server model I might talk about how a web request is routed from a client to a server where magic happens and some HTML is returned as a response. This isn't supposed to be condescending - it's a shorthand for "this is a black box where something else is going on, but talking about it will draw our focus away from the thing we're talking about here". Unremarkable, but the phrase got stuck in my head the other day and while turning it over it made me think about how learning has changed for developers.

When I was starting as a developer I hated magic. I dug into these black boxes and learned a great deal around the problem until I had a good working knowledge of hosting, networking, etc etc. At the time, you pretty much had to learn this stuff to get a decent local dev environment running. This was before frameworks like Rails made it relatively easy to run up a complete environment with a couple of console commands, and a long time before cloud hosts like Heroku allowed the application to be run in the wild with little more effort. Consequently, there was a huge technical barrier to entry for those learning to program. A barrier that now much lower.

This is in no way a post about how the kids today have it easy, or any other such nonsense. It's a musing on what new developers are learning and what that means for the older generation when it comes to mentoring the next generations (amusingly, my spellcheck wants that to read "tormenting the next generations"). It is now conceivable to have reasonably experienced developers (ie not juniors) who are highly competent, having worked in Rails environments pushed to cloud hosting, who have never had to worry about how anything under that works. No knowledge of linux, networking or other things from the sysadmin world. This has a significant impact on debugging problems and system design, which has a knock-on effect on production costs, maintenance and out of hours support rotas.

On the other hand, and before us more experienced folk feel smug, this focused learning means these developers are likely far better at the core language than we were at the same level of experience. If true, they will be better at their jobs than we were (for a narrow definition of "better", granted). However, it will also stunt career progress as a developer wanting to take the lead in designing or building a system generally needs a wider understanding of the subject area. Skills I learned early on were far less useful while I was a mid-level developer but that knowledge was pretty much essential when I moved into more senior work. This is a key area where our responsibility as more senior people comes in - it's not just juniors who need space to learn things.

In my experience, there is another benefit of this focused learning. Good developers who have learned their particular subject in-depth are aware of the edges of their knowledge. They are aware where, for them, "magic happens", and are capable of explaining that well. I'm not advocating siloed thinking or working, but learning how to define and articulate your problem space is an important skill as a developer - one that people who know everything (sigh) sometimes fail at. It's particularly helpful when discussing work and solutions with other, non-technical disciplines such as product and delivery managers.

Also worth remembering is that we learned around the subject out of necessity and back then it was much easier to get a really solid overview of everything. "Everything" has context of course - how many of us studied machine code, chip architecture and instruction sets, memory rings and so on? Some did, of course, but by no means all. Our window of learning took in what we needed. These days there are many more layers of abstraction in the hosting world as well as many more layers of complexity with SCSS processors, Javascript frameworks and transpilation, and on and on and on. The world today is far more complicated and while we may (MAY) have kept pace with new innovations, that is not the same thing as learning it all at once - especially since change continues to happen.

I have a feeling we're going to be seeing a lot more magic happening as the industry matures. We'll have more black boxes and more abstraction of important concepts - indeed making use magic components is a core part of cost saving strategies. Why on earth would you run up your own virtualisation environment when services like Linode exists, for instance? We are going to need to be increasingly aware of how we talk about technology - not just to those outside our discipline, but to those inside too. We're going to need to put an increasing amount of effort into making sure the next generation can get on - either by learning what they need around the subject, or by shaping our organisations to celebrate the new learning.

Return of Notes from a Gaming Table

It's back!

A million years ago, before I started posting my writing on Twitter, I used to write a gaming blog. For the last two years "restart gaming blog" has been on my list of New Year's Resolutions. Well, today I am restarting it. Behold: a new post on Notes from a Gaming Table.

And that is all.

From the darkest depths

I very rarely talk about my mental health. This isn't because I don't think it's important - when it comes to other people, I encourage them to talk and will happily lend a friendly ear if one is desired. I am, however, a hypocrite. Over the last year have I dabbled in burn-out, depression and what was, on reflection, probably chronic stress. At this stage I could probably write a book about how these things manifest for me, however this post is about climbing up after pulling myself clear. Apologies for the turns of phrase - I'm not a doctor and everyone's experiences with mental health are different. One of the reasons I don't talk about myself is the fear that someone who suffers with "proper" depression will call me out for just being a bit sad. I don't think that's the case, but I'm certain other people have a harder time with it than I do.

Anyway, this is about recovery. At rock bottom I needed a trigger to realise quite how bad things had become. For me, this was an argument with someone I hold very dear. It made me re-evaluate what was important in my life and decide that pushing ever onward was not an option - that I was threatening something inviolable. By this stage I had already put at risk my physical and mental health - neither things I am happy to lose for sure, but the slippery slope is not steep and it's easy to put something "at risk" without realising quite how much so. At this point several things happened at once, but the main thing is I pulled myself out of life and work. I then spent a week more or less asleep.

It took a month of doing almost nothing to start feeling vaguely right and slow the destructive cycles. Some of that was spent at home, some was spent with my parents and family. During this time I started programming again. Weird statement for someone with my job, but my role no longer revolves around making things. This is fine and acceptable, but one of the difficult parts of shifting into more strategic work is that you lose the emotional kick of being able to look at that thing you built today. Starting it again in my spare time gave me the chance to set goals and achieve them. I could look at my work and be proud of the progress I was making. I could give myself little kicks of positive emotion.

I wrote a lot of code over Christmas.

With some kind of positive feedback loop in my life, I started asking myself what makes me happy. If I walked away from my current life and started again, what would I do? Why do I want to get up in the morning? When I managed to pull back and analyse my life I was still lacking any real reason to want to engage with the world. My way of dealing with that is to directly face that question and pick it apart until I have the beginnings of an acceptable answer. I'm a middle class human living in an affluent country - I am aware that I have done well in the lottery of life and so I have options that are not available to others. I should be able to find a positive place.

I'm not going into the details of this, but suffice to say I am still working for the Civil Service. For the moment, my long-term goals are still in-line with working there. There are plenty of interesting challenges which will keep me engaged and enthused and which I hope I will approach in an effective and useful way. As an aside, this is the point I want to remind people that "no feedback is good feedback" is a terrible way of treating your colleagues. By all means say if something is bad, but if someone is doing a good job make sure they know. We shouldn't find our sense of self-worth through our work but while we seek enlightenment many of us do. Admittedly, I feel a bit of a rat writing that here as I know a few of my colleagues occasionally read my blog so if that's you tune this bit out and mutter something about fishing for compliments.

That gives me a direction for my life for the moment, however it doesn't help me much in the immediate future. For the short term I need to put some positive things in my life or remove some negative things. These need to be short-term, or at least have tight feedback loops. I'm not in a position to start a six month project that will only make me happy at the end. One of the key things that makes me unhappy at the moment is the deterioration my physical well-being, a combination of my weight and fitness. This is dangerous as losing weight can be soul-destroying. Given I'm living in London I decided to adopt something of a Londoner mentality and throw some money at the problem so I'm giving Alevere a try and so far I'm pleased to say that it is working. I can use my goals within this programme as a framework to trigger other nice things. I'm not sure what yet, but I've just started writing D&D again so I may well treat myself to props or whatever at key milestones.

I can also use the same milestones as a trigger for new, positive behaviours. Deciding these up front means that when the time comes I don't have to think so much and make it far more likely I'll get on and do. Hopefully that will let me ride any emotional wave and keep going, instead of stopping, losing momentum and having to restart again. For instance, I'm going to go back to Yoga when I reach a particular target weight. I haven't decided all of these things yet, but just thinking about what I can do is a positive exercise in and of itself. The timescale for all this is about three months so hopefully by the end I'll have rebuilt a positive life and figured out how to make myself happy in a non-destructive way.

I'm not better yet, not by a long shot. I am, however, moving away from the dangerous place I was in a couple of months ago. The word "triggered" is used far too much in modern parlance, but I can definitely say I know some of my own trigger points and still see them affecting me. Less so now, but the road to recovery is not short. I'm incredibly grateful to the people who have supported me in various ways through this time.

I'm writing this as a note to myself in the future. Maybe it'll be a me who is having a difficult time again and needs to remember that there is a turning point. Or maybe it's a me who is thinking back and wondering what the fuss was about - hello future me, you were not overreacting. If anyone else finds it helpful then I'm glad and please do reach out if you'd like to talk.

The year that was, 2017

Time to once again take stock of the last 12 months and that leads to an important question about 2017. Was it worse than 2016? For me, the answer is yes. Assorted burn-out issues have resulted in a significant decline in physical and mental health and a marked reduction in making things. That is something to fix for next year. Looking at the positives, I've survived a year in a new job and city and I've managed to catch up with a lot of friends from The Past. I've also managed to do Some Things, albeit Fewer Things than I'd have liked. Still, it hasn't all been Killing Floor 2 and Ultimate Chicken Horse.

• 12 posts on this blog (13 including this one)
• wrote a short story
• another year of the Year in Pictures site, up to 15 photographers
• over 385 contributions to various projects on github
• completed GMing a 2 year 5th ed D&D game to great critical acclaim
• got back in to playing D&D online
• started planning a new regular game
• created the university of bath RPG (post to follow on this)
• went on a photography course to learn how to use my camera properly
• 55+ pictures on my instagram account
• loads of photos on my flickr stream
• attended some cookery courses
• updated my website and Gareth's website
• lots of work with build pipelines and hosting
• plenty of game recording, with sadly little to no video editing
• a bit more voiceover work

I also hung up my abacus, finishing my time as a church treasurer and member of our church council with a financial presentation to the congregation. I've continued running the sound desk in the same church.

Resolution count: 3.5/10 - better than 2016. Still bad.

This coming year I'm going to try undoing the problems of this year - most notably fixing my health and writing more. I'm also starting a new role at work and I'm going to be spending more time going to see friends from university who have disappeared a bit. Here we go...

Hosting a Rails App on Cloud Foundry - first impressions

From time to time I have been known to write a bit of code and whenever one writes a web application, there is always the question of hosting. I've done my time in Ops and I can certainly deploy an application to a VPS and run the surrounding infrastructure to make it work - however, that all sounds like more work than I'm willing to put in. This is the world of Cloud hosting and I'd like to spend my time writing applications, not deployment scripts. What I want is something I can throw code at and have it sort itself out but for my own projects the price needs to be low so I'm not spending a ton of money every month on my own games.

This is an interesting niche as I don't have the same requirements for my own stuff as I would for professional hosting. Initially my requirements were:

• Very low monthly cost
• Rails 5
• Database (probably postgres)
• Ability to hook it into some kind of CI (ideally Codeship, as I'm already using that)

For my own projects I'm not that bothered about high capacity, or extensive DR. These are great, but are also expensive.

I'm going to end up on Heroku, because the free tier appears to do everything I want and more. However along the way I tried out Cloud Foundry so I thought it worth writing up how I got started.

Easy stuff first

I signed up for an account then created an org and a space on the dashboard. I also created a database within the space (no binding - it's better to do that with a manifest). This was all achievable via the web interface. The postgres service has the option of a free database, limited to 20mb storage.

Next, I installed the command line interface and logged in (cf l), choosing the space as the default.

Preparing the application

A Rails 5 application needs no additional configuration, beyond migrating it from sqlite to postgres. The easiest way to tie the application to the production database is via a manifest file. Mine looks like this:

---
applications:
- name: yip-helper
  random-route: true
  memory: 128M
  instances: 1
  path: .
  command: bundle exec rake db:migrate && bundle exec rails s -p $PORT
  services:
    - yip-postgres

The name becomes part of the subdomain on deployment. The memory is kept low to keep the costs down for a personal project. The service listed should match the name of the database created in the space, above. Stick this in the repository so it can be used with the CI later.

Now the application should be ready to deploy with a simple cf push.

Continuous integration

I use Codeship, and their docs worked fine for me with two modifications:

• I dropped the CF_APPLICATION envar from the script as it's defined in the manifest file
• My first deploy failed as it couldn't find the required gems - subsequent deploys worked fine, despite a warning about including the .bundle dir in my repo (which I didn't)

Problems

This all works with minimal fuss, however I'm going to end up going back to Heroku. I originally discounted it because it didn't play well with Docker (a requirement I've since abandoned). Also:

• Heroku encrypts traffic for free on their own domain, whereas Cloud Foundry doesn't have this option. I can pay $20/month to use my own domain and cert but this breaks my first requirement. I can understand them charging for additional domain hosting but honestly, securing their own subdomains should be a given.
• The Cloud Foundry free tier database is tiny. Paying for a database adds a lot to the monthly costs - this is true of all the hosting options I looked at - so a useful free tier offering is important.
• Heroku is better supported. In Codeship, for example, there is a plugin to support it whereas Cloud Foundry requires a custom script. It's a simple one, to be fair, but it's symptomatic.
• The Heroku tooling and web interface are nicer. Again, unsurprising given how much longer Heroku has been around. The Cloud Foundry tools are fine, but the doing the same things with Heroku is just easier.

So there it is. These are just my experiences, based on not a lot of time and with the intention of hosting for a personal project.

Remember remember

Let's talk about fire.

I've always loved bonfires. They are primal and evocative - beautiful pillars of light and movement shining bright in darkness. So what does that mean for photography? Taking a picture of a bonfire is both an interesting technical challenge and an exercise in trying to capture the emotion of the moment.

Breaking down the requirements in photographic terms, what do we have? The fire is moving so you need a fast shutter speed to capture the tongues of flame. The scene is dark, but with a bright light source so you need to control white balance. A fast shutter means a higher ISO, to make the camera more responsive to the available light. A wider aperture allows more light in, but also restricts the telephoto options. So a first guess leads to something like:

• High shutter speed
• High ISO
• Wide aperture

So something like this:



Nice and warm, but the body of the fire is over-exposed.

What happens if we drop close the aperture and rely on a crazy-high ISO to capture the available light?



Loses the warmth, but now we can see into the body of the fire.

I wonder how important the ISO is in practice? What is we keep a wide aperture and fast shutter and drop the ISO?



There is more definition on the tongues of flame here when we zoom right in. Pushing the ISO distorts the image so this is a good change.

I started taking these pictures trying to logically get to some decent manual settings. Out of interest I brought out my phone and took a photo then looked at the results:



Huh. Seems nice. Settings?

• Medium shutter speed
• Tiny ISO
• Wide aperture

So the phone decided aperture was the key to sorting out the light capture, and didn't think the movement of the fire was too much of a big deal. Next step was to try this collection of settings on my big camera:



Different again. More warmth, losing definition on the tongues of flame.

And for a last hurrah, how about a different framing - adding some zoom (which narrows the aperture) and correcting a bit with ISO and shutter:



I have my favourites. It's very interesting seeing the same fire depicted so differently just by fiddling with some settings. I'm still calibrating my mind - looking at the subject and trying to imagine the different photos possible with different settings. The first few here I can say I thought through logically but I'm not sure I'd have got to the settings chosen by the automatic mode on my phone. I suppose I'll have to go to more bonfires and keep practising. Such a shame.

All the photographs taken that evening, including some different attempts at fireworks, are in this album:

Getting off auto

I take a decent number of photographs and as well as running a monthly photograph website I am taking it increasingly seriously as a hobby. Most of my photographs are taken using a camera phone thanks to the convenience of it always being in my pocket and, although I've dabbled with manual settings and know (very roughly) what they do I've never really been able to get consistently decent shots on manual mode. Never one to have a hobby and not get nerdy about the details, this month I got myself a proper camera and went with my Dad on a course to learn how to get off auto.

Buying a camera is hard. There are a load of nearly identical models and so much comes down to personal preference at a time when one doesn't know enough to have a preference. Given how expensive lenses can be, this is also a decision about which ecosystem to buy in to. It's a similar lock-in to the Apple / Windows or iOS / Android world in that it's reversible, but change can be tricky, unfamiliar and expensive. I may write about this in the future, but for the moment my key learning was the big choice if buying an SLR is Nikon / Canon. I ended up choosing a Nikon 5600.

Not all the way off auto

I've played around with manual photography before and have a (very) basic understanding of what the various settings do. When switching to manual mode on the phones I've used, the interface has encouraged taking full control - that is handling aperture, shutter speed, ISO and white balance all at once. This has always been too many interconnected, unfamiliar settings to learn and embrace at the same time. Possibly the single most useful tip from this course was to view it differently - take control of one of these aspects and let the camera do most of the rest automatically. This gives priority to a particular setting and therefore set of techniques (indeed, the modes on a proper camera are called aperture or shutter priority so ... yeah).

Aperture

We focused on aperture, letting the camera adjust shutter speed automatically and making corrections for exposure with the ISO setting. A wider aperture (aka a smaller f stop setting) lets in more light, allowing a faster shutter speed. It also allows control of the depth of field, enabling the blurred background effect used to create useful negative space and draw attention to a specific area of a picture.

This is often seen in portraits. Some useful tips for portraits - the eyes are the most important bit and should therefore be in the sharpest focus. So far, fairly obvious. Less obvious is that the camera may well actually focus on the closest part of the face - the tip of the nose - so for a really good picture some manual adjustment is necessary.

Another counter-intuitive tip is that portrait pictures are best taken on maximum zoom or with a dedicated telephoto lens. This is because the longer focal length allows for a wider aperture and hence maximises the depth of field effect.

Results?

And lo, the results.

Or, for a particular example of the portrait stuff above:


We had a lot of fun walking around Lacock and practising the specific techniques that fell out of playing with this one setting and the day did a wonderful job of demystifying the bewildering array of buttons on a decent camera. I'm looking forward to learning more.