geekylucas

#whippet #wine (Taken with Instagram)

Sun, 07 Oct 2012 20:42:29 +1100

#whippet #wine (Taken with Instagram)

Horizontal scaling. #weber #bbq 🍗🍖 (Taken with Instagram)

Fri, 05 Oct 2012 23:53:58 +1000

Horizontal scaling. #weber #bbq 🍗🍖 (Taken with Instagram)

Ribs. 3-2-1 style (6hrs). Let’s go. (Taken with...

Sat, 01 Sep 2012 12:28:28 +1000

Ribs. 3-2-1 style (6hrs). Let’s go. (Taken with Instagram)

Pulled Pork #weber #bbq (Taken with Instagram)

Sat, 07 Jul 2012 13:39:43 +1000

Pulled Pork #weber #bbq (Taken with Instagram)

Some thoughts on the recent AWS outage (and outages in general)

Sun, 01 Jul 2012 20:32:00 +1000

1. Your most important data should exist somewhere outside AWS.

At the very least it should exist in another region. While most people have bought in the AWS model in a fairly big way I still think we need to be prepared for a time where AWS completely disappears. For most people I suppose this means that if your database is not being replicated out of AWS then data loss is something you may have to get comfortable with one day.

2. Power outages will happen

A lot of people are dumping on AWS right now for not having decent backup power. The thing is that it’s really hard for a hosting provider to respond to every type event which might cause power failure. So, you’re doing yourself a disservice if a power outage is not a scenario you can recover from. History has proven that the perfect combination of failures will happen from time to time. Even Rackspace is not immune to these things. Does anyone remember the great Texas Trucker incident of ‘07?

3. EBS is great, except when it’s not

Performance issues aside, EBS is mostly awesome. After AWS restored power yesterday our EBS volumes were frozen and inaccessible. That is a pretty special kind of frustration to experience. All your data is there but you’re waiting on a set of vague and ill-defined circumstances to occur before you can access it again. The correct response to this situation is not “wait”. Since there is an indeterminate amount before AWS gives you back your data you may as well start some sort of recovery process. In our case the recovery process finished first.

4. Plan B, C, etc

The more “writes” your system handles the less valuable EBS snapshots are as a recovery tool. Data can get stale really quickly. When Amazon restored access to EBS snapshots we were able to recover our most recent database backup and put it on a new server. However our data was around 40 minutes old. So, I then had to go dumpster diving through our offsite mysql replica and extract the last 40 minutes of data from the bin logs. Success!

AWS was still busy restoring EBS volumes for many hours after we were able to restore service so again, I think the lesson here is don’t wait. Execute a few recovery plans and see which wins.

5. RDS is great, except when it’s not

All I will say is that the ability to replicate data across regions or out of AWS is a critical missing feature from this service. I still really like RDS in theory, I go through phases of being quite enthused by it.

6. Finally, make sure you can talk to each other

Instant message services and IRC are pretty crap in emergency scenarios. If you have the option of FaceTime/Skype you will resolve things much quicker! Yesterday I had a FaceTime call going with Chippie and I have no doubt this shortened the time to recovery a lot.

From a few months back. #whippet (Taken with instagram)

Mon, 27 Feb 2012 08:34:54 +1100

From a few months back. #whippet (Taken with instagram)

Installing Devil RubyGem on MacOS (Lion)

Fri, 03 Feb 2012 16:50:37 +1100

Problem:

lucasbookair11: % gem install devil
Building native extensions.  This could take a while...
ERROR:  Error installing devil:
        ERROR: Failed to build gem native extension.

        /Users/lchan/.rbenv/versions/1.9.3-p0/bin/ruby extconf.rb
checking for main() in -lIL... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/Users/lchan/.rbenv/versions/1.9.3-p0/bin/ruby
        --with-ILlib
        --without-ILlib

Solution:

lucasbookair11: % brew install devil
lucasbookair11: % gem install devil --no-ri --no-rdoc

Two Corgis on a Treadmill

Mon, 30 Aug 2010 17:13:12 +1000

Two Corgis on a Treadmill

Introducing Snazzy Napper!

Wed, 25 Aug 2010 13:54:26 +1000

Introducing Snazzy Napper!

"I haven’t got a sign on the door that says “white people only”. I don’t care..."

Fri, 20 Aug 2010 09:46:07 +1000

“I haven’t got a sign on the door that says “white people only”. I don’t care if you’re black, brown, yellow… orientals make very good workers.”

- David Brent