How to Log Which Line Initiated a Query in Rails

Create a file (something like) config/initializers/active_record_log_subscriber.rb

…and paste this into its contents:

module LogQuerySource
  def debug(*args, &block)
    return unless super

    backtrace = Rails.backtrace_cleaner.clean caller

    relevant_caller_line = backtrace.detect do |caller_line|
      !caller_line.include?('/initializers/')
    end

    if relevant_caller_line
      logger.debug("  ↳ #{ relevant_caller_line.sub("#{ Rails.root }/", '') }")
    end
  end
end
ActiveRecord::LogSubscriber.send :prepend, LogQuerySource

Now, whenever a line of code initiates a SQL query the line will be logged.  This greatly improves the ease of tracking down inefficiencies such as like N + 1 queries.

If you would like to learn more about exactly how this mix-in works and how the original author went about figuring out how to add it, you can read more about it here.

 

Commenting and Documentation in Ruby and Rails

It’s always been frustrating to me that Ruby and Rails don’t really have strong templated standards on how to best document your code. There are several options available and it can all be very confusing and unclear.

There is one template I’ve found works best and its super simple:

# Brief explanation of what the code does.
#
# * *Args* :
# - ++ ->
# * *Returns* :
# -
# * *Raises* :
# - ++ ->
def my_method

end

While there are many other perfectly good documentation libraries this has the benefits of being simple to remember, very easy to read in code, and compiles out super nicely using

rake doc:app

Example (strongly based off this post):

# This is an example method commented the way I 
# like. It sums the three arguments and returns 
# that value.
#
# Lorem ipsum dolor sit amet, consectetur adipising,
# do eiusmod tempor incididunt ut labore et dolore. 
# Ut enim ad minim veniam, quis nostrud exercit
# laboris nisi ut aliquip ex ea commodo consequat.
#
# * *Args* :
#   - +apples+ -> the number of apples
#   - +oranges+ -> the number of oranges
#   - +pears+ -> the number of pears
# * *Returns* :
#   - the total number of fruit as an integer
# * *Raises* :
#   - +ArgumentError+ -> if any value is nil or negative
def sum_fruit(apples, oranges, pears)
  ...
end

Will render into something like this:


Other than this, I tend to try and follow the general conventions as outlined by the Rails contribution guidelines on commenting.

NotImplementedError Must Have Battered Wife Syndrome

I’ve been doing something very wrong my whole life.  I think it’s time to confess and seek help.

You see, like so many developers I see the NotImplementedError class in various languages and think, to my self “I haven’t had a chance to fully implement this class, so in my methods that my colleagues call that I didn’t finish yet, I will raise that exception to let them know.”

I am ashamed of myself.  And it’s time to get help.

You see the actual purpose of the NotImplementedError (as defined by the Ruby docs) is:

Raised when a feature is not implemented on the current platform. For example, methods depending on the fsync or fork system calls may raise this exception if the underlying operating system or Ruby runtime does not support them.

ERMERGERD!  I am a terrible person. Abusing this error class just because  my sub classes cannot meet the contractual obligation of the super class.

And I know I am not alone.

So whats the best practice, then?

Well, in Ruby’s case it turns out the answer is super simple; don’t raise an error at all! Instead, document the expectation and if a subclass fails to meet it a NameError, or often its subclass NoMethodError, will be raised automatically.

 

How to manually mark ActiveRecord attribute as dirty

Recently I had quite a conundrum. I was working on a project’s model that was using the ActiveRecord save method update_columns to avoid executing the callbacks on the model, however had the unexpected side-effect further down my stack when a method in the chain checked if a particular attribute changed, wasn’t working because as well as skipping validations and callbacks; it also bypasses the ActiveModel::Dirty logic!

This caused me quite a problem, as I feared that I would be forced to use a method of saving the record that triggered my callbacks (very undesirable).

Fortunately, there is a solution.  Because the ActiveModel::Dirty class is still loaded, you can manually mark with attributes are dirty by simply calling [attr_name]_will_change! before the change to track the attribute.

attribute_name_will_change!

Normally this is all done for you and now later on in the stack when you do something like

if attribute_name_changed?
  was = attribute_name_was
end

It will work, despite using a method to save that doesn’t automatically track the dirty attributes for you.

Actually the ActiveModel::Dirty module has some pretty cool stuff in it. Much of it never gets properly explored in the average Rails project, but it provides some hidden magic, like being able to fully revert an object and undo a save, providing a kind of diff of changes and some other really neat stuff.

A better way to benchmark

Today I found some magic.

include ActiveSupport::Benchmarkable

An ActiveSupport module that provides an AMAZINGLY elegant alternative to the classic Ruby Benchmark class; which I had always considered to be a bit ‘clunky’.

ActiveSupport::Benchmarkable provides you a simple method ‘benchmark’ which you can pass a few options and block through and output the benchmark result to the log. Wrap this block around expensive operations or possible bottlenecks to get a time reading for the operation.

For example, let’s say you thought your data transformattion method was taking too long; you could wrap it in a benchmark block.

<% benchmark 'v' do %>
  <%= expensive_record_transforming %>
<% end %>

That would write something like “Processing some data (143.6ms)” to the log, which you can now use to monitor or compare speed over time (make sure its not getting worse as data size increases) or compared to alternate methods for optimisation.

The ActiveSupport::Benchmarkable docs also indicate that, you may give an optional logger level (:debug, :info, :warn, :error) as the :level option. The default logger level value is :info.

<% benchmark 'Low-level files', level: :debug do %>
  <%= lowlevel_files_operation %>
<% end %>

Finally, you can pass true as the third argument to silence all log activity (other than the timing information) from inside the block. This is great for boiling down a noisy block to just a single statement that produces one log line:

<% benchmark 'Process data files', level: :info, silence: true do %>
  <%= expensive_and_chatty_files_operation %>
<% end %>

while these samples are indicative of what they would look like in a .html.erb file (view) – it also works in any other place you use the mix in.

I have personally used these to great success in controllers and models to help diagnose specific controller/model operations suspected to be slow.

This ActiveSupport module is an immensely powerful tool to have in a developers arsenal. I highly recommend it’s use.

IDisposable StopWatch to Benchmark code blocks (via Lambda or Delegate) in C#

Sometimes you need to time code execution for use *inside* your application. the StopWatch class was designed just for this purpose, but you end up with some pretty ugly code that looks like this:

Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
// Do something.
for (int i = 0; i < 1000; i++) { Thread.Sleep(1); }
// Stop timing.
stopwatch.Stop();
// Write result.
Console.WriteLine("Time elapsed: {0}", stopwatch.Elapsed);

This gets really messy, especially if the code is already pretty complicated. But with very little effort you can make the .Net StopWatch just as beautiful as the Benchmark class in Ruby.

public class DisposableStopwatch: IDisposable {
  private readonly Stopwatch sw;
  private readonly Action f;

  public DisposableStopwatch(Action f) {
    this.f = f;
    sw = Stopwatch.StartNew();
  }

  public void Dispose() {
    sw.Stop();
    f(sw.Elapsed);
  }

}

To use the new stopwatch:

using (new DisposableStopwatch(t => Console.WriteLine("{0} elapsed", t)) {
  // do stuff that I want to measure
}

This is the best solution I’ve ever seen for this problem! No extension (so that it can be used on many classes) and very clean and very simple!

SerializationException Running Npgsql Commands on .Net Entity Framework

Recently, while working on a project for a client I was writing a .Net application using the Entity Framework backed onto a PostgreSQL database. All of a sudden I got a System.Runtime.Serialization.SerializationException when I was trying to run my app, and the stack trace (weirdly) didn’t help.

It turns out that while I had added the Npgsql dependency to the project, it wasn’t enough. Entity Framework it seems, needs to have access to Npgsql.dll assembly information.  When EntityFramework tries to use the library, it won’t find it (despite being in the project) unless it is in the Global Assembly Cache (GAC).

The Global Assembly Cache (GAC) is a folder in Windows directory to store the .NET assemblies that are specifically designated to be shared by all applications executed on a system. Assemblies can be shared among multiple applications on the machine by registering them in global Assembly cache(GAC). Source

The best way to resolve this is to use the Npgsql installer that matches the version in your project from: https://github.com/npgsql/npgsql/releases. Those setups take care of registering Npgsql in GAC and set up the machine.config file to include the Npgsql db provider factory.

After I did this, everything worked as it should, and all was right with the world.

97 Things Every Programmer Should Know

PLEASE NOTE: This is a copy of the official list and I only copy it here for my own personal purposes (in case the list is lost), and not because I am trying to circumvent copyright or profiteer from it.

  1. Act with Prudence by Seb Rose
  2. Apply Functional Programming Principles by Edward Garson
  3. Ask “What Would the User Do?” (You Are not the User) by Giles Colborne
  4. Automate Your Coding Standard by Filip van Laenen
  5. Beauty Is in Simplicity by Jørn Ølmheim
  6. Before You Refactor by Rajith Attapattu
  7. Beware the Share by Udi Dahan
  8. The Boy Scout Rule by Uncle Bob
  9. Check Your Code First before Looking to Blame Others by Allan Kelly
  10. Choose Your Tools with Care by Giovanni Asproni
  11. Code in the Language of the Domain by Dan North
  12. Code Is Design by Ryan Brush
  13. Code Layout Matters by Steve Freeman
  14. Code Reviews by Mattias Karlsson
  15. Coding with Reason by Yechiel Kimchi
  16. A Comment on Comments by Cal Evans
  17. Comment Only What the Code Cannot Say by Kevlin Henney
  18. Continuous Learning by Clint Shank
  19. Convenience Is not an -ility by Gregor Hohpe
  20. Deploy Early and Often by Steve Berczuk
  21. Distinguish Business Exceptions from Technical by Dan Bergh Johnsson
  22. Do Lots of Deliberate Practice by Jon Jagger
  23. Domain-Specific Languages by Michael Hunger
  24. Don’t Be Afraid to Break Things by Mike Lewis
  25. Don’t Be Cute with Your Test Data by Rod Begbie
  26. Don’t Ignore that Error! by Pete Goodliffe
  27. Don’t Just Learn the Language, Understand its Culture by Anders Norås
  28. Don’t Nail Your Program into the Upright Position by Verity Stob
  29. Don’t Rely on “Magic Happens Here” by AlanGriffiths
  30. Don’t Repeat Yourself by Steve Smith
  31. Don’t Touch that Code! by Cal Evans
  32. Encapsulate Behavior, not Just State by Einar Landre
  33. Floating-point Numbers Aren’t Real by Chuck Allison
  34. Fulfill Your Ambitions with Open Source by Richard Monson-Haefel
  35. The Golden Rule of API Design by Michael Feathers
  36. The Guru Myth by Ryan Brush
  37. Hard Work Does not Pay Off by Olve Maudal
  38. How to Use a Bug Tracker by Matt Doar
  39. Improve Code by Removing It by Pete Goodliffe
  40. Install Me by Marcus Baker
  41. Inter-Process Communication Affects Application Response Time by Randy Stafford
  42. Keep the Build Clean by Johannes Brodwall
  43. Know How to Use Command-line Tools by Carroll Robinson
  44. Know Well More than Two Programming Languages by Russel Winder
  45. Know Your IDE by Heinz Kabutz
  46. Know Your Limits by Greg Colvin
  47. Know Your Next Commit by Dan Bergh Johnsson
  48. Large Interconnected Data Belongs to a Database by Diomidis Spinellis
  49. Learn Foreign Languages by Klaus Marquardt
  50. Learn to Estimate by Giovanni Asproni
  51. Learn to Say “Hello, World” by Thomas Guest
  52. Let Your Project Speak for Itself by Daniel Lindner
  53. The Linker Is not a Magical Program by Walter Bright
  54. The Longevity of Interim Solutions by Klaus Marquardt
  55. Make Interfaces Easy to Use Correctly and Hard to Use Incorrectly by Scott Meyers
  56. Make the Invisible More Visible by Jon Jagger
  57. Message Passing Leads to Better Scalability in Parallel Systems by Russel Winder
  58. A Message to the Future by Linda Rising
  59. Missing Opportunities for Polymorphism by Kirk Pepperdine
  60. News of the Weird: Testers Are Your Friends by Burk Hufnagel
  61. One Binary by Steve Freeman
  62. Only the Code Tells the Truth by Peter Sommerlad
  63. Own (and Refactor) the Build by Steve Berczuk
  64. Pair Program and Feel the Flow by Gudny Hauknes, Ann Katrin Gagnat, and Kari Røssland
  65. Prefer Domain-Specific Types to Primitive Types by Einar Landre
  66. Prevent Errors by Giles Colborne
  67. The Professional Programmer by Uncle Bob
  68. Put Everything Under Version Control by Diomidis Spinellis
  69. Put the Mouse Down and Step Away from the Keyboard by Burk Hufnagel
  70. Read Code by Karianne Berg
  71. Read the Humanities by Keith Braithwaite
  72. Reinvent the Wheel Often by Jason P Sage
  73. Resist the Temptation of the Singleton Pattern by Sam Saariste
  74. The Road to Performance Is Littered with Dirty Code Bombs by Kirk Pepperdine
  75. Simplicity Comes from Reduction by Paul W. Homer
  76. The Single Responsibility Principle by Uncle Bob
  77. Start from Yes by Alex Miller
  78. Step Back and Automate, Automate, Automate by Cay Horstmann
  79. Take Advantage of Code Analysis Tools by Sarah Mount
  80. Test for Required Behavior, not Incidental Behavior by Kevlin Henney
  81. Test Precisely and Concretely by Kevlin Henney
  82. Test While You Sleep (and over Weekends) by Rajith Attapattu
  83. Testing Is the Engineering Rigor of Software Development by Neal Ford
  84. Thinking in States by Niclas Nilsson
  85. Two Heads Are Often Better than One by Adrian Wible
  86. Two Wrongs Can Make a Right (and Are Difficult to Fix) by Allan Kelly
  87. Ubuntu Coding for Your Friends by Aslam Khan
  88. The Unix Tools Are Your Friends by Diomidis Spinellis
  89. Use the Right Algorithm and Data Structure by JC van Winkel
  90. Verbose Logging Will Disturb Your Sleep by Johannes Brodwall
  91. WET Dilutes Performance Bottlenecks by Kirk Pepperdine
  92. When Programmers and Testers Collaborate by Janet Gregory
  93. Write Code as If You Had to Support It for the Rest of Your Life by Yuriy Zubarev
  94. Write Small Functions Using Examples by Keith Braithwaite
  95. Write Tests for People by Gerard Meszaros
  96. You Gotta Care about the Code by Pete Goodliffe
  97. Your Customers Do not Mean What They Say by Nate Jackson

How to Install ‘therubyracer’ or ‘libv8’ gem(s) on OSX

Recently, I need to move some Rails projects I was working on to new computer and this needs me to install all the dependencies for these projects.  While using bundler to install the gems; I encountered the following error:

extconf failed, exit code 1
Gem files will remain installed in /Users/ash/.rvm/gems/ruby-2.2.1/gems/libv8-3.16.14.3 for inspection.
Results logged to /Users/ash/.rvm/gems/ruby-2.2.1/extensions/x86_64-darwin-14/2.2.0-static/libv8-3.16.14.3/gem_make.out

An error occurred while installing libv8 (3.16.14.3), and Bundler cannot continue.
Make sure that `gem install libv8 -v '3.16.14.3'` succeeds before bundling.

Fortunately, with homebrew fixing this (on OSX 10.11, El Capitan at least) worked perfectly. Simply execute these commands:

brew install v8
gem install therubyracer
gem install libv8 -v '3.16.14.3' -- --with-system-v8

Estabilishing ActiveRecord Database Connections in Ruby (But Without Rails)

Anyone who has even the smallest amount of experience developing with Ruby on Rails knows that Rails has some pretty sweet configuration conventions which make switching between environments very easy. Switching from development to production to testing is as easy as changing the RAILS_ENV variable. No doubt Rails does some dark magic behind the scenes to trivialise this. But what if you’re writing an app in Ruby, without the Rails to guide you?

ActiveRecord 101

Establishing a database connection in ActiveRecord without Rails is pretty basic:

require 'active_record'

ActiveRecord::Base.establish_connection({
  adapter:  'sqlite3',
  database: 'db/test.sqlite3'
})

It establishes a connection to the specified database with the specified connection  configuration. If you were using MySQL or PostgreSQL you would provide the relevant configuration such as username, password and host. With the connection established, we can now start consuming the connection however we like. For example, we may want to use funky Rails’ style models (so we can easily create and populate tables and reference them):

ActiveRecord::Schema.define do
  create_table :things do |t|
    t.integer :id, :null => false
    t.string  :name
  end
end

class Thing < ActiveRecord::Base
end

Thing.create({
  id:   0,
  name: 'Broomstick'
})

Care about the Environment

The other magic in Rails is the environment selection. The main benefit with this is that we can have isolated connection configurations and have our Ruby app automatically select the correct settings for us (using our environment variable ENV).

require 'active_record'

conf = case ENV['DB']
when 'conf1'
{
  adapter: 'sqlite3',
  database: 'db/mydb1.sqlite3'
}
when 'conf2'
{
  adapter: 'sqlite3',
  database: 'db/mydb2.sqlite3'
}
else
  raise 'export DB=conf[n]'
end

ActiveRecord::Base.establish_connection conf

Mind you, before this will work you first need to set the environment variable:

export DB=conf1 # or conf2

Now, depending on ENV[‘DB’], the code will open a connection to the corresponding database.

Best Practises

It’s never a good idea to store sensitive information (like database connection information and usernames and password of any kind) in your code base. Rails uses an external database configuration file called database.yml to solve this problem, so lets implement one ourselves.

database.yml:

conf1:
  adapter: sqlite3
  database: db/mydb1.sqlite3

conf2:
  adapter: sqlite3
  database: db/mydb2.sqlite3

Loading this YAML file is very simple:

require 'active_record'
conf = YAML.load_file('database.yml')
ActiveRecord::Base.establish_connection conf[ENV['DB']]

Now we can set our environment variable same as before, but our code will look in a YAML file for the proper database configuration automatically, without having to store database connection information directly inside our code.