Fun with Threads

During the release of rails 2.2 we made a lot of changes to ensure that we could dispatch requests in parallel by using multiple threads. Some like to describe this as ‘threadsafe’, but as I’ve argued before that’s a little too vague for me.

One of the key lessons we had in the very very early days of rails is that it is impossible to require code when you have multiple threads on the go. It takes a while for people to get their heads around this, typically they ask things like “Why not just put a mutex around require?”. I figured I should write this up for future reference as I haven’t seen it described anywhere else, and people are still learning this the hard way.

This isn’t an issue with autoload. Autoload makes the problem much worse by deferring the require statements to some ‘random’ moment in the future. If you call require from threaded code, you’re going to get bitten by this feature. Let’s start with a simple example:

We’ve all written code like this before, it’s the very basis of how ruby works! Now let’s imagine we want to run this code in multiple threads. Easy right? We’ll require ‘customer’ if it’s not already defined, then do the same thing.

If you run this enough times you’ll get an error like this:

NoMethodError: undefined method `scale!' for #<Customer:0x5f00dc>

WTF is that thing on about? The method is RIGHT THERE! This issue lead some less experienced people to say silly things like “When I used rails in a multi threaded environment methods kept appearing in the wrong thread”. But given how common and underdocumented this issue is, it’s hardly their fault.

The problem comes about because of the way constant definition works in ruby. It isn’t atomic. The second the interpreter has read the “class Customer” line of your file, it has defined the constant Customer, but the constant doesn’t have any methods, or a constructor. It’s like a baby-constant, and if you try to do anything with it, you’ll get all kinds of crazy errors. If you’re playing along at home and want to see this happen, the easiest way is to do this:

Now at first glance this is stupid behaviour, why on earth doesn’t ruby just avoid defining the constant until the very last end statement? Remember back to when you picked up the pick axe and it told you how to define class methods? There are a bunch of different types of syntax available. What about this one?

If Customer wasn’t defined immediately, that syntax couldn’t work! Class methods in ruby aren’t something magical, they’re just methods defined on that particular instance of a Class. Also it’d be impossible to have bi-directional references in your class declarations. ask the delphi guys how much fun that is

So the problem with require and autoload isn’t something about the implementation of how require works, or how autoload works. It’s an inherent limitation of the way ruby works. It’s completely impossible to define constants in one thread, without those constants being accidentally used before they’re ready.

Thankfully we knew about this problem going into the threadsafety project, and the vast majority of the work josh did was to catch all the places where we defined methods and classes, and do that work prior to the dispatcher starting up. This means you pay a penalty at start up, but your application will work correctly irrespective of which constants you reference and when you reference them. We intend to use autoload in non-threaded deployments for 2.3, and to support that we have to preload everything in a similar fashion.

One of the more challenging things with this issue is catching the problem. If all you ever test is a single method, then the first warm-up call will require everything and nothing will break. Hello world is an awful test case here.

So if you’re thinking of trying out jruby and using the nice native threads, be sure to audit every line of code you have for calls to require at runtime, including autoload declarations. If you’re using a framework or library which uses autoload, be sure to work around this problem by requiring everything explicitly ahead of time. If you don’t your code is only ‘thread safe’, and you’re just waiting for trouble.

If you’re a library author and would like a hand figuring out a solution for your case, drop us a line on the rails core list. We’d be happy to help.

Note: I’ve used gist to handle the snippets here, you noscript users will have to trust me :)

Countries and Controversies

The History of Country Select

Before rails 2.0 we received a large number of tickets with modifications to the list of countries in our built in country_select helper. Some of these were people with conflicting preferences, Great Britain vs United Kingdom. Others were questions about whether we should use the english spelling or the native ones, Spain vs Espana.

These patches were a constant distraction and I was concerned about the potential for stepping into controversial issues or legal matters. To my mind there were basically two options: synchronise our list of countries with some neutral source, or remove our list of countries entirely. Given the large number of people who relied on that functionality it seemed simpler at the time to use an ISO standard, and the ISO 3166 Long Names list seemed perfect.

To my mind the ISO list was ‘completely neutral’ and an ‘impartial alternative’, it let Rails have a list that was devoid of my own personal politics. We could avoid any suggestion that we were picking sides in border disputes or questions of recognition and nicely side step the definition of ‘country’ by simply appealing to the ISO.

We received a few patches since we made the switch, and rejected them all by saying “It’s not our place to decide what a country is, please take it up with the ISO”.

Where we find ourselves

However the names of two countries on that list have caused issues for people in recent months. The first was the inclusion of “Palestinian Territories, Occupied”, and the second is the name for Taiwan as “Taiwan, Province of China”. The second issue has recently burst into life again

Given the strength of feelings on the matter we decided to remove the country_select functionality altogether from future versions of rails. Clearly the ISO 3166 list isn’t the ‘completely neutral’ list that I hoped for, and I deeply regret the offense that I’ve caused.

This controversy has been especially frustrating for me because I actually agree with those people who have been upset by this issue. It’s difficult because despite my opinions on the matter, I’ve tried to be as neutral as possible, doing my best to pick the path of least resistance. Unfortunately, it seems that I’ve failed. So here’s my attempt to fix things.

I’m reluctant to make any deviations from the list provided by the ISO organisations as it’ll open the door to several other issues which we’re not qualified to answer. Just a short list of potential complications are:

• Transnistria
• South Ossetia
• Cyprus

So we went back to the original decision, and chose the other path. country_select is no longer in rails, and ideally that would be the end of the matter. But to allow users to avoid errors in their applications we added a plugin.

What’s the Way Forward?

As has been pointed out repeatedly, and angrily, this just moves the issue from one git repository to another. Despite the fact that it now warns you when you install it, people will blindly install the plugin, ignore the warning, and piss people off. So I figure there are a few options:

1. Choose another list of countries provided by some other authority
2. Remove the plugin and break everyone’s applications come 2.2, unless they want to use the list of alternative plugins which have already sprung up
3. Remove the taiwan entry from the plugin, so at least the offensive line doesn’t get displayed.
4. Follow some corporate guidelines such as IBM’s

I’d appreciate any feedback from the Taiwanese rails community about what they think we should do here.

Paris, in a few words

We’re having a blast in Paris. 5 weeks in the latin quarter was lovely. 5 months in le marais will be too.

Now to find some french lessons.

Rubinius runs Rails

Congratulations to the rubinius guys on getting our little web framework that could to start up and serve requests. It’s great to see so much experimentation and progress happening in one large and bazaar-like community!

Thread Safety

In this day of multi-core CPUs and parallel programming platforms a common meme amongst developers has become “Is library X thread safe?”. As reasonable as that sounds, it’s actually not a particularly interesting question to ask, and anyone who claims to have a useful answer, probably doesn’t know what’s going on. I say that because unless you say what you want to do in parallel, it’s meaningless to say whether or not a library is thread safe.

Take a popular definition of thread safety, this one’s from wikipedia:

A piece of code is thread-safe if it functions correctly during simultaneous execution by multiple threads.

Using that definition, is libevent thread-safe? The answer turns out to be a little more complicated than a yes or no answer, it depends what you’re trying to do in each of those threads.

What about Rails ? Well, strictly speaking Rails is currently thread-safe. It functions correctly during simultaneous execution, we just have this huge mutex around the entire request so we only handle one request at a time. What about Active Record? Well, yes… But it opens a connection per thread which will quickly exhaust your database server’s resources. Both of these restrictions present problems for people wanting to use rails in a highly threaded environment, so perhaps thread safety isn’t a particularly useful goal without further clarification.

It’s just not useful to say whether or not rails ( or any other librarary ) is ‘thread safe’ without saying what you want to do in parallel, the code may function correctly but run slower than it would in a single thread.

With rails there are several different concurrent use cases we could aim to support in our new thread-safety project

• Dispatch requests in parallel with a single thread per request
• Allow rendering to be performed in parallel for a single request (several threads each rendering a seperate partial for the same response)
• Make Active Record use a connection pool to prevent opening an excessive number of connections
• Allow a single Active Record object to be worked with concurrently in multiple threads.

Each of these goals will involved differing numbers of locks, and differing degrees of change to the current design. They’ll also improve (or degrade) performance by different factors for different benchmarks. Whatever the results of this process there will still be some locks around some methods, and those locks are bound to annoy someone.

Phrases like “W is completely threadsafe” or “X isn’t thread safe” should be a red flag to you. They’re a sign that the speaker doesn’t quite understand the subtlety of the question, and the irrelevance of the answer without further qualification.

The Paris Project

One of the nice things about the last two and a half years has been working from home. You avoid 90% of the dilbert-style office drama, and the commute is hard to beat. But the nicest thing about working from home is that, strictly speaking, I can work from any home, not just our little place.

So from July the 4th till the end of the year Anika and I will be living in Paris. You know, this paris:

We have an apartment in the Marais for most of our stay, and another in the Latin Quarter for the remainder. We intend to take the opportunity to travel around europe and soak up the french culture.

So if you’re doing something interesting with Ruby, are based in paris, and can handle a visit from a Kiwi speaking awful Franglais drop me a line.

Now running on Passenger

Today I had a few hours to spare and decided to try out Passenger. This blog is hardly a high traffic website, but it has some crazy RewriteRules that I figured would test the limits of the module.

Everything appears to be working perfectly, and it took less than 5 minutes to set everything up. I’m seriously impressed at how simple this was. I’ll confess to being skeptical at first, but so far the package lives up to the promises on its website.

If you notice anything broken let me know. Now I have to figure out how to kill the rest of my spare time.

Married

On Friday the 14th of March 2008 Anika and I ‘tied the knot’ in a small ceremony at the wellington registry office. Thanks to all our well wishers, especially the dozens and dozens of twitter messages I received.

PS – if I owe you an email or something, there’s my excuse!

On Git

If you’re a developer, and read other developers’ blogs, odds are you’ve heard of git. While some of the posts about git are almost absurdly exuberant, there’s no denying that it fixes some of the really hard problems with source control.

I’ve been using git-svn as a frontend for my rails development for around 4-5 months, I can’t imagine going without fast local branches, smart merges and rebases. So naturally I’m interested in migrating rails from subversion to git, as are the rest of the core team. Unfortunately it’s not as simple as running git-svnimport and updating some webpages. We rely on several pieces of subversion which aren’t quite so easy to replace.

Our scripts work with Subversion

Our subversion servers do a lot of traffic. The rake rails:freeze:edge task contributes to this, but so do people using svn:externals for their rails application. The amount of traffic the servers do is an indication that people like to use this functionality and we shouldn’t just turn it off one day. The svn command line client is extremely portable and available on almost every system. The git tool set doesn’t have the equivalent of an ‘svn export’, which is what the freeze tasks and plugin installers rely on.

Trac works better with svn than git

The ability to automatically close tickets with a commit message is really useful, as is the automatic hyperlinking that we can use in ticket comments. There are hacks to add git support to trac, but it will take some time to evaluate them. This stuff currently ‘just works’ with subversion.

Interim solution

In the meantime, I’m publishing my git-svn powered repository locally, at github and gitorious. If you want to use git to work on your rails patches, feel free to grab me in #rails-contrib to talk about your branches, and the steps to get them merged.

Unfortunately due to the way git-svn works, when your patches get applied to rails they’ll be squash merged and then rebased out of existence by git-svn. So once your branch gets accepted, you’ll have to throw it away or suffer thousands of merge conflicts. To minimise your pain make sure you have one branch per feature and ensure all branches can exist independently.

While the limitations of git-svn mean we’re only testing a subset of the git workflow, it’s important that we figure out kinks in a git workflow for rails, so please help out.

Migration plan

So in order for us to migrate from subversion to git, we need:

• A reliable read-only svn mirror based on our git repository
• A git post-commit hook which allows us to close tickets
• Something similar to our trac browser + ticket integration.

The post commit hook and browser aren’t particularly difficult, while gitweb is a little ugly, I’m sure we could make something work. The real challenge is the backwards compatible, read-only svn mirror. Evan had a post on this a while back but that involves initializing a brand new repository and apparently is prone to breaking from time to time. Rick has a simple file copying solution which we may end up using.

If you have ideas or experience with any of these issues, and the motivation to help out, please let us know by emailing the rails-core list or grabbing me in IRC.

"They"

The great thing about working with open source is that if you come across a bug, you can fix it yourself. It’s the whole underpinning of the success of the model. Every developer scratches their own itch, and the cumulative result is a vibrant and useful piece of software driven by an enthusiastic community. If you’ve been fortunate enough to be the steward of a successful open source project you’ve probably seen this in action.

If you’ve been working on a really successful open source project, you’ve probably seen something else happen.

In the early stages of your project, the people who hang around are willing and able to help. When they find a bug or some missing documentation, they write a patch to remedy the situation. Even those who can’t fix the bugs will put in a Herculean effort to help isolate it with a test case.

But after a while the ‘they-brigade’ shows up, and instead of offering to help they merely act indignant that ‘they’ (the open source project) haven’t done it already. Rather than spend 10 minutes writing a patch to improve the documentation of a project they’ll spend 20 minutes writing an outraged blog post demanding that ‘they’ fix the problem.

Most open source projects aren’t controlled by a secretive cabal of conspirators. If you notice something broken don’t assume that the maintainers are deliberately trying to fuck with you. It probably hasn’t hurt them yet, or they’ve chosen to spend their time on something else. Remember, you’re not paying for this, no-one’s violating their SLA. Instead of seething with rage, try to find a test case that isolates the bug you’ve found. Instead of flaming on a forum, try to sketch out the points that the improved documentation should cover.

You’ll find your own work much more fulfilling if you’ve taken the time to improve your tools. The fact that the wider community benefits is just a nice big cherry on top.

Rails

This started as a comment on Geoff’s post but seemed to justify knocking the dust off this thing.

Apparently there are some people who feel that it’s ‘impossible’ to get patches into rails and that the core team doesn’t communicate with its user base. As someone who spends a lot of my time helping others contribute to rails I find the ‘impression’ hard to reconcile with my own experience. We have an active mailing list and irc channel where discussion takes place almost every day.

In the 2.0 effort we’ve received patches from 177 different individuals ranging from minor typo fixes, through to entire new features. So if you’ve got some killer ideas to contribute, subscribe to the core list and start talking about them. 2.0 is almost here, but there’s plenty of scope for new or improved functionality in rails.next, and we’d love to hear from you.

Rails 2.0 has had several profiling-driven optimisations we found by benchmarking real applications rather than hello world apps. Named routes were a common source of slowness in big applications, so 2.0 has new code that makes them several times faster. Repeatedly parsing Dates and Times from database also contributed to performance problems, so we have code to cache the results and for good measure we made the parsing faster too.

I’m not saying we’ve solved all the problems, or that rails is now perfect. No framework is! If you have an idea for improving performance, and a profiler report showing it makes a big difference, join the irc channel and lets talk about it, we’re always open to ideas.

Like any open source project rails depends on you, the community, for contributions. If you have something you feel like fixing, jump on in!

Clever Caching

Besides our talk, my most valuable experience at RailsConf was talking with Tobi about the new caching strategy (‘Tobi caching’) he’s using at Shopify. There are a few parts which all work together nicely.

Etags Matter

As Joe explained, using good etags, can substantially reduce your bandwidth bill. In his case it was a 70% reduction. The take away from this is that you need to think about how you’re going to generate opaque cache coherency values for your actions. For a good intro to HTTP conditional gets, go read this tutorial by charles.

Expiry is a Pain

Anyone who’s had to write sweepers for for an application with heavy caching knows how frustrating it can be. After all, cache invalidation is one of the two hard things in computer science. If you could somehow avoid expiring all the ‘stuff’ you’re caching, your life would be much much easier.

Memcache is Smart

Memcache and the Memcache client libraries have plenty of smarts built into them, despite being ‘dumb by design’. The client libraries use clever hashing to know which server to talk to, this lets you run a cluster of caches without worrying too much about which keys live on which server.

The server also has its own smarts about what keys are important. When it needs the memory memcached will drop the least recently used values, thereby ensuring that your unused keys won’t be ‘wasting space’.

Mix it all together

So with that in mind, what can we do to improve our application’s performance, and simplify our application.

Forget about expiry

As mentioned before, expiry is a complete pain in the ass. So let’s not do it. The key to getting away with this is to pick a key which completely encapsulates the resource you’re caching, and also ensures that if anything relevant changes, the key changes. Take the case of this blog post, a simple key would be the permalink, however if we used that, we’d need to expire the cache every time someone commented, or I corrected a typo.

The no-expiry alternative would be for mephisto to keep a ‘version number’ associated with each post and increment it every time someone commented, or the post body changed. Once it was doing that, we could construct a key that looked like www.koziarski.net:clever-caching:#{version_number}. Every time the version number changed, we’d get a cache miss, and regenerate the content, but subsequent requests will be served out of memcache. No more expiry!

Now that we’ve saved all that CPU time, we should see if there’s a way we can save some bandwidth too.

Embrace Etags

Thankfully, our cache key has all the properties of an ETag, whenever something important changes, our cache key does. So lets use that as a basis of building our ETag by using the MD5 hash. The only reason I don’t advocate using the cache key itself, is that you may want to include sensitive data in the key. Now we can just chuck d444415a8228fbed44cfa7ef39f15d8b into the ETag header, and compare our key with the value of ‘If-None-Match’ from the request headers.

Conclusion

By doing this you get the bandwidth savings of HTTP caching, the performance boost of action caching, but without the difficult expiry code. You can avoid all the NFS related headaches of page caching, but still get most of the performance boost.

While the approach won’t suit every project, it could well suit yours. Finally, a snippet of sorts for those of you who think in code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

around_filter :cache_sensibly, :only=>:show

def cache_sensibly
  # compose the key using something we know matches our business
  cache_response(request.host, request.request_uri, @blog.version, @post.version) { yield }
end

private
def cache_response(*keys)
      key = keys * ':'
      # use the hash as an etag so we can cache on 
      # private data
      etag = MD5.hexdigest(key)
      
      
      # first handle HTTP, lets us avoid a memcache hit
      # and saves a huge amount of bandwidth to the client
      if request.env["HTTP_IF_NONE_MATCH"] == etag
        headers["X-Cache"] = "HTTP"
        head :not_modified
        return
      end
      response.headers["ETag"] = etag
      
      # Next check memcache
      if data = Cache.get(key)
        # render from the cached values
        headers["Content-Type"] = data[:content_type]
        headers["X-Cache"] = "HIT"
        render :text=>data[:content], :status=>data[:status]
      else
        # Finally, yield, indicate we've missed then cache the response
        headers["X-Cache"] = "MISS"
        yield
        Cache.put(key, {:content=>response.body, :status=>headers["Status"].to_i, :content_type=>(response.content_type || "text/html")})
      end
    end

What am I Missing?

My recent trip to Microsoft reintroduced me to a few goings on in “The Enterprise”, which is thankfully something of a dim memory for me. The one thing which always seemed like garbage to me is BPEL (and any other XML workflow tool). However there are a bunch of smart people who clearly think that there’s some merit to it. I’m fairly opinionated, it goes with the territory but I figure I should ask “what is it I’m missing”?

For the readers who have yet to learn about BPEL, check out this oracle tutorial.

One of the constructs BPEL gives you is the ability to specify conditionals:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

<switch>
  <case condition="bpws:getVariableData('FlightResponseAA','confirmationData','/confirmationData/Price') &lt;= bpws:getVariableData('FlightResponseDA','confirmationData','/confirmationData/Price')">
<!-- Select American Airlines -->
    <assign>
      <copy>
        <from variable="FlightResponseAA"/>
        <to variable="TravelResponse"/>
      </copy>
    </assign>
  </case>
  <otherwise>
<!-- Select Delta Airlines -->
    <assign>
      <copy>
        <from variable="FlightResponseDA"/>
        <to variable="TravelResponse"/>
      </copy>
    </assign>
  </otherwise>
</switch>

Of course we already have that:

1
2
3
4
5

if FlightResponseAA.confirmationData.Price <= FlightResponseDA.confirmationData.Price
  TravelResponse = FlightResponseAA
else
  TravelResponse = FlightResponseDA
end

It also lets you invoke processes:

1
2
3
4
5
6
7
8
9
10
11
12

<flow>
  <sequence>
<!-- Async invoke of the AA Web service and wait for the callback-->
    <invoke partnerLink="AmericanAirlines" portType="aln:FlightAvailabilityPT" operation="FlightAvailability" inputVariable="FlightDetails"/>
    <receive partnerLink="AmericanAirlines" portType="aln:FlightCallbackPT" operation="FlightTicketCallback" variable="FlightResponseAA"/>
  </sequence>
  <sequence>
<!-- Async invoke of the DA Web service and wait for the callback-->
    <invoke partnerLink="DeltaAirlines" portType="aln:FlightAvailabilityPT" operation="FlightAvailability" inputVariable="FlightDetails"/>
    <receive partnerLink="DeltaAirlines" portType="aln:FlightCallbackPT" operation="FlightTicketCallback" variable="FlightResponseDA"/>
  </sequence>
</flow>

But we already have that too:

1
2

FlightResponseAA = AmericanAirLines.FlightAvailability()
FlightResponseDA = DeltaAirlines.FlightAvailability()

So it seems to me that BPEL and other workflow tools simply take standard programming language constructs, wrap them up in XML and call it something other than programming. All that engineering effort could have been thrown into interesting research to advance the state of our industry, instead we get this mess…

So, what have I missed? Are workflow systems some kind of amazing solution to a really hard problem, or solution in search of a problem?

Microsoft Technology Summit 2007

Last week I had a whirlwind trip to Seattle to attend the Microsoft Technology Summit, which was intended as a gathering of a small group of technologists to discuss today’s technology issues and opportunities, as well discuss Microsoft’s role & future direction.

The Good

Overall the experience was pleasant and reasonably educational. The three most enjoyable parts of the conference were:

1. Hallway conversations with Program Managers for IE and IIS 7, they’re genuinely interested in helping improve the experience of Rails users on windows.
2. Informal conversations over meals or drinks with the other attendees.
3. Spending time with the team in the Microsoft Open Source Lab

It was great to see the new web.config file in IIS7, it seems like the MS guys really paid attention to what people liked about apache. Jim and John’s talk about dynamic languages on the CLR, was pretty interesting too, they’re both clearly sharp and some cool things are probably in the pipeline.

I was really impressed by the powershell demo, it’s like an REPL with classes and methods for almost everything on your system. I just can’t understand why they decided to take $_.

The Bad

The conference did have its downsides though, some of the sessions were clearly targeted more at people who were either currently using .NET or were likely to switch. This meant we often got demonstrations of IDE interaction rather than an explanation of the underlying technology.

Some of the speakers also seemed to think that if you were an ‘open source guy’ (or gal) you were some kind of foaming at the mouth lunatic. Myself and all of the other attendees are far more pragmatic than that, and it was annoying to be treated as some stallman-esque extremist.

The Ugly

The most frustrating session would have to be Don Box and Chris Anderson on “Why Microsoft Sucks”. Several of the funnier quotes from that session have already made their way onto the web, but there were others, such as:

“But if you’re Matz, or DHH, or Larry Wall, you’re screwed, because you don’t have time to build out this stack and then make it interoperate”.

Of course that’s not the real reason we don’t have a WS-* stack. The reality is that the entire set of standards is a steaming pile of complexity with an infinitesimally small value-add which manages to make even the most simple interactions an enormous engineering effort.

Don and Chris are clearly passionate, intelligent guys, but their talk came across as unjustifiably self-assured given how late Vista was, and the huge disaster that WS-* and SOA have inflicted on our industry.

Conclusion

All in all I’m grateful to Microsoft for inviting me to the summit, it was a great opportunity to meet relevant people within the company and the wider community. For more detailed coverage take a look at Ben Galbraith’s MTS07 articles or check Technorati

Full Disclosure: Microsoft paid my way to the summit and gave me some stuff while I was there. If you think that makes me a shill, you’re nuts but welcome to your opinion.