Hello, we are 29 Steps. We build software using Ruby.

I caught the MongoDB bug recently as part of my own exercise to learn more than 1 database system in my development life somewhat similar to ‘7 databases in 7 weeks’ but less severe regime.

I realised that it is possible to store file uploads within a mongodb instance using its gridfs file system so decided to have a go. I first setup carrierwave and carrierwave-mongoid but it seems that the latest versions of both gems do not work well with the mongoid version 3+ gems due to api changes. After several attempts and google searches I gave up and was in the process of rolling back mongoid gem to 2+ when I suddenly thought of Dragonfly the fantastic file uploader by Mark Evans of new bamboo

One of the coolest features of Dragonfly is the ability to have a single upload such as an image and being able to chain various processing options to it or to scale it on the fly. This beats the carrierwave approach of declaring versions in my opinion

To get this to work I have to do the following:

1. Declare dragonfly as a gem dependency in your gemfile. Put rack-cache above it if you wish to cache the assets
2. Create a dragonfly initialiser file documented here in the docs. Now Dragonfly::DataStorage::MongoDataStore has a require for the mongo gem so you need to update your gemfile to also include the mongo, bson and bson_ext gems. I put these before the dragonfly and rack-cache. To configure the datastore I decided to use the more verbose approach and declare everything manually as documented here. 
3. Within your model declare the relevant accessors. In my case I have a User model which needs to have its own avatar so I added in the following:
   

    field :avatar_image_uid
    image_accessor :avatar_image 
   

Restart your application server and you should have a working version of file uploads using GridFS with mongodb. Dragonfly also seems to deal with the streaming of the uploads back to the front end without any need to write your own rack middleware to deal with it.

Hope this helps someone with starting with file uploads in mongo. I am still learning about the system so any feedback is greatly welcomed.

I keep having to google this all the time so I thought I would remind myself here.

If you are using the AWS-S3 rubygem, you can use the AWS::S3::S3Object instance method exists? with the target bucket name to determine if the object in question exists or not.

For example:

  if AWS::S3::S3Object.exists? file_name, destination_target_name
     # do this if it exists
  else
     # do something else since it does not exist
   end

One of the most popular plugins to use for file uploading and processing is carrierwave. I have used it extensively in all my rails projects that require some form of file upload and it has never let me know - it always works out of the box and as expected.

One of the issues that has always interested me but I never took the time to investigate is - how do you define per user settings for each uploader instance? Normally the carrierwave settings are defined in advance in a rails initializer file and that pre determines the type of storage, where the file is uploaded to etc. In a recent project using carrierwave I had to upload user files to Amazon S3 using fog but each file had to go to the end user’s bucket.

What I did was to remove the initializer file from the rails config directory as that does not apply anymore. Within the uploader you would have to define the settings as instance methods. Thats the glue to the trick: define each setting as an instance method and then grab the user’s setting within the method

For instance to set the s3 bucket directory, we can do something like this:

In the example above, we have set the fog_directory to be based on the settings attribute of the user model on which this uploader is mounted on. The same can be applied to all the other setting such as ‘fog_host’, “fog_public’ 

You may be wondering thats great but where do I define each user’s s3 settings? The answer is you should not have to. When you are uploading into another user’s s3 bucket you would first need to add your settings as an ACL entry into their s3 account first. The example above assumes you have already done so and all you need is just your own s3 settings to upload to their s3 bucket.

Hope this helps someone.

In a recent project, I had to redirect users from certain parts of the world to a holding page based on their IP addresses. This could normally be achieved in Rails using something like request.remote_ip.

However the app is on Heroku and to make matters worse, it uses SSL which means any calls to request.remote_ip object only returns the IP of the ELB ( elastic load balancers) addresses which is normally in the range of ‘10.x.x.x’ which in turn returns an undefined location when passed to an underlying script which tries to determine the user’s location based on a reverse lookup.

After various solutions I came to the conclusion that this could only be achieved on the client side using a javascript solution. ipinfodb.com provides such a service which includes a free jquery script premade. All you need to do is sign up to the service to get an API key, download the jquery js file and within the js file, lists the countries which you are allowing access for. More examples can be found at their api page. I used their JSON api which comes with a ready made example in jquery.

This is my setup below as per their instructions but with the callback function modified as the example they provided did not include how to check the user’s country:

var allowed_countries = ['UK','USA'];

// geolocate object is provided by the jquery script
var visitorGeolocation = new geolocate(false, false);

var callback = function(){
  var country = visitorGeolocation.getField('countryCode');
  if($.inArray(country,allowed_countries) < 0){
    window.location.href='/global';
  }
);


visitorGeolocation.checkcookie(callback);

The solution works by making an API JSON request; storing the user’s location details into a cookie and then checking the country code from the cookie. If you are in the UK / EU please make sure you comply with the new EU cookie regulation before using the example. 

Hope this helps someone with the same issue I had. Sadly I still have not managed to come up with a Ruby / Rails solution due to the way Heroku works with regards to SSL and Elastic Load Balancers.

Snipplets, the syntax highlighting app I built a month ago, makes strong use of Pygmentizer to highlight user submitted code segments. It does so by making a POST request to an external pygmentizer app located on google app engines which then makes a successful request back to the app on completion of the highlighting.

While this serves the purpose of the app, it has some drawbacks. Snipplets relies on a Resque queue to wait for the request to be complete which means the user ends up with an inconsistent view of the codesnipplet; also what happens if the API service fails?

After watching the railscasts episode 207 which introduces pygments.rb I ported all the syntax highlighting code to using pygments.rb.

Whats kool about pygments.rb is that it still uses pygemntizer but the program itself is embedded within the gem and your app using ruby-python and libffi ( through the ffi gem ) which means any environment that supports or runs python can run pygments.rb and that includes Heroku.

However I ran into this peculiar bug whereby pygments.rb crashes the entire application server ( both Mongrel and Webrick as shown in the Railscasts) with a killed signal message on the console when running Pygments.highlight method.

If you are running into the same issue, try switching over to Thin which works for me without any issues. As yet I still cannot ascertain the error whether it is system dependent or not and ruby-debug did not thrown up any unusual bugs. Could have something to do with asynchronous processing but will look into it in a separate post.

If you are new to using pygments.rb and have just switched over you might notice that the line numbers have disappeared completely if you used to have line numbers enabled. This is off by default in the gem and can be enabled by passing the line numbers options to true inside the Pygments.highlight method.

Below is a short snipplet describing some of the workarounds for pygments.rb as discussed above:

The second issue you might run into when deploying onto Heroku is the version of RubyPython you are using. If you get errors such as ‘lexer cannot be found’ then it is related to RubyPython as pygments.rb cannot pick up on the Python interpreter. To do so simply create an initializer file ( I called mine ‘rubypython.rb’ and put it inside config/initializers) with the following code:

Essentially what the above does is to start a python interpreter session so that pygmentizer can run. I can’t seem to find any other way to avoid this else the entire app will fail. The other gotcha is that it only seems to work with RubyPython 0.5.1 so I have the gemfiles locked down to that version. I tried it with both RubyPython 0.5.3 and the latest 0.6.1 to no avail - the cedar stack on heroku just refuses to communicate with rubypython.

Hope this helps someone with similar dilemma as pygmentizer is a great syntax highlighting tool.

In one of my recent BDD endeavors I came across a test case whereby I have to make an assertion that a float value generated from an object method is equivalent to a certain value.

If you were to do an out right comparision such as ‘1.5.should == 1.5’ you will get an rspec error saying ‘expected 1.5 but received 1.5 instead’ . Check out this ruby forum post for the details: http://www.ruby-forum.com/topic/169330#742994

Only an approximation of the floating point values can make the specs pass, like so:

1.5.should be_close 1.5, 0.1

The should be_close statement is an RSpec expectation which takes 2 arguments: the value to compare against and a precision value. The default precision is 0. Essentially the line above states that the value of 1.5 should be approximately close to 1.5.

From my own understanding of the issue above, it is not a Ruby issue but a general issue with floating point arithmetic itself.

In the previous blog post, I demonstrated how to stub EM::HttpRequest calls using the built in EventMachine::MockHttpRequest module. While that is sufficient if you are sorely using EM::HttpRequest as your sole http client, it would mean refactoring the test code again if the client changes.

We want to make the stub calls as generic as possible so it is not tightly coupled to a specific http client implementation.

WebMock is the ideal library to utilise in this case. It is a http stubbing library that works with various popular Ruby Http clients including EM and Net::HTTP amongst others. It has a self explanatory interface and provides options to tailor your stub calls including setting request headers and body responses. In addition, it also allows you to match stub request calls based on regular expressions, which makes it even more flexible to adapt to your own test code if your application accesses various external API endpoints.

WebMock, by default, makes the external API request call so you would need to disable it using the following commands:

WebMock.disable_net_connect!

To re-enable it, simply use: WebMock.allow_net_connect! Since these commands may be used multiple times within your spec files, I tend to put them within methods inside spec_helper.rb

<script src=”https://gist.github.com/790144.js”> </script>

The WebMock.reset! command clears the WebMock request registry so it treats each stub request call as a new request. This comes in useful when you want to test the number of times a particular request has been made using the following matcher: WebMock.should have_requested(:get,<my url>.once

The stub request need to be initiated before the actual call is made. It takes the following format: stub_request(METHOD,<URL>||<REGEXP>).with(<REQUEST BODY>).to_return(HASH OF OPTIONS such as body,status,headers)

For example:

<script src=”https://gist.github.com/790149.js”> </script>

The syntax is self explanatory - thats the beauty of using WebMock. Essentially what the line above does is to stub out your request to @post_regexp which will be a url to your external api using the POST action; with the parameters of the request defined in the body hash ; and finally to return a hash of response which includes status code, header, and body.

You will need to include the stub call before making any actual requests else you will receive ‘Request not registered errors in the console’. The reason this occurs is that within the WebMock gem, it keeps an internal hash of url requests and if its not stubbed out will attempt to make an actual request.

There are some pitfalls to avoid when using WebMock in my own experience:

Firstly, if you are running a local service such as a SOLR server, in your WebMock.disconnect call, you need to pass in ‘allow_localhost => true’ else WebMock will just register all your local calls to be unregistered.

Secondly, within autotests , you might have to stub out the same request twice from my own experience. Not sure if thats a configuration issue on my end or the way autotests work.

Thirdly, if you are using older versions of Rspec, you need to explicitly include the following into your spec_helper.rb file to use the matchers:

  

 config.include WebMock::API
 config.include WebMock::Matchers

If you are unsure of how to structure your stub requests, I will be following this up with some more examples of using WebMock in the wild in my own projects.

In my efforts to improve upon the test coverage of the AmazeSNS gem, I came across an interesting problem: How do you mock EM:Http calls so that it does not call the SNS endpoint whenever the specs are run?

After a process of trial and error, the following is what I came up with which seems not only able to mock out the calls but also allows you to state what the response should be:

<script src=”http://gist.github.com/665582.js”> </script>

The example is an excerpt from an actual spec file from the AmazeSNS project. Its purpose is to test that the method calls do trigger the API access but without actually making the call. The EventMachine::MockHttpRequest object is a subclass of EventMachine::HttpRequest. It stores an internal hash of the request parameters such as url, method, headers and the data returned by the server in an actual call.

When you invoke the EventMachine::MockHttpRequest.use block, it automatically swaps out the HttpRequest object for the MockHttpRequest object. Within the block, you register the url using the EventMachine::MockHttpRequest.register method. This creates a Struct object on the fly with the params passed into the register method stored in an internal hash stored in the EventMachine::MockHttpRequest class.

Subsequent calls that trigger the API call will perform a hash lookup to grab the Struct object and its associated data defined in the data block. The data block is used to define the server response. You can set the response status through the ‘HTTP/1.0 XXX OK’ declaration to fit your interface. The data after the ‘Connection’ parameter is used to define the data returned from the server response. This can be plain text or an XML string and adjust the Content-Type parameter accordingly.

Within the before block, the declaration ‘EventMachine::MockHttpRequest.pass_through_requests = false’ ensures that the calls are made locally. This can be switched off by passing true. The EventMachine::MockHttpRequest.reset_counts! clears the internal counter of EventMachine::MockHttpRequest. When a request is made, this counter is increased by 1 and its used to track that the actual call did happen. EventMachine::MockHttpRequest.reset_registry! clears the internal hash as mentioned previously.

For an actual implementation please look at the specs in the AmazeSNS gem folder at http://github.com/cheeyeo/Amazon-SNS-Ruby

In a recent article on 37Signals blog, there is a great discussion on how to go about improving execution times of your test suite through Test::Unit.

Since I use Rspec and there are similarities between the way the two frameworks work, I set myself a challenge to see if I could incorporate those techniques into my own Rspec tests suite.

Firstly, to understand the mechanics of it all, you would need to read the original article on the 37signals blog. Essentially it boils down to two issues: memory leaks and how the garbage collector works.

For the first issue on memory leaks, the article recommends clearing out all your instance variables declared within code blocks in your tests but keep the reserved ones used by the framework itself. This is what I come up with as detailed in the gist below:

<script src=”https://gist.github.com/790094.js”></script>

Next, on the second issue of garbage collection, delay the running of the garbage collector after a certain period of time. Again please refer to the original blog post for a more detailed explanation:

<script src=”https://gist.github.com/790118.js”></script>

I have tried the above configuration without any additional tweaks and can confirm that the test suite does appear more responsive in Rspec especially with autotest. I used Spork as an additional DRB server to run the tests.

I hope this helps someone in trying to make their tests run faster in Rspec. Thanks to 37Signals for a great article and tips!

I was playing around with the CarrierWave gem in Rails 3 when an image upload using RMagick gem and ImageMagick caused a segmentation faul and crashed an entire application.

The only way I found which helped to resolve the segmentation fault issue was to reinstall ImageMagick through MacPorts. These are the steps I took to resolve it:

1. Uninstall ImageMagick through macports i.e. sudo port uninstall ImageMagick

2. Next edit the port file and add the ‘—disable —openmp’ to config.args ( sudo port edit ImageMagick )

I added it in after the previous disable option. On the mac this starts up vim so you will need to edit it in insert mode before saving.

3. sudo port install ImageMagick

4. Uninstall and re-install the RMagick gem.

That should fix the segmentation fault issue.  I tried the above in RVM running Rails 3.0.4 and ImageMagick version 6.6.4-10 with RMagick verison 2.13.1.

For more information on the issue check out this stack overflow post: Stackoverflow article link