Java The Hutt

Welcome to installment number 7 of the ruby realities series. Today was a day for nostalgia and made me remember how nice it is to have published and accurate API docs.
Allow me to elaborate. First some code...

class Loan < ActiveRecord::Base
  belongs_to :mailing_address,
             :class_name => "Address",
             :foreign_key => 'mailing_address_id'
  validates_presence_of :mailing_address
 
end
 
class Address < ActiveRecord::Base
 
  def empty?
    (self.line1.nil? || self.line1.empty?) &&
    (self.line2.nil? || self.line2.empty?) &&
    (self.city.nil? || self.city.empty?) &&
    (self.state.nil? || self.state.empty?) &&
    self.zip.nil?
  end
end

What we have here are two active record classes. There's more to them than what's shown here but this is what was needed to reproduce my problem "in the small" as Sam likes to say. Now some more code that demonstrates the problem...

loan = Loan.new
loan.build_mailing_address
loan.valid?
pp loan.errors.inspect

running that code provided me an output of:

@errors={"mailing_address"=>["can't be blank"]}

Huh? but I just created an instance of mailing address. As an experiment I took out the

validates_presence_of :mailing_address

and things were fine. I was able to create my mailing address and have it saved to the database (updating properties on the same related object is another story itself.. one for a later date).

Digging further it turns out that the presence of empty? in Address is causing my validation problem. Renaming it to is_emtpy?, putting my

validates_presence_of :mailing_address

back into the relation fixes the original code above. Obviously I'm using a method name that is already in use but isn't documented anywhere in the rails API. It's not part of ActiveRecord::Base or Object. Nowhere in the :belongs_to API doc does it talk about this method being overridden or used by a proxy, unless I'm missing the warning somewhere. This could very well be user error or a bug and I'm sure to find out soon as I've filed a bug with the RoR team. http://dev.rubyonrails.org/ticket/3646

Sometimes standards and good docs really are your best friends.

For anyone running a production site on rails today's reality tip is for you. I was in the process of cleaning up disk space and backing up production log files. I haven't setup a script to perform this task for me as of yet so every so often I'll copy off the production.log file to a historical log directory. Normally what I do is copy the file, pray nobody's hit the site, run "rake clear_logs" which truncates the log file. Well it's been awhile since I've done this and last night I forgot this procedure and just did a straight "mv production.log ../backup_dir".

The result of this folly is that today when I went to check the production.log file I saw that it wasn't there. Questioning the likelihood that the site's had zero hits since yesterday I hit the site and took a look again. Still no file. I figured that rails must not have liked my move so I needed it to realize that the file is gone and make a new one for me. I tried SIGHUP on my fastcgi processes but that didn't work either. I ended up having to kill my app and restart it to get my app logging to the log file again.

So in conclusion don't be stupid like me, always run "rake clear_logs" or better yet setup something to clear them for you daily and back them up. That's on my list of things to automate.

I've been working with paypal's developer sandbox and payment mechanisms as of late and while it's not been completely horrible I really wonder how someone else hasn't come along to eat paypal's lunch based on the site usability and clarity of documentation.

First it's not entirely clear when you visit paypal.com what exactly you need to do to get going. You end up clicking through various links not entirely certain that what you're looking for is contained within one area. Over time I've been able to get all the information I've needed via 3 means:

1) The paypal site (pages and PDF integration manuals)
2) The paypal hacks book (not sure this book is worth the price but it was a little helpful)
3) The developer forums (I didn't post or spend much time here but some google searched provided answers in this forum)

Now this could be just me but I didn't find what I thought would be the most common case which is "I have a site and I know how to code, show me all the steps, cut to the chase and give me a good example site to follow.". Paypal doesn't do this but the paypal hacks book did go into some of this a little bit though could've been better as it is just a collection of small hacks.

So at any rate, for anyone else out there that wishes to integrate paypal into their site I'm going to put down some of the "skinny" here that you need to understand in order to know how to proceed.

First off Paypal has a couple different mechanisms that you can utilize for processing payments:

1) A web services API
2) URLs that you call and that trigger callbacks to your site. These are known as IPN and PDT. IPN stands for "Instant payment notification" which is sort of ironic because it's asynchronous and is advertised to be mostly instant but can be delayed by up to 4 days. PDT, which stands for "Payment data transfer", is synchronous and will return transaction information if you provide paypal with a return URL after the transaction is complete. I'll get into more detail about the differences here in a minute.

I chose to go with #2 for what I interpreted to be the most expedient way to get this work done partially because ruby's support here seems to be kind of light and not commonly used yet so I figured HTTP POST and GET might be safer for now.
OK so #2 is my choice but do I use IPN or PDT? Let's go over how each one works roughly...

IPN

1. The user clicks on your payment button which posts a form to paypal's site containing information about the transaction (product description, item number, your business email, price, etc).
   
2. The user makes their payment at paypal's site. (The user isn't returned to your site unless you specify a URL to return to)
   
3. Paypal then calls back to a URL you provide to them, i.e. http://mysite.com/paypal_ipn_callback In this callback paypal provides all the information about the transaction so that you can verify things are correct and proceed to the next step(s) for your customer. This URL is intended to be a "behind the scenes" conversation between yourself and paypal.
   
4. When your site receives this callback you acknowledge the call with paypal by making a call back to verify the contents of the transaction.
   
5. Paypal either returns a VERIFIED or FAILED result to your call.
   
6. Proceed based on the results of the transaction (ship your product, contact the customer if there's a problem, etc)
   

Like I said earlier this callback from paypal to your site is asynchronous so if you'd like to make products available to users for immediate download this can present a problem especially if the delay is several days. For cases where you'd like to have instant downloads it seems like PDT is the better choice. I was hesitant at first about PDT because it seems like there were several warnings about IPN being the recommended and more reliable mechanism. It's possible to use a combination of both but first let's go over how PDT works..

PDT

1. The user clicks on your payment button which posts a form to paypal's site containing information about the transaction (product description, item number, your business email, price, etc).
   
2. The user makes their payment at paypal's site.
   
3. The user is returned to a URL you specify with the details of the transaction embedded into the URL. i.e. http://mysite.com/thank_you&txn_id=sjkjfa234j&blah=foo (This return URL is either given to paypal in your buy now button or in your account settings)
   
4. Your site takes the transaction ID, a PDT identifier that is given to you by paypal when you enabled PDT in your settings, and makes a call back to paypal to verify the transaction is valid.
   
5. Paypal returns immediately with a status message and, assuming the transaction is valid, the details of the transaction.
   
6. You then proceed to process the results of the transaction and deliver your product.
   

So as you can see the mechanisms are very similar with the major difference being the response time guarantee. PDT is synchronous but you are warned that it may not be available and that the use of IPN for a backup and verification mechanism is recommended.

So now that we've got an idea of how the mechanisms work how do we actually use this stuff? The first thing to do is to signup for a developer account and create a sandbox. This painful mechanism is just a mock of the real paypal site and what your account would be like when you're ready to go live. The main reason that it's painful is that the session times out what seems to be every 5 minutes. So while you're working on your code you end up having to log back in and relaunch your sandbox.

OK, you've got your developer account, created a sandbox, launched it, and are ready to start testing transactions. What do I need to do next? Well depending on if you're using IPN, PDT, or both you need to do some configuration. This configuration
resides under the "My account" tab in the "profile" subtab. Here you'll find the settings for IPN and PDT under "Instant Payment Notification Preferences" and "Website Payment Preferences" respectively.

After you've set things up you can create your code to call paypal, receive callbacks, and process your transactions. There's certainly more details in there, for example you're going to want to create another account in your sandbox that you can use for making purchases and have paypal generate a fake CC number to store in that account. This was just meant to get to the salient points of how the paypal mechanisms work and what's required of someone writing code. I can certainly go into more detail and even post my ruby code for handling IPN and PDT as the existing library out there for rails focuses solely on IPN and is basically a few convenience methods for acknowledging an IPN callback from paypal.

Hope this is helpful to someone out there embarking upon paypal integration.

I've been debating the merits of purchasing a new powerbook over the last several months and have come close a couple of times to hitting that "add to cart" button after "configuring the powerbook of my dreams", but I just had a hard time parting with so much money for a laptop that I knew was not going to be as fast as a machine 1/2 its price. So I waited, hoping that the rumor of Intel powerbooks in January would be a reality.

Well this morning I woke up like a kid on xmas morning and ran down to my computer to figure out how I can get up to the microsecond updates about what Steve Jobs was announcing at MacWorld. Aimee teased me for being a nerd as I sat in an IRC chatroom dedicated to the live rebroadcast of Steve Jobs' keynote. For a bit I wasn't certain that it was going to be announced. First it was iLife and apps, then the new iMac. Finally, saving the best till the end in the "one more thing" part of the keynote my xmas wish came true. A new powerbook with an Intel chip. Well, almost. A macbook but I like the name powerbook better :-). I didn't order right away as I plotted and planned as to the best way to go about my purchase. Turns out signing up for the Apple Developer Connection and making my purchase through there is cheaper than just buying it outright. Let's go through the math:

$2,499 for the high end MacBook on apple.com

The same MacBook on the ADC store is $1,999 + $500 to join ADC

So right there it's the same exact price but you also get access to early access releases of OSX, JDK builds, etc. But wait! There's more!

That's right, if you act right now you can pickup an iPod and Cinema display as well as upgrades on your MacBook for less than what you'd pay at the regular apple.com store. So while I didn't slurge for a cinema display (The 24" Dell appears to be a better monitor), I did pickup an iPod video, bumped the RAM to 2GB, made the HD a 100GB 7200 RPM, all for less than it costs on the regular store. Add to that you now pay lower sales tax on the lower prices and you've got a pretty compelling reason to join the ADC and buy your new MacBook that way. Out the door the MacBook, RAM upgrade, HD upgrade, and 30GB iPod, and tax came to $2,815.98 + $500 (ADC) = $3,315.98 which compares to $3,477.83 for the same items purchased without purchasing the ADC subscription first. So there you have it, you can get some more crap and upgrades and save a few bucks on it.

Ironically enough Safari was hanging on me while I was in the middle of placing my order on my current 800MHZ Ti PB so I swung my chair over to the windows box at my desk and completed my order from there. I swear this Ti knew what I was doing and was fighting for its life. Well for now it's still my daily machine until sometime in February when my new machine arrives. Can't wait.

I came across my first need for using model inheritance yesterday and so I dug a little deeper into active record and the only support currently is for single table inheritance. That means that all classes in a hierarchy have their attributes stored in the same table. For example:

class Dog < Animal
end

class Animal < ActiveRecord::Base
end

The table definition for these two classes using rails migration mechanism, is:

    create_table :Animals do |t|
            t.column :type,     :string
            t.column :fetches,    :boolean
            t.column :runs,    :boolean
    end  

The type column is used by rails to determine what object type to instantiate with the data in a given row. Fetches is intended to be a property on Dog while runs is intended to be a property on Animal. Turns out in reality that both of these properties will be present on both of these types because there's no way to specify any sort of encapsulation of these properties.

I actually gave it a shot by declaring the specific attr_accessors on each class, like this:

class Dog < Animal
  attr_accessor :fetch
end

That turned out to be a mistake as declaring those attributes yielded an instance variable named @fetch and my property was empty when I did a dog.inspect

#< Dog:0x22b3114 @new_record=true, @errors=# < ActiveRecord::Errors:0x22b2fac @errors={}, @base=# < Dog:
0x22b3114 ...>>, @attributes={"fetch"=>nil, "runs"=>nil}, @fetch="true">

So be careful out there if you use STI. Turns out you can be very OO when using it. Hopefully I'm missing something here but it appears that there needs to be a way to declare what fields go in what class in the hierarchy.

A short one here. I was working on a new action and view when during my POST I received the following exception:

NoMethodError (You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occured while evaluating nil.[]):

The source of this error was this line:

      if params[:commit] == 'Cancel'

Which confused me because params is the hash that's created by default containing all of the form elements I'm posting. A quick look over my code and I saw that I had created a local variable named 'params' in a branch of logic that doesn't even get executed on a POST. So my code looks similar to this:

def my_action

  if request.get?
    params = Hash.new
    # do stuff
  else
    if params[:commit] == 'Cancel' # <-- NoMethodError on a nil object here
      #do stuff
    end
  end
end

Renaming my local variable to something else fixed the problem so this means that despite never being executed the local variable declared 'params' is overriding the default created by rails. I'm still a newbie to ruby and rails but this doesn't seem correct to me as the 'else' clause above never executes the initialization of the new Hash. Perhaps some ruby expert out there can tell me why this is the case, or have I found a bug in rails?

I didn't expect my next post to be so quick but this cropped up tonight while I was working on my app.
This is just a quick post for anyone else that may need to use a date_select more than once on a page. Here's some code:

< %=date_select(:borrower_profile, :date_of_birth,   :order => [:month, :day, :year], :start_year => Time.new.years_ago(7).to_date.year, 
    :end_year => Time.new.years_ago(104).to_date.year, :include_blank => true, :index => index)% >

In the above snippet creates a set of select widgets to obtain a birthdate with rails' date_select helper method. The problem is that I'm using this helper to obtain birthdates for many users which means I need some sort of index value to track which birthday is attached to which user.

So basically I need HTML like the following to appear in my page:

< select name="borrower_profile[2][date_of_birth(2i)]" > <-- month
< select name="borrower_profile[2][date_of_birth(3i)]" > <-- day
< select name="borrower_profile[2][date_of_birth(1i)]" > <-- year

The :index => index part of my tag in the first code snippet is what is supposed to provide the index value in my select name value "borrower_profile[2]" (2 is the index value).

As it is rails 1.0 and earlier will always ignore :index and generate: "borrower_profile[date_of_birth(2i)]", etc

As it turns out the date_helper method doesn't recognize the :index parameter which many, if not all, other form helper methods do recognize (text_field, select, radio_button, etc). This is most likely just an oversight. At any rate I can't wait for this to be fixed as I need this right now so I dug into rails and fixed it myself.

With some help on IRC I figured out the easiest way to create a patch for rails and have it as part of my app until I can submit it or file a bug to have it fixed in a future release. Anyway, the fix is as follows (This assumes you want to apply the patch from your app and not your rails installation. If not then you can just make the changes outlined below in your rails installation):

1) Paste the following code into a file called date_helper.rb and place it into your application's lib directory.

require "date"

module ActionView
  module Helpers
    class InstanceTag #:nodoc:
      include DateHelper
      
      alias :__to_date_select_tag :to_date_select_tag
      
      def to_date_select_tag(options = {})
        defaults = { :discard_type => true }
        options  = defaults.merge(options)

        # begin patch
        if options[:index]
          @index = options[:index]
          options_with_prefix = Proc.new { |position| options.merge(:prefix => "#{@object_name}[#{@index}][#{@method_name}(#{position}i)]") }
        else
          options_with_prefix = Proc.new { |position| options.merge(:prefix => "#{@object_name}[#{@method_name}(#{position}i)]") }
        end
        # end patch

        date     = options[:include_blank] ? (value || 0) : (value || Date.today)

        date_select = ''
        options[:order]   = [:month, :year, :day] if options[:month_before_year] # For backwards compatibility
        options[:order] ||= [:year, :month, :day]

        position = {:year => 1, :month => 2, :day => 3}

        discard = {}
        discard[:year]  = true if options[:discard_year]
        discard[:month] = true if options[:discard_month]
        discard[:day]   = true if options[:discard_day] or options[:discard_month]

        options[:order].each do |param|
          date_select << self.send("select_#{param}", date, options_with_prefix.call(position[param])) unless discard[param]
        end

        date_select
      end
    end
  end
end

2) Place a require 'date_helper' in your environment.rb file
3) restart your webserver
4) use the date_select helper function with the :index option and you'll see your field indexed properly. yay!

Enjoy!

In my last post I covered how newbies could be tripped up with symbols and their usage within various hashes. Today I'm going to talk about issues related to tracking down errors when things go wrong. An advantage of rails that is touted by many is the time savings of "convention over configuration" and refers to XML configuration files in J2EE with the tagline "XML situps". Now I completely buy the value of convention over explicit configuration with the flexibility to override, it is certainly a time saver. Although I can say with 100% certainty that I've never spent a full day writing an XML configuration file of any sort, the same cannot be said with respect to debugging a runtime problem in rails. I have a few examples of battle scars in my short 3 month travels with RoR. Let's go through each one:

1) I had a action method in my controller that I'd made private by accident while refactoring. (For all you javaheads out there the way ruby works is that you define access modifiers on their own line and they act as sort of a toggle switch, meaning that when you declare private, all methods below are private until another access modifier is defined, i.e. public. For those of you that are wincing, I agree and don't like the convention myself but have now gotten used to it) At any rate, when I ran my functional test for the controller it was failing on my assertion for a redirect. The action that I'd accidentally made private performed a redirect. The test harness was informing me that an HTTP 200 (success) was being returned which puzzled me and led to me to an hour of checking my parameters, session variables, as well as the underlying model components that were part of the action. Of course in the end it turned out that I'd accidentally forgotten to put a public identifier back into my code while refactoring some private methods. Unfortunately the test harness gave me a bad error which wasted an hour to and hour and a half of my time. It should've given me a 404 error as it shouldn't have been able to find the action.

2) This one is related to my lack of ruby knowledge at the time but the error is that I had and 'include' and a 'require' inside application_helper. I was including a module as well as requiring the module, which is apparently a no-no. This code worked in .13.1. and didn't work in .14.x. The error message I received was nothing even remotely related to the actual issue.

3) In my app configuration file included in the general rails 'environment.rb' file I had a path specified that did not exist. Rails provided no error message on startup and I had to find it by adding my app code to a new app until it broke. How's that for some debugging situps?

4) My rails application wouldn't startup in production mode. The error messages were nonexistent in the rails log files and in the apache env. and only showed up in lighttpd as:

2005-11-27 10:51:04: (mod_fastcgi.c.2196) unexpected end-of-file (perhaps the fastcgi process died): pid: 8002 fcgi-fd: 9 remote-fd: 8 
2005-11-27 10:51:04: (mod_fastcgi.c.2958) child exited, pid: 8002 status: 1 
2005-11-27 10:51:04: (mod_fastcgi.c.3005) response not sent, request sent: 812 connection-fd: 8 fcgi-fd: 9 
2005-11-27 11:16:22: (mod_fastcgi.c.1532) connect failed: 9 Connection refused 61 0 /tmp/ruby-fastcgi.BB1C1FB6-5F76-11DA-83BB-000393B4C164-239-0000058F6C5C1CC0.socket-1

Turns out I had a syntax error in production.rb, an errant keystroke. Looking back now it's obvious to me to look there when it was just the production environment that was failing, but at the time you don't know the lay of the land all that well and where to look when trouble strikes. Logs are normally where you go but again I was failed by rails' lack of verbosity.

5) This time I was using rails' handy migration feature to make changes to my DB schema and migrate my data. Only it was failing with only the following error to guide me:

michael-kovacs-powerbook-g4:~/loanback_src/trunk mkovacs$ rake migrate
(in /Users/mkovacs/loanback_src/trunk)
rake aborted!
cannot convert nil into String

Well as it turns out I had commented out the RAILSENV value in 'enviroment.rb' so that it would grab the value from the shell environment variable and this was the error that rails gave when it couldn't find a value for RAILSENV. No clear error message, not even a line number or a stack trace,

Again, I think that there's some really great things in rails, specifically the ease of testing, the thought of end to end application development and the things that are needed to build, deploy, and advance your application, but there is certainly room for improvement on a few fronts and error messages from rails when something goes wrong is #1 in my book.

To that end I ended up modifying my installation of rails to be a little more verbose on startup. I haven't have time to go back and do a clean implementation that has flags for runtime options but the relevant code to look at is the rails Initializer class, and specifically the process method where the list of things to do on startup resides. I also added debugging information to where the DB connection gets obtained...

    def process
      set_load_path
      STDERR.print "setting connection adapters..."
      set_connection_adapters

      require_frameworks
      load_environment

      initialize_database
      initialize_logger
      initialize_framework_logging
      initialize_framework_views
      initialize_routing
      initialize_dependency_mechanism
      initialize_breakpoints
      initialize_whiny_nils
      
      initialize_framework_settings
      
      # Support for legacy configuration style where the environment
      # could overwrite anything set from the defaults/global through
      # the individual base class configurations.
      load_environment
      
      load_framework_info

      load_plugins
    end

    def initialize_database
      return unless configuration.frameworks.include?(:active_record)
      STDERR.print "\n\ninitializing database... " + configuration.database_configuration.inspect.to_s
      ActiveRecord::Base.configurations = configuration.database_configuration
      ActiveRecord::Base.establish_connection
    end

You could add debug messages to wherever you'd like to monitor the startup execution of whatever you're interested in.
This is all very quick and dirty stuff but it's saved my skin a few times since some of the earlier problems and I figure other newbies like myself would be intimidated to try and figure out where rails is doing all of its magic. Well the startup time magic appears to all happen in

 /usr/lib/ruby/gems/1.8/gems/rails-X.XX.X/lib/initializer.rb 

which is called from boot.rb in the config directory of your app. At some point I'm sure someone will add something that does what I did only in a clean manner instead of a crude hack or if not I may get around to writing it myself and submitting it.