Need to move S3 Buckets or Accounts?

So when crusher first started last year our engineer who was experimenting with S3 decided to open up his own account and use his credit card. Then crush3r.com went live using S3 and his account. For the past 7 months he has been paying our S3 bill, which has been no more than a couple dollars. Then he started doing other projects using his S3 account, so we couldn’t take his account for crusher use. So we had this task of taking over our crusher bucket. Which is what we did last week. I found that there were no good tools for downloading entire accounts, or uploading, etc .. so I wrote some code using the aws-s3 gem.  It’s pretty rough code, because I didn’t spend more than an hour working on it, and I just used script/console to run and didn’t write the code for any other use other than this one-time port.
Our key structure is formatted as such “xxxx/xxxx/file_name.blah”, so every key is guaranteed to have 2 directories.

I substituted all the crush3r specific information with CAPS.  Go, downloads everything to a local machine, and then put uploads it all.  In case it can help you with a starting point, here is the code :

  @@root = "LOCAL_FILE_DIR"
  @@new_bucket = "NEW_BUCKET_NAME"
  def self.go
    require 'aws/s3'
    AWS::S3::Base.establish_connection!(
      :access_key_id => "OLD_ACCESS_KEY_ID",
      :secret_access_key => "OLD_SECRET_ACCESS_KEY"
    )
    crusher_bucket = AWS::S3::Bucket.find("OLD_BUCKET_NAME")
    objects = crusher_bucket.objects
    begin
      objects.each do |obj|
        begin
          dirs = obj.key.split("/")
          root = @@root
          (0..(dirs.size-2)).each do |pos|
            FileUtils.mkdir root + dirs[pos] unless File.directory?(root + dirs[pos])
            root = root + "/" + dirs[pos] + "/"
          end
          file_name = "#{@@root}#{obj.key}"
          unless File.exists?(file_name)
            puts "WRITING : #{file_name}"
            open(file_name, 'w') do |file|
              obj.value do |chunk|
                file.write chunk
              end
            end
          else
            # puts "SKIPPING : #{file_name}"
          end
        rescue
          puts "#{obj.key} is messed up"
        end
      end
      objects = crusher_bucket.objects(:marker => objects.last.key)
    end while objects.size > 0
  end

  def self.put
    require 'aws/s3'
    AWS::S3::Base.establish_connection!(
      :access_key_id => "NEW_ACCESS_KEY_ID",
      :secret_access_key => "NEW_SECRET_ACCESS_KEY"
    )
    dirs = ["ARRAY_LIST_OF_LOCAL_DIRS"]
    dirs.each do |dir_name|
      names = Dir.entries(dir_name)
      names.delete_at(0) if names[0] == "."
      names.delete_at(0) if names[0] == ".."
      names.delete_at(0) if names[0] == ".DS_Store"
      left_path = dir_name.gsub(/LOCAL_FILE_DIR/, "")
      names.each do |name|
        begin
          key_name = left_path + name
          unless AWS::S3::S3Object.exists?(key_name, @@new_bucket) then
            AWS::S3::S3Object.store(key_name, open(dir_name + name), @@new_bucket, :access => :public_read)
            puts "WRITING : #{key_name}"
          else
            # puts "SKIPPING : #{key_name}"
          end
        rescue
          puts "MESSED UP : #{key_name}"
        end
      end
    end
  end