Ruby S3 V3 Read Object to Memory

1 December 2020

Demystifying AWS S3 uploads in Ruby on Rails

Amazon Web Services include S3 (Simple Storage System) which Amazon describes every bit "an object storage service" that provides "scalability, availability, security and performance". While there are many types of objects that can be stored on S3, in this article we will focus on storing a JSON file that volition exist shared between ii physically separated systems. On one side is a consumer-facing website taking orders from customers. On the other side is a 3rd political party fulfillment warehouse which is shipping those orders. An API might be a preferred interface betwixt these companies' systems, yet for this case, the fulfillment visitor does non offer that, and then we are uploading JSON files to their AWS S3 account.

Accessing Amazon S3 programmatically is becoming more commonplace, and if you accept not already washed this, the documentation can seem overwhelming to digest. In this post, I promise to provide some insight from my feel to aid you apace get upwardly to speed on the AWS S3 skill set when using Ruby on Rails.

Before jumping into the code, though, you should take a look at the documentation for S3. A high-level overview can be found here: Amazon S3 FAQs. This is not a conventional starting bespeak, only this FAQ is well organized, and scanning through it will cut the fourth dimension yous need to understand enough S3 to get-go working with it. We are non roofing how to sign upwards for AWS, how to select the form of S3, and prepare a saucepan. You will need to do these earlier starting to write whatsoever code. I propose that you lot follow Amazon'south S3 setup documentation that will lead yous through the setup process.

For this mail, you just need to come up abroad from this introductory research with the understanding that Amazon S3 is a system of buckets that contain objects. In more familiar terms, this translates to buckets being like folders, and objects existence like files. Furthermore, the buckets have globally unique names that compose a URL to allow access to the objects via HTTP, only like an API endpoint. Uploading the objects volition be washed by us programmatically using an account that is fix on AWS for such admission. Amazon describes that hither. When you lot create this business relationship, exist sure to capture the access central and surreptitious cardinal.

I created the instance code using ruby version ii.7.i and rails version 6.0.3. You will want to use something similar to follow along. Amazon has provided a gem for usa to access S3. Add together the gem to your Gemfile as shown in the following line.

              gem 'aws-sdk-s3', '~> 1.78'                          

As previously mentioned, our example app volition be an order treatment procedure. Nosotros do not want our customers waiting for the warehouse to get the order, and then we are going to do this processing asynchronously using something like SideKiq. Such a chore is shown below. It is fairly simple lawmaking logic: get the order information, build a document object containing the order information, and send that to AWS S3. Nosotros are focusing on the S3 interaction in these examples, so I am non roofing Sidekiq or any functionality that might have us off this subject.

              class SendOrderJob < ApplicationJob   queue_as :send_orders    def perform(order_id)     order = fetch_order(order_id)     certificate = build_document(society)     send_to_s3(certificate, order_id)   end    private    def send_to_s3(document, order_id)     ShipmentOrder::SendShipmentOrderDocument       .new(         document: document,         file_name: file_name(order_id),         aws_client: aws_client       ).send_document   stop    def aws_client     @aws_client ||= ShipmentOrder::AwsClient.new()   end    def filename(order_id)     #create a filename using the order id   end    def fetch_order(order_id)     # fetch the order data   end    def build_document(order)     # format the order information   end end                          

In the above code nosotros are getting the aws_client information needed to talk to S3 from a course whose code is shown below. All of the interactions with S3 will exist washed via this ShipmentOrder::AwsClient class.

              module ShipmentOrder   class AwsClient     def initialize       config_data = get_config_data #from a secrets file     end      def s3_client   	 @s3_client ||= Aws::S3::Client.new(     	   region: 'united states-east-1',     	   credentials: credentials   	 )     end      def s3_bucket   	 resources = Aws::S3::Resource.new(customer: s3_client)   	 resources.bucket('my_bucket_name')     finish      private      def credentials   	 Aws::Credentials.new(     	   config_data.access_key,     	   config_data.secret_key   	 )     end   finish end                          

The key have abroad from the higher up lawmaking is that we are creating an Aws::S3::Client object using our access key and cloak-and-dagger cardinal that are in an Aws::Credentials object. These keys need to match the keys you lot generated when you created the account on AWS that will access the AWS S3 bucket. We volition use the Client object to connect to AWS. Finally, we have the bucket as an AWS::S3::Resource object that we will employ to put our shipment society documents into.

Now we have all the pieces to push a file to AWS S3. We will do that in the code beneath which we instantiated and made a telephone call to in the SendOrderJob shown earlier.

The lawmaking below is fairly straightforward. Nosotros utilise the aws_client to send the order certificate to S3 referencing the bucket resources that information technology is to be placed in. Once the put_object call has completed we will get back a response object from AWS which contains an etag element if we are successful. Anything else is considered an error and our code should alert someone who can take activity.

              module ShipmentOrder   class SendShipmentOrderDocument     attr_reader :aws_client, :document, :file_name      def initialize(document:, file_name:, aws_client:)       @document = certificate.to_s       @file_name = file_name       @aws_client = aws_client     stop      def send_document       response = aws_client.s3_client.put_object(         bucket: aws_client.s3_bucket,         key: file_name,         body: document,         content_type: 'text/json'       )       if response.etag         return true       else         return false       terminate     end   end end                          

To test this, nosotros need to write a test like the 1 in the following lawmaking. In this example, we are setting up some false information for our filename and document. Then we ready a mock on our s3_client so that an case double is returned with the beliefs nosotros want. And so in the exam we set the behavior of the put_object method to always return an etag value of 'ok'. While that does non truly test the functionality of AWS S3 receiving a certificate, that's OK, as nosotros actually desire to check that the lawmaking on our side is always working. Y'all should also have separate tests effectually creating the filename and the certificate, as well.

              require 'rails_helper'  RSpec.draw ShipmentOrder::SendShipmentOrderDocument, type: :service do   let(:filename) { 'file_order_id.json' }   let(:document) { 'my_document' }    allow(:sender) do     ShipmentOrder::SendShipmentOrderDocument.new(       document: document,       file_name: filename,       aws_client: aws_client     )   finish    permit(:aws_client) { ShipmentOrder::AwsClient.new }   allow(:s3_client) { instance_double(Aws::S3::Client) }    before practice     let(aws_client).to receive(:s3_client).and_return(s3_client)   end    describe '#send_document' practice     it 'calls AWS to put a document in the S3 Saucepan' do       permit(s3_client).to receive(:put_object).with(         saucepan: aws_client.s3_bucket,         key: filename,         content_type: 'text/json',         trunk: certificate       ).and_return({ etag: 'ok' })       response = sender.send_document        expect(response[:etag]).to eq('ok')     terminate   stop end                          

In this commodity, nosotros have created code to upload a file to AWS S3 and take written tests to embrace that code. While in that location is a lot to learn about working with S3, hopefully, you lot are at present up to speed on uploading files. From here, you lot should be able to figure out reading files, via get_object and deleting files with delete_object. I hope this article will encourage yous to comprise AWS S3 in your Ruby on Rail app.

Want to learn most the types of products we build?

Check out our projects

troyerhoublive1955.blogspot.com

Source: https://teamgaslight.com/blog/demystifying-AWS-S3-uploads-ruby-on-rails

Share this post