Amazon S3, Cloud, Integrations

Microsoft Flow SFTP Connector and Couchdrop for Amazon S3 Bucket Storage Connections

This Blog post was written and published by a third party, please see below for full information about the author.

At the time of writing this post, there is not a native Amazon S3 connector for Microsoft Flow. In the course of creating a Flow to automate static website updates, I found a fantastic service called Couchdrop, which when used in conjunction with the Microsoft Flow SFTP connector, allows Flow to access and manipulate files and folders in AWS S3 buckets. If you have seen some of my other posts, I am sure you know that I am a big fan of SFTP with Flow.

In this case, Couchdrop serves as middleware that translates SFTP commands from Flow into S3 API commands. The limits of the “Freebie” version of Couchdrop are as follows, but you can still accomplish quite a bit (depending on the nature of the files you are moving):

  • 10 transactions per month – a “transaction” is synonymous to a connection or session (if you move multiple files in one session, it counts as only a single transaction)
  • 1 user and 1 storage endpoint

If you purchase the service, the transaction limit is removed and the user/storage endpoint limits are relaxed. In my case, the free tier ended up being perfect (my Flow only checks and updates an S3-hosted static site once per week, so 4 transactions per month for checking and 6 left over for manipulating files). I am very impressed with the functionality of the free Couchdrop offering (kudos to the team for making this available and to their awesome support team for promptly answering my question about “transactions”).

In this post, I will describe the configuration required to allow Flow to interface with S3 buckets, including the Amazon (AWS) S3 setup, Couchdrop setup, and Microsoft Flow SFTP connector setup.

Amazon S3 (and IAM) Setup

The Amazon AWS setup is relatively simple. I recommend creating 1 API user (IAM) per S3 bucket to keep things compartmentalized and secure. If you have more than 1 bucket, you will end up needing a Couchdrop plan besides the free one (1 endpoint limit).

Assuming you already have a bucket setup, let’s look at setting up the IAM user. This user’s credentials will be added in the storage endpoint settings in Couchdrop.

1) Browse to the IAM console in AWS

2) Select Users -> Add User

3) Enter a username, select Programmatic access, then Next

4) Select the “Attach existing policies directly” option, then click the Create Policy button

5) In the new window, select the JSON button

6) I used one of Amazon’s examples. The name of the bucket this policy grants read/write access to is http://www.mywebsite.com:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::www.mywebsite.com"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::www.mywebsite.com/*"
        }
    ]
}

7) Click Review policy, name the policy, then click Create policy

8) Once the policy is created, return to your original IAM window, click the refresh button above the policy list, and use search to find the policy you just created – tick the box next to the policy and click Next: Tags

9) Add tags as appropriate, click Next: Review and then Next: Finish

10) Lastly, capture the Access key ID and Secret access key – you will use these in the Couchdrop setup

Couchdrop Setup

The Couchdrop setup is even easier than the AWS setup. We will configure Couchdrop to connect to our S3 bucket (http://www.mywebsite.com) using the IAM user we created above. Then, we will create a user (key) in Couchdrop that Microsoft Flow will use to connect with the service. Technically, you could use your default Couchdrop key/account but for security, I recommend creating a second account.

1) Navigate to your Couchdrop storage area and click Add Bucket if needed

2) Add your AWS S3 bucket as follows, using the Access key IDSecret access key, and bucket name – once done, click Save Settings

Couchdrop storage settings for Amazon S3 bucket connectivity

3) Next, add an SFTP user to the Couchdrop account – navigate to the Couchdrop Keys area

4) Click Add Account and once the account appears in the list, click “Edit”

5) Change the password (I like to randomly generate one that is longer than the default), select Read/Write for Permissions, and click Save

Couchdrop key or account settings for the SFTP interface

6) Take note of the username and password you set in Couchdrop – these are the credentials we will use in the Microsoft Flow SFTP connector

Microsoft Flow SFTP Connector Setup

The final step is setting up an SFTP Connector in Flow that points to your Couchdrop setup. At this point, the Flow setup is just like any other SFTP connector setup (thanks to Couchdrop). Once done, Flow “speaks” SFTP to Couchdrop and Couchdrop translates that into S3 API commands for Amazon storage buckets.

1) Add an SFTP Action to your flow (using the Flow Editor) and create a new SFTP Connector

2) Configure the connector as follows (add the username/password from Couchdrop) and specify sftp.couchdrop.io as the Host server address

Microsoft Flow SFTP connector settings to access Couchdrop

3) Add other logic to your flow that leverages the SFTP connection; you can use actions like “List file in root folder”, “List files in folder”, “Copy file”, “Delete file”, etc. Note, Flow does not reuse SFTP connections across actions, so each counts against your monthly Couchdrop transaction limit.

Closing thoughts

Depending on your needs, Couchdrop is a phenomenal way to interface Microsoft Flow with Amazon S3. If you can get by with the “Freebie” tier, the value is even more appealing. Even if you opt for the paid version ($4 per month at the time of writing), your monthly cost will be lower than if you spin up an Amazon EC2 instance and use something like s3fs-fuse to bridge this gap. The cheapest ec2 instances are ~$4-5/month and this does not include the setup/configuration (i.e. sftp and s3fs-fuse) and ongoing maintenance required to remain secure and operational. I would personally choose Couchdrop every time.

It is worth noting that Microsoft Azure offers blob storage and CDN service (similarly to Amazon Web Services), and there IS a Flow connector for Azure blob storage. If you are not bound to Amazon, hosting your static site with Azure and interfacing with it using the native Flow connector is an option as well.

About the Author

The author of this post, Neil Sabol, originally posted this blog at blog.neilsabol.site on 2019-01-26. 

Neil is an avid fan of IT and is an huge proponent of open source solutions in the enterprise and enjoys providing outstanding customer support for various applications and platforms. Neil’s favorite technologies (and interests) include: Amazon Web Services, Apache (webserver), cPanel/WHM, F5 (LTM and APM), Hugo, Jamf, Linux, SCCM, Windows, and WordPress. Some of my ancillary technology interests include 2 factor authentication, Microsoft Office 365, and scripting (Bash, JavaScript, PHP, PowerShell).

Please head to blog.neilsabol.site for more posts similar to the above and to learn more about Neil.

Original link to the above content, https://blog.neilsabol.site/post/microsoft-ms-flow-sftp-connector-couchdrop-amazon-s3-bucket-storage/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s