OIC (Oracle Integration Cloud): SFTP use case

Use case: I recently came over a use case where the customer was searching for an SFTP server solution on cloud with RBAC (Role Based Access Control) where files would be placed by several vendors (in their appropriate directories) and once the files are processed, customer wants to persist them for a very long time.

The typical items that we suggest to our clients are either of the following (in no particular order):

1. SFTP Go — Install this for HA on a few virtual machines (VMs) with a load balancer in front of it.
2. Oracle MFT — An Oracle WebLogic product named Oracle Managed File Transfer Cloud Service is offered as a Marketplace image. Customers may choose the the source and destination, in addition to build orchestrations, or pipelines, for transferring data from a specific source to a target. It interfaces with OCI IAM and has RBAC.
3. The default SFTP server which Windows.
4. Oracle Integration Cloud

In this blog, we will see how OICs’ (Oracle Integration Cloud) SFTP server can be used, what are the limitations and how we can architect the solution in a way to turn the limitation to our advantage to save cost.

Why Oracle Integration Cloud (OIC)?

Will begin by understanding why OIC is required for an SFTP requirement before advancing on to the solution?

Storage: Oracle Integration Cloud (OIC), also offers an integrated SFTP Server with a free storage space of 500 GB per OIC Instance.

How to Access: The same procedure that we follow to access any other SFTP server can be used to access this one using WinSCP or another application.

Security: Access to this can be via Username/password or Public Key Authentication or Multi-Level Authentication.

Access Control: Users will be created in OCI IDCS/IAM and then permissions to the directories on OIC SFTP can be given from OIC console.

Now that OCI and OIC have taken care of all of the above basics, clients can simply use this product without being concerned over maintaining it. As you may have observed, the storage capacity is limited to 500GB. Now, let’s explore into how we can create a solution to overcome this whilst simultaneously saving money.

Solution: There are two parts for this entire solution:

a. Vendors will be placing files on their respective SFTP directories on OIC SFTP server with access only to their directories. The files placed are to be read/processed by their application.

b. The files are to be stored for unknown amount of time, as their internal teams need this for some other purposes.

Below is the detailed explanation of the solution:

1. Point 1 from the above figure is just the flow where Vendors will be accessing OIC’s SFTP server through WinSCP via Public Internet where directories specific to Vendor are created with access restriction.

Directories for Vendors on OIC SFTP Server with permissions created at Directory level

Files placed by Vendors in their respective directories

2. Point 2 is the integration — ReadFiles from Vendor Directories → This integration simply reads the files from Vendor specific directories and pass it to the application (or) insert in a database (based on their requirement). Upon successfully reading the files I am also pushing these files on to another directory “archival” in the same OIC SFTP Server (just to say that this is processed already, need not be processed again and can be removed from OIC SFTP server).

Reading Files from “vendor01” directory on OIC SFTP Server

Processed Files moved to “archival” directory on OIC SFTP server

3. Push File from OIC SFTP Server to OCI Object Storage → This integration will push the files from OIC SFTP “archival” directory (“archival” is just the name I gave to the directory in OIC SFTP Server) to OCI Object Storage Bucket.

OIC Integration to push the files from OIC SFTP Server “archival” directory to OCI Object Storage

Files moved from “archival” directory of OIC SFTP Server to OCI Object Storage bucket of “Archival” Storage tier

4. Push Files from OCI Object Storage Bucket to OIC SFTP Directory → In some cases there might be an ask from customer to reprocess certain file/s. In such cases, to reprocess the files you need to move the file/s from OCI Object Storage bucket to OIC SFTP server, so that Integration from Step2 can process the file. But, customer has to take a decision whether he would need to reprocess the file immediately when the requirement arises or if it is fine to process even after a couple of hours. The reason being, in the last step we have moved the file to OCI Object Storage “Archive” Tier, where to reprocess you need to restore the file first from Archive tier to Standard and then push it to OIC SFTP server.

OIC Integration to restore Archived Objects on OCI Object Storage Bucket

Objects restoring from “Archive” tier to “Standard”

NOTE: Here I chose OCI Object Storage “Archive” tier to save the storage costs.