OCI Logs + SIEM: Download Latest Logs (Python SDK)
Every customer especially those in BFSI sector looks for Monitoring the Audit logs for which every cloud has their own tool. However, customers who are using other clouds or using third party SIEM (Security information and event management) tools from the On-Prem world to monitor logs would need integration from respective cloud to their SIEM.
In this blog we will see what are the different ways in which third party SIEM providers can access OCI Audit logs. But our stress would be more around using custom Python SDK to fetch these logs.
Option 1: OCI Streaming
Most of the customers follow this process of pushing logs to OCI Streaming Service (OSS) which is backed by Kafka. SIEM providers then consume logs from OCI Streams using default Kafka APIs. Please find this blog where a colleague of mine has explained on how to integrate OCI Logs into IBM QRadar.
Option 2: OCI Object Storage
Using OCI Object Storage as the log aggregator on cloud and then third party SIEM providers read the latest logs from OCI Object Storage bucket.
There are two steps to achieve this:
- Push logs to OCI Object Storage: You can follow this blog on how to push all the logs to OCI Object Storage.
- Third Party SIEM providers fetch the latest logs from OCI Object Storage Bucket: This can be achieved via OCI REST APIs or via OCI SDKs. In this blog I will be discussing on the custom Python SDK which I have built to pull the latest logs from the OCI Object Storage using Python SDK.
Python SDK
To achieve this use case of fetching only the logs uploaded to OCI Object Storage bucket after the last retrieval I have built a custom Python SDK. Calling it custom because I made few changes to the Python SDKs provided by Oracle.
Custom Script1 (Download Files & Archive them): This custom script will list objects in the OCI Object Storage bucket whose storage tier is “Standard”, download them to a specified location and then update storage tier of the objects to “Archive”.
NOTE: I am updating the storage tier of the object to “Archive”, so that it will be easy for me to identify which are the objects which I have already consumed and also to help me with cost optimization.
You can find the Python code for the same in my GitHub over here.
Custom Script2 (Download Files based on time interval E.g.: last 10mins): This custom script will list objects in the OCI Object Storage bucket whose storage tier is “Standard” that are uploaded in the last 10 minutes only and lets you download them to a specified location.
NOTE: You can set the time duration based on your requirement, for the purpose of this blog I have considered files uploaded in the last 10minutes and I will be scheduling this python code to execute every 10mins.
You can find the Python code for the same in my GitHub over here.
!!Happy Reading!!
