---
title: "Malware Scanning Service Lessons Learned"
canonical: "https://resourcecenter.kenect.com/space/Messaging/179437644/Malware%20Scanning%20Service%20Lessons%20Learned"
format: markdown
---
## **Introduction**

In our efforts to ensure that all files managed by our resource-service are safe and free from malware, we decided to implement an automated malware scanning solution. For this, we adopted the architecture recommended by Google in their guide, ["Automate Malware Scanning for Documents Uploaded to Cloud Storage."](https://cloud.google.com/architecture/automate-malware-scanning-for-documents-uploaded-to-cloud-storage) This write-up details our implementation experience, challenges encountered, and lessons learned.

![image-20240904-151903.png](media://d62ab20f-c3a1-43de-8493-154c08a540b4)

## **Overview of Google's Recommended Architecture**

Google's recommended architecture for automated malware scanning in Cloud Storage offers a scalable and efficient way to handle potentially malicious files uploaded to storage buckets. The architecture leverages multiple Google Cloud services to automate the detection and handling of malware in a streamlined workflow:

- **Cloud Storage:** Serves as the entry point where users upload documents requiring malware scanning.
- **Eventarc:** Captures the upload events from Cloud Storage and routes them to the malware scanning service.
- **Cloud Run Services:** A containerized malware scanning service receives the event from Eventarc, scans the files using antivirus tools, and determines if they are safe.
- **Pub/Sub:** Utilized for event notifications and communication between services.

### **Workflow:**

1. A file is uploaded to a Cloud Storage bucket.
2. Eventarc captures the upload event and triggers a Cloud Run service.
3. The Cloud Run service processes the file, performing a malware scan.
4. Based on the scan results, the file is moved either to a clean or quarantine storage bucket.

### **Benefits:**

- **Scalability:** Capable of handling a high volume of file uploads efficiently.
- **Flexibility:** Allows for custom malware scanning tools and configurations using containerized services.
- **Integration:** Seamlessly integrates with other Google Cloud services to automate the scanning process end-to-end.

## **Challenges Encountered and Mitigations**

### **1. Eventarc’s Ten-Second Acknowledgement Limitation:**

- **Challenge:** Eventarc requires that events are acknowledged within ten seconds. When initially implemented using Spring Boot for the malware scanning service, the startup time exceeded this limit, leading to repeated event retries.
- **Initial Approach:** Developed the malware scanning service using Spring Boot.
- **Issue:** Spring Boot's longer startup times caused Eventarc to miss the acknowledgment deadline. Additionally, file scan times can also take an excess of ten seconds to complete.
- **Mitigation:** Switched to using a Cloud Function to handle Eventarc events to publish to a topic we control. This would allow us to control the acknowledgement deadline. However, new challenges arose due to the lack of control over container image builds for cloud functions.
- **Final Decision:** Moved to a Cloud Run service using a lightweight Golang implementation to ensure quick startup to meet the ten-second acknowledgment requirement and ability to control the container build and storage.

### **2. Cloud Functions and Container Image Control:**

- **Challenge:** Deploying new Cloud Functions led to automatic creation of container images managed by Google, conflicting with our practice of building images once and reusing them across environments.  Our existing cloud functions are first generation and appear to be grandfathered and can be upgraded to the latest runtimes without triggering the container requirement.
- **Issue:** This automatic process did not align with our deployment standards, where we prefer to have full control over the build and storage of container images. Specifically, we build images once and store in our development environment and be used in all environments..
- **Mitigation:** Transitioned to Cloud Run services, which allowed us to control the container image builds per our standards. This change enabled us to manage container images in alignment with our existing CI/CD practices.

### **3. Terraform and Container Image Reference Issues:**

- **Challenge:** During the deployment of new Cloud Run services, `terraform apply` would update the service configuration without changing the container image, leading to unintended behavior.
- **Issue:** Terraform did not recognize changes to container images if the image reference was specified using a mutable tag that did not change.
- **Workaround:** The command `gcloud run deploy` was used to manually update the service with the latest container image. While this approach works, the Terraform state is not updated resulting in subsequent applies reverting the image used.
- **Mitigation:** Adjusted the Terraform configuration to reference the container image digest instead of a mutable tag. This ensured that the correct image version was deployed consistently.
- **Final Approach:** Integrated CircleCI to extract the image digest from the container built in the pipeline and used this digest in the Terraform configuration, ensuring accurate and consistent deployments.

## **Recommendations**

- **Avoid using Spring Boot with Eventarc:** Due to Eventarc's requirement for a ten-second acknowledgment deadline, using frameworks with long startup times, like Spring Boot, can cause issues. Consider using lightweight runtimes or languages, such as Golang, to ensure timely event handling.
- **Prefer Cloud Run over Cloud Functions for new projects:** Deployments often require container images, which are more easily managed in Cloud Run. Cloud Functions, especially in newer generations, rely on Google-controlled container builds, which may not align with custom build practices.
- **Specify container image digests in Terraform configurations:** To avoid unintended updates and ensure consistent deployments, use the container image digest rather than a mutable tag in Terraform. This approach maintains state consistency and reduces configuration drift.