AWS Lambda Use case Questions ❓

📍Use case: We are having a very big file , that our Lambda need to work on — now , we have many options like place it in S3/Tmp/File System Config ..! which path we need to choose and why ?

Answer :

Please note down all the answers for below questions first

Question 01 : We need to ask — what is the size of the File

Question 02: Is it okay if Latency present — Lambda takes a bit more time for the Execution

Question 03: Do we need store the processed files ?

Question 04: How much money you can spend ? Less or More ?

Now , we have answers for all the above questions — Now we will look into the what

✅ We will choose S3

Pros:

• Scalability: S3 is highly scalable and can handle very large files without any issues.
• Cost-Effective: S3 storage is relatively inexpensive.
• Accessibility: The file can be accessed by multiple Lambda functions or other AWS services.
• Durability and Availability: S3 provides high durability and availability for stored files.

Cons:

• Latency: There may be some latency involved in downloading large files from S3 during the Lambda execution.

Use Case:

• Best for large files that need to be processed but do not need to be stored within the Lambda execution environment.

✅ We will choose /tmp folder

Pros:

• Low Latency: The /tmp directory is local to the Lambda execution environment, so read/write operations are fast.
• Simple Implementation: No need for additional services or configurations.

Cons:

• Storage Limitations: The /tmp directory has a storage limit of 10 GB.
• Ephemeral Storage: The data in /tmp is lost when the Lambda function execution ends.

Use Case:

• Suitable for temporary storage of smaller files (up to 10 GB ) during processing.

✅ We will choose File System Config

Pros:

• Persistent Storage: Data in EFS persists beyond the life of the Lambda execution.
• Scalability: EFS can handle large amounts of data, with a soft limit of 1 TB per Lambda function but can be extended upon request.
• Shared Access: Multiple Lambda functions can access the same file system.

Cons:

• Cost: EFS can be more expensive compared to S3, especially for large amounts of data.
• Configuration Complexity: Requires setup and management of the EFS file system.

Use Case:

• Suitable for scenarios where large files need to be processed and possibly shared between multiple Lambda functions.

Choose the Right Option 🤞

File Size and Execution Time:

• For files larger than 512 MB, S3 or EFS are better choices.
• For smaller files or temporary storage during processing, the /tmp directory is suitable.

Persistence and Sharing Needs:

• If the data needs to persist and be shared across multiple Lambda executions, EFS is a good option.
• For temporary and single-execution data, /tmp or S3 can be used.

Latency and Cost Considerations:

• S3 offers a cost-effective and highly available solution with slight latency.
• EFS offers persistent and scalable storage at a higher cost.
• /tmp is fast and simple but limited in size.

🥷Enjoy your Learning and Please comment if you feel — any other similar questions we can add to this page..!

Thank you much for reading📍

“ Yours Love ( @lisireddy across all the platforms )