Data-Engineer-Associate Exam Dumps - Amazon Web Services AWS Certified Data Engineer Questions and Answers

Option A is correct because Amazon Kinesis Data Streams is designed to collect and process large streams of data records in real time and is optimized for rapid, continuous data intake and aggregation. AWS states that Kinesis Data Streams is a fully managed, serverless data streaming service for ingesting streaming data at any scale and highlights event sources such as clickstream and application activity data. AWS also notes that Kinesis can pair with AWS Lambda for low-latency event processing. This directly matches the requirement to ingest millions of customer activity events with minimum latency in a scalable and durable way.

Option B is batch processing every 10 minutes, so it does not satisfy the real-time requirement. Option C depends on files landing in S3 first, which is also not the lowest-latency ingest path. Option D can support streaming, but a managed Kafka cluster with self-managed consumers introduces more operational overhead and is generally less cost-effective here than a native Kinesis + Lambda pipeline for an AWS-centric event-ingestion workload. Therefore, Kinesis Data Streams + Lambda + Redshift is the best answer.

Amazon Kinesis Data Firehose is a fully managed data ingestion service that enables reliable and scalable delivery of streaming and batch data into Amazon S3 with minimal operational overhead. This directly satisfies the requirement for a fully managed AWS ingestion service while avoiding the need to provision, scale, or manage infrastructure.

By using the Amazon Kinesis Agent on the on-premises servers, the company can automatically forward daily data files to Kinesis Data Firehose. Firehose handles buffering, retry logic, scaling, and delivery without requiring administrative effort. Delivering the data to Amazon S3 allows seamless integration with Amazon Athena, which natively queries data stored in S3 without requiring data movement or transformation.

Enabling Amazon S3 versioning protects files from accidental deletion by preserving previous versions of objects. This aligns with AWS best practices for data durability and governance, especially for analytics workloads and compliance requirements.

Other options introduce unnecessary operational complexity. AWS DataSync with Amazon EFS is not optimized for Athena-based analytics. AWS Glue jobs and Amazon RDS are unsuitable for file-based analytical access. A self-managed Apache Kafka solution with Amazon MSK significantly increases operational overhead.

Therefore, option B is the most efficient, scalable, and operationally optimal solution according to AWS Certified Data Engineer – Associate best practices.

The requirement is to update the AWS Glue Data Catalog incrementally based on S3 events. Using an S3 event-based approach is the most automated and operationally efficient solution.

A. Create an S3 event-based AWS Glue crawler:

An event-based Glue crawler can automatically update the Data Catalog when new data arrives in the S3 bucket. This ensures incremental updates with minimal operational overhead.

Option A is correct. The MERGE INTO statement is used to conditionally update, insert, or delete rows based on a match condition between a target table and a source table. In this statement, the target table is accounts and the source table is monthly_accounts_update. The join condition is t.customer = s.customer. Because the statement uses WHEN MATCHED THEN DELETE, every row in accounts that has a matching customer value in monthly_accounts_update will be deleted from the target table.

AWS documentation for MERGE INTO states that it conditionally updates, deletes, or inserts rows into an Apache Iceberg table, and the syntax explicitly includes WHEN MATCHED THEN DELETE. AWS Glue guidance and examples for Iceberg also show MERGE INTO as a supported pattern for row-level changes in Glue ETL workflows. This means the syntax is valid for supported Glue-Iceberg use cases, so option D is incorrect. Options B and C are also incorrect because the statement does not retain only matching rows and does not delete the entire table. It deletes only the rows in the target table that satisfy the match condition.

The data engineer needs to ensure that the data in an Amazon EMR cluster is encrypted both at rest and in transit. The data in Amazon S3 is already encrypted using an AWS KMS key. To meet the requirements, the most suitable solution is to create an EMR security configuration that specifies the correct KMS key for at-rest encryption and use the PEM file for in-transit encryption.

Option C: Create an Amazon EMR security configuration. Specify the appropriate AWS KMS key for at-rest encryption for the S3 bucket. Specify the Amazon S3 path of the PEM file for in-transit encryption. Use the security configuration during EMR cluster creation.This option configures encryption for both data at rest (using KMS keys) and data in transit (using the PEM file for SSL/TLS encryption). This approach ensures that data is fully protected during storage and transfer.

Options A, B, and D either involve creating unnecessary additional security configurations or make inaccurate assumptions about the way encryption configurations are attached.

Options A and B are correct because both are managed, serverless orchestration approaches for AWS Glue ETL pipelines.

AWS Step Functions is a serverless orchestration service that can coordinate AWS services, including Glue jobs, as a sequence of steps. AWS documentation states that Step Functions supports built-in error handling with Retry and Catch, which directly fits the requirement to coordinate ETL jobs and handle failures automatically. AWS Prescriptive Guidance also shows ETL pipelines that move data from Amazon S3 through AWS Glue and onward in a Step Functions workflow.

AWS Glue workflows are also correct because AWS documentation states that Glue workflows can create and visualize complex ETL activities involving multiple jobs, crawlers, and triggers. They provide a graph of dependencies and can manage chained execution of interdependent tasks. That matches the requirement for serverless ETL orchestration.

Option C is not serverless because it requires an always-on EC2 instance. Option D can start jobs from events, but it is not the best service for managing complex chained dependencies and workflow state. Option E is primarily a CI/CD orchestration service, not the best fit for ETL job orchestration.

When processing many small files, the overhead of managing partitions can degrade performance. Rather than scaling workers manually, enabling AWS Glue auto scaling dynamically adjusts worker count based on resource requirements, which is cost-effective and addresses the performance concern.

“AWS Glue auto scaling automatically adds or removes workers based on the workload, which is efficient for handling performance bottlenecks when working with many small files.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

Manually scaling up workers or increasing their size (A, B) can be more expensive and less adaptive.

Using an Amazon EventBridge rule to run an AWS Glue job or invoke an AWS Glue workflow job every 15 minutes are two possible solutions that will meet the requirements. AWS Glue is a serverless ETL service that can process and load data from various sources to various targets, including Amazon Redshift. AWS Glue can handle different data formats, such as CSV, JSON, and Parquet, and also support schema evolution, meaning it can adapt to changes in the data schema over time. AWS Glue can also leverage Apache Spark to perform distributed processing and transformation of large datasets. AWS Glue integrates with Amazon EventBridge, which is a serverless event bus service that can trigger actions based on rules and schedules. By using an Amazon EventBridge rule, you can invoke an AWS Glue job or workflow every 15 minutes, and configure the job or workflow to run an AWS Glue crawler and then load the data into the Amazon Redshift tables. This way, you can build a cost-effective and scalable ETL pipeline that can handle data from 10 source systems and function correctly despite changes to the data schema.

The other options are not solutions that will meet the requirements. Option C, configuring an AWS Lambda function to invoke an AWS Glue crawler when a file is loaded into the S3 bucket, and creating a second Lambda function to run the AWS Glue job, is not a feasible solution, as it would require a lot of Lambda invocations and coordination. AWS Lambda has some limits on the execution time, memory, and concurrency, which can affect the performance and reliability of the ETL pipeline. Option D, configuring an AWS Lambda function to invoke an AWS Glue workflow when a file is loaded into the S3 bucket, is not a necessary solution, as you can use an Amazon EventBridge rule to invoke the AWS Glue workflow directly, without the need for a Lambda function. Option E, configuring an AWS Lambda function to invoke an AWS Glue job when a file is loaded into the S3 bucket, and configuring the AWS Glue job to put smaller partitions of the DataFrame into an Amazon Kinesis Data Firehose delivery stream, is not a cost-effective solution, as it would incur additional costs for Lambda invocations and data delivery. Moreover, using Amazon Kinesis Data Firehose to load data into Amazon Redshift is not suitable for frequent and small batches of data, as it can cause performance issues and data fragmentation. References:

AWS Glue

Amazon EventBridge

Using AWS Glue to run ETL jobs against non-native JDBC data sources

[AWS Lambda quotas]

[Amazon Kinesis Data Firehose quotas]

To achieve the most cost-effective storage solution, the data engineer needs to use an S3 Lifecycle policy that transitions objects to lower-cost storage classes based on their access patterns, and deletes them when they are no longer needed. The storage classes should also provide high availability, which means they should be resilient to the loss of data in a single Availability Zone1. Therefore, the solution must include the following steps:

Transition objects to S3 Standard-Infrequent Access (S3 Standard-IA) after 6 months. S3 Standard-IA is designed for data that is accessed less frequently, but requires rapid access when needed. It offers the same high durability, throughput, and low latency as S3 Standard, but with a lower storage cost and a retrieval fee2. Therefore, it is suitable for data files that are accessed once or twice each month. S3 Standard-IA also provides high availability, as it stores data redundantly across multiple Availability Zones1.

Transfer objects to S3 Glacier Deep Archive after 2 years. S3 Glacier Deep Archive is the lowest-cost storage class that offers secure and durable storage for data that is rarely accessed and can tolerate a 12-hour retrieval time. It is ideal for long-term archiving and digital preservation3. Therefore, it is suitable for data files that are accessed only once or twice each year. S3 Glacier Deep Archive also provides high availability, as it stores data across at least three geographically dispersed Availability Zones1.

Delete objects when they are no longer needed. The data engineer can specify an expiration action in the S3 Lifecycle policy to delete objects after a certain period of time. This will reduce the storage cost and comply with any data retention policies.

Option C is the only solution that includes all these steps. Therefore, option C is the correct answer.

Option A is incorrect because it transitions objects to S3 One Zone-Infrequent Access (S3 One Zone-IA) after 6 months. S3 One Zone-IA is similar to S3 Standard-IA, but it stores data in a single Availability Zone. This means it has a lower availability and durability than S3 Standard-IA, and it is not resilient to the loss of data in a single Availability Zone1. Therefore, it does not provide high availability as required.

Option B is incorrect because it transfers objects to S3 Glacier Flexible Retrieval after 2 years. S3 Glacier Flexible Retrieval is a storage class that offers secure and durable storage for data that is accessed infrequently and can tolerate a retrieval time of minutes to hours. It is more expensive than S3 Glacier Deep Archive, and it is not suitable for data that is accessed only once or twice each year3. Therefore, it is not the most cost-effective option.

Option D is incorrect because it combines the errors of option A and B. It transitions objects to S3 One Zone-IA after 6 months, which does not provide high availability, and it transfers objects to S3 Glacier Flexible Retrieval after 2 years, which is not the most cost-effective option.

1: Amazon S3 storage classes - Amazon Simple Storage Service

2: Amazon S3 Standard-Infrequent Access (S3 Standard-IA) - Amazon Simple Storage Service

3: Amazon S3 Glacier and S3 Glacier Deep Archive - Amazon Simple Storage Service

[4]: Expiring objects - Amazon Simple Storage Service

[5]: Managing your storage lifecycle - Amazon Simple Storage Service

[6]: Examples of S3 Lifecycle configuration - Amazon Simple Storage Service

[7]: Amazon S3 Lifecycle further optimizes storage cost savings with new features - What’s New with AWS