Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Dumps - Databricks Certification Questions and Answers

When you convert a large pyspark.pandas (aka Pandas API on Spark) DataFrame to a local Pandas DataFrame using .toPandas(), Spark collects all partitions to the driver.

From the Spark documentation:

“Be careful when converting large datasets to Pandas. The entire dataset will be pulled into the driver’s memory.”

Thus, for large datasets, this can cause memory overflow or out-of-memory errors on the driver.

Final Answer: D

Adaptive Query Execution (AQE) is a Spark SQL feature introduced to dynamically optimize queries at runtime based on actual data statistics collected during execution.

Key benefits include:

Runtime plan adaptation: Spark adjusts the physical plan after some stages complete.

Skew handling: Automatically splits skewed partitions to balance work distribution.

Join strategy optimization: Dynamically switches between shuffle join and broadcast join depending on partition sizes.

Coalescing shuffle partitions: Reduces the number of small tasks for better performance.

Example configuration:

spark.conf.set("spark.sql.adaptive.enabled", True)

This enables AQE globally in Spark 3.5.

Why the other options are incorrect:

A: AQE adapts during runtime, not only before execution.

B: Task distribution is a base Spark feature, not specific to AQE.

C: AQE specifically addresses runtime skew and join adjustments.

To sort a PySpark DataFrame by multiple columns with mixed sort directions, the correct usage is:

python

CopyEdit

df.orderBy("age", "salary", ascending=[True, False])

age will be sorted in ascending order

salary will be sorted in descending order

The orderBy() and sort() methods in PySpark accept a list of booleans to specify the sort direction for each column.

Documentation Reference: PySpark API - DataFrame.orderBy

Broadcast joins work by sending the smaller DataFrame to all executors, eliminating the shuffle of the larger DataFrame.

From Spark documentation:

“Broadcast joins are efficient when one DataFrame is small enough to fit in memory. Spark avoids shuffling the larger table.”

DataFrame B (1 GB) fits within the default threshold and should be broadcasted.

It eliminates the need to shuffle the large DataFrame A.

Final Answer: B

Spark automatically uses a broadcast hash join when one side of the join is small enough to fit within the broadcast threshold.

Default threshold:

spark.sql.autoBroadcastJoinThreshold = 10MB (as of Spark 3.5)

Since df2 is 8 MB, Spark automatically broadcasts it to all executors. This avoids a shuffle on the large dataset (df1) and speeds up the join.

Why the other options are incorrect:

A: 8 MB < 10 MB threshold → broadcast join is efficient.

B: AQE is not required for broadcast joins; it’s a static optimization.

C: Broadcast hints are optional — Spark infers automatically.

dropDuplicates() with no column list removes duplicates based on all columns.

It's the most efficient and semantically correct way to deduplicate records that are completely identical across all fields.

From the PySpark documentation:

dropDuplicates(): Return a new DataFrame with duplicate rows removed, considering all columns if none are specified.

— Source: PySpark DataFrame.dropDuplicates() API

The correct way to overwrite an existing file using the DataFrameWriter is:

df.write.mode("overwrite").json("path/to/file")

Option D is also technically valid, but Option A is the most concise and idiomatic PySpark syntax.

The split() function in PySpark splits strings into an array based on a given delimiter.

Then, .getItem(index) extracts a specific element from the array.

Correct usage:

from pyspark.sql.functions import split, col

customerDF = customerDF \

withColumn("username", split(col("email"), "@").getItem(0)) \

withColumn("domain", split(col("email"), "@").getItem(1))

This creates two new columns derived from the email field:

"username" → text before @

"domain" → text after @

Why the other options are incorrect:

B: regexp_replace only replaces text; does not split into multiple columns.

C: .select() cannot alias multiple derived columns like this.

D: Column objects are not native Python strings; cannot use standard .split().

In Structured Streaming, checkpoints store state information (offsets, progress, and metadata) needed to resume a stream after a failure or restart.

Correct usage:

Set the checkpointLocation option when writing the streaming output:

streaming_df.writeStream \

format("delta") \

option("checkpointLocation", "/path/to/checkpoint/dir") \

start("/path/to/output")

Spark uses this checkpoint directory to recover progress automatically and maintain exactly-once semantics.

Why the other options are incorrect:

A/D: recoveryLocation is not a valid Spark configuration option.

B: Checkpointing must be configured in writeStream, not during readStream.

Apache Spark follows a lazy evaluation model, meaning transformations (like filter(), select(), map()) are not executed immediately. Instead, they build a logical plan (lineage graph) that represents the sequence of operations to be applied.

Execution only begins when an action (e.g., count(), collect(), save(), show()) is called. At that point, Spark’s engine:

Optimizes the logical plan into a physical plan.

Divides it into stages and tasks.

Executes them across the cluster.

This design helps Spark optimize execution paths and avoid unnecessary computations.

Why the other options are incorrect:

A: Transformations do not execute immediately; they are deferred.

B: Optimization happens during job execution (after an action), not during transformations.

D: Execution starts automatically once an action is triggered, no manual intervention needed.