Posts

Spark SQL Dive (1) - When do Spark SQL runtime parameters take affects in AQE (Spark 3)?
We iterate the functionality of the selected Spark SQL runtime parameters in Spark 3
May 23, 2023 3 min read
Spark Setup (3) - TPCH and TPCDS benchmarks
Setup TPCH and TPCDS in a local Spark cluster on MacOS
May 20, 2023 4 min read
How to remote debug Spark via Intellij?
the setup for debug Spark programming running remotely in the cluster via Intellij
May 3, 2023 2 min read
How to spark-submit to a remote Spark cluster via Intellij?
the setup for submit Spark programs to a remote Spark cluster via Intellij
May 3, 2023 1 min read
Spark Setup (2) - local deployment for Spark, Hadoop(Yarn), Hive
The local deployment for Spark, Yarn, Hive to enable local debug in a (fake) distributed environment
May 3, 2023 4 min read
Spark Setup (1) - building and deploying Apache Spark from scratch
How to build and deploy Apache Spark from source code on Linux or Mac (including M1/M2)?
Last updated on Apr 28, 2023 2 min read
Spark Dev (1) - debug Spark program on M1/M2 chip
How to debug Spark program locally on Mac M1/M2 (without deploying a Spark locally)?
Last updated on Apr 26, 2023 2 min read
About my name
Why “Chenghao”?
Oct 13, 2017 1 min read
git clone, checkout to the spark-version & make export DSS_CONFIG=tpch-kit/dbgen run via spark shell cd $SPARK_HOME ./bin/spark-shell \ --master "local[4]" \ --conf spark.
Jan 1, 0001 2 min read