PySpark_Training_Repository/README.md

Python PySpark Training Repository

Installation

• Python 3.10
  • pyspark=3.1.1
  • findspark
  • pyspark-test
• Spark 3.1.1
• Hadoop 3.3.6
• Java JDK 11
• (not mandatory) Anaconda for conda

Run Python Test

• path from src/test_pyspark_training
• pytest -k test_

Run pylint for code check

Run Python doc with Sphinx