PySpark_Training_Repository/README.md

Python PySpark Training Repository

Author: Yûki VACHOT

Updated: 10/01/24

CONTENT TABLE

---

Installation

python -m venv

• Python 3.11.7
• Spark 3.5.0 with Hadoop 3.0.0
• winutils.exe, .pdb and hadoop.dll
• Java JDK 17

---

Run Python PySpark

• python init.py

---

Run Python Test

• path from src/test_pyspark_training
• pytest -k test_

---

Run pylint for code check

---

Run Python doc with Sphinx

---