PySpark_Training_Repository/README.md

Python PySpark Training Repository

Author: Yûki VACHOT

Updated: 10/01/24

CONTENT TABLE

---

Installation

python -m venv

• Python 3.11.7
• Spark 3.5.0 with Hadoop 3.0.0
• winutils.exe, .pdb and hadoop.dll
• Java JDK 17
• pygraphviz install in x86
  • pip install --global-option=build_ext --global-option="-IC:\Program Files (x86)\Graphviz\include" --global-option="-LC:\Program Files (x86)\Graphviz\lib" pygraphviz

---

Run Python PySpark

• python init.py

---

Run Python Test

• path from src/test_pyspark_training
• pytest -k test_

---

Run pylint for code check

---

Run Python doc with Sphinx

---