yvachot/PySpark_Training_Repository

No description

Find a file

Yûki VACHOT 8f6699ecc6 PipeGraph Decorator		2024-01-12 17:28:45 +01:00
.idea	Add env python for Spark	2024-01-11 14:01:57 +01:00
assets	PipeGraph Decorator	2024-01-12 17:28:45 +01:00
src	PipeGraph Decorator	2024-01-12 17:28:45 +01:00
.gitignore	Init	2024-01-05 13:05:39 +01:00
.pylintrc	Add env python for Spark	2024-01-11 14:01:57 +01:00
init.py	Add env python for Spark	2024-01-11 14:01:57 +01:00
README.md	PipeGraph Decorator	2024-01-12 17:28:45 +01:00
requirements.txt	PipeGraph Decorator	2024-01-12 17:28:45 +01:00
spark_check.py	Add env python for Spark	2024-01-11 14:01:57 +01:00

README.md

Python PySpark Training Repository

Author: Yûki VACHOT

Updated: 10/01/24

CONTENT TABLE

Installation

python -m venv

Python 3.11.7
Spark 3.5.0 with Hadoop 3.0.0
winutils.exe, .pdb and hadoop.dll
Java JDK 17
pygraphviz install in x86
- pip install --global-option=build_ext --global-option="-IC:\Program Files (x86)\Graphviz\include" --global-option="-LC:\Program Files (x86)\Graphviz\lib" pygraphviz

Run Python PySpark

python init.py

Run Python Test

path from src/test_pyspark_training
pytest -k test_

Run pylint for code check

Run Python doc with Sphinx