31 lines
No EOL
679 B
Markdown
31 lines
No EOL
679 B
Markdown
Python PySpark Training Repository
|
|
==============
|
|
**Author:** *Yûki VACHOT*
|
|
|
|
**Updated:** **10/01/24**
|
|
# CONTENT TABLE
|
|
|
|
|
|
---
|
|
# Installation
|
|
|
|
`python -m venv `
|
|
|
|
- [Python 3.11.7](https://www.python.org/downloads/)
|
|
- [Spark 3.5.0 with Hadoop 3.0.0](https://spark.apache.org/downloads.html)
|
|
- [winutils.exe, .pdb and hadoop.dll](https://github.com/steveloughran/winutils/tree/master/hadoop-3.0.0/bin)
|
|
- [Java JDK 17](https://www.azul.com/downloads/?version=java-17-lts&package=jdk#zulu)
|
|
---
|
|
# Run Python PySpark
|
|
- `python init.py`
|
|
---
|
|
# Run Python Test
|
|
- path from src/test_pyspark_training
|
|
- `pytest -k test_`
|
|
---
|
|
# Run pylint for code check
|
|
|
|
---
|
|
# Run Python doc with Sphinx
|
|
|
|
--- |