1 using pycharm, pylint, pytest, and vcs to design a … introduction... · 2020. 10. 30. ·...
TRANSCRIPT
![Page 1: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/1.jpg)
USING PYCHARM, PYLINT,PYTEST, AND VCS TO DESIGNA PROJECT PIPELINEAntonio Luca Alfeo
1
![Page 2: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/2.jpg)
OVERVIEW1. Installation
2. Work with Pycharm projects
I. Configure the interpreter
II. Requirements management
III. Project navigation and run
3. Code quality checking
I. Pylint
II. Plugin installation and usage
4. Code correctness checking
I. Enabling and configuring Pytest in Pycharm
II. Testing methods and usage
5. Version Control on Pycharm
I. Local history
II. Connect to GitHub
III. Github: share a project
IV. Github: commit and push
V. Github: clone a project
6. Recap and more resources
7. Exercise
8. Python basics to design a structured project pipeline
I. Tabular data with pandas.dataframes
II. Dataframe manipulation
III. Data segregation with Sklearn
IV. Model hyperarametrization
V. From JSON to Object
VI. Performance evaluation
VII. Model deployment
VIII. Project profiling
9. Recap and more resources
10. Exercise
2
![Page 3: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/3.jpg)
3
INSTALLATION
![Page 4: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/4.jpg)
INSTALLATION 1/3
1. Install Git using only the default settings in the installation process
• https://git-scm.com/downloads
2. Install PyCharm
• Linux -> https://download.jetbrains.com/python/pycharm-community-anaconda-2020.2.3.tar.gz
• Windows -> https://download.jetbrains.com/python/pycharm-community-anaconda-2020.2.3.exe
• MacOs -> https://download.jetbrains.com/python/pycharm-community-anaconda-2020.2.3.dmg
4
![Page 5: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/5.jpg)
INSTALLATION 2/3
During PyCharm installation enable “open folder as project”
5
![Page 6: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/6.jpg)
INSTALLATION 3/3
• Accept the JetBrainsPrivacy Policy
• Choose the UI theme youprefer
• Do not install anyfeatured plugin
• Install Miniconda: includes
the conda environmentmanager, Python, thepackages they dependon, and a small numberof other useful packages(e.g. pip).
• Start using PyCharm
6
![Page 7: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/7.jpg)
7
WORK WITH PYCHARM PROJECTS
![Page 8: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/8.jpg)
1. Unzip “simpleClassifier.rar”
2. Open the resulting folder “as a PyCharm project”
OPEN A PROJECT
8
![Page 9: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/9.jpg)
Project view
Menu Toolbar
Editor
Run/Debug buttons
Console and Tool windows
GUI
9
![Page 10: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/10.jpg)
INTERPRETER CONFIGURATION• File > Settings > Project > Python interpreter > Show all
• > Conda Environment > New environment
• Location and conda executable may have slightly different paths (the first part) according to your miniconda3 location.
• Apply > Ok
10
1
2
3
4
56
7
8
![Page 11: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/11.jpg)
WARNING
It can take some minutes to complete the project setting. A restart may be required before moving to
the next step!
11
![Page 12: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/12.jpg)
PROJECT REQUIREMENTS 1/2• The requirements are all the packages that our software needs to
run properly. Those can be installed with a PyCharm plugin.
• Double click on “requirements.txt” in the Project View > Install Plugin
• Once the plugin is installed click on “Install requirements”, select alland click install.
12
![Page 13: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/13.jpg)
• if everything goes smoothly you might see the package installationstatus progress at the bottom of the GUI.
• Once the package installation is done a notification appears, but itis still NOT possible to go to next step, at least until PyCharm hasfinished "Updating skeletons".
• Each package can also be installed via GUI or with the Terminal byusing
• “pip install <name_lib>” to install a single package
• “pip install –r requirements.txt” to install the packages on therequirements.txt
PROJECT REQUIREMENTS 2/2
13
![Page 14: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/14.jpg)
CHECK THE PACKAGES AVAILABLE
You can see (add, eliminate and upgrade*) the packages and libraries available in your virtual environment by checking :
File > Settings > Project > Python Interpreter
14
*
![Page 15: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/15.jpg)
CONFIGURE THE FIRST RUN
1. Click “Add configuration”
2. Select “Python”
15
3. Select “main.py” in your project folder.
4. Click OK
![Page 16: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/16.jpg)
RUN IT!
The confusion matrix!Txt with evaluation score!
Click RUN
16
![Page 17: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/17.jpg)
17
CODE QUALITY CHECKING
![Page 18: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/18.jpg)
CODE QUALITY CHECKING
Pylint is a source-code, bug and quality checker for the Python programming language. It checks:
• if the code is compliant with Python's PEP8 standard
• if each module is properly imported and used
• if declared interfaces are truly implemented
• if there is duplicated code
18
![Page 19: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/19.jpg)
PYLINT PLUGIN INSTALLATION
1. File > Settings
> Plugin
2. Search for
“pylint”
3. Install Pylint
plugin
4. Restart
Pycharm
1
2
3
19
4
![Page 20: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/20.jpg)
PYLINT PLUGIN SETUP
1. File > Settings > Pylint
2. Insert the path to the pylint
executable
3. Click “Test”
4. Check the result
5. Click “Apply” and “Ok”. You
can now run “pylint” by
clicking the green arrow in
the Pylint Tool Windows
1
23
45
20
![Page 21: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/21.jpg)
PYLINT TOOL WINDOW USAGE
21
1. Double click on a Python
file in the Project View
2. Open the Pylint Tool
Windows
3. Click the green arrow
1
2
3
![Page 23: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/23.jpg)
23
CODE CORRECTNESS CHECKING
![Page 24: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/24.jpg)
CODE CORRECTNESS CHECKING
PyCharm supports pytest, providing many testing features such as:
• Dedicated test runner.
• Code completion for test subject.
• Code navigation.
• Detailed failing assert reports.
• Multiprocessing test execution.
24
![Page 25: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/25.jpg)
ENABLE PYTEST
1
2
1. File > Settings > Python Integrated Tools and
2. Set “pytest” as the “Default test runner”
25
![Page 26: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/26.jpg)
CREATE A NEW TEST1. Put the cursor on the bracket of a method declaration, right-click and
select Go To > Test
2. Create a new test script
3. Give it a directory, a name, and a method to test
1
2
26
3
![Page 27: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/27.jpg)
BUILD AND RUN A TEST 1/3
1. Fill the test function and the assert statement. If this statement is true
the test is PASSED!
2. Click “Edit pytest for …”
27
![Page 28: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/28.jpg)
BUILD AND RUN A TEST 2/3
28
3. Reduce the verbosity of the pytest
outcome to improve readability. Put the
following Additional Arguments:
“ -q --disable-warnings --tb=no “
-q, quiet output
--disable-warnings, don't show most of the warnings
--tb=no, don't show any traceback
4. Click OK and Run the test
3
4
![Page 29: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/29.jpg)
BUILD AND RUN A TEST 3/3
29
![Page 30: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/30.jpg)
TEST PARAMETRIZATION
The Pytest framework makes it easy to write tests to support multiple and complex functional testing for applications and libraries. Pytestenables test parametrization at several levels, such as:
• @pytest.fixture provide a fixed baseline (e.g. an instance of a class) upon which tests can reliably and repeatedly execute. Moreover, via scope controls it is possible to check how often a fixture gets called.
• @pytest.mark.parametrize allows the definition of multiple sets of arguments and fixtures for testing purpose.
30
![Page 31: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/31.jpg)
BUILD AND RUN MANY TESTS
1. Prepare a set of values to
test
2. Test those by passing
them to the test function
3. Now you have one
evaluation for each test
value!
31
1
2
3
![Page 33: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/33.jpg)
33
VERSION CONTROL ON PYCHARM
![Page 34: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/34.jpg)
VERSION CONTROL
In Pycharm the Version Control
can be accessed locally:
1. via VCS > Local History
2. Stores only local and recent
change in the project
highlighting them
34
1
2
![Page 35: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/35.jpg)
ENABLE VCI TO USE GITHUB
Github is way more convenient especially when a project is developed in group.
After VC integration is enabled (1 and 2), PyCharm will ask you whether you want to share project settings files via VCS.
In the dialog you can choose ”Always Add” to synchronize project settings with other repository users who work with PyCharm.
1
2
35
![Page 36: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/36.jpg)
36
ENABLE VCI TO USE GITHUB
Login into GitHub, requesting a new access token with your login and password (1). Then, authorize JetBrain IDE integration (2 and 3).
1 2 3
![Page 37: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/37.jpg)
SHARE A PROJECT 1/21. When connection to GitHub has been established, the Share Project on
GitHub dialog appears (VCS > Import Version Control > Share …).
2. Name the repository and click share
37
1
2
3
![Page 38: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/38.jpg)
SHARE A PROJECT 2/2
38
1. Being the initial commit, of the
project every file is highlighted
in red (they are not in the
VCS).
2. Click Add
3. PyCharm will notify once the
process is finalized 1
2
3
![Page 39: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/39.jpg)
… ON GITHUB
Let’s say you have a central repository for your project, the
contribution of each coder will be highlighted by the commits she/he
made. In the next slide you will see how to “sign” your commit º
39
![Page 40: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/40.jpg)
COMMIT AND PUSHUpdate (pull) Push
Commit
Color code:
- Blue: modified
- Green: commit done
- Red: not in the VCS
- White: push done
After the first commit here
there will be the “before
and after” comparison
Explain clearly the
change provided
º
40
![Page 41: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/41.jpg)
CLONE FROM GIT 1/2
1. Go to GitHub and annotate the URL of the repository you are
interested in.
2. From the main menu, choose VCS > Get from Version Control, and
choose GitHub on the left.
1
2
41
![Page 42: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/42.jpg)
CLONE FROM GIT 2/2
1. Specify the URL of the repository that you want to clone (the one you
annotated from GitHub).
2. In the Directory field, enter the path to the folder where your local Git
repository will be created.
3. Click Clone. If you want to create a project based on these sources,
click Yes in the confirmation dialog. PyCharm will automatically set Git
root mapping to the project root directory.
1
2
3
42
![Page 43: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/43.jpg)
43
![Page 44: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/44.jpg)
RECAPPyCharm is an integrated development environment (IDE) widelyused in the Python community. PyCharm provides:
• Project and code navigation: create a Python project by opening afolder as a project or even from scratch, by creating or importingthe python files. Navigate the project with specialized project views,jump between files, classes, methods and usages.
• Python refactoring: rename, extract methods, introduce variablesand constants
• Configurable python interpreter and virtual environment support.
• Coding assistance and analysis: code completion, syntax and errorhighlighting, quick fixes, guided import, integrated Pythondebugger, linting and unit testing with code coverage.
• PyCharm provides version control integration, a unified userinterface for many VCS such as GithHub, supporting themanagement of commit, push, pull, change lists, cloning… andeven branch, merge, and conflict via Git.
44
![Page 45: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/45.jpg)
45
MORE RESOURCES…
![Page 46: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/46.jpg)
ON GETTING STARTED WITH PYCHARM
PYCHARM DOCS
• Miniconda can be installed at any time from Tools > Install Miniconda3
• https://www.jetbrains.com/help/pycharm/quick-start-guide.html
• https://www.jetbrains.com/help/pycharm/opening-your-project-for-the-first-time.html
• https://www.jetbrains.com/help/pycharm/configuring-python-interpreter.html
• https://www.youtube.com/watch?v=hyoS5W_1kBQ&list=PLQ176FUIyIUbDwdvWZNuB7FrCc6hHregy
CREATE AND MANAGE REQUIREMENTS
• https://www.jetbrains.com/help/pycharm/managing-dependencies.html#create-requirements
A PYCHARM BOOK
• https://www.academia.edu/42195195/Hands_On_Application_Development_with_PyCharm_Accelerate_your_Python_applications_using_practical_coding_techniques_in_PyCharm?auto=download
46
![Page 47: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/47.jpg)
ON LINTING
• https://www.python.org/dev/peps/pep-0008/
• https://plugins.jetbrains.com/plugin/11084-pylint
• Pylint is fully customizable: modify your pylintrc to customize which errors or conventions are important to you.
47
![Page 48: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/48.jpg)
ON TESTING
• https://www.jetbrains.com/help/pycharm/pytest.html
• https://www.jetbrains.com/pycharm/guide/technologies/pytest/
• pytest.mark allows you setting the metadata on your test functions. Here you have some builtin markers different from pytest.mark.parametrize:
• @pytest.mark.skip: always skip a test function
• @pytest.mark.skipif: skip a test function if a certain condition is met
• @pytest.mark.xfail: produce an expected failure outcome if a certain condition is met
48
![Page 49: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/49.jpg)
ON VERSION CONTROL
PYCHARM DOCS
• https://www.jetbrains.com/help/pycharm/enabling-version-control.html
• https://www.jetbrains.com/help/pycharm/manage-projects-hosted-on-github.html
• https://www.jetbrains.com/help/pycharm/contribute-to-projects.html
VIDEO TUTORIAL
• https://www.youtube.com/watch?v=jFnYQbUZQlA
• https://www.youtube.com/watch?v=_w9XWHDSSa4
• https://www.youtube.com/watch?v=AHkiCKG-JhM
49
![Page 50: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/50.jpg)
50
![Page 51: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/51.jpg)
EXERCISE
1. Use the IrisClassifier.py by changing the number of training epochs,i.e. "max_iter" in the MLPClassifier() documentation page. Howdoes the performance change by using 50 epochs?
2. Perform code linting.
3. Test if the performance is greater than 0.75 with number of trainingepochs equal to 5, 10, 25, 50, 100, 200. How many of them fail?
4. Make commit and push on your personal GitHub repository.
51
![Page 52: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/52.jpg)
52
A POSSIBLE SOLUTION
from typing import Setfrom irisclassifier import IrisClassifierimport pytest
epochs: Set[int] = {5, 10, 25, 50, 100, 200}
@pytest.mark.parametrize('epoch', epochs)def test_evaluations(epoch):
i = IrisClassifier(epoch)i.ingestion()i.segregation()i.train()res = i.evaluation()assert res > 0.75
![Page 53: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/53.jpg)
53
![Page 54: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/54.jpg)
54
PYTHON BASICS TO DESIGN A PROJECT PIPELINE
![Page 55: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/55.jpg)
SO MANY PACKAGES AVAILABLE• Pytest and Pylint: to write better code!
• Pandas: read data from a broad range of sources like CSV, SQLdatabases, and Excel files.
• Json and Joblib: to save/read models and configurations to/fromfiles.
• Sci-kit_learn: machine learning library with a broad range ofclustering, regression and classification algorithms. It provides alsodata preprocessing, pipelining and performance analysis modules.
• Matplotlib: graphic visualization libraries.
• NumPy: to apply mathematical functions and scientificcomputation to multi-dimensional data
…and much more.
55
![Page 56: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/56.jpg)
PROJECT ORGANIZATION
• Consider that each class in your project should provideone or more functionalities that work with inputs andconfigurations to generate outputs. Inputs,configurations and outputs have to be stored indedicated folders. Each class should be able tosave/load data and configurations from those folders.
• Let’s start with a very simple example. We aim atclassify a yearly set of time series. We need at least 3folders:
• data -> tabular data, from raw to segregated form
• configs -> configurations and models
• results -> outcomes, figures and scores obtained
56
![Page 57: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/57.jpg)
57
1. UPLOAD/SAVE TABULAR DATA WITH PANDAS.DATAFRAME
from pandas import read_csv
# Read csvdata1 = read_csv('data/hotspotUrbanMobility-1.csv')# Print dataframe shapeprint(data1.shape)# Read csvdata2 = read_csv('data/hotspotUrbanMobility-2.csv')# Print dataframe shapeprint(data2.shape)# Append dataframesdata1 = data1.append(data2, ignore_index=True)print(data1.shape)# Save the new datafame as csvdata1.to_csv('data/completeDataset.csv', index=False)
Path of the csv to read
Dataframe shape
consist of its numer of
rows and column
Append 2 dataframes
and ignore the indexes
that pandas provides
Save the dataframe
![Page 58: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/58.jpg)
2. DATAFRAME MANIPULATION WITH PANDAS.DATAFRAME
58
from pandas import read_csv
# Read csvdata = read_csv('data/completeDataset.csv')# Check the content of the dataframeprint(data.describe())# Remove column with constant valuesdata = data.drop('h24', axis=1)# Remove anomalous instancesdata = data.loc[data['Anomalous'] < 1]print(data.describe())# Save as csvdata.to_csv('data/preprocessedDataset.csv', index=False)
Provides statistics for
each column in the
dataframe
Drop the column (axis=1)
whose label is ‘h24’
Loc select instances by
using a boolean array.
![Page 59: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/59.jpg)
59
3. DATA SEGREGATION WITH PANDAS AND SCIKIT-LEARN
from pandas import read_csvfrom sklearn.model_selection import train_test_split
# Read csvdata = read_csv('data/preprocessedDataset.csv')print(data.describe())# Random split 75% as training and 25% as testingtraining_data, testing_data, training_labels, testing_labels =
train_test_split(data.iloc[:, 4:len(data.columns)], data.iloc[:, 1])# Save as csvtraining_data.to_csv('data/trainingData.csv', index=False)testing_data.to_csv('data/testingData.csv', index=False)training_labels.to_csv('data/trainingLabels.csv', index=False)testing_labels.to_csv('data/testingLabels.csv', index=False)
Split randomly 75% and
25% of the dataset
(default), once it knows
which are the target
and the data.
iloc provides
integer-location
based indexing
![Page 60: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/60.jpg)
60
4. ML MODEL PARAMETRIZATION VIA GRID SEARCH
from sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import GridSearchCVfrom pandas import read_csvfrom numpy import ravelimport json
# Read the datatraining_data = read_csv('data/trainingData.csv')training_labels = read_csv('data/trainingLabels.csv')# setup Grid Search for a randomForestClassifierrf = RandomForestClassifier(random_state=0)# values to testparameters = {'n_estimators': (1, 2, 3)}# apply grid searchgs = GridSearchCV(rf, parameters)gs.fit(training_data, ravel(training_labels))# save best configurationconfig_path = 'config/modelConfiguration.json'with open(config_path, 'w') as f:
json.dump(gs.best_params_, f)
The ML model
(random_state is used for
reproducibility)
N_estimator is the name
of the hyperparameter of
RandomForestClassifier
Use ravel to transform
dataframe to narray
Save the best
hyperparameter as JSON
![Page 61: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/61.jpg)
61
5. FROM JSON TO OBJECT
import json
class Params():def __init__(self, n_estimators):
self.n_estimators = n_estimators
# load best configurationconfig_path = 'config/modelConfiguration.json'with open(config_path, 'r') as f:
params = json.load(f)
params_object = Params(**params)
Create a new Object,
and pass the JSON
dictionary as a map to
convert JSON data into a
custom Python Object
Load the JSON .
Remember, it should be
compliant to the
expected json schema
i.e. pass the validation
Use a simple class to get
the model’s parameters
![Page 62: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/62.jpg)
62
from numpy import ravelfrom pandas import read_csvfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import accuracy_scoreimport json
# Read the datatraining_data = read_csv('data/trainingData.csv')training_labels = read_csv('data/trainingLabels.csv')testing_data = read_csv('data/testingData.csv')testing_labels = read_csv('data/testingLabels.csv')# train the modelmodel = RandomForestClassifier(n_estimators=3, random_state=0).fit(training_data, ravel(training_labels))# compute labels on testing datalabels = model.predict(testing_data)# evaluate accuracy scorescore = accuracy_score(ravel(testing_labels), labels)# report the result on a jsonresults = {}results['accuracy'] = float(score)json_str = json.dumps(results, indent=4)with open('results/testReport.json', 'w') as file:
file.write(json_str)
You can use many different
performances measures
Define a set for your results
and save them as JSON
6. ML MODEL PERFORMANCE ASSESSMENT
![Page 63: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/63.jpg)
7. MODEL DEPLOYMENT WITH JOBLIB
63
from pandas import read_csvfrom sklearn.ensemble import RandomForestClassifierfrom numpy import ravelimport joblib
# Read the training datatraining_data = read_csv('data/trainingData.csv')training_labels = read_csv('data/trainingLabels.csv')# train the modelinitial_model = RandomForestClassifier(n_estimators=3, random_state=0).fit(training_data, ravel(training_labels))# save the modeljoblib.dump(initial_model, 'config/fitted_model.sav')#...# Read new datatesting_data = read_csv('data/testingData.csv')testing_labels = read_csv('data/testingLabels.csv')# load the modelmodel = joblib.load('config/fitted_model.sav')# Evaluate the initial model on new dataprint('Evaluation score:')print(model.score(testing_data, ravel(testing_labels)))
Save a fitted model to file
Load a fitted model from file
Use the model with new data
![Page 64: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/64.jpg)
1. Install the packages via Terminal by using:
• Pip install cProfile
• Pip install snakeviz
2. Than execute:
• python -m cProfile -o test.profile main.py
• snakeviz test.profile
3. Open the link in the Terminal windows:
64
8. PROFILING YOUR PROJECT WITH CPROFILE AND SNAKEVIZ
The main of your
project
The output of your
project profiling
![Page 65: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/65.jpg)
65
8. PROFILING YOUR PROJECT WITH CPROFILE AND SNAKEVIZ
Use the interactive interface to navigate the profiling output or search for
a specific method to understand its usage in your project. More info in
the official documentation
![Page 66: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/66.jpg)
66
RECAP AND MORE RESOURCES…
![Page 67: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/67.jpg)
MANIPULATE TABULAR DATA WITH PANDAS.DATAFRAME
• Load csv as a dataframe, save a dataframe as csv
• Shape, head and quick description of a dataframe
• Dataframe concatenation and append
• Data sort by index or by value
• Data resampling
• Dataframe selection by value or index
• Dataframe columns/row addition and elimination
• Replace by value, drop duplicate rows, group by value
• Drop NAN values , fill NAN value
• Data interpolation and signal smoothing via moving average
…and much more
67
![Page 68: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/68.jpg)
GRAPHS AND VISUALIZATION WITH MATPLOTLIB.PYPLOT
• Plots and save a figure
• Histogram e barplot
• Boxplots
• Scatterplot
• Single e multiple graphs
…and much more
68
![Page 69: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/69.jpg)
DATA PRE-PROCESSING WITH SCIKIT-LEARN
• Data discretization
• Data normalization with minmax or standard scaler
• Features computation and extraction with NumPy and Scipy
• Split data in train and testing set
…and much more
69
![Page 70: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/70.jpg)
ML MODEL IMPLEMENTATION WITH SCIKIT-LEARN
• MLPclassifier (classification)
• RandomForestClassifier (classification)
• DecisionTreeRegressor (regression)
• MLPregressor (regression)
• k-means (partition based clustering)
• DBSCAN (density based clustering)
• IsolationForest (anomaly detection)
• OneClassSVM (anomaly detection)
…and much more
70
![Page 71: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/71.jpg)
ARCHITECTURE EVALUATION
• Choose the right performance metric
• Performance visualization
• K-Folds and Stratified-K-Fold cross-validation
• Hyperparameters search via grid search
• Import/export objects parametrization with json
• Validate the json schema
• Dump/load fitted model with joblib
• How much time does an instruction take to execute?
• Official Pycharm profiling tool (just for Pro version)
…and much more
71
![Page 72: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/72.jpg)
72
![Page 73: 1 USING PYCHARM, PYLINT, PYTEST, AND VCS TO DESIGN A … Introduction... · 2020. 10. 30. · Pylint is a source-code, bug and quality checker for the Python programming language](https://reader035.vdocuments.us/reader035/viewer/2022071410/610572e7db9d960ae46140ce/html5/thumbnails/73.jpg)
EXERCIZE• Given the csvs used in this lab, build a few classes that:
• read the csvs “1” and “2” and append them
• remove the columns with constant values
• select only data from Cluster 0 (‘Cluster’ = 0)
• generate training and testing sets (respectively 75% and 25% of the
whole dataset) using only columns ‘h1’-‘h23’ as data and the column
‘Anomalous’ as target
• build a ML model using isolationForest
• train the model with the training data
• predict the anomalous label with the testing data
• compute the accuracy of your model. Consider that with your labels
you have 1 for anomalous instances whereas isolationForest return -1 for
anomalous instances.
• save the ML trained model
• load the trained model and use it with the “New” data
• print the resulting accuracy on screen.
73