This article assumes that you already have a project on GitHub that you want to package and publish.
Step 0: Obtaining a project license
Before doing anything else, since your project is going to be open source, it should have a license. Which license you get depends on how the project package is used. Some common licenses for open source projects are MIT or BSD.
To add a license to your project, simply refer to the steps in the following link and add the LICENSE file to the root directory in your project library:/en/articles/adding-a-license-to-a-repository
Step 1: Getting your code ready to go
To package your project, you need to do some preparatory work:
- Get your project structure in place correctly. Typically, the root directory of a project library contains a folder named after the project name, and the core code of the project should be located in this folder. Outside of this folder is the rest of the code needed to run and build packages (tests, documentation, etc.).
- The core folder should contain one (or more) module(s) and a __init__.py file containing the classes/functions you wish to make accessible to end users. This file may also contain package versions for end-user access.
- Ideally, the logging package should be used to set up a reasonable logging system (rather than outputting in prints).
- Ideally, your core code should be assigned to one or more classes.
from .estimate import Estimator
Using __init__.py as an example, if Estimator is the class that the end user will access (which is defined in the file)
import logging class LogMixin(object): @property def logger(self): name = '.'.join([self.__module__, self.__class__.__name__]) FORMAT = '%(name)s:%(levelname)s:%(message)s' (format=FORMAT, level=) logger = (name) return logger
Take the example of a logging system: the LogMixin class can be used in any other class
Step 2: Use the packaging tool to create
After you have a set of structures for your project, you should add files to the root directory of your project repository. This helps automate all release and versioning processes. Here is an example (source code:/nathan-toubiana/scitime/blob/master/）。
from setuptools import setup
from os import path
DIR = ((__file__))
INSTALL_PACKAGES = open((DIR, '')).read().splitlines()
with open((DIR, '')) as f:
README = ()
setup(
name='scitime',
packages=['scitime'],
description="Training time estimator for scikit-learn algorithms",
long_description=README,
long_description_content_type='text/markdown',
install_requires=INSTALL_PACKAGES,
version='0.0.2',
url='/nathan-toubiana/scitime',
author='Gabriel Lerner & Nathan Toubiana',
author_email='@',
keywords=['machine-learning', 'scikit-learn', 'training-time'],
tests_require=[
'pytest',
'pytest-cov',
'pytest-sugar'
],
package_data={
# include json and pkl files
'': ['*.json', 'models/*.pkl', 'models/*.json'],
},
include_package_data=True,
python_requires='>=3'
)
Examples of documents
A few notes:
- If your package has dependencies, an easy way to deal with them is to add the dependencies via the install_requires parameter in the configuration file (if the list is long, you can point to a file as before).
- If you want metadata to be downloaded (from the project repository) when anyone installs the package, you should add this metadata via the package_data parameter.
- For more information about the setup() function, see:/en/latest/
Note: Steps 3 through 6 are optional (but highly recommended), but if you want to publish your package right now, you can skip straight to step 7.
Step 3: Set up local tests and check test coverage
At this point it's not done, your project should also have unit tests. While there are many frameworks that can help you do this, an easy way is to use pytest. all tests should be in a dedicated folder (e.g. a folder called tests/ or testing). Place all the test files you need in this folder in order to include as much of your core code as possible. Here is an example of how to write unit tests. There is also a test file for SciTime here.
Once in place, you can test locally by running python -m pytest from the root of the project repository.
After creating tests, you should also be able to estimate coverage. This is important because you want to test as much of the amount of code in your project as possible (to minimize unexpected bugs).
Many frameworks can also be used to calculate coverage, for SciTime we use codecov. You can determine the minimum coverage threshold allowed by creating . file to determine the minimum coverage threshold allowed, and by creating a .coveragerc file to determine which files to include in the coverage analysis.
comment: false coverage: status: project: default: target: auto threshold: 10% patch: default: target: auto threshold: 10%
. Example of a file
[run] branch = True source = scitime include = */scitime/* omit = */_data.py */
Example of a coveragerc file
Step 4: Standardize syntax and code style
You also need to make sure that your code follows the PEP8 guidelines (i.e., that it has a standard style and is syntactically correct). Again, there are many tools to help you with this. Here we used flake8.
Step 5: Create a reasonable document
Now that your project is tested and structured, it's time to add a reasonable amount of documentation. The first step is to have a good readme file, which will show up on the root of your Github project repository. Once that's done, it's even better to add the following:
- Pull request and issue templates: When creating a new Pull request or issue, these files can provide you with templates for your descriptions based on your needs.
- Pull request creation step:/en/articles/creating-a-pull-request-template-for-your-repository
- issue Create step:/en/articles/manually-creating-a-single-issue-template-for-your-repository
- Pull request template:/nathan-toubiana/scitime/blob/master/.github/PULL_REQUEST_TEMPLATE.md
- issue template:/nathan-toubiana/scitime/tree/master/.github/ISSUE_TEMPLATE
- Contribution guide. The contribution guide should simply state how you would like external users to help you improve the package. see the contribution guide for Scitime:/nathan-toubiana/scitime/blob/master/.github/。
- For guidelines, see Scitime's guidelines:/nathan-toubiana/scitime/blob/master/.github/CODE_OF_CONDUCT.md
- Labels and descriptions (see screenshot below)
- Tags in readme files (Recommend a good article on how to use tags.)/how-to-use-badges-to-stop-feeling-like-a-noob-d4e6600d37d2）。
Since the readme file should be fairly comprehensive, there is usually a more detailed documentation. You can do this with sphinx and then manage the documentation on readthedocs. Files related to documentation are usually in the docs/ folder. sphinx and readthedocs related tutorials:
/en/stable/intro/。
Example of a project library with labels and instructions
Step 6: Create Continuous Integration
At this point, your project is not far from being release-ready. However, having to update documentation, run tests, and check for style and coverage after each commit can seem a bit overwhelming. Luckily, continuous integration (CI) can help. You can use a GitHub webhook to automate all of this after each commit. Here's a set of CI tools we use in SciTime:
- For running tests, we use travis ci and appveyor (for testing on Windows platforms). For Travis CI, in addition to setting up a webhook on the project repository, you must create a . file where you can not only run tests, but also upload updated coverage output and check styles and formats. By creating a file, appveyor can do the same.
- There are also dedicated webhooks for codecov and readthdocs.
language: python python: - "3.6" # command to install dependencies install: - pip install -r - pip install flake8 - pip install pytest-cov - pip install codecov # command to run tests script: - python -m pytest --cov=scitime - ./build_tools/flake_diff.sh after_success: - codecov
. Example of a file: note that for each commit, the test needs to be done along with a check for test coverage. But there is also a flake8 check (the logic is then defined in the flake_diff.sh file: /nathan-toubiana/scitime/blob/master/build_tools/flake_diff.sh)
environment: matrix: - PYTHON: "C:Python36-x64" install: # We need wheel installed to build wheels - "%PYTHON% -m pip install -r " - "%PYTHON% -m pip install pytest==3.2.1" build: off test_script: - "%PYTHON% -m pytest"
Example file: Here we only run the test
This will make the whole process of updating the project library much easier.
Sample commit history with webhook integration
Step 7: Create your first release and publication
At this point, the package you are about to release should look similar to the following:
your_package/ __init__.py your_module.py docs/ tests/ .coveragerc . LICENSE .github/ CODE_OF_CONDUCT.md PULL_REQUEST_TEMPLATE.md ISSUE_TEMPLATE/
Now it's time to release! The first thing to do is to create your first release on GitHub - this is in order to keep track of the project's status at a given point in time, and a new release needs to be created every time a version changes.
Once you're done, the only thing left to do is to publish the package. The most common platforms for distributing python packages are PyPI and Conda, and we'll describe how to distribute with both below:
- For PyPI, you first need to create an account and then perform some steps with twine: /pypi-publish-python-package/. This should be fairly simple, and Pypi also provides a test environment that you can use before actually deploying it.PyPI in general consists of creating the source code (python sdist) and uploading it using twine (twine upload dist/*). Once this is done, there should be a PyPI page corresponding to your package, and anyone should be able to install your package by running the pip command.
- For Conda, we recommend distributing your packages through conda forge, which is a community that helps you distribute and maintain packages through the conda channel. You can add packages to the community by following these steps:/#add_recipe, then you will be added to the conda forge Github organization and will be able to maintain your packages very easily, and then anyone can install your packages by running the conda command.
Done!
By now, your package should be out and available to anyone who wants to use it! While most of the work is done, you still need to maintain your project, and you'll need to make some updates: this roughly means changing the version every time you make a major change, creating a new release, and doing step 7 again.
This is the whole content of this article.