Contributing to Pygments¶
Thanks for your interest in contributing! Please read the following guidelines.
The code is distributed under the BSD 2-clause license. Contributors making pull requests must agree that they are able and willing to put their contributions under that license.
General contribution checklist¶
Run the test suite with
tox, and ensure it passes.
Make sure to add a test for your new functionality, and where applicable, write documentation. See below on how to test lexers.
Use the standard importing convention:
from token import Punctuation
How to add a lexer¶
To add a lexer, you have to perform the following steps:
Select a matching module under
pygments/lexers, or create a new module for your lexer class.
We encourage you to put your lexer class into its own module, unless it’s a very small derivative of an already existing lexer.
Next, make sure the lexer is known from outside the module. All modules in the
__all__. For example,
__all__ = ['BrainfuckLexer', 'BefungeLexer', ...]
Add the name of your lexer class to this list (or create the list if your lexer is the only class in the module).
Finally the lexer can be made publicly known by rebuilding the lexer mapping.
$ tox -e mapfiles
How lexers are tested¶
To add a new lexer test, create a file with just your code snippet
tests/snippets/<lexer_alias>/. Then run
tox -- --update-goldens <filename.txt> to auto-populate the
currently expected tokens. Check that they look good and check in the
Lexer tests are run with
tox, like all other tests. While
working on a lexer, you can also run only the tests for that lexer
tox -- tests/snippets/language-name/ and/or
tox -- tests/examplefiles/language-name/.
Running the test suite with
tox will run lexers on the test
inputs, and check that the output matches the expected tokens. If you
are improving a lexer, it is normal that the token output changes. To
update the expected token output for the tests, again use
tox -- --update-goldens <filename.txt>. Review the changes and
check that they are as intended, then commit them along with your
proposed code change.
Large test files should go in
tests/examplefiles. This works
snippets, but the token output is stored in a separate
file. Output can also be regenerated with
Goals & non-goals of Pygments¶
Pygments supports all supported Python versions as per the Python Developer’s Guide. Additionally, the default Python version of the latest stable version of RHEL, Ubuntu LTS, and Debian are supported, even if they’re officially EOL. Supporting other end-of-life versions is a non-goal of Pygments.
Pygments does not attempt to validate the input. Accepting code that is not legal for a given language is acceptable if it simplifies the codebase and does not result in surprising behavior. For instance, in C89, accepting // based comments would be fine because de-facto all compilers supported it, and having a separate lexer for it would not be worth it.