MediaWiki Custom Pygments: Difference between revisions

From Leo's Notes
This page was last edited on 19 May 2020, at 19:48.
No edit summary
 
(10 intermediate revisions by the same user not shown)
Line 1: Line 1:
To create a custom Pygments lexer for MediaWiki, do the following:
You can define your own Pygments lexer for MediaWiki. This allows custom syntax highlighting for languages that are not part of the default set of lexers in the built-in pygments binary.


# Clone the Pygments project in the {{code|extensions/SyntaxHighlight_GeSHi/pygments}} directory. This should replace the {{code|pygmentize}} binary with the python script.
== Installation ==
# Create your lexer in the {{code|extensions/SyntaxHighlight_GeSHi/pygments/pygments/lexers}} directory.
Pygments requires python. The system you're running MediaWiki should have access to either python2 or python3.
# Update the map by running {{code|make mapfiles}} in {{code|extensions/SyntaxHighlight_GeSHi/pygments}}
# Update the {{code|SyntaxHighlight_GeSHi.lexers.php}} file to contain the new lexer alias.


Replace the built-in pygments binary used by the SyntaxHighlight extension by following the following steps:
# Download the latest version of Pygments at https://bitbucket.org/birkenfeld/pygments-main.  (https://bitbucket.org/birkenfeld/pygments-main/get/2.4.2.tar.gz)
# Extract it to {{code|extensions/pygments}}
# Update {{code|LocalSettings.php}}, set {{code|1=$wgPygmentizePath = "$IP/extensions/pygments/pygmentize";}}.
To add a custom lexer:
# Create your lexer in the {{code|extensions/pygments/pygments/lexers}} directory
# Update the mapfile by running {{code|Make mapfiles}} in the {{code|extensions/pygments}} directory
# Update the list of lexers by running {{code|extensions/SyntaxHighlight_GeSHi/maintenance/updateLexerList.php}}
On CentOS 8, support for python2 ended and there is no longer 'python2' or 'python3' binaries. Instead, it now uses {{code|platform-python}}. Ensure that the {{code|pygments}} script references {{code|/usr/libexec/platform-python}} for pygments to work.
=== Custom Terminal Lexer ===
A simple lexer used on this wiki is for the Terminal output. It has the following lexer code which is just a couple regexes.
A simple lexer used on this wiki is for the Terminal output. It has the following lexer code which is just a couple regexes.


Line 12: Line 23:
"""
"""
     pygments.lexers.terminal
     pygments.lexers.terminal
     ------------------------
     16:41, 23 November 2018 (UTC)16:41, 23 November 2018 (UTC)16:41, 23 November 2018 (UTC)[[User:Leo|Leo]] ([[User talk:Leo|talk]]) 16:41, 23 November 2018 (UTC)


     Lexers for terminal output
     Lexers for terminal output
Line 41: Line 52:
         'root': [
         'root': [
             (r'^##.*\n', Comment),
             (r'^##.*\n', Comment),
             (r'^(\[*[a-zA-Z0-9\@:~\/\-\]*]*[\$#]{1})', Name, 'command'),
             (r'^(\[*[a-zA-Z0-9\@:~\/\-\]*]*[\$#%]{1})', Name, 'command'),
            (r'^(\[+[a-zA-Z0-9\@:~\/\-\]+]*[\$#%]?)', Name, 'command'),
             (r'.*\n', Text)
             (r'.*\n', Text)
         ],
         ],
Line 50: Line 62:
                 (r'\n', String, '#pop')
                 (r'\n', String, '#pop')
         ]
         ]
    }
}}
== Troubleshooting ==
=== Issue Updating Mapfiles ===
If you get the following error while updating the mapfiles and you are using Python 2:
{{highlight|lang=terminal|code=
# make mapfiles
...
Traceback (most recent call last):
  File "_mapping.py", line 491, in <module>
    module = __import__(module_name, None, None, [''])
  File "../../pygments/lexers/make.py", line 18, in <module>
    from pygments.lexers.shell import BashLexer
  File "../../pygments/lexers/shell.py", line 140
    EXTRA_KEYWORDS = {'srun'}
                            ^
SyntaxError: invalid syntax
make: *** [mapfiles] Error 1
}}
Edit the {{code|lexers/shell.py}} and comment out line containing {{code|1=EXTRA_KEYWORDS = {'srun'} }}.
== Custom Terminal Lexer Issues ==
=== Support Anaconda environment prefixes ===
{{highlight|lang=terminal|code=
(base) $ conda deactivate
}}
=== Support '>' prompts. ===
{{highlight|lang=terminal|code=
> something in python or Rlang console
something
}}
=== Shell prompt shows partially as command if it contains a space ===
{{highlight|lang=terminal|code=
[root@kube ~]# kubectl get services --namespace=kube-system kubernetes-dashboard
NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP  PORT(S)        AGE
kubernetes-dashboard  ClusterIP  10.108.158.128  <none>        443/TCP        2m36s
}}
=== Support heredoc ===
{{highlight|lang=terminal|code=
# cat <<EOF > /somewhere/something.txt
[something something]
this should all be grey text.
everything until the line with
a single EOF.
This should be grey too.
EOF
# this should be a command
## and comments should work
# cat /somewhere/something.txt
[something something]
...
}}
=== Fixed: Shell prompt's leading space should not be part of the command. ===
{{highlight|lang=terminal|code=
# something something
something
}}


    }
=== Fixed: The ESXi default shell and zsh {{code|%}} shell are not highlighting properly  ===
 
{{highlight|lang=terminal|code=
[root@server:/vmfs/volumes/ee8ff6f0-af4dbc98] sh ./ghettoVCB.sh -f backup_vms
 
leo@nas:/data/vm% ls
backup_vms  esxi-backup/  ghettoVCB.conf  ghettoVCB.sh*
}}
 
=== Fixed: Empty # will cause the next # to be a comment, even if they are on different lines: ===
{{highlight|lang=terminal|code=
# gpart create -s GPT /dev/ada1
# gpart add -b 2048 -s 1953113520 -t freebsd-zfs -l disk01 /dev/ada1
#
# gpart create -s GPT /dev/ada2
# gpart add -b 2048 -s 1953113520 -t freebsd-zfs -l disk02 /dev/ada2
#
# gpart create -s GPT /dev/ada3
# gpart add -b 2048 -s 1953113520 -t freebsd-zfs -l disk03 /dev/ada3
}}
}}




[[Category:Code]]
[[Category:Code]]

Latest revision as of 19:48, 19 May 2020

You can define your own Pygments lexer for MediaWiki. This allows custom syntax highlighting for languages that are not part of the default set of lexers in the built-in pygments binary.

Installation

Pygments requires python. The system you're running MediaWiki should have access to either python2 or python3.

Replace the built-in pygments binary used by the SyntaxHighlight extension by following the following steps:

  1. Download the latest version of Pygments at https://bitbucket.org/birkenfeld/pygments-main. (https://bitbucket.org/birkenfeld/pygments-main/get/2.4.2.tar.gz)
  2. Extract it to extensions/pygments
  3. Update LocalSettings.php, set $wgPygmentizePath = "$IP/extensions/pygments/pygmentize";.

To add a custom lexer:

  1. Create your lexer in the extensions/pygments/pygments/lexers directory
  2. Update the mapfile by running Make mapfiles in the extensions/pygments directory
  3. Update the list of lexers by running extensions/SyntaxHighlight_GeSHi/maintenance/updateLexerList.php

On CentOS 8, support for python2 ended and there is no longer 'python2' or 'python3' binaries. Instead, it now uses platform-python. Ensure that the pygments script references /usr/libexec/platform-python for pygments to work.

Custom Terminal Lexer

A simple lexer used on this wiki is for the Terminal output. It has the following lexer code which is just a couple regexes.

# -*- coding: utf-8 -*-
"""
    pygments.lexers.terminal
    16:41, 23 November 2018 (UTC)16:41, 23 November 2018 (UTC)16:41, 23 November 2018 (UTC)[[User:Leo|Leo]] ([[User talk:Leo|talk]]) 16:41, 23 November 2018 (UTC)

    Lexers for terminal output

    :copyright:  Leo's stuff
    :license:  Leo's stuff
"""

import re

from pygments.lexer import RegexLexer, bygroups
from pygments.token import Text, Comment, Operator, Keyword, Name, String, \
    Number, Punctuation, Error

__all__ = ['TerminalLexer']

class TerminalLexer(RegexLexer):
    """
    For terminal output

    .. versionadded:: 0.1
    """

    name = 'terminal'
    aliases = ['terminal']

    tokens = {
        'root': [
            (r'^##.*\n', Comment),
            (r'^(\[*[a-zA-Z0-9\@:~\/\-\]*]*[\$#%]{1})', Name, 'command'),
            (r'^(\[+[a-zA-Z0-9\@:~\/\-\]+]*[\$#%]?)', Name, 'command'),
            (r'.*\n', Text)
        ],

        'command': [
                (r'[^\\\n]+', String),
                (r'\\\n', String),
                (r'\n', String, '#pop')
        ]
    }

Troubleshooting

Issue Updating Mapfiles

If you get the following error while updating the mapfiles and you are using Python 2:

# make mapfiles
...
Traceback (most recent call last):
  File "_mapping.py", line 491, in <module>
    module = __import__(module_name, None, None, [''])
  File "../../pygments/lexers/make.py", line 18, in <module>
    from pygments.lexers.shell import BashLexer
  File "../../pygments/lexers/shell.py", line 140
    EXTRA_KEYWORDS = {'srun'}
                            ^
SyntaxError: invalid syntax
make: *** [mapfiles] Error 1

Edit the lexers/shell.py and comment out line containing EXTRA_KEYWORDS = {'srun'}.



Custom Terminal Lexer Issues

Support Anaconda environment prefixes

(base) $ conda deactivate

Support '>' prompts.

> something in python or Rlang console
something

Shell prompt shows partially as command if it contains a space

[root@kube ~]# kubectl get services --namespace=kube-system kubernetes-dashboard
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   ClusterIP   10.108.158.128   <none>        443/TCP         2m36s

Support heredoc

# cat <<EOF > /somewhere/something.txt
[something something]
this should all be grey text.
everything until the line with
a single EOF.
This should be grey too.
EOF
# this should be a command
## and comments should work
# cat /somewhere/something.txt
[something something]
...


Fixed: Shell prompt's leading space should not be part of the command.

# something something
something

Fixed: The ESXi default shell and zsh % shell are not highlighting properly

[root@server:/vmfs/volumes/ee8ff6f0-af4dbc98] sh ./ghettoVCB.sh -f backup_vms

leo@nas:/data/vm% ls
backup_vms  esxi-backup/  ghettoVCB.conf  ghettoVCB.sh*

Fixed: Empty # will cause the next # to be a comment, even if they are on different lines:

# gpart create -s GPT /dev/ada1
# gpart add -b 2048 -s 1953113520 -t freebsd-zfs -l disk01 /dev/ada1
#
# gpart create -s GPT /dev/ada2
# gpart add -b 2048 -s 1953113520 -t freebsd-zfs -l disk02 /dev/ada2
#
# gpart create -s GPT /dev/ada3
# gpart add -b 2048 -s 1953113520 -t freebsd-zfs -l disk03 /dev/ada3