easy contributable internationalization process with sphinx (pycon apac 2015 in taiwan)
TRANSCRIPT
Easy contributable internationalization process with Sphinx
Takayuki Shimizukawa
Sphinx co-maintainer
Sphinx-users.jp
1
Who am I
@shimizukawa
1. Sphinx co-maintainer
2. Sphinx-users.jp
3. PyCon JP committee
BePROUD co, ltd.2
Early Bird Registration: 210 tickets.
Speaker Call for Proposals: until July 15th.
4
170 tickets gone
Agenda
1. Sphinx introduction & Setup for i18n
2. A easy way and Non easy ways to translate
3. Sphinx i18n feature
4. Automated translation process with several services
5. Tips, tricks, traps
5
What is Sphinx?
Sphinx is a documentation generator
Sphinx generates doc as several output formats from the reST text
markup
7
1. Sphinx introduction & Setup for i18n
Sphinx
reSTreSTreStructuredTe
xt(reST)
reST Parser
HTML Builder
ePub Builder
LaTeX Builder texlive
HTMLtheme
Favorite Editor
Many docs are written by SphinxFor examples
Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …
Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish
8
1. Sphinx introduction & Setup for i18n
Relation between Sphinx and Docutils
Sphinx based upon Docutils library
Docutils/Sphinx supports reStructuredText it called reST, it's an extensible markup(PEP287 / accepted at 2002)
9
1. Sphinx introduction & Setup for i18n
Sphinx
reST Parser
HTML Builder
ePub Builder
LaTeX Builder
HTMLtheme
docutils
Comparing Docutils and Sphinx
Docutils Sphinx
1. Single page
2. Link to others with full name
3. Cross reference in a page
4. Writers: html, latex, man, xml, ...
5. Standard reST markup specs
1. Multiple Pages
2. toctree to connect all pages under single tree structure
3. Cross reference over each other pages
4. Additional writers: html(w/ themes), pdf(latex), texinfo, epub, …
5. Additional markup specs:autodoc directive and so on
10
1. Sphinx introduction & Setup for i18n
SPHINX
docutils
HTML Builder HTML theme(Jinja2)
gettext Builder
*.pot
*.po
I18N
*.mo OmegaT
Pootle
Transifex
Translation Tools, Services
Favorite Editor
Sphinx i18n feature (built-in)
11
1. Sphinx introduction & Setup for i18n
reST Parser(directive / role)
doctree(Intermediat
e)
reSTreSTreStructuredText(reST)
Sphinx internals and externals
12
1. Sphinx introduction & Setup for i18n
SPHINX
docutils
Sp
hin
x拡
張d
irect
ive /
role
reSTreSTreStructuredText(reST)
Several formats
HTML Builder
EPUB Builder
LaTeX Builder
HTML theme(Jinja2)
code highlighter(Pygments)
man, texinfo, text, ... Builder
gettext Builder
XML, man, texinfo, text,winhelp, qthelp, ...
TeXLiveor else
*.pot
*.po
I18N
*.mo OmegaT
Pootle
Transifex
Translation Tools, Services
Favorite Editor
Sphinx Extension Builder
Custom Theme
epub3, docx, dash, ...
Sp
hin
x E
xte
nsi
on
dir
ect
ive /
role
reST Parser(directive / role)
doctree(Intermediat
e)
$ pip install sphinx
Support tools for translation
How to install Sphinx with i18n tools
To build a sphinx document
$ pip install sphinx-intl$ pip install transifex-client=0.8
13
1. Sphinx introduction & Setup for i18n
$ git clone https://github.com/shimizukawa/deepthought.git$ cd deepthought/doc$ make html...Build finished. The HTML pages are in _build/html.
"make html" command generates html files into _build/html.
make html once
14
1. Sphinx introduction & Setup for i18n
Files & setup conf.py$ tree /path/to/deepthought+- deep_thought| +- __init__.py| +- api.py| +- calc.py| +- utils.py+- doc| +- _build/| | +- html/ | +- _static/| +- _template/| +- conf.py| +- index.py| +- make.bat| +- Makefile+- setup.py
15
1. Sphinx introduction & Setup for i18n
Document source
Document build output
Target library for doc1. ...2. 3. language = 'zh_tw'4. locale_dirs = ['locale']5.
doc/conf.py
What is internationalization?
17
2. Easy contributable internationalization process with Sphinx
I18N
Not a easy way (example 1/3)
The manual are provided only in the HTML files
Rewriting reST files
19
2. Easy contributable internationalization process with Sphinx
Not a easy way (example 2/3)
The manual are generated fromreST files and
docstrings in the source files
Rewriting reST filesRewriting docstrins in the source files.
20
2. Easy contributable internationalization process with Sphinx
Not a easy way (example 3/3)
OK, we are engineers. We can use git!
Learn gitLearn GitHubgit clone to get the codeTranslation (Yes! This is what I want to
do!)git commitgit pullSometimes resolve conflictgit push
21
2. Easy contributable internationalization process with Sphinx
Not easy vs easy
Not a easy way Easy way
1. Learn git2. Learn GitHub3. git clone to get the
code4. Translation5. git commit, pull,
push6. Resolve conflict7. Build html your self
1. No git2. No Github3. No file4. Translation5. Update
Automatically6. No conflict7. You can get a HTML
output w/o hand-build.
22
2. Easy contributable internationalization process with Sphinx
Two i18n features of Sphinx
Output pot files:from reST
Input po files:to generate translated HTML
24
3. Sphinx i18n feature
reST
pot
reST
html
po
Translation flow
Generate pot
Translate pot into po
Generate Translated HTML
25
3. Sphinx i18n feature
reST
pot
reST
html
po
pot po
Translator
Pleasetranslate
Translate!Thanks for yourContribution!
#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr ""
msgid "For example:"msgstr ""
26
Output pot files 3. Sphinx i18n feature
$ make gettext...Build finished. The message catalogs are in _build/gettext.
$ ls _build/gettextapi.pot examples.pot generated.pot index.pot
generated.pot
reST
pot
Preparing po files to translate
doc+- _build/| +- gettext/| +- api.pot| +- examples.pot| +- generated.pot| +- index.pot+- locale/
doc+- _build/+- locale/ +- zh_tw/ | +- LC_MESSAGES/ | +- api.po | +- examples.po | +- generated.po | +- index.po +- ja/
copy and renamefor each langs
Tra
nsla
te th
em
Translator
pot po
27
3. Sphinx i18n feature
28
Preparing po files to translate
$ sphinx-intl update -p _build/gettext -l zh_twCreate: locale/zh_tw/LC_MESSAGES/api.poCreate: locale/zh_tw/LC_MESSAGES/examples.poCreate: locale/zh_tw/LC_MESSAGES/generated.poCreate: locale/zh_tw/LC_MESSAGES/index.po
At first, sphinx-intl copy pot files and rename them
pot po
sphinx-intl
$ make gettext$ sphinx-intl update -p _build/gettext -l zh_twNot Changed: locale/zh_tw/LC_MESSAGES/api.poUpdated: locale/zh_tw/LC_MESSAGES/examples.po +3, -1Not Changed: locale/zh_tw/LC_MESSAGES/generated.po
After change the document, sphinx-intl update differences
3. Sphinx i18n feature
Translate po files
#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr ""
generated.po
pot po
sphinx-intl
Translator
#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr "msgstr " 序列化 ``obj`` 要 JSON 格式 :class:`str`."
generated.po
Translate by using Vim, Emacs, OmegaT, ...
29
3. Sphinx i18n feature
Input po files
30
3. Sphinx i18n feature
reST
html
po
Translated
$ make html...Build finished. The HTML pages are in _build/html.
Big picture
31
3. Sphinx i18n feature
reST
pot
html
po
make gettext
sphinx-intl
Translator
make html
Doc Author
TranslationCatalog
Translatedcatalog
Entire process to translate sphinx docs
33
4. Automated translation process with several services
reST
pot
html
po
make gettext
sphinx-intl
make html
Doc Author
TranslationCatalog
Translatedcatalog
Translator
Translator
Translator
Autor / Translator
upload
Translator
clone
Translator
To Be
34
4. Automated translation process with several services
reST
pot
html
po
make gettext
sphinx-intl
make html
Doc Author
TranslationCatalog
Translatedcatalog
upload
Translators
To BeAutomated
To BeParallel
clone
Translation tool types
Vim / Emacs (Editors)Edit local filesTranslation support plugins are available.
OmegaT (Translation Tools)Edit local files.Translation support features.
Transifex (Services)Edit onlineTranslation support
features.
35
4. Automated translation process with several services
po
Translators
To BeParallel
Be Parallel by using Transifex
Transifex provides...As API:Upload potDownload po
As Web console:GlossaryTranslation memoryRecommendationAuto-translation
36
4. Automated translation process with several services
po
Translators
Parallel
pot
Upload
pot
Auto Update
sphinx-intltransifex-client
po transifex-client
Download
Translation on Transifex web console
37
4. Automated translation process with several services
Original Text
Translated Text(you should keep reST syntax)
Suggestions fromTranslation Memory (TM)
Original(pot)
Translation(po)
Copy orig to translation
Machine translation
Save
Review (if needed)
Translators
Parallel
1
2
4
3
5
6
To Be Automated
38
4. Automated translation process with several services
po
Translators
Parallel
pot
Upload
pot
Auto Update
sphinx-intltransifex-client
po transifex-client
Download
reST
html
make gettext
make html
Doc Author
upload
To BeAutomated
clone
To Be Automated
39
4. Automated translation process with several services
pot
Upload sphinx-intltransifex-client
po transifex-client
Download
reST
html
make gettext
make htmlupload
clone1
2 3
456
To BeAutomated
The procedure for automation
1. clone repository2. make gettext3. Upload pot4. Download po5. make html6. Upload html
40
4. Automated translation process with several services
pot
Upload sphinx-intltransifex-client
po transifex-client
Download
reST
html
make gettext
make htmlupload
clone1
2 3
456
To BeAutomate
d
1. pip install sphinx sphinx-intl transifex-client==0.82. git clone https://github.com/shimizukawa/deepthought.git3. cd deepthought/doc4. sphinx-intl create-transifexrc # create ~/.transifexrc5. sphinx-intl create-txconfig # create .tx/config6. make gettext7. sphinx-intl -p _build/gettext update-txconfig-resources
# update .tx/config8. tx push -s # push pot files to
transifex9. tx pull -l zh_tw # pull po files from
transifex10. make html SPHINXOPTS="-D language=zh_tw"
Shell commands for automation
run.sh
$ export SPHINXINTL_TRANSIFEX_USERNAME=mice$ export SPHINXINTL_TRANSIFEX_PASSWORD=42$ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7$ export SPHINXINTL_POT_DIR=_build/gettext$ run.sh
41
4. Automated translation process with several services
The drone.io
42
WebHook
Deploy
Clone repositoryRun shell script
The drone.io is a continuous integration service.
4. Automated translation process with several services
GitHub + drone.io + S3
43
GitHub
Amazon S3
Transifex 1. Clone repository2. make gettext.3. Upload pot4. Download po5. make html6. Upload html
4. Automated translation process with several services
2
1
3
Be Automated
44
pot
Upload sphinx-intltransifex-client
po transifex-client
Download
reST
html
make gettext
make htmlupload
clone1
2 3
456
To BeAutomated
UploadpotDownload
poupload
WebHookclone
1
5
6make html
make gettext2
3
4
1WebHook
Automated
4. Automated translation process with several services
Automated by drone.io
45
UploadpotDownload
poupload
WebHookclone
1
5
6make html
make gettext2
3
4
1WebHook
Automated
4. Automated translation process with several services
View from doc author
Doc Author doesn't require annoying procedure.
UpdateTranslation Source
Doc Author
Notify
See
Commit
46
4. Automated translation process with several services
View from doc translators
No gitNo fileNo conflictUpdate AutomaticallyThey can get a translated HTML output w/o hand-
build.
Translators
Parallel
See
TranslateTranslated Pages
47
4. Automated translation process with several services
The entire automated process
48
UpdateTranslation Source
Doc Author
Translators
Parallel
Notify
See
See
Publish
Translate
Commit
DownloadTranslations
Notify
Automated
4. Automated translation process with several services
TIP: Drone.io limits 15mins for a build
Drone.io limits 15mins for a build.Install from wheel package may help you
50
5. Tips, tricks, traps
1. curl -L -s https://example.com/wheelhouse.tgz | tar vzxf -
2. export PIP_FIND_LINKS=./wheelhouse3. pip install sphinx sphinx-intl transifex-client==0.8run.sh
ex. https://bitbucket.org/sphinxjp/docutils-translation/
TRAP: Version of transifex-client
transifex-client 0.11b3 is not stable (for me)Especially for Windows users
If you have encountered trouble with using newer version, please try:
51
5. Tips, tricks, traps
$ pip install "transifex-client=0.8"
Drone.io only list-up repositories which you have admin permission.
Prepare a empty repository to create a drone.io project.
52
TRICK: Preparing Drone.io project5. Tips, tricks, traps
Examples
Sphinx-1.3 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc13/admin
Sphinx-1.4 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc14/admin
Docutils doc for "ja" translationhttps://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin
53
5. Tips, tricks, traps