Preparing for i18n right now

dachary · July 1, 2017, 9:24am

Bonjour,

My understanding of the discussions about i18n during the past two weeks is that the SecureDrop team is willing to add i18n as soon as possible. They are however focused on publishing the 0.4 release which is going to feature freeze after the last pending PRs are merged. It means the release/0.4 branch will be created next week and we can start sending i18n pull requests against the develop branch without disturbing the stable release.

Since the SecureDrop team attention won’t be focused on i18n and their bandwidth will be limited, we (as a community), need a strategy to avoid wasting good work. This is not a problem specific to SecureDrop, it is pretty much the same with every project when trying to introduce a significant change that is not the top priority of the core team driving the development.

I think it would be enough to make progress as follows:

create a topic branch for i18n with a patch series made of minimal commit, each passing the CI
work with this topic branch to provide translations in at least one language
send a PR for the first commit in the patch series and wait for the SecureDrop team to merge it on their own time. And submit again, one commit, when it is merged.
rebase at least once a day the patch series so it stays up to date with the develop branch

I realize this is more work on our part (us the community but we’re not the bottleneck and there are many of us. I don’t mind if things take a very long time to land upstream, as long as we have a strategy ensuring it does not go to waste.

What do you think ?

dachary · July 1, 2017, 2:23pm

After discussing with heartsucker today I created the tiniest pull request I could think of at https://github.com/freedomofpress/securedrop/pull/1924 (adding a requirement ;-). Hopefully that will be merged for 0.4.1 ?

Next step will be to split heartsucker initial pull request ( https://github.com/freedomofpress/securedrop/pull/1103/files ) into smaller ones and test it manually.

dachary · July 1, 2017, 10:02pm

Round trip with weblate almost complete: import successful to http://weblate.securedrop.club . Now for export there needs to be a git repository to push edits.

dachary · July 1, 2017, 11:15pm

Weblate has a cli:

$ pip3 install wcl
$ wlc --url http://weblate.securedrop.club/api/ list-projects
name: SecureDrop
slug: securedrop
source_language: en
url: http://weblate.securedrop.club/api/projects/securedrop/
web: http://securedrop.org
web_url: http://example.com/projects/securedrop/

dachary · July 1, 2017, 11:16pm

Using the same method as phpmyadmin the SecureDrop documentation can also be translated via weblate so it is all in the same place (website strings and localized documentation). Not sure I understand how they do it just yet but it looks simple.

dachary · July 2, 2017, 9:31am

Sent a mail to michal@cihar.com asking for the scripts to create the POT from RST files. They should be in https://github.com/phpmyadmin/localized_docs/ but I can’t find them. He replied they are created with sphinx-build -b gettext… which seems fairly straightforward.

dachary · July 2, 2017, 2:12pm

And the first translation was pushed to git by weblate. Looks like this is actually going to work

dachary · July 2, 2017, 5:42pm

I’m optimistic this will work out fine, it has been a good week-end

create a topic branch for i18n with a patch series made of minimal commit, each passing the CI : https://github.com/dachary/securedrop/commits/wip-i18n-base
work with this topic branch to provide translations in at least one language : TBD
send a PR for the first commit in the patch series and wait for the SecureDrop team to merge it on their own time. And submit again, one commit, when it is merged. : https://github.com/freedomofpress/securedrop/pull/1924

I’m not too worried about the translation workflow now, weblate is good enough.

dachary · July 4, 2017, 5:29pm

@heartsucker I was reading the documentation to better understand best_match and it looks like the behaviour is somewhat imperfect in werkzeug. Combined with the undocumented assumption of flask-babel about the translation directory it got me worried about the maintenance of i18n in flask. Do you have an opinion about that ?

heartsucker · July 5, 2017, 7:35am

This is what I’ve done in the past. I can’t remember what my old PR did.

github.com

berlin-leaks/berlinleaks.org/blob/master/website/app.py#L55-L67


@babel.localeselector
def app_localeselector():
if request.path.startswith('/en/'):
    return 'en'
elif request.path.startswith('/de/'):
    return 'de'
else:
    keys = app_config.locales.keys()


    for locale in map(lambda x: x[0].lower(), sorted(request.accept_languages, key=lambda x: x[1], reverse=True)):
        if locale[:2] in keys:
            return locale[:2]
    return 'de'

heartsucker · July 5, 2017, 7:36am

We could store the locale in the session for SD or use a query string param. The initial would default to English with someone clicking the locale they want to switch it.

dachary · July 5, 2017, 10:03am

I see you manually parsed the Accept-Language. At least there is no complicated match going on and problems are easier to diagnose

As a first minimal step to load the locale, I used best_match. I’ll most likely revert to your method if it gives me trouble. The user has no control over it, this is very limited. I did not want to merge the commit focusing on loading the locale with the commit that implements a user interface to select the language (via query arguments or via a select in the HTML page).

Also, instead of explicitly listing the locales in the config file, I chose to only define LOCALE which set the default. The list of available locales is defined by exploring the content of the translations directory. My motivation was primarily to reduce the change of config.py.example to the strict minimum because it needs to be carefully upgraded / verified. If there was a list of available locales, including human readable labels, it would make it more complicated. Both can be defined elsewhere (the human readable strings are part of the GUI, the list of available translations can be read from the directory, restricting the list of translations displayed to the user can also be part of the GUI configuration).

This is not carved in stone, reason why I did not elaborate more in the commit messages. Please let me know if you have concerns

heartsucker · July 5, 2017, 10:59am

Yes, but you’d want to have a mapping of en_US to English (American) or whatever for the buttons a user clicks. Don’t put that in config.py, but somewhere else that’s hard coded. The app I pulled it from had a config.py that is totally unlike what SD uses.

dachary · July 5, 2017, 2:18pm

The minimal version of i18n works.

The next step is to go over each source/template file and gettextize it. It may be good to l10n in parallel, just to see how manageable the fixed size font / layout is and what compromises it requires. It would be better to run into blockers before all files are processed. Since @heartsucker already did the work for German not too long ago I’m not very worried.

It would go like this:

pick a source/template
apply the gettext changes from https://github.com/freedomofpress/securedrop/pull/1103/files
./manage.py translate --extract-update # on the development VM
commit & push to the commit to the wip-i18n-test which is what weblate is watching
translate at http://weblate.securedrop.club
git pull wip-i18n-test # which is where weblate merges the translated .po
./manage.py translate --compile # on the development VM
navigate to manually check the strings do not break the layout

and repeat until it’s done. In firefox Preference -> Content -> Languages can be used to select the language by moving the prefered language in front of the list. It changes the Accept-Language header accordingly and source/journalist pages will respond to the change on every request.

dachary · July 10, 2017, 11:29pm

While making progress with the french translation of the journalist interface it is most useful to verify the output using screenshots automatically generated (see the implementation). Here is an example that show where the display is not right (Edition and Suppression are too close to each other):

It was generated with:

ACCEPT_LANGUAGES=fr-fr pytest -v tests/pages-layout/test_journalist.py

dachary · July 12, 2017, 8:09am

@heartsucker your original commit is a great time saver. It is so much easier to follow your footsteps than to figure out how to proceeed It also turns out the majority of the templates gettext() are still valid too, which is a nice suprise. Moving on to javascript i18n now and I would not know where to begin if it was not for the work you did.

heartsucker · July 12, 2017, 9:21am

Hey thanks. Yeah, that was some fun little hack to work around the embedded strings in the JS.

dachary · July 12, 2017, 10:21am

I suspect it will need to be reworked but it does the job right now Here is how it looks now if you’re curious: https://github.com/dachary/securedrop/commit/e9783582c539e194fb9e751db37601e79899fc88 . To ease testing I derived the functional tests that create a text file containing the translated alert (cannot conveniently be captured by a screenshot): https://github.com/dachary/securedrop/commit/dbd9652c46f60a7b5b761969627410c1ecd1cf02#diff-52101396e6b39f1de6c1d3c6b3409888R301

I also simplified the workflow so it creates all screenshots / txt files for supported locales with one call:

pytest -v tests/pages-layout/test_journalist.py

dachary · July 13, 2017, 8:01am

All journalist templates localized in french. Moving on the source templates now. Here is how they look now. It took longer than I thought. It should be a little quicker for the source templates if all the i18n is complete (I think it is but … ;-). Here is how the automated screenshots look like. There are 26 different screenshots, these show how they vary depending on the application state / javascript enabled or not etc.

dachary · July 16, 2017, 1:44pm

The French translation is complete. And the German translation was resurrected from @heartsucker past efforts: about 50% of the ~200 strings are still good.

It is probably time to call for new translators. The weblate workflow is good. Hopefully good enough to be understandable with a one page documentation targeting a new translator.