Preparing for i18n right now

dachary · July 1, 2017, 11:16pm

Using the same method as phpmyadmin the SecureDrop documentation can also be translated via weblate so it is all in the same place (website strings and localized documentation). Not sure I understand how they do it just yet but it looks simple.

dachary · July 2, 2017, 9:31am

Sent a mail to michal@cihar.com asking for the scripts to create the POT from RST files. They should be in https://github.com/phpmyadmin/localized_docs/ but I can’t find them. He replied they are created with sphinx-build -b gettext… which seems fairly straightforward.

dachary · July 2, 2017, 2:12pm

And the first translation was pushed to git by weblate. Looks like this is actually going to work

dachary · July 2, 2017, 5:42pm

I’m optimistic this will work out fine, it has been a good week-end

create a topic branch for i18n with a patch series made of minimal commit, each passing the CI : https://github.com/dachary/securedrop/commits/wip-i18n-base
work with this topic branch to provide translations in at least one language : TBD
send a PR for the first commit in the patch series and wait for the SecureDrop team to merge it on their own time. And submit again, one commit, when it is merged. : https://github.com/freedomofpress/securedrop/pull/1924

I’m not too worried about the translation workflow now, weblate is good enough.

dachary · July 4, 2017, 5:29pm

@heartsucker I was reading the documentation to better understand best_match and it looks like the behaviour is somewhat imperfect in werkzeug. Combined with the undocumented assumption of flask-babel about the translation directory it got me worried about the maintenance of i18n in flask. Do you have an opinion about that ?

heartsucker · July 5, 2017, 7:35am

This is what I’ve done in the past. I can’t remember what my old PR did.

github.com

berlin-leaks/berlinleaks.org/blob/master/website/app.py#L55-L67


@babel.localeselector
def app_localeselector():
if request.path.startswith('/en/'):
    return 'en'
elif request.path.startswith('/de/'):
    return 'de'
else:
    keys = app_config.locales.keys()


    for locale in map(lambda x: x[0].lower(), sorted(request.accept_languages, key=lambda x: x[1], reverse=True)):
        if locale[:2] in keys:
            return locale[:2]
    return 'de'

heartsucker · July 5, 2017, 7:36am

We could store the locale in the session for SD or use a query string param. The initial would default to English with someone clicking the locale they want to switch it.

dachary · July 5, 2017, 10:03am

I see you manually parsed the Accept-Language. At least there is no complicated match going on and problems are easier to diagnose

As a first minimal step to load the locale, I used best_match. I’ll most likely revert to your method if it gives me trouble. The user has no control over it, this is very limited. I did not want to merge the commit focusing on loading the locale with the commit that implements a user interface to select the language (via query arguments or via a select in the HTML page).

Also, instead of explicitly listing the locales in the config file, I chose to only define LOCALE which set the default. The list of available locales is defined by exploring the content of the translations directory. My motivation was primarily to reduce the change of config.py.example to the strict minimum because it needs to be carefully upgraded / verified. If there was a list of available locales, including human readable labels, it would make it more complicated. Both can be defined elsewhere (the human readable strings are part of the GUI, the list of available translations can be read from the directory, restricting the list of translations displayed to the user can also be part of the GUI configuration).

This is not carved in stone, reason why I did not elaborate more in the commit messages. Please let me know if you have concerns

heartsucker · July 5, 2017, 10:59am

Yes, but you’d want to have a mapping of en_US to English (American) or whatever for the buttons a user clicks. Don’t put that in config.py, but somewhere else that’s hard coded. The app I pulled it from had a config.py that is totally unlike what SD uses.

dachary · July 5, 2017, 2:18pm

The minimal version of i18n works.

The next step is to go over each source/template file and gettextize it. It may be good to l10n in parallel, just to see how manageable the fixed size font / layout is and what compromises it requires. It would be better to run into blockers before all files are processed. Since @heartsucker already did the work for German not too long ago I’m not very worried.

It would go like this:

pick a source/template
apply the gettext changes from https://github.com/freedomofpress/securedrop/pull/1103/files
./manage.py translate --extract-update # on the development VM
commit & push to the commit to the wip-i18n-test which is what weblate is watching
translate at http://weblate.securedrop.club
git pull wip-i18n-test # which is where weblate merges the translated .po
./manage.py translate --compile # on the development VM
navigate to manually check the strings do not break the layout

and repeat until it’s done. In firefox Preference -> Content -> Languages can be used to select the language by moving the prefered language in front of the list. It changes the Accept-Language header accordingly and source/journalist pages will respond to the change on every request.

dachary · July 10, 2017, 11:29pm

While making progress with the french translation of the journalist interface it is most useful to verify the output using screenshots automatically generated (see the implementation). Here is an example that show where the display is not right (Edition and Suppression are too close to each other):

It was generated with:

ACCEPT_LANGUAGES=fr-fr pytest -v tests/pages-layout/test_journalist.py

dachary · July 12, 2017, 8:09am

@heartsucker your original commit is a great time saver. It is so much easier to follow your footsteps than to figure out how to proceeed It also turns out the majority of the templates gettext() are still valid too, which is a nice suprise. Moving on to javascript i18n now and I would not know where to begin if it was not for the work you did.

heartsucker · July 12, 2017, 9:21am

Hey thanks. Yeah, that was some fun little hack to work around the embedded strings in the JS.

dachary · July 12, 2017, 10:21am

I suspect it will need to be reworked but it does the job right now Here is how it looks now if you’re curious: https://github.com/dachary/securedrop/commit/e9783582c539e194fb9e751db37601e79899fc88 . To ease testing I derived the functional tests that create a text file containing the translated alert (cannot conveniently be captured by a screenshot): https://github.com/dachary/securedrop/commit/dbd9652c46f60a7b5b761969627410c1ecd1cf02#diff-52101396e6b39f1de6c1d3c6b3409888R301

I also simplified the workflow so it creates all screenshots / txt files for supported locales with one call:

pytest -v tests/pages-layout/test_journalist.py

dachary · July 13, 2017, 8:01am

All journalist templates localized in french. Moving on the source templates now. Here is how they look now. It took longer than I thought. It should be a little quicker for the source templates if all the i18n is complete (I think it is but … ;-). Here is how the automated screenshots look like. There are 26 different screenshots, these show how they vary depending on the application state / javascript enabled or not etc.

dachary · July 16, 2017, 1:44pm

The French translation is complete. And the German translation was resurrected from @heartsucker past efforts: about 50% of the ~200 strings are still good.

It is probably time to call for new translators. The weblate workflow is good. Hopefully good enough to be understandable with a one page documentation targeting a new translator.

byeskille · July 19, 2017, 7:56am

Thank you so much for ramping up the speed on getting translations to SecureDrop!

I’m involved in running the SecureDrop instance for the Norwegian Broadcasting Corp (NRK), and as most of our readers use Norwegian we have been discussing how we could help out in order to make translations happen.

So could of course at least contribute with translations to Norwegian. I could also try to get some more people to help out in order to spread the burden and also get reviewers of the translated strings.

Maybe a stupid question: Has it been discussed if the interfaces should set locale based on browser/OS settings alone, or also letting the user choose translation (i.e. bilingual sites)?

dachary · July 19, 2017, 8:02am

Excellent question It has been discussed in the past indeed. In the current implementation, for the sake of simplicity, it is based on the Browser language preference as sent in the Accept-Languages header. It would however be convenient to let the users choose the language they want. Adding a drop down menu with supported languages to the journalist & source interface, just what you would expect on any internationalized web site, seems the best option.

dachary · July 19, 2017, 8:50am

A post was split to a new topic: Norwegian translation

heartsucker · July 19, 2017, 12:52pm

@dachary I think the best choice would be to use the following

Use a configured default locale
Override that with Accept-Language
Override that with a manual selection (set in the cookie)

I think #3 is super important because if there are multiple translations, and something is ambiguous, the user should be able to switch to another language to help them figure out what was actually meant. Also, it drives me INSANE when websites force me to use the language they think I want. The selectors I used in my old SD PR provide this mechanism.