=============================================
Internationalization & Localization Framework
=============================================

Internationalization (also known as i18n) is the process of designing software so that it
can be easily (i.e. without any source code modifications) adapted to various languages
and regions. Localization (also known as L10n) is the process of adapting
internationalized software for a specific language or region (e.g. by translating text,
formatting date or time).

Basics
======

Locale class
------------

Instances of ``\Neos\Flow\I18n\Locale`` class are fundamental for the whole i18n and
L10n functionality. They are used to specify what language should be used for translation,
how date and time should be formatted, and so on. They can be treated as simple wrappers
for locale identifiers (like *de* or *pl_PL*). Many methods from the i18n framework accept
Locale objects as a optional parameter - if not provided, the default ``Locale`` instance
for a Flow installation will be used.

You can create a ``Locale`` object for any valid locale identifier (specified by RFC
4646), even if it is not explicitly meant to be supported by the current Flow
installation (i.e. there are no localized resources for this locale). This can be useful,
because Flow uses the `Common Locale Data Repository`_ (CLDR), so each Flow installation
knows how to localize numbers, date, time and so on to almost any language and region on
the world.

Additionally Flow creates a special collection of available ``Locale`` objects. Those can
either be configured explicitly in settings via ``Neos.Flow.i18n.availableLocales`` or they
are automatically generated by scanning the filesystem for any localized resources. You can
use the i18n service API to obtain these verified ``Locale`` objects.

.. note::

  You can configure which folders Flow should scan for finding available locales through
  the ``Neos.Flow.i18n.scan.includePaths`` setting. This is useful to restrict the scanning
  to specific paths when you have a big file structure in your package ``Resources``.
  You can also exclude folders through ``Neos.Flow.i18n.scan.excludePatterns``.
  By default the ``Public`` and ``Private/Translations`` folders, except 'node_modules',
  'bower_components' and any folder starting with a dot will be scanned.

Locales are organized in a hierarchy. For example, *en* is a parent of *en_US* which is a
parent of *en_US_POSIX*. Thanks to the hierarchical relation resources can be
automatically shared between related resources. For example, when you request a *foobar*
item for *en_US* locale, and it does not exist, but the item does exist for the *en*
locale, it will be used.

Common Locale Data Repository
-----------------------------

Flow comes bundled with the CLDR (*Common Locale Data Repository*). It's an Unicode
project with the aim to provide a systematic representation of data used for the
localization process (like formatting numbers or date and time). The i18n framework
provides a convenient API to access this data.

.. note::

  For now Flow covers only a subset of the CLDR data. For example, only the Gregorian
  calendar is supported for date and time formatting or parsing.

Detecting user locale
---------------------

The ``Detector`` class can be used for matching one of the available locales with locales
accepted by the user. For example, you can provide the *AcceptLanguage* HTTP header to the
``detectLocaleFromHttpHeader()`` method, which will analyze the header and return the best
matching ``Locale`` object. Also methods exist which accept a locale identifier or
template ``Locale`` object as a parameter and will return a best match.

Translating text
================

Translator class
----------------

The ``\Neos\Flow\I18n\Translator`` class is the central place for the translation
related functionality. Two translation modes can be used: translating by original label or
by ID. ``Translator`` also supports plural forms and placeholders.

For ``translateByOriginalLabel()`` you need to provide the original (untranslated, source)
message to be used for searching the translated message. It makes view templates more
readable.

``translateById()`` expects you to provide the systematic ID (like *user.notRegistered*)
of a message.

Both methods accept the following optional arguments:

* ``arguments`` - array of values which will replace corresponding placeholders
* ``quantity`` - integer or decimal number used for finding the correct plural form
* ``sourceName`` - name of source catalog to read the translation from.
* ``packageKey`` of the package the source catalog is contained in.

.. hint::

  Translation by label is very easy and readable, but if you ever want to change the
  original text, you are in trouble. The use of IDs gives you more flexibility in that
  respect.

  Another issue: some labels do not contain their context, like "Name". What is meant
  here, a person's name or a category label? This can be solved by using IDs that convey
  the context (note that both could be "Name" in the final output):

  * party.person.fullName
  * blog.category.name

  We therefore recommend to use ``translationById()`` in your code.

Plural forms
------------

The ``Translator`` supports plural forms. English has only two plural forms: *singular*
and *plurals* but the CLDR defines six plural forms: *zero*, *one*, *two*, *few*, *many*,
*other*. Though english only uses *one* and *other*, different languages use more forms
(like *one*, *few*, and *other* for Polish) or less forms (like only *other* for
Japanese).

Sets of rules exist for every language defining which plural form should be used for a
particular quantity of a noun. If no rules match, the implicit *other* rule is assumed.
This is the only form existing in every language.

If the catalogs with translated messages define different translations for particular
plural forms, the correct form can be obtained by the ``Translator`` class. You just need
to provide the ``quantity`` parameter - an integer or decimal number which specifies the
quantity of a noun in the sentence being translated.

Placeholders
------------

Translated messages (labels) can contain placeholders - special markers denoting he place
where to insert a particular value and optional configuration on how to format it.

The syntax of placeholders is very simple::

	{id[,formatter[,attribute1[,attribute2...]]]}

where:

* *id* is an integer used to index the arguments to insert
* *formatter* (optional) is a name of one of the Formatters_ to use for formatting the argument
  (if no formatter is given the provided argument will be cast to string)
* *attributes* (optional) are strings directly passed to the ``Formatter``. What they do
  depends on the concrete ``Formatter`` which is being used, but generally they are used
  to specify formatting more precisely.

Some examples:

.. code-block:: none

  {0}
  {0,number,decimal}
  {1,datetime,time,full}

1. The first example would output the first argument (indexing starts with 0), simply
   string-casted.
2. The second example would use ``NumberFormatter`` (which would receive one attribute:
   *decimal*) to format first argument.
3. The third example would output the second argument formatted by the
   ``DatetimeFormatter``, which would receive two attributes: *time* and *full* (they
   stand for format *type* and *length*, accordingly).

Formatters
----------

A ``Formatter`` is a class implementing the
``\Neos\Flow\I18n\Formatter\FormatterInterface``. A formatter can be used to format a
value of particular type: to convert it to string in locale-aware manner. For example, the
number *1234.567* would be formatted for French locale as *1 234,567*. It is possible to
define more elements than just the position and symbols of separators.

Together with placeholders, formatters provide robust and easy way to place formatted
values in strings. But formatters can be used directly (i.e. not in placeholder, but in
your class by injection), providing you more control over the results of formatting.

The following formatters are available in Flow by default:

``\Neos\Flow\I18n\Formatter\NumberFormatter``
  Formats integers or floats in order to display them as strings in localized manner.
  Uses patterns obtained from CLDR for specified locale (pattern defines such elements
  like minimal and maximal size of decimal part, symbol for decimal and group separator,
  etc.). You can indirectly define a pattern by providing format type (first additional
  attribute in placeholder) as *decimal* or *percent*. You can also manually set the
  pattern if you use this class directly (i.e. not in placeholder, but in your class by
  injection).
``\Neos\Flow\I18n\Formatter\DatetimeFormatter``
  Formats date and / or time part of PHP ``\DateTime`` object. Supports most of very
  extensive pattern syntax from CLDR. Has three format types: *date*, *time*, and
  *datetime*. You can also manually set the pattern if you use this class directly.

The following parameters are generally accepted by Formatters' methods:

* ``locale`` - formatting result depends on the localization, which is defined by provided
  ``Locale`` object
* ``formatLength`` (optional) - CLDR provides different formats for *full*, *long*,
  *medium*, *short*, and *default* length

Every formatter provides few methods, one for each format type. For example,
``NumberFormatter`` has methods ``formatDecimalNumber()`` - for formatting decimals and
integers - and ``formatPercentNumber()`` - for percentage (parsed value is automatically
multiplied by 100).

You can create your own formatter class which will be available for use in
placeholders. Just make sure your class implements the
``\Neos\Flow\I18n\Formatter\FormatterInterface``. Use the fully qualified class name,
without the leading backslash, as formatter name::

  {0,Acme\Foobar\Formatter\SampleFormatter}

Translation Providers
---------------------

Translation providers are classes implementing the ``TranslationProviderInterface``. They
are used by the ``Translator`` class for accessing actual data from translation files
(message catalogs).

A ``TranslationProvider``'s task is to read (understand) the concrete format of catalogs.
Flow comes with one translation provider by default: the ``XliffTranslationProvider``. It
supports translations stored in `XLIFF message catalogs`_, supports plural forms, and
both translation modes.

You can create and use your own translation provider which reads the file format you need,
like *PO*, *YAML* or even *PHP* arrays. Just implement the interface mentioned earlier and
use the *Objects.yaml* configuration file to set your translation provider to be injected
into the ``Translator``.
Please keep in mind that you have to take care of overrides yourself as this is within the
responsibilities of the translation provider.

Fluid ViewHelper
----------------

There is a ``TranslateViewHelper`` for Fluid. It covers all ``Translator``
features: it supports both translation modes, plural forms, and placeholders.
In the simplest case, the ``TranslateViewHelper`` can be used like this:

.. code-block:: xml

  <f:translate id="label.id"/>

It will output the translation with the ID "label.id" (corresponding to the
trans-unit id in XLIFF files).

The ``TranslateViewHelper`` also accepts all optional parameters the ``Translator`` does.

.. code-block:: xml

  <f:translate id="label.id" source="someLabelsCatalog" arguments="{0: 'foo', 1: '99.9'}"/>

It will translate the label using *someLabelsCatalog*. Then it will insert string casted
value "*foo*" in place of *{0}* and localized formatted *99.9* in place of *{1,number}*.

Translation by label is also possible:

.. code-block:: xml

  <f:translate>Unregistered User</f:translate>

It will output the translation assigned to *user.unregistered* key.

When the translation for particular label or ID is not found, value placed between
``<f:translate>`` and ``</f:translate>`` tags will be displayed.

Localizing validation error messages
====================================

Flow comes with a bundle of translations for all basic validator error messages. To make use
of these translations, you have to adjust your templates to make use of the ``TranslateViewHelper``.

.. code-block:: xml

  <f:validation.results for="{property}">
	<f:for each="{validationResults.errors}" as="error">
		{error -> f:translate(id: error.code, arguments: error.arguments, package: 'Neos.Flow', source: 'ValidationErrors')}
	</f:for>
  </f:validation.results>

If you want to change the validation messages, you can use your own package and override the labels there.
See the "XLIFF file overrides" section below.

.. tip::

	If you want to have different messages depending on the property, for example if you want to
	be more elaborate about specific validation errors depending on context, you could add the property
	to the translate key and provide your own translations.

Localizing resources
====================

Resources can be localized easily in Flow. The only thing you need to do is to put a
locale identifier just before the extension. For example, *foobar.png* can be localized as
*foobar.en.png*, *foobar.de_DE.png*, and so on. This works with any resource type when
working with the Flow ResourceManagement.

Just use the ``getLocalizedFilename()`` of the i18n ``Service`` singleton to obtain a
localized resource path by providing a path to the non-localized file and a ``Locale``
object. The method will return a path to the best matching localized version of the file.

Fluid ViewHelper
----------------

The ``ResourceViewHelper`` will by default use locale-specific versions of any resources
you ask for. If you want to avoid that you can disable that:

.. code-block:: xml

  {f:uri.resource(path: 'header.png', localize: 0)}


Validating and parsing input
============================

Validators
----------

A validator is a class implementing ``ValidatorInterface`` and is used by the Flow
Validation Framework for assuring correctness of user input. Flow provides two validators
that utilize i18n functionality:

``\Neos\Flow\Validation\Validator\NumberValidator``
  Validates decimal and integer numbers provided as strings (e.g. from user's input).
``\Neos\Flow\Validation\Validator\DateTimeValidator``
  Validates date, time, or both date and time provided as strings.

Both validators accept the following options: *locale*, *strictMode*, *formatType*,
*formatLength*.

These validators are working on top of the parsers API. Please refer to the Parsers_
documentation for details about functionality and accepted options.

Parsers
-------

A Parsers' task is to read user input of particular type (e.g. number, date, time), with
respect to the localization used and return it in a form that can be further processed.
The following parsers are available in Flow:

``\Neos\Flow\I18n\Parser\NumberParser``
  Accepts strings with integer or decimal number and converts it to a float.
``\Neos\Flow\I18n\Parser\DatetimeParser``
  Accepts strings with date, time or both date and time and returns an array with date /
  time elements (like day, hour, timezone, etc.) which were successfully recognized.

The following parameters are generally accepted by parsers' methods:

* *locale* - formatting results depend on the localization, which is defined by the
  provided ``Locale`` object
* *formatLength* - CLDR provides different formats for *full*, *long*, *medium*, *short*,
  and *default* length
* *strictMode* - whether to work in *strict* or *lenient* mode

Parsers are complement to Formatters_. Every parser provides a few methods, one for each
format type. Additionally each parser has a method which accepts a custom format
(pattern). You can provide your own pattern and it will be used for matching input. The
syntax of patterns depends on particular parser and is the same for a corresponding
formatter (e.g. ``NumberParser`` and ``NumberFormatter`` support the same pattern syntax).

Parsers can work in two modes: *strict* and *lenient*. In *strict* mode, the parsed value
has to conform the pattern exactly (even literals are important). In *lenient* mode, the
pattern is only a "base". Everything that can be ignored will be ignored, some
simplifications in the pattern are done. The parser tries to do it's best to read the
value.

XLIFF message catalogs
======================

The primary source of translations in Flow are XLIFF message catalogs. `XLIFF
<http://en.wikipedia.org/wiki/XLIFF>`_, the *XML Localisation Interchange File Format* is
an `OASIS-blessed <https://www.oasis-open.org/committees/xliff>`_ standard format for
translations.

.. note::

  In a nutshell an XLIFF document contains one or more ``<file>`` elements. Each file
  element usually corresponds to a source (file or database table) and contains the source
  of the localizable data. Once translated, the corresponding localized data for one, and
  only one, locale is added.

  Localizable data are stored in ``<trans-unit>`` elements. The ``<trans-unit>`` contains
  a ``<source>`` element to store the source text and a (non-mandatory) ``<target>``
  element to store the translated text.

File locations and naming
-------------------------

Each Flow package may contain any number of XLIFF files. The location for these files is
the *Resources/Private/Translations* folder. The files there can be named at will,
but keep in mind that *Main* is the default catalog name. The target locale is then added
as a directory hierarchy in between. The minimum needed to provide message catalogs for the
*en* and *de* locales thus would be:


.. code-block:: text

  Resources/
    Private/
      Translations/
        en/
          Main.xlf
        de/
          Main.xlf

XLIFF file creation
-------------------

It is possible to create initial translation files for a given language. With Flow command

.. code-block:: bash

  ./flow kickstart:translation --package-key Some.Package --source-language-key en --target-language-keys "de,fr"

the files for the default language *english* in the package *Some.Package* will be created as well as the translation
files for *german* and *french*. Already existing files will not be overwritten. Translations that do not yet exist are
generated based on the default language.

A minimal XLIFF file looks like this:

.. code-block:: xml

	<?xml version="1.0"?>
	<xliff version="1.2" xmlns="urn:oasis:names:tc:xliff:document:1.2">
		<file original="" source-language="da" target-language="fr" datatype="plaintext">
			<body>
				<trans-unit id="danish.celebrity">
					<source>Skarhøj</source>
					<target>Sarkosh</target>
				</trans-unit>
			</body>
		</file>
	</xliff>

If possible you should set up your editor to use the XLIFF 1.2 strict schema to validate
the files you are working on.

.. note::

  When using ``translationById()`` the framework will check the catalog's source language
  against the currently needed locale and use the ``<source>`` element if no ``<target>``
  element is found. This eliminates the need to duplicate messages in catalogs where
  source and target language are the same.

  But you may still ask yourself *do I really need to duplicate all the strings
  in XLIFF files?* The answer is *you should*. Using target allows to fix typos
  or change wording without breaking translation by label for all other languages.

.. admonition:: How to create meaningful XLIFF ids

  When using the recommended way of translating by id, it is even more important to use
  meaningful identifiers. Our suggestion is to group identifiers and use dot notation
  to build a hierarchy that is meaningful and intuitive::

    settings.account.keepLoggedIn
    settings.display.compactControls
    book.title
    book.author
    …

Labels may contain placeholders to be replaced with given arguments during
output. Earlier we saw an example use of the TranslateViewHelper:

.. code-block:: xml

	<f:translate id="label.id" arguments="{0: 'foo', 1: '99.9'}"/>

The corresponding XLIFF files will contain placeholders in the source and target strings:

.. code-block:: xml

	<trans-unit id="some.label">
		<source>Untranslated {0} and {1,number}</source>
		<target>Übersetzung mit {1,number} und {0}</target>
	</trans-unit>

As you can see, placeholders may be reordered in translations if needed.

Plural forms in XLIFF files
---------------------------

Plural forms are also supported in XLIFF. The following example defines a string
in two forms that will be used depending on the count:

.. code-block:: xml

	<group id="some.label" restype="x-gettext-plurals">
		<trans-unit id="some.label[0]">
			<source>This is only {0} item.</source>
			<target>Dies ist nur {0} Element.</target>
		</trans-unit>
		<trans-unit  id="some.label[1]">
			<source>These are {0} items.</source>
			<target>Dies sind {0} Elemente.</target>
		</trans-unit>
	</group>

Please be aware that the number of the available plural forms depends on the language!
If you want to find out which plural forms are available for a locale you can have a
look at *Neos.Flow/Resources/Private/I18n/CLDR/Sources/supplemental/plurals.xml*

XLIFF file translation
----------------------

To translate XLIFF files you can use any text editor, but translation is a lot easier
using one the available translation tools. To name two of them: Virtaal is a free and
open-source tool for offline use and Pootle (both from the `Translate Toolkit
<http://toolkit.translatehouse.org>`_ project) is a web-based
translation server.

XLIFF can also easily be converted to *PO* file format, edited by well known *PO* editors
(like *Poedit*, which supports plural forms), and converted back to *XLIFF* format. The
*xliff2po* and *po2xliff* tools from the *Translate Toolkit* project can convert without
information loss.

XLIFF file overrides
--------------------

As of Flow 4.2, XLIFF files are no longer solely identified by their location in the file system.
Instead, the ``<file>``'s ``product-name`` and ``original`` attributes are evaluated to the known
``package`` and ``source`` properties, if given. The actual location in the file system is only taken
into account if this information is missing and mainly for backwards compatibility.

This allows for an override mechanism, which comes in two levels:

Package-based overrides
^^^^^^^^^^^^^^^^^^^^^^^

Translation files are assembled by collecting labels along the composer dependency graph. This means that
as long a package depends (directly or indirectly) on another package, it can override or enrich the other package's
XLIFF files by using the other package's ``product-name`` and ``original`` values.

.. note::

  If you have trouble overriding another package's translations, please check your ``composer.json`` if you correctly
  declared that package as a dependency.

Global translations overrides
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In case translations are provided by another source than packages (e.g. via import from a third party system),
a global translation path can be declared and is evaluated with highest priority in that it overrides all translations
provided by packages. The default value for this is ``Data/Translations`` and can be changed via the configuration
parameter

.. code-block:: yaml

  Neos:
    Flow:
      i18n:
        globalTranslationPath: '%FLOW_PATH_DATA%Translations/'

**Example**

``Packages/Framework/Neos.Flow/Resources/Private/Translations/en/ValidationErrors.xlf``

.. code-block:: xml

  <file original="" product-name="Neos.Flow" source-language="en" datatype="plaintext">
    <body>
      <trans-unit id="1221551320" xml:space="preserve">
        <source>Only regular characters (a to z, umlauts, ...) and numbers are allowed.</source>
      </trans-unit>
    </body>
  </file>

``Packages/Application/Acme.Package/Resources/Private/Translations/en/ValidationErrors.xlf``

.. code-block:: xml

  <file original="ValidationErrors" product-name="Neos.Flow" source-language="en" datatype="plaintext">
    <body>
      <trans-unit id="1221551320" xml:space="preserve">
        <source>Whatever translation more appropriate to your domain comes to your mind.</source>
      </trans-unit>
    </body>
  </file>

.. note::

  In case of undetected labels, please make sure the ``original`` and ``product-name`` attributes are properly set
  (or not at all, if the file resides in the matching directory). Since these fields are used to detect overrides,
  they are now meaningful and cannot be filled arbitrarily any more.
