#2 bring harmonization in total source strings and messages to produce global statistics
Merged by jibecfed. Opened by jibecfed.
stats  into  master

Download 2.patch

problem

we had no global statistics, and while doing it, I realized that for some files, the total source strings were inconsistent. I assume this is a lack of updates.

solution

  1. Get the dirname of the translation files
  2. Remove from dirname the language if present
  3. Use the couple (['src', 'dirname']) to calculate the max source messages and words
  4. Update current data
  5. Display a summary

result

 * CSV loaded  we now have 44298 rows
 * * duplicated headers are removed  we now have 43899 rows
 * * duplicated po files are removed  we now have 43588 rows
 * Deduplication is done  we now have 43588 rows
 * * remove pot files  we now have 43381 rows
 * * remove gmo files  we now have 39208 rows
 * * remove files with 'totalMessage'=0  we now have 39148 rows
 * Removal is done  we now have 39148 rows
 * * remove if lang endswith encoding values  we now have 39132 rows
 * * remove if lang contains a point  we now have 39007 rows
 * bcp47 data are guessed  we now have 39007 rows
 * * remove if len(territory)>2  we now have 38881 rows
 * * remove if len(language)>3  we now have 26288 rows
 * * remove if language.isdigit()  we now have 23984 rows
 * bcp47 data are cleaned  we now have 23984 rows
 * cldr data are added  we now have 23984 rows
 * * remove languages non existing in CLDR  we now have 23087 rows
 * cldr data are cleaned  we now have 23087 rows
 * cldr consistency are done  we now have 23087 rows
 * data harmonization are done  we now have 23087 rows
We have:
  * number of upstream sources: 265
  * number of distinct lang-script-territory: 268
  * number of languages: 175
  * translation files: 23087
This represents:
  * Total messages: 205663
  * Total words: 1288494

FYI: @petersen @suanand

Pull-Request has been merged by jibecfed

Metadata