#1 add data cleanup and cldr
Merged by jibecfed. Opened by jibecfed.
pandas  into  master

Download 1.patch

problem

the output of ./build.py is difficult to handle and contains many false positives

solution

the data cleanups and manipulation are now automated using the python pandas library
the CLDR data are used to add english names

comment

$ ./build_stats.py 
 * CSV loaded  we now have 44298 rows
 * * duplicated headers are removed  we now have 43899 rows
 * * duplicated po files are removed  we now have 43588 rows
 * Deduplication is done  we now have 43588 rows
 * * remove pot files  we now have 43381 rows
 * * remove gmo files  we now have 39208 rows
 * * remove files with 'totalMessage'=0  we now have 39148 rows
 * Removal is done  we now have 39148 rows
 * * remove if lang endswith encoding values  we now have 39132 rows
 * * remove if lang contains a point  we now have 39007 rows
 * bcp47 data are guessed  we now have 39007 rows
 * * remove if len(region)>2  we now have 38881 rows
 * * remove if len(language)>3  we now have 26288 rows
 * * remove if language.isdigit()  we now have 23984 rows
 * bcp47 data are cleaned  we now have 23984 rows
 * cldr data are added  we now have 23984 rows
 * * remove languages non existing in CLDR  we now have 23087 rows
 * cldr data are cleaned  we now have 23087 rows

2 new commits added

  • add build_stats in readme
  • add language, region and territory name from cldr

2 new commits added

  • add f30 results
  • use results folder

@petersen @suanand : can you please look at the build_stats.py and give your feedback?

for example, I remove files ending with ".big5", ".cp936", etc. Is it indeed encoding only?

finale result file is: results/f30/3.result.csv

@petersen @suanand : can you please look at the build_stats.py and give your feedback?

for example, I remove files ending with ".big5", ".cp936", etc. Is it indeed encoding only?

finale result file is: results/f30/3.result.csv

1 new commit added

  • add CLDR ideas

2 new commits added

  • add territory-language consistency check using population data
  • improve documentation

1 new commit added

  • rename *region* in *territory*

10 new commits added

  • rename *region* in *territory*
  • add territory-language consistency check using population data
  • improve documentation
  • add CLDR ideas
  • add f30 results
  • use results folder
  • add build_stats in readme
  • add language, region and territory name from cldr
  • add CLDR data
  • use pandas to clean data

Pull-Request has been merged by jibecfed

Metadata