Child Directed Speech test 2019-01-19

«Re-generate tests (with trees production) for POCE & CDS for all baseline parse&WSD variants»

Link Grammar 5.5.1, test_grammar updated 2018-10-19 , server 94.130.238.118
This notebook is shared as static Child-Directed-Speech-2019-01-19.html.
The "All tests" table is shared as 'short_table.txt' in the Child-Directed-Speech-2019-01-19 directory.
Previous tests: Child-Directed-Speech-2019-01-03.html, Child-Directed-Speech-2018-12-05.html, Child-Directed-Speech-2018-10-24.html,
Child-Directed-Speech-2018-10-19.html, Child-Directed-Speech-2018-08-14.html, Child-Directed-Speech-2018-08-06.html.

Basic settings

In [1]:
import os, sys, time
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path: sys.path.append(module_path)
from src.grammar_learner.utl import UTC
from src.grammar_learner.read_files import check_dir
from src.grammar_learner.write_files import list2file
from src.grammar_learner.widgets import html_table
from src.grammar_learner.pqa_table import table_rows, wide_rows
tmpath = module_path + '/tmp/'
check_dir(tmpath, True, 'none')
table = []
long_table = []
start = time.time()
out_dir = module_path + '/output/Child-Directed-Speech-' + str(UTC())[:10]
print(UTC(), ':: out_dir:\n', out_dir)
runs = (1,1)

2019-01-19 13:56:47 UTC :: out_dir:
 /home/obaskov/94/language-learning/output/Child-Directed-Speech-2019-01-19

Corpus test settings

In [2]:
kwargs = {
    'left_wall'     :   ''          ,
    'period'        :   False       ,
    'min_word_count':           1   ,
    'min_link_count':           1   ,
    'min_co-occurrence_count':  1   ,
    'word_space'    :   'vectors'   ,
    'clustering'    :   ['kmeans', 'kmeans++', 10],
    'cluster_range' :   [30,120,3,3],
    'cluster_criteria'  : 'silhouette',
    'clustering_metric' : ['silhouette', 'cosine'],
    'cluster_level' :   1           ,
    'rules_merge'       :   0.8     , # grammar rules merge threshold
    'rules_aggregation' :   0.1     , # grammar rules aggregation threshold
    'top_level'         :   0.01    , # top-level rules generalization threshold
    'tmpath'        :   tmpath      , 
    'verbose'       :   'log+'       ,
    'template_path' :   'poc-turtle',
    'linkage_limit' :   1000        }
lines = [
    [33, 'CDS' , 'LG-E-551'           ,0,0, 'none'], 
    [34, 'CDS' , 'LG-E-551'           ,0,0, 'rules' ], 
    [35, 'CDS' , 'R=6-W=6:R-MW=+1:R'    ,0,0, 'none'], 
    [36, 'CDS' , 'R=6-W=6:R-MW=+1:R'    ,0,0, 'rules' ]]
cp = rp = module_path + '/data/CDS/LG-E-clean'  # clean: both files, 100% parsed
cp = rp  # test corpus path = reference_path

ULL Project Plan ⇒ Parses ⇒ lines 39-42, by columns (K-N), (O-Q)

Connectors-DRK-Connectors

In [3]:
%%capture
kwargs['context'] = 1
kwargs['word_space'] = 'vectors'
kwargs['clustering'] = 'kmeans'
kwargs['cluster_range'] = [30,120,3,3]
kwargs['grammar_rules'] = 1
average21, long21, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average21, long21, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average21)
long_table.extend(long21)
In [4]:
display(html_table([header] + average21))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonecDRKc770.17100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonecDRKc670.1258%41%0.42

Connectors-DRK-Disjuncts

In [5]:
%%capture
kwargs['context'] = 1
kwargs['word_space'] = 'vectors'
kwargs['clustering'] = 'kmeans'
kwargs['grammar_rules'] = 2
average22, long22, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average22, long22, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average22)
long_table.extend(long22)
In [6]:
display(html_table([header] + average22))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonecDRKd810.16100%88%0.89
34CDSLG-E-551 --- --- rulescDRKd790.16100%82%0.83
35CDSR=6-W=6:R-MW=+1:R --- --- nonecDRKd650.1358%41%0.42
36CDSR=6-W=6:R-MW=+1:R --- --- rulescDRKd590.1258%40%0.41

Disjuncts-DRK-Disjuncts

In [7]:
%%capture
kwargs['context'] = 2
kwargs['word_space'] = 'vectors'
kwargs['clustering'] = 'kmeans'
kwargs['grammar_rules'] = 2
average23, long23, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average23, long23, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average23)
long_table.extend(long23)
In [8]:
display(html_table([header] + average23))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonedDRKd690.33100%89%0.91
34CDSLG-E-551 --- --- rulesdDRKd310.33100%76%0.77
35CDSR=6-W=6:R-MW=+1:R --- --- nonedDRKd900.558%41%0.42
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdDRKd900.558%41%0.42

Disjuncts-ILE-Disjuncts

In [9]:
%%capture
kwargs['context'] = 2
kwargs['word_space'] = 'discrete'
kwargs['clustering'] = 'group'
kwargs['grammar_rules'] = 2
average24, long24, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average24, long24, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average24)
long_table.extend(long24)
In [10]:
display(html_table([header] + average24))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonedILEd2997 --- 100%97%0.98
34CDSLG-E-551 --- --- rulesdILEd1874 --- 100%96%0.97
35CDSR=6-W=6:R-MW=+1:R --- --- nonedILEd3558 --- 0%0%0.00
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdILEd3129 --- 0%0%0.00

Disjuncts-ALE-Disjuncts

In [11]:
kwargs['word_space'] = 'sparse'
kwargs['min_word_count'] = 1

Linkage/affinity: ward/euclidean; 50/400 clusters

In [12]:
%%capture
kwargs['clustering'] = ['agglomerative', 'ward']
kwargs['cluster_range'] = 50
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [13]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonedALEd50 --- 100%87%0.88
34CDSLG-E-551 --- --- rulesdALEd16 --- 100%77%0.78
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd50 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd49 --- 58%40%0.41
In [14]:
%%capture
kwargs['cluster_range'] = 400
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [15]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonedALEd400 --- 100%94%0.95
34CDSLG-E-551 --- --- rulesdALEd294 --- 100%95%0.95
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd400 --- 58%41%0.42
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd291 --- 58%41%0.43

Linkage/affinity: complete/cosine; 50/400 clusters

In [16]:
%%capture
kwargs['clustering'] = ['agglomerative', 'complete', 'cosine']
kwargs['cluster_range'] = 50
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [17]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonedALEd50 --- 100%75%0.76
34CDSLG-E-551 --- --- rulesdALEd50 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd50 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd50 --- 58%40%0.41
In [18]:
%%capture
kwargs['cluster_range'] = 400
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [19]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonedALEd400 --- 100%78%0.79
34CDSLG-E-551 --- --- rulesdALEd108 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd400 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd400 --- 58%40%0.41

Linkage/affinity: average/cosine; 50/400 clusters

In [20]:
%%capture
kwargs['cluster_range'] = 50
kwargs['clustering'] = ['agglomerative', 'complete', 'cosine']
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [21]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonedALEd50 --- 100%75%0.76
34CDSLG-E-551 --- --- rulesdALEd50 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd50 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd50 --- 58%40%0.41
In [22]:
%%capture
kwargs['cluster_range'] = 400
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [23]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonedALEd400 --- 100%78%0.79
34CDSLG-E-551 --- --- rulesdALEd108 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd400 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd400 --- 58%40%0.41

All tests

In [24]:
display(html_table([header]+long_table))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-E-551 --- --- nonecDRKc 77 0.17100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonecDRKc 67 0.1258%41%0.42
33CDSLG-E-551 --- --- nonecDRKd 81 0.16100%88%0.89
34CDSLG-E-551 --- --- rulescDRKd 79 0.16100%82%0.83
35CDSR=6-W=6:R-MW=+1:R --- --- nonecDRKd 65 0.1358%41%0.42
36CDSR=6-W=6:R-MW=+1:R --- --- rulescDRKd 59 0.1258%40%0.41
33CDSLG-E-551 --- --- nonedDRKd 69 0.33100%89%0.91
34CDSLG-E-551 --- --- rulesdDRKd 31 0.33100%76%0.77
35CDSR=6-W=6:R-MW=+1:R --- --- nonedDRKd 90 0.558%41%0.42
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdDRKd 90 0.558%41%0.42
33CDSLG-E-551 --- --- nonedILEd 2997 --- 100%97%0.98
34CDSLG-E-551 --- --- rulesdILEd 1874 --- 100%96%0.97
35CDSR=6-W=6:R-MW=+1:R --- --- nonedILEd 3558 --- 0%0%0.00
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdILEd 3129 --- 0%0%0.00
33CDSLG-E-551 --- --- nonedALEd 50 --- 100%87%0.88
34CDSLG-E-551 --- --- rulesdALEd 16 --- 100%77%0.78
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 50 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 49 --- 58%40%0.41
33CDSLG-E-551 --- --- nonedALEd 400 --- 100%94%0.95
34CDSLG-E-551 --- --- rulesdALEd 294 --- 100%95%0.95
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 400 --- 58%41%0.42
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 291 --- 58%41%0.43
33CDSLG-E-551 --- --- nonedALEd 50 --- 100%75%0.76
34CDSLG-E-551 --- --- rulesdALEd 50 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 50 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 50 --- 58%40%0.41
33CDSLG-E-551 --- --- nonedALEd 400 --- 100%78%0.79
34CDSLG-E-551 --- --- rulesdALEd 108 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 400 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 400 --- 58%40%0.41
33CDSLG-E-551 --- --- nonedALEd 50 --- 100%75%0.76
34CDSLG-E-551 --- --- rulesdALEd 50 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 50 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 50 --- 58%40%0.41
33CDSLG-E-551 --- --- nonedALEd 400 --- 100%78%0.79
34CDSLG-E-551 --- --- rulesdALEd 108 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 400 --- 58%40%0.41
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 400 --- 58%40%0.41
In [25]:
print(UTC(), ':: finished, elapsed', str(round((time.time()-start)/3600.0, 1)), 'hours')
table_str = list2file(table, out_dir+'/short_table.txt')
if runs == (1,1):
    print('Results saved to', out_dir + '/short_table.txt')
else:
    long_table_str = list2file(long_table, out_dir+'/long_table.txt')
    print('Average results saved to', out_dir + '/short_table.txt\n'
          'Detailed results for every run saved to', out_dir + '/long_table.txt')

2019-01-19 16:50:35 UTC :: finished, elapsed 2.9 hours
Results saved to /home/obaskov/94/language-learning/output/Child-Directed-Speech-2019-01-19/short_table.txt