Child Directed Speech test 2018-12-21_

«Re-generate tests (with trees production) for POCE & CDS for all baseline parse&WSD variants»

Link Grammar 5.5.1, test_grammar updated 2018-10-19 , server 94.130.238.118
This notebook is shared as static Child-Directed-Speech-2018-12-21.html;
The "All tests" table is shared as 'short_table.txt' in Child-Directed-Speech-2018-12-21 directory.
Previous tests: Child-Directed-Speech-2018-12-05.html, Child-Directed-Speech-2018-10-24.html,
Child-Directed-Speech-2018-10-19.html, Child-Directed-Speech-2018-08-14.html, Child-Directed-Speech-2018-08-06.html.

Basic settings

In [1]:
import os, sys, time
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path: sys.path.append(module_path)
from src.grammar_learner.utl import UTC
from src.grammar_learner.read_files import check_dir
from src.grammar_learner.write_files import list2file
from src.grammar_learner.widgets import html_table
from src.grammar_learner.pqa_table import table_rows, wide_rows
tmpath = module_path + '/tmp/'
check_dir(tmpath, True, 'none')
table = []
long_table = []
start = time.time()
out_dir = module_path + '/output/Child-Directed-Speech-' + str(UTC())[:10]
print(UTC(), ':: out_dir:\n', out_dir)
runs = (1,1)

2018-12-21 18:31:31 UTC :: out_dir:
 /home/obaskov/94/language-learning/output/Child-Directed-Speech-2018-12-21

Corpus test settings

In [2]:
kwargs = {
    'left_wall'     :   ''          ,
    'period'        :   False       ,
    'min_word_count':           1   ,
    'min_link_count':           1   ,
    'min_co-occurrence_count':  1   ,
    'word_space'    :   'vectors'   ,
    'clustering'    :   ['kmeans', 'kmeans++', 10],
    'cluster_range' :   [30,120,3,3],
    'cluster_criteria'  : 'silhouette',
    'clustering_metric' : ['silhouette', 'cosine'],
    'cluster_level' :   1           ,
    'rules_merge'       :   0.8     , # grammar rules merge threshold
    'rules_aggregation' :   0.1     , # grammar rules aggregation threshold
    'top_level'         :   0.01    , # top-level rules generalization threshold
    'tmpath'        :   tmpath      , 
    'verbose'       :   'log+'       ,
    'template_path' :   'poc-turtle',
    'linkage_limit' :   1000        }
lines = [
    [33, 'CDS' , 'LG-English'           ,0,0, 'none'], 
    [34, 'CDS' , 'LG-English'           ,0,0, 'rules' ], 
    [35, 'CDS' , 'R=6-W=6:R-MW=+1:R'    ,0,0, 'none'], 
    [36, 'CDS' , 'R=6-W=6:R-MW=+1:R'    ,0,0, 'rules' ]]
cp = rp = module_path + '/data/CDS/LG-E-clean'  # clean: both files, 100% parsed
cp = rp  # test corpus path = reference_path

ULL Project Plan ⇒ Parses ⇒ lines 39-42, by columns (K-N), (O-Q)

Connectors-DRK-Connectors

In [3]:
%%capture
kwargs['context'] = 1
kwargs['word_space'] = 'vectors'
kwargs['clustering'] = 'kmeans'
kwargs['cluster_range'] = [30,120,3,3]
kwargs['grammar_rules'] = 1
average21, long21, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average21, long21, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average21)
long_table.extend(long21)
In [4]:
display(html_table([header] + average21))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonecDRKc850.1799%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonecDRKc630.1389%49%0.52

Connectors-DRK-Disjuncts

In [5]:
%%capture
kwargs['context'] = 1
kwargs['word_space'] = 'vectors'
kwargs['clustering'] = 'kmeans'
kwargs['grammar_rules'] = 2
average22, long22, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average22, long22, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average22)
long_table.extend(long22)
In [6]:
display(html_table([header] + average22))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonecDRKd840.17100%89%0.91
34CDSLG-English --- --- rulescDRKd90.17100%74%0.75
35CDSR=6-W=6:R-MW=+1:R --- --- nonecDRKd620.1288%48%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulescDRKd700.1388%48%0.51

Disjuncts-DRK-Disjuncts

In [7]:
%%capture
kwargs['context'] = 2
kwargs['word_space'] = 'vectors'
kwargs['clustering'] = 'kmeans'
kwargs['grammar_rules'] = 2
average23, long23, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average23, long23, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average23)
long_table.extend(long23)
In [8]:
display(html_table([header] + average23))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonedDRKd760.34100%91%0.92
34CDSLG-English --- --- rulesdDRKd310.34100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedDRKd860.586%48%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdDRKd910.584%46%0.49

Disjuncts-ILE-Disjuncts

In [9]:
%%capture
kwargs['context'] = 2
kwargs['word_space'] = 'discrete'
kwargs['clustering'] = 'group'
kwargs['grammar_rules'] = 2
average24, long24, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average24, long24, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average24)
long_table.extend(long24)
In [10]:
display(html_table([header] + average24))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonedILEd2980 --- 100%98%0.98
34CDSLG-English --- --- rulesdILEd1820 --- 100%97%0.98
35CDSR=6-W=6:R-MW=+1:R --- --- nonedILEd3558 --- 69%43%0.46
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdILEd3129 --- 72%44%0.46

Disjuncts-ALE-Disjuncts

In [11]:
kwargs['word_space'] = 'sparse'
kwargs['min_word_count'] = 1

Linkage/affinity: ward/euclidean; 50/400 clusters

In [12]:
%%capture
kwargs['clustering'] = ['agglomerative', 'ward']
kwargs['cluster_range'] = 50
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [13]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonedALEd50 --- 100%88%0.89
34CDSLG-English --- --- rulesdALEd13 --- 100%78%0.79
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd50 --- 85%47%0.50
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd49 --- 87%48%0.50
In [14]:
%%capture
kwargs['cluster_range'] = 400
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [15]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonedALEd400 --- 100%98%0.98
34CDSLG-English --- --- rulesdALEd281 --- 100%95%0.96
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd400 --- 72%44%0.46
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd291 --- 81%47%0.50

Linkage/affinity: complete/cosine; 50/400 clusters

In [16]:
%%capture
kwargs['clustering'] = ['agglomerative', 'complete', 'cosine']
kwargs['cluster_range'] = 50
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [17]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonedALEd50 --- 100%75%0.76
34CDSLG-English --- --- rulesdALEd50 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd50 --- 89%49%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd50 --- 89%49%0.51
In [18]:
%%capture
kwargs['cluster_range'] = 400
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [19]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonedALEd400 --- 100%76%0.77
34CDSLG-English --- --- rulesdALEd125 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd400 --- 89%49%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd400 --- 89%49%0.51

Linkage/affinity: average/cosine; 50/400 clusters

In [20]:
%%capture
kwargs['cluster_range'] = 50
kwargs['clustering'] = ['agglomerative', 'complete', 'cosine']
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [21]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonedALEd50 --- 100%75%0.76
34CDSLG-English --- --- rulesdALEd50 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd50 --- 89%49%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd50 --- 89%49%0.51
In [22]:
%%capture
kwargs['cluster_range'] = 400
average, long, header = table_rows(lines, out_dir, cp, rp, runs, **kwargs)
# average, long, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
table.extend(average)
long_table.extend(long)
In [23]:
display(html_table([header] + average))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonedALEd400 --- 100%76%0.77
34CDSLG-English --- --- rulesdALEd125 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd400 --- 89%49%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd400 --- 89%49%0.51

All tests

In [24]:
display(html_table([header]+long_table))
LineCorpusParsingLWRWGen.SpaceRulesSilhouettePAPQF1
33CDSLG-English --- --- nonecDRKc 85 0.1799%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonecDRKc 63 0.1389%49%0.52
33CDSLG-English --- --- nonecDRKd 84 0.17100%89%0.91
34CDSLG-English --- --- rulescDRKd 9 0.17100%74%0.75
35CDSR=6-W=6:R-MW=+1:R --- --- nonecDRKd 62 0.1288%48%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulescDRKd 70 0.1388%48%0.51
33CDSLG-English --- --- nonedDRKd 76 0.34100%91%0.92
34CDSLG-English --- --- rulesdDRKd 31 0.34100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedDRKd 86 0.586%48%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdDRKd 91 0.584%46%0.49
33CDSLG-English --- --- nonedILEd 2980 --- 100%98%0.98
34CDSLG-English --- --- rulesdILEd 1820 --- 100%97%0.98
35CDSR=6-W=6:R-MW=+1:R --- --- nonedILEd 3558 --- 69%43%0.46
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdILEd 3129 --- 72%44%0.46
33CDSLG-English --- --- nonedALEd 50 --- 100%88%0.89
34CDSLG-English --- --- rulesdALEd 13 --- 100%78%0.79
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 50 --- 85%47%0.50
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 49 --- 87%48%0.50
33CDSLG-English --- --- nonedALEd 400 --- 100%98%0.98
34CDSLG-English --- --- rulesdALEd 281 --- 100%95%0.96
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 400 --- 72%44%0.46
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 291 --- 81%47%0.50
33CDSLG-English --- --- nonedALEd 50 --- 100%75%0.76
34CDSLG-English --- --- rulesdALEd 50 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 50 --- 89%49%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 50 --- 89%49%0.51
33CDSLG-English --- --- nonedALEd 400 --- 100%76%0.77
34CDSLG-English --- --- rulesdALEd 125 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 400 --- 89%49%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 400 --- 89%49%0.51
33CDSLG-English --- --- nonedALEd 50 --- 100%75%0.76
34CDSLG-English --- --- rulesdALEd 50 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 50 --- 89%49%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 50 --- 89%49%0.51
33CDSLG-English --- --- nonedALEd 400 --- 100%76%0.77
34CDSLG-English --- --- rulesdALEd 125 --- 100%75%0.76
35CDSR=6-W=6:R-MW=+1:R --- --- nonedALEd 400 --- 89%49%0.51
36CDSR=6-W=6:R-MW=+1:R --- --- rulesdALEd 400 --- 89%49%0.51
In [25]:
print(UTC(), ':: finished, elapsed', str(round((time.time()-start)/3600.0, 1)), 'hours')
table_str = list2file(table, out_dir+'/short_table.txt')
if runs == (1,1):
    print('Results saved to', out_dir + '/short_table.txt')
else:
    long_table_str = list2file(long_table, out_dir+'/long_table.txt')
    print('Average results saved to', out_dir + '/short_table.txt\n'
          'Detailed results for every run saved to', out_dir + '/long_table.txt')

2018-12-21 21:10:11 UTC :: finished, elapsed 2.6 hours
Results saved to /home/obaskov/94/language-learning/output/Child-Directed-Speech-2018-12-21/short_table.txt