Nameexact_genome_1512568488_42_4483_0
Workunit2593640
Created7 Dec 2017, 10:09:15 UTC
Sent7 Dec 2017, 10:11:50 UTC
Report deadline12 Dec 2017, 7:18:13 UTC
Received7 Dec 2017, 23:52:35 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID64142
Run time3 hours 59 min 56 sec
CPU time2 hours 32 min 48 sec
Validate stateValid
Credit1,219.86
Device peak FLOPS3.08 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size155.91 MB
Peak swap size158.86 MB
Peak disk usage1.85 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
33337 on epoch:    23, epoch:   26/100, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 179.1541442871s, input_fired_time: 1.8970521688, output_fired_time: 1.8826538324, propagate_forward_time: 68.3893203735, propagate_backward_time: 99.8646621704, weight_update_time: 0.9279951453, other_time: 7.1204528809
epoch time: 15.1705665588s, input_fired_time: 0.2637054026, output_fired_time: 0.0000000000, propagate_forward_time: 14.1494579315, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7574033737
validation[          , genome  4483] predictions:    9625/  10000 (96.25%), best:    9649/10000 (96.49%), error:      1255.73767, best error:      1254.33337 on epoch:    23, epoch:   27/100, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

epoch time: 220.0237731934s, input_fired_time: 2.9654240608, output_fired_time: 3.6954660416, propagate_forward_time: 80.9933929443, propagate_backward_time: 123.7665100098, weight_update_time: 0.1935616434, other_time: 8.6029815674
epoch time: 19.4809513092s, input_fired_time: 0.1567925513, output_fired_time: 0.0000000000, propagate_forward_time: 17.2843742371, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0397853851
validation[          , genome  4483] predictions:    9592/  10000 (95.92%), best:    9649/10000 (96.49%), error:      1314.35046, best error:      1254.33337 on epoch:    23, epoch:   28/100, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 247.8560943604s, input_fired_time: 4.3180799484, output_fired_time: 2.5842657089, propagate_forward_time: 92.9052658081, propagate_backward_time: 140.7253875732, weight_update_time: 0.2093187571, other_time: 7.3231048584
epoch time: 20.5235939026s, input_fired_time: 0.4847666621, output_fired_time: 0.0000000000, propagate_forward_time: 18.6668663025, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3719615936
validation[          , genome  4483] predictions:    9627/  10000 (96.27%), best:    9649/10000 (96.49%), error:      1260.37903, best error:      1254.33337 on epoch:    23, epoch:   29/100, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 260.7324218750s, input_fired_time: 3.6965303421, output_fired_time: 3.3665027618, propagate_forward_time: 94.8558502197, propagate_backward_time: 150.2475738525, weight_update_time: 0.2875789404, other_time: 8.5659637451
epoch time: 24.1279144287s, input_fired_time: 0.6420760751, output_fired_time: 0.0000000000, propagate_forward_time: 21.1171302795, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.3687076569
validation[          , genome  4483] predictions:    9620/  10000 (96.20%), best:    9649/10000 (96.49%), error:      1334.93274, best error:      1254.33337 on epoch:    23, epoch:   30/100, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

epoch time: 166.5030364990s, input_fired_time: 2.4281063080, output_fired_time: 1.5011143684, propagate_forward_time: 62.8501014709, propagate_backward_time: 94.3041687012, weight_update_time: 0.1607821137, other_time: 5.4195556641
epoch time: 9.4289560318s, input_fired_time: 0.1702784002, output_fired_time: 0.0000000000, propagate_forward_time: 8.5581092834, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7005681992
validation[          , genome  4483] predictions:    9541/  10000 (95.41%), best:    9649/10000 (96.49%), error:      1413.14294, best error:      1254.33337 on epoch:    23, epoch:   31/100, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 114.4089813232s, input_fired_time: 1.6984263659, output_fired_time: 1.4635007381, propagate_forward_time: 41.8952522278, propagate_backward_time: 65.2849655151, weight_update_time: 0.2225249708, other_time: 4.0668411255
epoch time: 9.4936294556s, input_fired_time: 0.1991481185, output_fired_time: 0.0000000000, propagate_forward_time: 8.4529829025, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.8414983749
validation[          , genome  4483] predictions:    9553/  10000 (95.53%), best:    9649/10000 (96.49%), error:      1492.84265, best error:      1254.33337 on epoch:    23, epoch:   32/100, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 116.3108444214s, input_fired_time: 1.6629053354, output_fired_time: 1.3865849972, propagate_forward_time: 42.7942276001, propagate_backward_time: 66.1734161377, weight_update_time: 0.1758831143, other_time: 4.2937164307
epoch time: 9.2203903198s, input_fired_time: 0.2633541226, output_fired_time: 0.0000000000, propagate_forward_time: 8.3854598999, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5715761185
validation[          , genome  4483] predictions:    9633/  10000 (96.33%), best:    9633/10000 (96.33%), error:      1241.82776, best error:      1241.82776 on epoch:    33, epoch:   33/100, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 115.1707992554s, input_fired_time: 1.5714156628, output_fired_time: 1.7753499746, propagate_forward_time: 42.0092506409, propagate_backward_time: 65.8304138184, weight_update_time: 0.1290938109, other_time: 3.9843597412
epoch time: 9.0552320480s, input_fired_time: 0.2292584628, output_fired_time: 0.0000000000, propagate_forward_time: 8.0378694534, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7881040573
validation[          , genome  4483] predictions:    9580/  10000 (95.80%), best:    9633/10000 (96.33%), error:      1361.85876, best error:      1241.82776 on epoch:    33, epoch:   34/100, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 120.1775512695s, input_fired_time: 1.8663071394, output_fired_time: 1.4624994993, propagate_forward_time: 44.4157485962, propagate_backward_time: 68.4525985718, weight_update_time: 0.2373273075, other_time: 3.9803924561
epoch time: 8.9746875763s, input_fired_time: 0.1555416733, output_fired_time: 0.0000000000, propagate_forward_time: 8.0036249161, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.8155212402
validation[          , genome  4483] predictions:    9556/  10000 (95.56%), best:    9633/10000 (96.33%), error:      1412.87280, best error:      1241.82776 on epoch:    33, epoch:   35/100, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 115.6642761230s, input_fired_time: 1.7438520193, output_fired_time: 1.5240139961, propagate_forward_time: 42.6373748779, propagate_backward_time: 65.7784500122, weight_update_time: 0.1756399572, other_time: 3.9805831909
epoch time: 8.3575630188s, input_fired_time: 0.2441917211, output_fired_time: 0.0000000000, propagate_forward_time: 7.4117226601, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7016482353
validation[          , genome  4483] predictions:    9632/  10000 (96.32%), best:    9632/10000 (96.32%), error:      1229.16003, best error:      1229.16003 on epoch:    36, epoch:   36/100, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 114.1456069946s, input_fired_time: 1.7256264687, output_fired_time: 1.2755278349, propagate_forward_time: 42.0858650208, propagate_backward_time: 65.2576980591, weight_update_time: 0.1740278155, other_time: 3.8008956909
epoch time: 9.1021461487s, input_fired_time: 0.2283940911, output_fired_time: 0.0000000000, propagate_forward_time: 8.2318191528, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6419324875
validation[          , genome  4483] predictions:    9581/  10000 (95.81%), best:    9632/10000 (96.32%), error:      1382.84387, best error:      1229.16003 on epoch:    36, epoch:   37/100, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 108.1414794922s, input_fired_time: 2.0265102386, output_fired_time: 1.4950932264, propagate_forward_time: 39.6637535095, propagate_backward_time: 61.0105972290, weight_update_time: 0.2609287202, other_time: 3.9455184937
epoch time: 7.1757826805s, input_fired_time: 0.1953545958, output_fired_time: 0.0000000000, propagate_forward_time: 6.4559574127, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5244708061
validation[          , genome  4483] predictions:    9620/  10000 (96.20%), best:    9632/10000 (96.32%), error:      1325.40747, best error:      1229.16003 on epoch:    36, epoch:   38/100, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 96.8840713501s, input_fired_time: 1.6306073666, output_fired_time: 1.5278391838, propagate_forward_time: 34.9831161499, propagate_backward_time: 55.5075225830, weight_update_time: 0.1634571105, other_time: 3.2349853516
epoch time: 8.4073772430s, input_fired_time: 0.1466168016, output_fired_time: 0.0000000000, propagate_forward_time: 7.5988636017, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6618967056
validation[          , genome  4483] predictions:    9615/  10000 (96.15%), best:    9632/10000 (96.32%), error:      1296.24268, best error:      1229.16003 on epoch:    36, epoch:   39/100, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 88.7855224609s, input_fired_time: 1.1830012798, output_fired_time: 1.0337475538, propagate_forward_time: 32.3931427002, propagate_backward_time: 51.1445617676, weight_update_time: 0.1178905368, other_time: 3.0310745239
epoch time: 8.0303258896s, input_fired_time: 0.1480320692, output_fired_time: 0.0000000000, propagate_forward_time: 7.2222352028, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6600584984
validation[          , genome  4483] predictions:    9595/  10000 (95.95%), best:    9632/10000 (96.32%), error:      1322.89270, best error:      1229.16003 on epoch:    36, epoch:   40/100, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 101.8699951172s, input_fired_time: 1.4632698298, output_fired_time: 1.1958686113, propagate_forward_time: 37.3334922791, propagate_backward_time: 58.3147697449, weight_update_time: 0.2112762779, other_time: 3.5625915527
epoch time: 6.6362662315s, input_fired_time: 0.1547629237, output_fired_time: 0.0000000000, propagate_forward_time: 6.0023722649, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4791312218
validation[          , genome  4483] predictions:    9639/  10000 (96.39%), best:    9632/10000 (96.32%), error:      1268.14734, best error:      1229.16003 on epoch:    36, epoch:   41/100, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 93.5447921753s, input_fired_time: 1.4697403908, output_fired_time: 1.2873566151, propagate_forward_time: 34.5563011169, propagate_backward_time: 53.0277824402, weight_update_time: 0.1628785580, other_time: 3.2036132813
epoch time: 7.5413646698s, input_fired_time: 0.1393129826, output_fired_time: 0.0000000000, propagate_forward_time: 6.9114975929, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4905543327
validation[          , genome  4483] predictions:    9554/  10000 (95.54%), best:    9632/10000 (96.32%), error:      1446.43848, best error:      1229.16003 on epoch:    36, epoch:   42/100, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 94.0181503296s, input_fired_time: 1.5048904419, output_fired_time: 1.3467311859, propagate_forward_time: 34.2683067322, propagate_backward_time: 53.9048881531, weight_update_time: 0.1181486547, other_time: 2.9933319092
epoch time: 7.3430633545s, input_fired_time: 0.2171324790, output_fired_time: 0.0000000000, propagate_forward_time: 6.5822224617, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5437083244
validation[          , genome  4483] predictions:    9660/  10000 (96.60%), best:    9660/10000 (96.60%), error:      1149.71130, best error:      1149.71130 on epoch:    43, epoch:   43/100, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 100.2486190796s, input_fired_time: 1.5915857553, output_fired_time: 1.4769585133, propagate_forward_time: 36.7958106995, propagate_backward_time: 57.0614891052, weight_update_time: 0.1929258704, other_time: 3.3227767944
epoch time: 6.9049806595s, input_fired_time: 0.1433814317, output_fired_time: 0.0000000000, propagate_forward_time: 6.2773060799, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4842929840
validation[          , genome  4483] predictions:    9607/  10000 (96.07%), best:    9660/10000 (96.60%), error:      1292.30994, best error:      1149.71130 on epoch:    43, epoch:   44/100, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 94.1298446655s, input_fired_time: 1.3812402487, output_fired_time: 1.3676123619, propagate_forward_time: 34.8579597473, propagate_backward_time: 53.2599678040, weight_update_time: 0.1960495412, other_time: 3.2630615234
epoch time: 6.8603587151s, input_fired_time: 0.1407629848, output_fired_time: 0.0000000000, propagate_forward_time: 6.1716551781, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5479407310
validation[          , genome  4483] predictions:    9523/  10000 (95.23%), best:    9660/10000 (96.60%), error:      1556.00989, best error:      1149.71130 on epoch:    43, epoch:   45/100, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 95.6164093018s, input_fired_time: 1.6245341301, output_fired_time: 1.2349612713, propagate_forward_time: 34.3175315857, propagate_backward_time: 55.1697654724, weight_update_time: 0.1158235446, other_time: 3.2696151733
epoch time: 8.7026834488s, input_fired_time: 0.1665550172, output_fired_time: 0.0000000000, propagate_forward_time: 7.8339400291, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7021880150
validation[          , genome  4483] predictions:    9624/  10000 (96.24%), best:    9660/10000 (96.60%), error:      1254.14758, best error:      1149.71130 on epoch:    43, epoch:   46/100, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 164.5792541504s, input_fired_time: 2.7275769711, output_fired_time: 1.9252859354, propagate_forward_time: 63.1210556030, propagate_backward_time: 91.7400588989, weight_update_time: 0.4884061217, other_time: 5.0652770996
epoch time: 17.0364608765s, input_fired_time: 0.9090039730, output_fired_time: 0.0000000000, propagate_forward_time: 14.2613925934, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8660640717
validation[          , genome  4483] predictions:    9577/  10000 (95.77%), best:    9660/10000 (96.60%), error:      1411.40613, best error:      1149.71130 on epoch:    43, epoch:   47/100, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 202.0422821045s, input_fired_time: 2.8905181885, output_fired_time: 3.1650516987, propagate_forward_time: 74.3732681274, propagate_backward_time: 115.3824157715, weight_update_time: 0.2404880524, other_time: 6.2310256958
epoch time: 16.1145210266s, input_fired_time: 0.3462595940, output_fired_time: 0.0000000000, propagate_forward_time: 14.2157335281, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5525274277
validation[          , genome  4483] predictions:    9597/  10000 (95.97%), best:    9660/10000 (96.60%), error:      1276.61438, best error:      1149.71130 on epoch:    43, epoch:   48/100, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 223.9316101074s, input_fired_time: 3.3328106403, output_fired_time: 3.6866683960, propagate_forward_time: 82.0331726074, propagate_backward_time: 127.8532867432, weight_update_time: 0.2250996977, other_time: 7.0256805420
epoch time: 20.1313476563s, input_fired_time: 0.3000977337, output_fired_time: 0.0000000000, propagate_forward_time: 17.9341335297, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.8971157074
validation[          , genome  4483] predictions:    9628/  10000 (96.28%), best:    9660/10000 (96.60%), error:      1275.18469, best error:      1149.71130 on epoch:    43, epoch:   49/100, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 266.7472839355s, input_fired_time: 3.2600936890, output_fired_time: 3.3627407551, propagate_forward_time: 102.5043258667, propagate_backward_time: 148.1178588867, weight_update_time: 0.3813027740, other_time: 9.5022583008
epoch time: 25.7952136993s, input_fired_time: 0.4839162827, output_fired_time: 0.0000000000, propagate_forward_time: 24.0691795349, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2421188354
validation[          , genome  4483] predictions:    9618/  10000 (96.18%), best:    9660/10000 (96.60%), error:      1283.00354, best error:      1149.71130 on epoch:    43, epoch:   50/100, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

epoch time: 302.4579162598s, input_fired_time: 5.0264401436, output_fired_time: 4.7176046371, propagate_forward_time: 111.9506301880, propagate_backward_time: 171.0066375732, weight_update_time: 0.6319637895, other_time: 9.7566375732
epoch time: 24.0510101318s, input_fired_time: 0.6737201810, output_fired_time: 0.0000000000, propagate_forward_time: 22.1974754333, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.1798152924
validation[          , genome  4483] predictions:    9638/  10000 (96.38%), best:    9660/10000 (96.60%), error:      1226.35669, best error:      1149.71130 on epoch:    43, epoch:   51/100, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488

epoch time: 301.1885070801s, input_fired_time: 5.1855983734, output_fired_time: 3.9189081192, propagate_forward_time: 112.4208526611, propagate_backward_time: 169.2968139648, weight_update_time: 0.4752489924, other_time: 10.3663177490
epoch time: 25.0343170166s, input_fired_time: 0.8480331898, output_fired_time: 0.0000000000, propagate_forward_time: 22.7248382568, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4614448547
validation[          , genome  4483] predictions:    9635/  10000 (96.35%), best:    9660/10000 (96.60%), error:      1232.62988, best error:      1149.71130 on epoch:    43, epoch:   52/100, mu: 0.9559730291, learning_rate: 0.0008680347, weight_decay: 0.0000347214

epoch time: 295.8641357422s, input_fired_time: 4.9539365768, output_fired_time: 3.2568302155, propagate_forward_time: 107.0404510498, propagate_backward_time: 171.0873565674, weight_update_time: 0.5536260605, other_time: 9.5255432129
epoch time: 23.1462535858s, input_fired_time: 0.2827968895, output_fired_time: 0.0000000000, propagate_forward_time: 20.8478603363, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0155963898
validation[          , genome  4483] predictions:    9578/  10000 (95.78%), best:    9660/10000 (96.60%), error:      1396.61377, best error:      1149.71130 on epoch:    43, epoch:   53/100, mu: 0.9576743841, learning_rate: 0.0008246329, weight_decay: 0.0000329853

epoch time: 244.1794738770s, input_fired_time: 3.6017341614, output_fired_time: 2.9123215675, propagate_forward_time: 89.0190048218, propagate_backward_time: 138.7102508545, weight_update_time: 0.6943438053, other_time: 9.9361724854
epoch time: 18.7582988739s, input_fired_time: 0.2519339025, output_fired_time: 0.0000000000, propagate_forward_time: 16.3677234650, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.1386413574
validation[          , genome  4483] predictions:    9603/  10000 (96.03%), best:    9660/10000 (96.60%), error:      1282.34314, best error:      1149.71130 on epoch:    43, epoch:   54/100, mu: 0.9592906833, learning_rate: 0.0007834013, weight_decay: 0.0000313361

epoch time: 221.2200775146s, input_fired_time: 3.4407136440, output_fired_time: 2.4650342464, propagate_forward_time: 81.3417205811, propagate_backward_time: 126.6389389038, weight_update_time: 0.4741670489, other_time: 7.3336715698
epoch time: 6.8596425056s, input_fired_time: 0.1467980593, output_fired_time: 0.0000000000, propagate_forward_time: 6.2215356827, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4913086891
validation[          , genome  4483] predictions:    9634/  10000 (96.34%), best:    9660/10000 (96.60%), error:      1212.45178, best error:      1149.71130 on epoch:    43, epoch:   55/100, mu: 0.9608261585, learning_rate: 0.0007442312, weight_decay: 0.0000297693

epoch time: 90.7143478394s, input_fired_time: 1.3307443857, output_fired_time: 1.2064100504, propagate_forward_time: 33.0949325562, propagate_backward_time: 51.8246116638, weight_update_time: 0.1214065328, other_time: 3.2576484680
epoch time: 7.0180406570s, input_fired_time: 0.2264423966, output_fired_time: 0.0000000000, propagate_forward_time: 6.2610368729, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5305614471
validation[          , genome  4483] predictions:    9605/  10000 (96.05%), best:    9660/10000 (96.60%), error:      1322.77222, best error:      1149.71130 on epoch:    43, epoch:   56/100, mu: 0.9622848630, learning_rate: 0.0007070196, weight_decay: 0.0000282808

epoch time: 87.8569564819s, input_fired_time: 1.3572783470, output_fired_time: 1.2458243370, propagate_forward_time: 32.1117362976, propagate_backward_time: 50.1344833374, weight_update_time: 0.1842270195, other_time: 3.0076332092
epoch time: 7.7361955643s, input_fired_time: 0.1825384349, output_fired_time: 0.0000000000, propagate_forward_time: 6.8918414116, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6618156433
validation[          , genome  4483] predictions:    9620/  10000 (96.20%), best:    9660/10000 (96.60%), error:      1226.61182, best error:      1149.71130 on epoch:    43, epoch:   57/100, mu: 0.9636706114, learning_rate: 0.0006716687, weight_decay: 0.0000268668

epoch time: 93.8757400513s, input_fired_time: 1.3882311583, output_fired_time: 1.3305516243, propagate_forward_time: 34.5017127991, propagate_backward_time: 53.6301956177, weight_update_time: 0.1183088869, other_time: 3.0250511169
epoch time: 8.6836509705s, input_fired_time: 0.1848630011, output_fired_time: 0.0000000000, propagate_forward_time: 7.8890876770, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6097002029
validation[          , genome  4483] predictions:    9585/  10000 (95.85%), best:    9660/10000 (96.60%), error:      1340.39636, best error:      1149.71130 on epoch:    43, epoch:   58/100, mu: 0.9649870992, learning_rate: 0.0006380852, weight_decay: 0.0000255234

epoch time: 94.4703598022s, input_fired_time: 1.3394600153, output_fired_time: 1.1768742800, propagate_forward_time: 35.0249557495, propagate_backward_time: 53.8707885742, weight_update_time: 0.1234545410, other_time: 3.0582809448
epoch time: 6.7976241112s, input_fired_time: 0.1489080489, output_fired_time: 0.0000000000, propagate_forward_time: 6.1032886505, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5454273224
validation[          , genome  4483] predictions:    9576/  10000 (95.76%), best:    9660/10000 (96.60%), error:      1413.49402, best error:      1149.71130 on epoch:    43, epoch:   59/100, mu: 0.9662377238, learning_rate: 0.0006061810, weight_decay: 0.0000242472

epoch time: 91.0589294434s, input_fired_time: 1.4842568636, output_fired_time: 1.3038709164, propagate_forward_time: 32.8528747559, propagate_backward_time: 52.1927337646, weight_update_time: 0.1193313822, other_time: 3.2251892090
epoch time: 6.8707337379s, input_fired_time: 0.1522842646, output_fired_time: 0.0000000000, propagate_forward_time: 6.1902909279, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5281586647
validation[          , genome  4483] predictions:    9647/  10000 (96.47%), best:    9660/10000 (96.60%), error:      1184.76941, best error:      1149.71130 on epoch:    43, epoch:   60/100, mu: 0.9674258232, learning_rate: 0.0005758719, weight_decay: 0.0000230349

epoch time: 95.4028396606s, input_fired_time: 1.3263998032, output_fired_time: 1.1619389057, propagate_forward_time: 35.2365684509, propagate_backward_time: 54.6459007263, weight_update_time: 0.1194266006, other_time: 3.0320281982
epoch time: 6.7025690079s, input_fired_time: 0.1342093050, output_fired_time: 0.0000000000, propagate_forward_time: 5.9912633896, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5770964622
validation[          , genome  4483] predictions:    9637/  10000 (96.37%), best:    9660/10000 (96.60%), error:      1257.42920, best error:      1149.71130 on epoch:    43, epoch:   61/100, mu: 0.9685545564, learning_rate: 0.0005470783, weight_decay: 0.0000218831

epoch time: 95.1477890015s, input_fired_time: 1.4926131964, output_fired_time: 1.0765169859, propagate_forward_time: 34.6570320129, propagate_backward_time: 54.6332855225, weight_update_time: 0.1227464229, other_time: 3.2883415222
epoch time: 7.1777529716s, input_fired_time: 0.1517692059, output_fired_time: 0.0000000000, propagate_forward_time: 6.4977245331, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5282592773
validation[          , genome  4483] predictions:    9610/  10000 (96.10%), best:    9660/10000 (96.60%), error:      1239.97461, best error:      1149.71130 on epoch:    43, epoch:   62/100, mu: 0.9696268439, learning_rate: 0.0005197244, weight_decay: 0.0000207890

epoch time: 91.5898361206s, input_fired_time: 1.4125249386, output_fired_time: 1.1456141472, propagate_forward_time: 33.4775276184, propagate_backward_time: 52.3898849487, weight_update_time: 0.1232052296, other_time: 3.1642875671
epoch time: 6.8993544579s, input_fired_time: 0.1657650024, output_fired_time: 0.0000000000, propagate_forward_time: 6.2144532204, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5191364288
validation[          , genome  4483] predictions:    9632/  10000 (96.32%), best:    9660/10000 (96.60%), error:      1208.76526, best error:      1149.71130 on epoch:    43, epoch:   63/100, mu: 0.9706454873, learning_rate: 0.0004937382, weight_decay: 0.0000197495

epoch time: 89.0620880127s, input_fired_time: 1.3344492912, output_fired_time: 1.1498275995, propagate_forward_time: 32.6187744141, propagate_backward_time: 50.9073410034, weight_update_time: 0.1202596426, other_time: 3.0516967773
epoch time: 7.1809959412s, input_fired_time: 0.1524369121, output_fired_time: 0.0000000000, propagate_forward_time: 6.4835491180, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5450100899
validation[          , genome  4483] predictions:    9585/  10000 (95.85%), best:    9660/10000 (96.60%), error:      1331.86462, best error:      1149.71130 on epoch:    43, epoch:   64/100, mu: 0.9716132283, learning_rate: 0.0004690512, weight_decay: 0.0000187621

epoch time: 97.5953445435s, input_fired_time: 1.5634925365, output_fired_time: 1.3650714159, propagate_forward_time: 35.5983772278, propagate_backward_time: 55.6284790039, weight_update_time: 0.1195238903, other_time: 3.4399223328
epoch time: 6.0922102928s, input_fired_time: 0.1524261087, output_fired_time: 0.0000000000, propagate_forward_time: 5.4635705948, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4762134552
validation[          , genome  4483] predictions:    9517/  10000 (95.17%), best:    9660/10000 (96.60%), error:      1515.36707, best error:      1149.71130 on epoch:    43, epoch:   65/100, mu: 0.9725325704, learning_rate: 0.0004455987, weight_decay: 0.0000178240

epoch time: 93.9635314941s, input_fired_time: 1.2252670527, output_fired_time: 1.2272485495, propagate_forward_time: 34.5635223389, propagate_backward_time: 53.6092453003, weight_update_time: 0.1182085127, other_time: 3.3382492065
epoch time: 7.4496865273s, input_fired_time: 0.1517497003, output_fired_time: 0.0000000000, propagate_forward_time: 6.7211437225, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5767931938
validation[          , genome  4483] predictions:    9624/  10000 (96.24%), best:    9660/10000 (96.60%), error:      1211.25769, best error:      1149.71130 on epoch:    43, epoch:   66/100, mu: 0.9734059572, learning_rate: 0.0004233187, weight_decay: 0.0000169328

epoch time: 98.1358032227s, input_fired_time: 1.4961723089, output_fired_time: 1.3434101343, propagate_forward_time: 35.3874626160, propagate_backward_time: 56.4423408508, weight_update_time: 0.1336093992, other_time: 3.4664230347
epoch time: 7.1182394028s, input_fired_time: 0.1513375193, output_fired_time: 0.0000000000, propagate_forward_time: 6.3969273567, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5699744225
validation[          , genome  4483] predictions:    9634/  10000 (96.34%), best:    9660/10000 (96.60%), error:      1257.94617, best error:      1149.71130 on epoch:    43, epoch:   67/100, mu: 0.9742356539, learning_rate: 0.0004021528, weight_decay: 0.0000160861

epoch time: 91.3445053101s, input_fired_time: 1.2686932087, output_fired_time: 1.0848784447, propagate_forward_time: 33.6532249451, propagate_backward_time: 52.2810249329, weight_update_time: 0.1194557399, other_time: 3.0566864014
epoch time: 6.6577301025s, input_fired_time: 0.1504175067, output_fired_time: 0.0000000000, propagate_forward_time: 6.0049324036, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5023803711
validation[          , genome  4483] predictions:    9555/  10000 (95.55%), best:    9660/10000 (96.60%), error:      1464.18140, best error:      1149.71130 on epoch:    43, epoch:   68/100, mu: 0.9750238657, learning_rate: 0.0003820452, weight_decay: 0.0000152818

epoch time: 87.7861328125s, input_fired_time: 1.4473010302, output_fired_time: 1.1832989454, propagate_forward_time: 32.2856903076, propagate_backward_time: 50.0108222961, weight_update_time: 0.1179540381, other_time: 2.8590202332
epoch time: 6.6559576988s, input_fired_time: 0.1614201069, output_fired_time: 0.0000000000, propagate_forward_time: 6.0072932243, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4872441292
validation[          , genome  4483] predictions:    9622/  10000 (96.22%), best:    9660/10000 (96.60%), error:      1261.35107, best error:      1149.71130 on epoch:    43, epoch:   69/100, mu: 0.9757726789, learning_rate: 0.0003629429, weight_decay: 0.0000145177

epoch time: 85.9009399414s, input_fired_time: 1.2749795914, output_fired_time: 1.2237854004, propagate_forward_time: 31.3824005127, propagate_backward_time: 49.1662406921, weight_update_time: 0.1191066206, other_time: 2.8535346985
epoch time: 7.9238166809s, input_fired_time: 0.2928608954, output_fired_time: 0.0000000000, propagate_forward_time: 7.0269088745, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6040468216
validation[          , genome  4483] predictions:    9587/  10000 (95.87%), best:    9660/10000 (96.60%), error:      1378.94556, best error:      1149.71130 on epoch:    43, epoch:   70/100, mu: 0.9764840603, learning_rate: 0.0003447958, weight_decay: 0.0000137918

epoch time: 90.2746429443s, input_fired_time: 1.2473230362, output_fired_time: 1.1293916702, propagate_forward_time: 32.8327903748, propagate_backward_time: 51.9116439819, weight_update_time: 0.1339309663, other_time: 3.1534919739
epoch time: 7.4436073303s, input_fired_time: 0.1791663468, output_fired_time: 0.0000000000, propagate_forward_time: 6.6516337395, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6128072739
validation[          , genome  4483] predictions:    9614/  10000 (96.14%), best:    9660/10000 (96.60%), error:      1273.18262, best error:      1149.71130 on epoch:    43, epoch:   71/100, mu: 0.9771598577, learning_rate: 0.0003275560, weight_decay: 0.0000131022

epoch time: 96.1454467773s, input_fired_time: 1.5729529858, output_fired_time: 1.4732872248, propagate_forward_time: 34.7328109741, propagate_backward_time: 55.1336441040, weight_update_time: 0.2120177150, other_time: 3.2327499390
epoch time: 7.3392114639s, input_fired_time: 0.1443296820, output_fired_time: 0.0000000000, propagate_forward_time: 6.6950268745, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4998550415
validation[          , genome  4483] predictions:    9532/  10000 (95.32%), best:    9660/10000 (96.60%), error:      1535.22363, best error:      1149.71130 on epoch:    43, epoch:   72/100, mu: 0.9778018594, learning_rate: 0.0003111782, weight_decay: 0.0000124471

epoch time: 99.7294769287s, input_fired_time: 1.4467418194, output_fired_time: 1.4421167374, propagate_forward_time: 36.4602508545, propagate_backward_time: 56.7690734863, weight_update_time: 0.1980950981, other_time: 3.6112976074
epoch time: 7.5512161255s, input_fired_time: 0.1669820398, output_fired_time: 0.0000000000, propagate_forward_time: 6.8305144310, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5537195206
validation[          , genome  4483] predictions:    9623/  10000 (96.23%), best:    9660/10000 (96.60%), error:      1230.83972, best error:      1149.71130 on epoch:    43, epoch:   73/100, mu: 0.9784117937, learning_rate: 0.0002956192, weight_decay: 0.0000118248

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/mnist_training_50k.bin'
boincified validation filename: '../../projects/csgrid.org_csg/mnist_validation_10k.bin'
boincified testing filename: '../../projects/csgrid.org_csg/mnist_testing_new.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_42_4483.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_42_4483_0_r1719403357_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.978991
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000280838
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 1.12335e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 74
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 43
read number_validation_images: 10000
read best_validation_predictions: 9660
read best_validation_error: 1149.71
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 4483
read normal distribution: '1 0 0'
generator_str: '2073957330'
read generator: 2073957330
read generated_by_map:
1 enable_edge 1
reading 22 nodes.
reading 43 edges.
number input nodes: 1
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 4923 images.
reading image set with 5742 images.
reading image set with 4958 images.
reading image set with 5131 images.
reading image set with 4842 images.
reading image set with 4421 images.
reading image set with 4918 images.
reading image set with 5265 images.
reading image set with 4851 images.
reading image set with 4949 images.
image_size: 1x28x28 = 784
read 50000 images.
    class    0: 4923
    class    1: 5742
    class    2: 4958
    class    3: 5131
    class    4: 4842
    class    5: 4421
    class    6: 4918
    class    7: 5265
    class    8: 4851
    class    9: 4949
calculating averages and standard deviations for images
average pixel value for channel 0: 0.130530551075935
pixel variance for channel 0: 0.0948646068572998
pixel standard deviation for channel 0: 0.308000981807709
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 980 images.
reading image set with 1135 images.
reading image set with 1032 images.
reading image set with 1010 images.
reading image set with 982 images.
reading image set with 892 images.
reading image set with 958 images.
reading image set with 1028 images.
reading image set with 974 images.
reading image set with 1009 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 980
    class    1: 1135
    class    2: 1032
    class    3: 1010
    class    4: 982
    class    5: 892
    class    6: 958
    class    7: 1028
    class    8: 974
    class    9: 1009
loaded images
starting backpropagation!
epoch time: 88.8132019042969s, input_fired_time: 1.32425212860107, output_fired_time: 1.24439978599548, propagate_forward_time: 32.3514556884766, propagate_backward_time: 50.707462310791, weight_update_time: 0.118734560906887, other_time: 3.18563461303711
epoch time: 7.19423675537109s, input_fired_time: 0.137469470500946, output_fired_time: 0, propagate_forward_time: 6.51558303833008, propagate_backward_time: 0, weight_update_time: 0, other_time: 0.541184425354004
validation[          , genome  4483] predictions:    9600/  10000 (96.00%), best:    9660/10000 (96.60%), error:      1277.63647, best error:      1149.71130 on epoch:    43, epoch:   74/100, mu: 0.9789912105, learning_rate: 0.0002808383, weight_decay: 0.0000112335

epoch time: 83.8244628906s, input_fired_time: 1.2775045633, output_fired_time: 1.1168357134, propagate_forward_time: 30.4633922577, propagate_backward_time: 48.1157112122, weight_update_time: 0.1332706213, other_time: 2.8510169983
epoch time: 6.7620573044s, input_fired_time: 0.2199893147, output_fired_time: 0.0000000000, propagate_forward_time: 6.0513534546, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4907145500
validation[          , genome  4483] predictions:    9628/  10000 (96.28%), best:    9660/10000 (96.60%), error:      1175.38831, best error:      1149.71130 on epoch:    43, epoch:   75/100, mu: 0.9795416594, learning_rate: 0.0002667964, weight_decay: 0.0000106719

epoch time: 88.0268020630s, input_fired_time: 1.3489843607, output_fired_time: 1.1356340647, propagate_forward_time: 32.2955207825, propagate_backward_time: 50.1247100830, weight_update_time: 0.1796611845, other_time: 3.1219520569
epoch time: 7.7985749245s, input_fired_time: 0.2388435900, output_fired_time: 0.0000000000, propagate_forward_time: 6.9265084267, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6332230568
validation[          , genome  4483] predictions:    9623/  10000 (96.23%), best:    9660/10000 (96.60%), error:      1250.81677, best error:      1149.71130 on epoch:    43, epoch:   76/100, mu: 0.9800645709, learning_rate: 0.0002534565, weight_decay: 0.0000101383

epoch time: 84.2117080688s, input_fired_time: 1.4267877340, output_fired_time: 1.1411890984, propagate_forward_time: 30.7091369629, propagate_backward_time: 48.1906509399, weight_update_time: 0.1176798195, other_time: 2.7439422607
epoch time: 6.7650671005s, input_fired_time: 0.1479058117, output_fired_time: 0.0000000000, propagate_forward_time: 6.1281604767, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4890007973
validation[          , genome  4483] predictions:    9644/  10000 (96.44%), best:    9660/10000 (96.60%), error:      1201.24878, best error:      1149.71130 on epoch:    43, epoch:   77/100, mu: 0.9805613160, learning_rate: 0.0002407837, weight_decay: 0.0000096314

epoch time: 86.8781356812s, input_fired_time: 1.3136907816, output_fired_time: 1.2225254774, propagate_forward_time: 31.6600990295, propagate_backward_time: 49.7830772400, weight_update_time: 0.1306147724, other_time: 2.8987426758
epoch time: 7.5887265205s, input_fired_time: 0.1648329198, output_fired_time: 0.0000000000, propagate_forward_time: 6.8503026962, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5735907555
validation[          , genome  4483] predictions:    9592/  10000 (95.92%), best:    9660/10000 (96.60%), error:      1334.27917, best error:      1149.71130 on epoch:    43, epoch:   78/100, mu: 0.9810332656, learning_rate: 0.0002287445, weight_decay: 0.0000091498

epoch time: 88.9423599243s, input_fired_time: 1.3226052523, output_fired_time: 1.2356214523, propagate_forward_time: 32.2066192627, propagate_backward_time: 50.8398704529, weight_update_time: 0.1317033917, other_time: 3.3376426697
epoch time: 6.3461694717s, input_fired_time: 0.1439934969, output_fired_time: 0.0000000000, propagate_forward_time: 5.7282123566, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4739637375
validation[          , genome  4483] predictions:    9548/  10000 (95.48%), best:    9660/10000 (96.60%), error:      1515.70605, best error:      1149.71130 on epoch:    43, epoch:   79/100, mu: 0.9814816117, learning_rate: 0.0002173073, weight_decay: 0.0000086923

epoch time: 107.9742279053s, input_fired_time: 1.4750272036, output_fired_time: 1.2712259293, propagate_forward_time: 39.7534828186, propagate_backward_time: 62.0279579163, weight_update_time: 0.2041313797, other_time: 3.4465293884
epoch time: 11.4073915482s, input_fired_time: 0.1781595647, output_fired_time: 0.0000000000, propagate_forward_time: 10.5264520645, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7027797699
validation[          , genome  4483] predictions:    9602/  10000 (96.02%), best:    9660/10000 (96.60%), error:      1334.66541, best error:      1149.71130 on epoch:    43, epoch:   80/100, mu: 0.9819075465, learning_rate: 0.0002064419, weight_decay: 0.0000082577

epoch time: 125.8190689087s, input_fired_time: 1.6441909075, output_fired_time: 1.5238852501, propagate_forward_time: 47.0420684814, propagate_backward_time: 71.3282318115, weight_update_time: 0.1459012777, other_time: 4.2806930542
epoch time: 9.8284473419s, input_fired_time: 0.1655706912, output_fired_time: 0.0000000000, propagate_forward_time: 8.8802013397, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7826757431
validation[          , genome  4483] predictions:    9576/  10000 (95.76%), best:    9660/10000 (96.60%), error:      1338.91077, best error:      1149.71130 on epoch:    43, epoch:   81/100, mu: 0.9823121428, learning_rate: 0.0001961198, weight_decay: 0.0000078448

epoch time: 125.8001632690s, input_fired_time: 1.3653066158, output_fired_time: 1.4462362528, propagate_forward_time: 47.0507354736, propagate_backward_time: 71.6387710571, weight_update_time: 0.1305486560, other_time: 4.2991180420
epoch time: 10.0506315231s, input_fired_time: 0.4273688197, output_fired_time: 0.0000000000, propagate_forward_time: 9.0768556595, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5464067459
validation[          , genome  4483] predictions:    9611/  10000 (96.11%), best:    9660/10000 (96.60%), error:      1264.91382, best error:      1149.71130 on epoch:    43, epoch:   82/100, mu: 0.9826965332, learning_rate: 0.0001863138, weight_decay: 0.0000074526

epoch time: 124.5096511841s, input_fired_time: 2.3662431240, output_fired_time: 1.2451201677, propagate_forward_time: 45.9860191345, propagate_backward_time: 70.4374465942, weight_update_time: 0.1614520997, other_time: 4.4748306274
epoch time: 10.1637935638s, input_fired_time: 0.2209442258, output_fired_time: 0.0000000000, propagate_forward_time: 9.3985195160, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5443296432
validation[          , genome  4483] predictions:    9567/  10000 (95.67%), best:    9660/10000 (96.60%), error:      1426.85400, best error:      1149.71130 on epoch:    43, epoch:   83/100, mu: 0.9830617309, learning_rate: 0.0001769981, weight_decay: 0.0000070799

epoch time: 124.1191177368s, input_fired_time: 1.5796021223, output_fired_time: 1.6803903580, propagate_forward_time: 45.9335670471, propagate_backward_time: 70.8776473999, weight_update_time: 0.3802562356, other_time: 4.0479049683
epoch time: 10.1414270401s, input_fired_time: 0.1593164802, output_fired_time: 0.0000000000, propagate_forward_time: 9.1715965271, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.8105144501
validation[          , genome  4483] predictions:    9559/  10000 (95.59%), best:    9660/10000 (96.60%), error:      1481.09094, best error:      1149.71130 on epoch:    43, epoch:   84/100, mu: 0.9834086299, learning_rate: 0.0001681482, weight_decay: 0.0000067259

epoch time: 129.1344604492s, input_fired_time: 1.8032360077, output_fired_time: 1.3004561663, propagate_forward_time: 47.4185485840, propagate_backward_time: 74.0833587646, weight_update_time: 0.1622334421, other_time: 4.5288619995
epoch time: 11.8819303513s, input_fired_time: 0.2089365274, output_fired_time: 0.0000000000, propagate_forward_time: 10.7985277176, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.8744659424
validation[          , genome  4483] predictions:    9622/  10000 (96.22%), best:    9660/10000 (96.60%), error:      1230.80591, best error:      1149.71130 on epoch:    43, epoch:   85/100, mu: 0.9837381840, learning_rate: 0.0001597408, weight_decay: 0.0000063896

epoch time: 131.8483276367s, input_fired_time: 2.0325639248, output_fired_time: 1.5077407360, propagate_forward_time: 48.6453285217, propagate_backward_time: 73.9931335449, weight_update_time: 0.1846285909, other_time: 5.6695709229
epoch time: 10.9601211548s, input_fired_time: 0.2089918703, output_fired_time: 0.0000000000, propagate_forward_time: 10.1278018951, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6233272552
validation[          , genome  4483] predictions:    9627/  10000 (96.27%), best:    9660/10000 (96.60%), error:      1187.68164, best error:      1149.71130 on epoch:    43, epoch:   86/100, mu: 0.9840512872, learning_rate: 0.0001517538, weight_decay: 0.0000060702

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/mnist_training_50k.bin'
boincified validation filename: '../../projects/csgrid.org_csg/mnist_validation_10k.bin'
boincified testing filename: '../../projects/csgrid.org_csg/mnist_testing_new.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_42_4483.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_42_4483_0_r1719403357_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.984349
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000144166
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 5.76664e-06
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 87
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 43
read number_validation_images: 10000
read best_validation_predictions: 9660
read best_validation_error: 1149.71
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 4483
read normal distribution: '1 0 0'
generator_str: '2061162936'
read generator: 2061162936
read generated_by_map:
1 enable_edge 1
reading 22 nodes.
reading 43 edges.
number input nodes: 1
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 4923 images.
reading image set with 5742 images.
reading image set with 4958 images.
reading image set with 5131 images.
reading image set with 4842 images.
reading image set with 4421 images.
reading image set with 4918 images.
reading image set with 5265 images.
reading image set with 4851 images.
reading image set with 4949 images.
image_size: 1x28x28 = 784
read 50000 images.
    class    0: 4923
    class    1: 5742
    class    2: 4958
    class    3: 5131
    class    4: 4842
    class    5: 4421
    class    6: 4918
    class    7: 5265
    class    8: 4851
    class    9: 4949
calculating averages and standard deviations for images
average pixel value for channel 0: 0.130530551075935
pixel variance for channel 0: 0.0948646068572998
pixel standard deviation for channel 0: 0.308000981807709
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 980 images.
reading image set with 1135 images.
reading image set with 1032 images.
reading image set with 1010 images.
reading image set with 982 images.
reading image set with 892 images.
reading image set with 958 images.
reading image set with 1028 images.
reading image set with 974 images.
reading image set with 1009 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 980
    class    1: 1135
    class    2: 1032
    class    3: 1010
    class    4: 982
    class    5: 892
    class    6: 958
    class    7: 1028
    class    8: 974
    class    9: 1009
loaded images
starting backpropagation!
epoch time: 136.449234008789s, input_fired_time: 1.79822182655334, output_fired_time: 1.48407852649689, propagate_forward_time: 52.0484580993652, propagate_backward_time: 76.1393051147461, weight_update_time: 0.427050054073334, other_time: 4.97916412353516
epoch time: 11.7165908813477s, input_fired_time: 0.261138707399368, output_fired_time: 0, propagate_forward_time: 10.0541687011719, propagate_backward_time: 0, weight_update_time: 0, other_time: 1.40128326416016
validation[          , genome  4483] predictions:    9614/  10000 (96.14%), best:    9660/10000 (96.60%), error:      1271.88916, best error:      1149.71130 on epoch:    43, epoch:   87/100, mu: 0.9843487144, learning_rate: 0.0001441661, weight_decay: 0.0000057666

epoch time: 154.2129058838s, input_fired_time: 2.7755334377, output_fired_time: 1.4467879534, propagate_forward_time: 55.0198059082, propagate_backward_time: 89.5722198486, weight_update_time: 0.6620715261, other_time: 5.3985595703
epoch time: 13.4211606979s, input_fired_time: 0.2066843063, output_fired_time: 0.0000000000, propagate_forward_time: 12.2963247299, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9181518555
validation[          , genome  4483] predictions:    9549/  10000 (95.49%), best:    9660/10000 (96.60%), error:      1432.51831, best error:      1149.71130 on epoch:    43, epoch:   88/100, mu: 0.9846313000, learning_rate: 0.0001369578, weight_decay: 0.0000054783

epoch time: 161.0474700928s, input_fired_time: 1.5313510895, output_fired_time: 2.1470899582, propagate_forward_time: 58.9461593628, propagate_backward_time: 92.6479415894, weight_update_time: 0.1304788142, other_time: 5.7749176025
epoch time: 14.6552581787s, input_fired_time: 0.1768064350, output_fired_time: 0.0000000000, propagate_forward_time: 13.5606431961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9178085327
validation[          , genome  4483] predictions:    9617/  10000 (96.17%), best:    9660/10000 (96.60%), error:      1289.96265, best error:      1149.71130 on epoch:    43, epoch:   89/100, mu: 0.9848997593, learning_rate: 0.0001301099, weight_decay: 0.0000052044

epoch time: 166.2064514160s, input_fired_time: 2.2714803219, output_fired_time: 3.2463662624, propagate_forward_time: 62.5610618591, propagate_backward_time: 93.4970169067, weight_update_time: 0.2339687794, other_time: 4.6305160522
epoch time: 14.9537448883s, input_fired_time: 0.4558296204, output_fired_time: 0.0000000000, propagate_forward_time: 13.5182046890, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9797105789
validation[          , genome  4483] predictions:    9422/  10000 (94.22%), best:    9660/10000 (96.60%), error:      1898.97156, best error:      1149.71130 on epoch:    43, epoch:   90/100, mu: 0.9851547480, learning_rate: 0.0001236044, weight_decay: 0.0000049442

epoch time: 162.7062835693s, input_fired_time: 3.2353680134, output_fired_time: 1.8936889172, propagate_forward_time: 59.9698333740, propagate_backward_time: 91.5342178345, weight_update_time: 0.4274817705, other_time: 6.0731735229
epoch time: 11.8464002609s, input_fired_time: 0.1781775802, output_fired_time: 0.0000000000, propagate_forward_time: 10.7049598694, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9632625580
validation[          , genome  4483] predictions:    9579/  10000 (95.79%), best:    9660/10000 (96.60%), error:      1332.09290, best error:      1149.71130 on epoch:    43, epoch:   91/100, mu: 0.9853969812, learning_rate: 0.0001174242, weight_decay: 0.0000046970

epoch time: 162.1838989258s, input_fired_time: 2.6880321503, output_fired_time: 2.4157927036, propagate_forward_time: 58.9337615967, propagate_backward_time: 93.0975189209, weight_update_time: 0.1928384453, other_time: 5.0487976074
epoch time: 12.9924230576s, input_fired_time: 0.1775611490, output_fired_time: 0.0000000000, propagate_forward_time: 11.7899303436, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0249319077
validation[          , genome  4483] predictions:    9606/  10000 (96.06%), best:    9660/10000 (96.60%), error:      1275.07849, best error:      1149.71130 on epoch:    43, epoch:   92/100, mu: 0.9856271148, learning_rate: 0.0001115530, weight_decay: 0.0000044621

epoch time: 156.2468566895s, input_fired_time: 2.4068725109, output_fired_time: 1.5242173672, propagate_forward_time: 59.5255470276, propagate_backward_time: 87.1375045776, weight_update_time: 0.1303380281, other_time: 5.6527175903
epoch time: 11.0071392059s, input_fired_time: 0.1619663090, output_fired_time: 0.0000000000, propagate_forward_time: 10.2604303360, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5847425461
validation[          , genome  4483] predictions:    9633/  10000 (96.33%), best:    9660/10000 (96.60%), error:      1250.44885, best error:      1149.71130 on epoch:    43, epoch:   93/100, mu: 0.9858457446, learning_rate: 0.0001059753, weight_decay: 0.0000042390

epoch time: 138.1104431152s, input_fired_time: 1.9830472469, output_fired_time: 1.5142173767, propagate_forward_time: 51.8477973938, propagate_backward_time: 78.4928665161, weight_update_time: 0.6450901031, other_time: 4.2725143433
epoch time: 10.0197105408s, input_fired_time: 0.1765867621, output_fired_time: 0.0000000000, propagate_forward_time: 9.3050022125, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5381212234
validation[          , genome  4483] predictions:    9593/  10000 (95.93%), best:    9660/10000 (96.60%), error:      1347.40369, best error:      1149.71130 on epoch:    43, epoch:   94/100, mu: 0.9860534668, learning_rate: 0.0001006766, weight_decay: 0.0000040271

epoch time: 127.2547607422s, input_fired_time: 1.3786664009, output_fired_time: 1.6449661255, propagate_forward_time: 48.4649581909, propagate_backward_time: 72.3365020752, weight_update_time: 0.1302995533, other_time: 3.4296646118
epoch time: 11.2946538925s, input_fired_time: 0.2083160579, output_fired_time: 0.0000000000, propagate_forward_time: 10.4738254547, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6125125885
validation[          , genome  4483] predictions:    9601/  10000 (96.01%), best:    9660/10000 (96.60%), error:      1317.61243, best error:      1149.71130 on epoch:    43, epoch:   95/100, mu: 0.9862508178, learning_rate: 0.0000956427, weight_decay: 0.0000038257

epoch time: 125.5672988892s, input_fired_time: 1.6420152187, output_fired_time: 1.6257861853, propagate_forward_time: 46.7469482422, propagate_backward_time: 71.2488632202, weight_update_time: 0.1302719712, other_time: 4.3036880493
epoch time: 10.1750593185s, input_fired_time: 0.4118310213, output_fired_time: 0.0000000000, propagate_forward_time: 8.9269762039, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.8362522125
validation[          , genome  4483] predictions:    9595/  10000 (95.95%), best:    9660/10000 (96.60%), error:      1349.13611, best error:      1149.71130 on epoch:    43, epoch:   96/100, mu: 0.9864382744, learning_rate: 0.0000908606, weight_decay: 0.0000036344

epoch time: 125.6617507935s, input_fired_time: 1.7851636410, output_fired_time: 1.2712259293, propagate_forward_time: 46.2163238525, propagate_backward_time: 72.2578964233, weight_update_time: 0.1457530707, other_time: 4.1311416626
epoch time: 9.9310255051s, input_fired_time: 0.1567319781, output_fired_time: 0.0000000000, propagate_forward_time: 8.7516651154, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0226287842
validation[          , genome  4483] predictions:    9552/  10000 (95.52%), best:    9660/10000 (96.60%), error:      1460.73401, best error:      1149.71130 on epoch:    43, epoch:   97/100, mu: 0.9866163731, learning_rate: 0.0000863176, weight_decay: 0.0000034527

epoch time: 125.3861236572s, input_fired_time: 2.2067291737, output_fired_time: 1.8184850216, propagate_forward_time: 46.4338951111, propagate_backward_time: 71.3748550415, weight_update_time: 0.1301173121, other_time: 3.5521621704
epoch time: 10.6387052536s, input_fired_time: 0.1635493785, output_fired_time: 0.0000000000, propagate_forward_time: 9.4373054504, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.0378503799
validation[          , genome  4483] predictions:    9612/  10000 (96.12%), best:    9660/10000 (96.60%), error:      1267.91846, best error:      1149.71130 on epoch:    43, epoch:   98/100, mu: 0.9867855310, learning_rate: 0.0000820017, weight_decay: 0.0000032801

epoch time: 122.2911911011s, input_fired_time: 1.8193489313, output_fired_time: 1.4308531284, propagate_forward_time: 45.9561195374, propagate_backward_time: 69.7431259155, weight_update_time: 0.1447899938, other_time: 3.3417434692
epoch time: 6.2008471489s, input_fired_time: 0.1412754059, output_fired_time: 0.0000000000, propagate_forward_time: 5.5995707512, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4600009918
validation[          , genome  4483] predictions:    9584/  10000 (95.84%), best:    9660/10000 (96.60%), error:      1298.51904, best error:      1149.71130 on epoch:    43, epoch:   99/100, mu: 0.9869462252, learning_rate: 0.0000779016, weight_decay: 0.0000031161

epoch time: 75.4534606934s, input_fired_time: 1.1666288376, output_fired_time: 0.9853222370, propagate_forward_time: 27.3680133820, propagate_backward_time: 43.2801780701, weight_update_time: 0.1123178303, other_time: 2.6533241272
epoch time: 6.1683988571s, input_fired_time: 0.1442364454, output_fired_time: 0.0000000000, propagate_forward_time: 5.5627832413, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4613790512
validation[          , genome  4483] predictions:    9623/  10000 (96.23%), best:    9660/10000 (96.60%), error:      1164.90430, best error:      1149.71130 on epoch:    43, epoch:  100/100, mu: 0.9870989323, learning_rate: 0.0000740065, weight_decay: 0.0000029603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 30.3275203705s, input_fired_time: 0.7275121212, output_fired_time: 0.0000000000, propagate_forward_time: 27.1991653442, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.4008426666
best training[          , genome  4483] predictions:   49003/  50000 (98.01%), best:    9660/10000 (96.60%), error:      3622.50879, best error:      1149.71130 on epoch:    43, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 5.9114227295s, input_fired_time: 0.1356131583, output_fired_time: 0.0000000000, propagate_forward_time: 5.2996101379, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4761996269
   testing[          , genome  4483] predictions:    9700/  10000 (97.00%), best:    9660/10000 (96.60%), error:      1025.28625, best error:      1149.71130 on epoch:    43, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
backpropagation finished successfully!
00:08:55 (5436): called boinc_finish(0)

</stderr_txt>
]]>