Nameexact_genome_1512568488_43_4468_0
Workunit2593654
Created7 Dec 2017, 10:09:20 UTC
Sent7 Dec 2017, 10:12:12 UTC
Report deadline11 Dec 2017, 18:11:49 UTC
Received7 Dec 2017, 23:52:35 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID64142
Run time3 hours 32 min 11 sec
CPU time2 hours 19 min 59 sec
Validate stateValid
Credit1,083.27
Device peak FLOPS3.08 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size153.28 MB
Peak swap size156.33 MB
Peak disk usage1.58 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
on epoch:    24, epoch:   26/100, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 232.1827545166s, input_fired_time: 2.3748466969, output_fired_time: 1.8603639603, propagate_forward_time: 90.2380981445, propagate_backward_time: 129.1622009277, weight_update_time: 1.2499550581, other_time: 8.5472412109
epoch time: 19.1922531128s, input_fired_time: 0.1342640966, output_fired_time: 0.0000000000, propagate_forward_time: 17.6214466095, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.4365425110
validation[          , genome  4468] predictions:    9578/  10000 (95.78%), best:    9631/10000 (96.31%), error:      1308.67139, best error:      1190.88782 on epoch:    24, epoch:   27/100, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

epoch time: 210.1373443604s, input_fired_time: 1.3368773460, output_fired_time: 2.0563449860, propagate_forward_time: 80.6134414673, propagate_backward_time: 119.5314407349, weight_update_time: 0.1232610270, other_time: 6.5992279053
epoch time: 6.4507451057s, input_fired_time: 0.0828771293, output_fired_time: 0.0000000000, propagate_forward_time: 5.9230504036, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4448175430
validation[          , genome  4468] predictions:    9528/  10000 (95.28%), best:    9631/10000 (96.31%), error:      1551.66589, best error:      1190.88782 on epoch:    24, epoch:   28/100, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 88.9155120850s, input_fired_time: 0.7710619569, output_fired_time: 0.5745140910, propagate_forward_time: 33.8088569641, propagate_backward_time: 51.0299720764, weight_update_time: 0.1310989708, other_time: 2.7311019897
epoch time: 9.0758132935s, input_fired_time: 0.0805249363, output_fired_time: 0.0000000000, propagate_forward_time: 8.5135440826, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4817438126
validation[          , genome  4468] predictions:    9611/  10000 (96.11%), best:    9631/10000 (96.31%), error:      1247.85120, best error:      1190.88782 on epoch:    24, epoch:   29/100, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 95.6807785034s, input_fired_time: 0.7141938210, output_fired_time: 0.8169168234, propagate_forward_time: 36.3836097717, propagate_backward_time: 54.9022369385, weight_update_time: 0.1320694834, other_time: 2.8638191223
epoch time: 7.5399599075s, input_fired_time: 0.0802899376, output_fired_time: 0.0000000000, propagate_forward_time: 6.9605197906, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4991502762
validation[          , genome  4468] predictions:    9634/  10000 (96.34%), best:    9631/10000 (96.31%), error:      1201.44604, best error:      1190.88782 on epoch:    24, epoch:   30/100, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

epoch time: 101.7143020630s, input_fired_time: 0.8875208497, output_fired_time: 0.8331885338, propagate_forward_time: 38.7050399780, propagate_backward_time: 58.2292556763, weight_update_time: 0.1483857930, other_time: 3.0592956543
epoch time: 8.9460039139s, input_fired_time: 0.1462098211, output_fired_time: 0.0000000000, propagate_forward_time: 8.3479633331, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4518308640
validation[          , genome  4468] predictions:    9621/  10000 (96.21%), best:    9631/10000 (96.31%), error:      1271.29688, best error:      1190.88782 on epoch:    24, epoch:   31/100, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 98.9007797241s, input_fired_time: 0.8724397421, output_fired_time: 0.7323576212, propagate_forward_time: 37.3666076660, propagate_backward_time: 56.9685287476, weight_update_time: 0.1368144602, other_time: 2.9608459473
epoch time: 7.4138698578s, input_fired_time: 0.0846512467, output_fired_time: 0.0000000000, propagate_forward_time: 6.8532829285, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4759354591
validation[          , genome  4468] predictions:    9581/  10000 (95.81%), best:    9631/10000 (96.31%), error:      1408.72058, best error:      1190.88782 on epoch:    24, epoch:   32/100, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 107.2223510742s, input_fired_time: 1.0347573757, output_fired_time: 0.7687307000, propagate_forward_time: 40.5819740295, propagate_backward_time: 61.7150077820, weight_update_time: 0.1219015867, other_time: 3.1218757629
epoch time: 9.1831188202s, input_fired_time: 0.1010688990, output_fired_time: 0.0000000000, propagate_forward_time: 8.3382968903, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7437534332
validation[          , genome  4468] predictions:    9577/  10000 (95.77%), best:    9631/10000 (96.31%), error:      1380.08130, best error:      1190.88782 on epoch:    24, epoch:   33/100, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 102.9247131348s, input_fired_time: 0.8624026775, output_fired_time: 0.6905325651, propagate_forward_time: 39.0279464722, propagate_backward_time: 58.9618797302, weight_update_time: 0.1795915067, other_time: 3.3819541931
epoch time: 8.5540714264s, input_fired_time: 0.1491262019, output_fired_time: 0.0000000000, propagate_forward_time: 7.7842445374, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6207008362
validation[          , genome  4468] predictions:    9655/  10000 (96.55%), best:    9655/10000 (96.55%), error:      1150.94116, best error:      1150.94116 on epoch:    34, epoch:   34/100, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 99.9154205322s, input_fired_time: 0.8330522776, output_fired_time: 0.6740386486, propagate_forward_time: 37.7922668457, propagate_backward_time: 57.4214897156, weight_update_time: 0.2009588629, other_time: 3.1945686340
epoch time: 8.5143241882s, input_fired_time: 0.0853250697, output_fired_time: 0.0000000000, propagate_forward_time: 7.9022808075, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5267181396
validation[          , genome  4468] predictions:    9611/  10000 (96.11%), best:    9655/10000 (96.55%), error:      1256.29053, best error:      1150.94116 on epoch:    34, epoch:   35/100, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 102.1636352539s, input_fired_time: 0.7842164040, output_fired_time: 0.7493525743, propagate_forward_time: 38.6798629761, propagate_backward_time: 58.6955680847, weight_update_time: 0.2007070184, other_time: 3.2546348572
epoch time: 7.6337261200s, input_fired_time: 0.1161701456, output_fired_time: 0.0000000000, propagate_forward_time: 7.0234384537, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4941177368
validation[          , genome  4468] predictions:    9618/  10000 (96.18%), best:    9655/10000 (96.55%), error:      1294.91980, best error:      1150.94116 on epoch:    34, epoch:   36/100, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 106.2984848022s, input_fired_time: 0.9293082952, output_fired_time: 0.8621448874, propagate_forward_time: 40.5842742920, propagate_backward_time: 60.5992240906, weight_update_time: 0.1223209128, other_time: 3.3235359192
epoch time: 8.6475009918s, input_fired_time: 0.1000904068, output_fired_time: 0.0000000000, propagate_forward_time: 7.9014434814, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6459674835
validation[          , genome  4468] predictions:    9570/  10000 (95.70%), best:    9655/10000 (96.55%), error:      1454.63477, best error:      1150.94116 on epoch:    34, epoch:   37/100, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 100.0831527710s, input_fired_time: 0.8133423328, output_fired_time: 0.7195632458, propagate_forward_time: 38.0986785889, propagate_backward_time: 57.4483985901, weight_update_time: 0.1374042481, other_time: 3.0031700134
epoch time: 8.8543729782s, input_fired_time: 0.1013689116, output_fired_time: 0.0000000000, propagate_forward_time: 7.9760518074, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7769522667
validation[          , genome  4468] predictions:    9630/  10000 (96.30%), best:    9655/10000 (96.55%), error:      1254.93542, best error:      1150.94116 on epoch:    34, epoch:   38/100, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 99.8922805786s, input_fired_time: 0.7809591889, output_fired_time: 0.8403922319, propagate_forward_time: 37.9679222107, propagate_backward_time: 57.4476509094, weight_update_time: 0.1527342647, other_time: 2.8553543091
epoch time: 8.7848081589s, input_fired_time: 0.0848919153, output_fired_time: 0.0000000000, propagate_forward_time: 8.1425867081, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5573291779
validation[          , genome  4468] predictions:    9614/  10000 (96.14%), best:    9655/10000 (96.55%), error:      1210.62463, best error:      1150.94116 on epoch:    34, epoch:   39/100, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 101.2648849487s, input_fired_time: 0.8114192486, output_fired_time: 0.8223522305, propagate_forward_time: 38.5584411621, propagate_backward_time: 57.9307594299, weight_update_time: 0.1520363539, other_time: 3.1419181824
epoch time: 7.9955773354s, input_fired_time: 0.0857315660, output_fired_time: 0.0000000000, propagate_forward_time: 7.2990570068, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6107888222
validation[          , genome  4468] predictions:    9651/  10000 (96.51%), best:    9651/10000 (96.51%), error:      1136.24841, best error:      1136.24841 on epoch:    40, epoch:   40/100, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 98.1961822510s, input_fired_time: 0.7646199465, output_fired_time: 0.9323043823, propagate_forward_time: 36.9845199585, propagate_backward_time: 56.5310516357, weight_update_time: 0.1522701979, other_time: 2.9836883545
epoch time: 8.6902265549s, input_fired_time: 0.0847603232, output_fired_time: 0.0000000000, propagate_forward_time: 7.8824539185, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7230119705
validation[          , genome  4468] predictions:    9579/  10000 (95.79%), best:    9651/10000 (96.51%), error:      1367.08618, best error:      1136.24841 on epoch:    40, epoch:   41/100, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 96.0132827759s, input_fired_time: 0.7590262294, output_fired_time: 0.8628571033, propagate_forward_time: 36.3645172119, propagate_backward_time: 55.0871849060, weight_update_time: 0.1192278787, other_time: 2.9397010803
epoch time: 7.0275912285s, input_fired_time: 0.0843435675, output_fired_time: 0.0000000000, propagate_forward_time: 6.5075736046, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4356741905
validation[          , genome  4468] predictions:    9592/  10000 (95.92%), best:    9651/10000 (96.51%), error:      1311.22559, best error:      1136.24841 on epoch:    40, epoch:   42/100, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 94.5121231079s, input_fired_time: 0.7862214446, output_fired_time: 0.7492201328, propagate_forward_time: 35.8623237610, propagate_backward_time: 54.1243667603, weight_update_time: 0.1507083029, other_time: 2.9899864197
epoch time: 7.8938302994s, input_fired_time: 0.0855232030, output_fired_time: 0.0000000000, propagate_forward_time: 7.2211070061, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5872001648
validation[          , genome  4468] predictions:    9571/  10000 (95.71%), best:    9651/10000 (96.51%), error:      1380.14026, best error:      1136.24841 on epoch:    40, epoch:   43/100, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 97.2263259888s, input_fired_time: 0.6986365914, output_fired_time: 0.7727299929, propagate_forward_time: 37.0966339111, propagate_backward_time: 55.6258926392, weight_update_time: 0.1356624365, other_time: 3.0324325562
epoch time: 7.5192217827s, input_fired_time: 0.0968739837, output_fired_time: 0.0000000000, propagate_forward_time: 6.9088835716, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5134644508
validation[          , genome  4468] predictions:    9550/  10000 (95.50%), best:    9651/10000 (96.51%), error:      1444.89893, best error:      1136.24841 on epoch:    40, epoch:   44/100, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 93.8100967407s, input_fired_time: 0.9158672690, output_fired_time: 0.7049311399, propagate_forward_time: 35.2950897217, propagate_backward_time: 54.0190505981, weight_update_time: 0.1189201921, other_time: 2.8751525879
epoch time: 8.0021753311s, input_fired_time: 0.0861007869, output_fired_time: 0.0000000000, propagate_forward_time: 7.3767051697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5393695831
validation[          , genome  4468] predictions:    9634/  10000 (96.34%), best:    9651/10000 (96.51%), error:      1251.25098, best error:      1136.24841 on epoch:    40, epoch:   45/100, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 89.6459274292s, input_fired_time: 0.8442756534, output_fired_time: 0.8118919134, propagate_forward_time: 34.2940673828, propagate_backward_time: 51.0476760864, weight_update_time: 0.1184819043, other_time: 2.6480178833
epoch time: 8.9993324280s, input_fired_time: 0.0754990429, output_fired_time: 0.0000000000, propagate_forward_time: 8.4252347946, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4985990524
validation[          , genome  4468] predictions:    9652/  10000 (96.52%), best:    9652/10000 (96.52%), error:      1113.42310, best error:      1113.42310 on epoch:    46, epoch:   46/100, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 86.0415649414s, input_fired_time: 0.8465585709, output_fired_time: 0.5659769773, propagate_forward_time: 32.3380126953, propagate_backward_time: 49.4810447693, weight_update_time: 0.1761524230, other_time: 2.8099708557
epoch time: 6.6954021454s, input_fired_time: 0.0856901184, output_fired_time: 0.0000000000, propagate_forward_time: 6.1682267189, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4414854050
validation[          , genome  4468] predictions:    9675/  10000 (96.75%), best:    9675/10000 (96.75%), error:      1073.01453, best error:      1073.01453 on epoch:    47, epoch:   47/100, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 87.9929046631s, input_fired_time: 0.7436054945, output_fired_time: 0.6045135856, propagate_forward_time: 33.7318840027, propagate_backward_time: 50.2081527710, weight_update_time: 0.1149572656, other_time: 2.7047462463
epoch time: 7.1172714233s, input_fired_time: 0.0778205022, output_fired_time: 0.0000000000, propagate_forward_time: 6.6380271912, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4014239311
validation[          , genome  4468] predictions:    9658/  10000 (96.58%), best:    9675/10000 (96.75%), error:      1095.12781, best error:      1073.01453 on epoch:    47, epoch:   48/100, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 84.3046188354s, input_fired_time: 0.7647147775, output_fired_time: 0.5983446836, propagate_forward_time: 31.5282039642, propagate_backward_time: 48.6589508057, weight_update_time: 0.1166084781, other_time: 2.7544021606
epoch time: 6.8962750435s, input_fired_time: 0.0846845359, output_fired_time: 0.0000000000, propagate_forward_time: 6.3718175888, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4397730827
validation[          , genome  4468] predictions:    9673/  10000 (96.73%), best:    9675/10000 (96.75%), error:      1098.29199, best error:      1073.01453 on epoch:    47, epoch:   49/100, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 80.5766448975s, input_fired_time: 0.6506283283, output_fired_time: 0.6838586926, propagate_forward_time: 30.6788215637, propagate_backward_time: 46.0958366394, weight_update_time: 0.1150305197, other_time: 2.4674987793
epoch time: 5.9369287491s, input_fired_time: 0.0796125457, output_fired_time: 0.0000000000, propagate_forward_time: 5.4470663071, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4102497101
validation[          , genome  4468] predictions:    9597/  10000 (95.97%), best:    9675/10000 (96.75%), error:      1323.55640, best error:      1073.01453 on epoch:    47, epoch:   50/100, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

epoch time: 81.0997238159s, input_fired_time: 0.6642626524, output_fired_time: 0.6486941576, propagate_forward_time: 30.8822822571, propagate_backward_time: 46.3906822205, weight_update_time: 0.1139250398, other_time: 2.5138015747
epoch time: 7.4375505447s, input_fired_time: 0.1533692479, output_fired_time: 0.0000000000, propagate_forward_time: 6.8044695854, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4797115326
validation[          , genome  4468] predictions:    9628/  10000 (96.28%), best:    9675/10000 (96.75%), error:      1220.09424, best error:      1073.01453 on epoch:    47, epoch:   51/100, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488

epoch time: 84.7950210571s, input_fired_time: 0.7617653012, output_fired_time: 0.6662676334, propagate_forward_time: 32.1657905579, propagate_backward_time: 48.5978660583, weight_update_time: 0.1156873628, other_time: 2.6033325195
epoch time: 6.3206529617s, input_fired_time: 0.0993423462, output_fired_time: 0.0000000000, propagate_forward_time: 5.7874875069, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4338231087
validation[          , genome  4468] predictions:    9658/  10000 (96.58%), best:    9675/10000 (96.75%), error:      1110.64270, best error:      1073.01453 on epoch:    47, epoch:   52/100, mu: 0.9559730291, learning_rate: 0.0008680347, weight_decay: 0.0000347214

epoch time: 79.5424499512s, input_fired_time: 0.6757020354, output_fired_time: 0.6493819952, propagate_forward_time: 30.1326866150, propagate_backward_time: 45.5170898438, weight_update_time: 0.1176191643, other_time: 2.5675849915
epoch time: 6.6010518074s, input_fired_time: 0.0776930153, output_fired_time: 0.0000000000, propagate_forward_time: 6.0763134956, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4470453262
validation[          , genome  4468] predictions:    9636/  10000 (96.36%), best:    9675/10000 (96.75%), error:      1210.14307, best error:      1073.01453 on epoch:    47, epoch:   53/100, mu: 0.9576743841, learning_rate: 0.0008246329, weight_decay: 0.0000329853

epoch time: 87.1183853149s, input_fired_time: 0.6632205248, output_fired_time: 0.6316282153, propagate_forward_time: 33.0629348755, propagate_backward_time: 49.9927406311, weight_update_time: 0.1461695433, other_time: 2.7678565979
epoch time: 6.8116278648s, input_fired_time: 0.0793985426, output_fired_time: 0.0000000000, propagate_forward_time: 6.2756252289, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4566040039
validation[          , genome  4468] predictions:    9655/  10000 (96.55%), best:    9675/10000 (96.75%), error:      1154.21094, best error:      1073.01453 on epoch:    47, epoch:   54/100, mu: 0.9592906833, learning_rate: 0.0007834013, weight_decay: 0.0000313361

epoch time: 86.6636886597s, input_fired_time: 0.7494859099, output_fired_time: 0.6091231704, propagate_forward_time: 32.5071411133, propagate_backward_time: 49.9044570923, weight_update_time: 0.1783181876, other_time: 2.8934783936
epoch time: 6.5966258049s, input_fired_time: 0.0858590752, output_fired_time: 0.0000000000, propagate_forward_time: 6.0539650917, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4568014145
validation[          , genome  4468] predictions:    9650/  10000 (96.50%), best:    9675/10000 (96.75%), error:      1134.77844, best error:      1073.01453 on epoch:    47, epoch:   55/100, mu: 0.9608261585, learning_rate: 0.0007442312, weight_decay: 0.0000297693

epoch time: 88.0773696899s, input_fired_time: 0.7186608315, output_fired_time: 0.6175825000, propagate_forward_time: 33.4346275330, propagate_backward_time: 50.7743301392, weight_update_time: 0.1163472831, other_time: 2.5321693420
epoch time: 13.8544445038s, input_fired_time: 0.1831670702, output_fired_time: 0.0000000000, propagate_forward_time: 12.3381042480, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.3331727982
validation[          , genome  4468] predictions:    9637/  10000 (96.37%), best:    9675/10000 (96.75%), error:      1211.95520, best error:      1073.01453 on epoch:    47, epoch:   56/100, mu: 0.9622848630, learning_rate: 0.0007070196, weight_decay: 0.0000282808

epoch time: 157.4197540283s, input_fired_time: 1.2420805693, output_fired_time: 1.9588370323, propagate_forward_time: 60.4117507935, propagate_backward_time: 88.5488815308, weight_update_time: 0.1560942233, other_time: 5.2582092285
epoch time: 13.8944559097s, input_fired_time: 0.3386960030, output_fired_time: 0.0000000000, propagate_forward_time: 12.8485326767, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7072267532
validation[          , genome  4468] predictions:    9614/  10000 (96.14%), best:    9675/10000 (96.75%), error:      1294.85449, best error:      1073.01453 on epoch:    47, epoch:   57/100, mu: 0.9636706114, learning_rate: 0.0006716687, weight_decay: 0.0000268668

epoch time: 163.0358734131s, input_fired_time: 1.5838115215, output_fired_time: 1.0964447260, propagate_forward_time: 63.2656898499, propagate_backward_time: 92.0526809692, weight_update_time: 0.1873956770, other_time: 5.0372390747
epoch time: 14.2309532166s, input_fired_time: 0.4167256057, output_fired_time: 0.0000000000, propagate_forward_time: 13.0149860382, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7992420197
validation[          , genome  4468] predictions:    9650/  10000 (96.50%), best:    9675/10000 (96.75%), error:      1153.54590, best error:      1073.01453 on epoch:    47, epoch:   58/100, mu: 0.9649870992, learning_rate: 0.0006380852, weight_decay: 0.0000255234

epoch time: 171.1879577637s, input_fired_time: 1.2811489105, output_fired_time: 0.7814002037, propagate_forward_time: 64.2659988403, propagate_backward_time: 100.1970138550, weight_update_time: 0.1248604804, other_time: 4.6623992920
epoch time: 14.2256708145s, input_fired_time: 0.1209963337, output_fired_time: 0.0000000000, propagate_forward_time: 13.5225629807, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5821113586
validation[          , genome  4468] predictions:    9584/  10000 (95.84%), best:    9675/10000 (96.75%), error:      1376.30029, best error:      1073.01453 on epoch:    47, epoch:   59/100, mu: 0.9662377238, learning_rate: 0.0006061810, weight_decay: 0.0000242472

epoch time: 183.4253997803s, input_fired_time: 1.3662623167, output_fired_time: 1.0176922083, propagate_forward_time: 73.4168090820, propagate_backward_time: 102.3565292358, weight_update_time: 0.3903532922, other_time: 5.2681198120
epoch time: 17.8079929352s, input_fired_time: 0.2602672875, output_fired_time: 0.0000000000, propagate_forward_time: 16.6073894501, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9403362274
validation[          , genome  4468] predictions:    9632/  10000 (96.32%), best:    9675/10000 (96.75%), error:      1201.94006, best error:      1073.01453 on epoch:    47, epoch:   60/100, mu: 0.9674258232, learning_rate: 0.0005758719, weight_decay: 0.0000230349

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/mnist_training_50k.bin'
boincified validation filename: '../../projects/csgrid.org_csg/mnist_validation_10k.bin'
boincified testing filename: '../../projects/csgrid.org_csg/mnist_testing_new.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_43_4468.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_43_4468_0_r1691656081_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.968555
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000547078
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 2.18831e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 61
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 47
read number_validation_images: 10000
read best_validation_predictions: 9675
read best_validation_error: 1073.01
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 4468
read normal distribution: '1 0.791580021381378 -1.03413057327271'
generator_str: '773626434'
read generator: 773626434
read generated_by_map:
1 change_size 1
reading 16 nodes.
reading 29 edges.
number input nodes: 1
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 4923 images.
reading image set with 5742 images.
reading image set with 4958 images.
reading image set with 5131 images.
reading image set with 4842 images.
reading image set with 4421 images.
reading image set with 4918 images.
reading image set with 5265 images.
reading image set with 4851 images.
reading image set with 4949 images.
image_size: 1x28x28 = 784
read 50000 images.
    class    0: 4923
    class    1: 5742
    class    2: 4958
    class    3: 5131
    class    4: 4842
    class    5: 4421
    class    6: 4918
    class    7: 5265
    class    8: 4851
    class    9: 4949
calculating averages and standard deviations for images
average pixel value for channel 0: 0.130530551075935
pixel variance for channel 0: 0.0948646068572998
pixel standard deviation for channel 0: 0.308000981807709
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 980 images.
reading image set with 1135 images.
reading image set with 1032 images.
reading image set with 1010 images.
reading image set with 982 images.
reading image set with 892 images.
reading image set with 958 images.
reading image set with 1028 images.
reading image set with 974 images.
reading image set with 1009 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 980
    class    1: 1135
    class    2: 1032
    class    3: 1010
    class    4: 982
    class    5: 892
    class    6: 958
    class    7: 1028
    class    8: 974
    class    9: 1009
loaded images
starting backpropagation!
epoch time: 133.56950378418s, input_fired_time: 0.946619808673859, output_fired_time: 1.33077573776245, propagate_forward_time: 51.3457336425781, propagate_backward_time: 75.8451690673828, weight_update_time: 0.171180725097656, other_time: 4.1011962890625
epoch time: 10.4106779098511s, input_fired_time: 0.0884555950760841, output_fired_time: 0, propagate_forward_time: 9.75726413726807, propagate_backward_time: 0, weight_update_time: 0, other_time: 0.564958572387695
validation[          , genome  4468] predictions:    9629/  10000 (96.29%), best:    9675/10000 (96.75%), error:      1170.30859, best error:      1073.01453 on epoch:    47, epoch:   61/100, mu: 0.9685545564, learning_rate: 0.0005470783, weight_decay: 0.0000218831

epoch time: 135.6585083008s, input_fired_time: 0.7755922675, output_fired_time: 1.0967837572, propagate_forward_time: 49.7568969727, propagate_backward_time: 79.4326858521, weight_update_time: 0.1236594766, other_time: 4.5965499878
epoch time: 10.8405179977s, input_fired_time: 0.1040945277, output_fired_time: 0.0000000000, propagate_forward_time: 10.1643733978, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5720500946
validation[          , genome  4468] predictions:    9648/  10000 (96.48%), best:    9675/10000 (96.75%), error:      1127.50220, best error:      1073.01453 on epoch:    47, epoch:   62/100, mu: 0.9696268439, learning_rate: 0.0005197244, weight_decay: 0.0000207890

epoch time: 123.6843185425s, input_fired_time: 0.9895241857, output_fired_time: 0.9412379265, propagate_forward_time: 48.7561492920, propagate_backward_time: 69.6172103882, weight_update_time: 0.3737293482, other_time: 3.3801956177
epoch time: 10.5207862854s, input_fired_time: 0.1053812280, output_fired_time: 0.0000000000, propagate_forward_time: 9.6770629883, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7383422852
validation[          , genome  4468] predictions:    9606/  10000 (96.06%), best:    9675/10000 (96.75%), error:      1312.36365, best error:      1073.01453 on epoch:    47, epoch:   63/100, mu: 0.9706454873, learning_rate: 0.0004937382, weight_decay: 0.0000197495

epoch time: 124.4600601196s, input_fired_time: 1.6081067324, output_fired_time: 1.2215430737, propagate_forward_time: 48.5637321472, propagate_backward_time: 69.7392883301, weight_update_time: 0.1238709539, other_time: 3.3273925781
epoch time: 9.4528493881s, input_fired_time: 0.0887725875, output_fired_time: 0.0000000000, propagate_forward_time: 8.8937149048, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4703617096
validation[          , genome  4468] predictions:    9611/  10000 (96.11%), best:    9675/10000 (96.75%), error:      1242.26233, best error:      1073.01453 on epoch:    47, epoch:   64/100, mu: 0.9716132283, learning_rate: 0.0004690512, weight_decay: 0.0000187621

epoch time: 115.8982391357s, input_fired_time: 0.9838464260, output_fired_time: 0.9424596429, propagate_forward_time: 42.8860664368, propagate_backward_time: 67.3732223511, weight_update_time: 0.1397991627, other_time: 3.7126388550
epoch time: 9.2223253250s, input_fired_time: 0.0897443891, output_fired_time: 0.0000000000, propagate_forward_time: 8.4047403336, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7278404236
validation[          , genome  4468] predictions:    9632/  10000 (96.32%), best:    9675/10000 (96.75%), error:      1224.10242, best error:      1073.01453 on epoch:    47, epoch:   65/100, mu: 0.9725325704, learning_rate: 0.0004455987, weight_decay: 0.0000178240

epoch time: 92.6781234741s, input_fired_time: 0.6944704652, output_fired_time: 0.5821119547, propagate_forward_time: 35.3977470398, propagate_backward_time: 52.7016181946, weight_update_time: 0.1315124631, other_time: 3.3021697998
epoch time: 5.6971631050s, input_fired_time: 0.0764540657, output_fired_time: 0.0000000000, propagate_forward_time: 5.2117171288, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4089918137
validation[          , genome  4468] predictions:    9655/  10000 (96.55%), best:    9675/10000 (96.75%), error:      1160.96448, best error:      1073.01453 on epoch:    47, epoch:   66/100, mu: 0.9734059572, learning_rate: 0.0004233187, weight_decay: 0.0000169328

epoch time: 76.7718353271s, input_fired_time: 0.7058597803, output_fired_time: 0.5891212821, propagate_forward_time: 28.9125232697, propagate_backward_time: 44.0492591858, weight_update_time: 0.1118690297, other_time: 2.5150794983
epoch time: 5.6425657272s, input_fired_time: 0.0785108060, output_fired_time: 0.0000000000, propagate_forward_time: 5.1696181297, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3944368362
validation[          , genome  4468] predictions:    9644/  10000 (96.44%), best:    9675/10000 (96.75%), error:      1163.53137, best error:      1073.01453 on epoch:    47, epoch:   67/100, mu: 0.9742356539, learning_rate: 0.0004021528, weight_decay: 0.0000160861

epoch time: 79.0453109741s, input_fired_time: 0.6721140146, output_fired_time: 0.6173371077, propagate_forward_time: 29.8812999725, propagate_backward_time: 45.2514533997, weight_update_time: 0.1101568788, other_time: 2.6231040955
epoch time: 6.0948166847s, input_fired_time: 0.0766404346, output_fired_time: 0.0000000000, propagate_forward_time: 5.5859117508, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4322643280
validation[          , genome  4468] predictions:    9655/  10000 (96.55%), best:    9675/10000 (96.75%), error:      1140.73401, best error:      1073.01453 on epoch:    47, epoch:   68/100, mu: 0.9750238657, learning_rate: 0.0003820452, weight_decay: 0.0000152818

epoch time: 76.9655532837s, input_fired_time: 0.7246903777, output_fired_time: 0.5787928700, propagate_forward_time: 28.7265529633, propagate_backward_time: 44.4608879089, weight_update_time: 0.1093089357, other_time: 2.4746208191
epoch time: 6.3086476326s, input_fired_time: 0.0760715902, output_fired_time: 0.0000000000, propagate_forward_time: 5.7827782631, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4497976303
validation[          , genome  4468] predictions:    9639/  10000 (96.39%), best:    9675/10000 (96.75%), error:      1185.25134, best error:      1073.01453 on epoch:    47, epoch:   69/100, mu: 0.9757726789, learning_rate: 0.0003629429, weight_decay: 0.0000145177

epoch time: 76.3474578857s, input_fired_time: 0.6907438636, output_fired_time: 0.6226792336, propagate_forward_time: 28.6426219940, propagate_backward_time: 44.0587310791, weight_update_time: 0.1109366491, other_time: 2.3326797485
epoch time: 6.1839494705s, input_fired_time: 0.0804160163, output_fired_time: 0.0000000000, propagate_forward_time: 5.6524806023, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4510526657
validation[          , genome  4468] predictions:    9660/  10000 (96.60%), best:    9675/10000 (96.75%), error:      1096.48889, best error:      1073.01453 on epoch:    47, epoch:   70/100, mu: 0.9764840603, learning_rate: 0.0003447958, weight_decay: 0.0000137918

epoch time: 76.0291976929s, input_fired_time: 0.6946193576, output_fired_time: 0.6702618003, propagate_forward_time: 28.7107677460, propagate_backward_time: 43.5447044373, weight_update_time: 0.1084283292, other_time: 2.4088401794
epoch time: 6.3845100403s, input_fired_time: 0.1289792806, output_fired_time: 0.0000000000, propagate_forward_time: 5.8397388458, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4157919884
validation[          , genome  4468] predictions:    9640/  10000 (96.40%), best:    9675/10000 (96.75%), error:      1175.76807, best error:      1073.01453 on epoch:    47, epoch:   71/100, mu: 0.9771598577, learning_rate: 0.0003275560, weight_decay: 0.0000131022

epoch time: 75.9590148926s, input_fired_time: 0.6641059518, output_fired_time: 0.6237341166, propagate_forward_time: 28.7312717438, propagate_backward_time: 43.5046424866, weight_update_time: 0.1088302732, other_time: 2.4352607727
epoch time: 6.8611869812s, input_fired_time: 0.0789588094, output_fired_time: 0.0000000000, propagate_forward_time: 6.3558773994, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4263505936
validation[          , genome  4468] predictions:    9638/  10000 (96.38%), best:    9675/10000 (96.75%), error:      1138.46802, best error:      1073.01453 on epoch:    47, epoch:   72/100, mu: 0.9778018594, learning_rate: 0.0003111782, weight_decay: 0.0000124471

epoch time: 76.2694549561s, input_fired_time: 0.6671996713, output_fired_time: 0.5784835219, propagate_forward_time: 28.6754379272, propagate_backward_time: 43.9278831482, weight_update_time: 0.1089679897, other_time: 2.4204521179
epoch time: 6.5286417007s, input_fired_time: 0.0974880159, output_fired_time: 0.0000000000, propagate_forward_time: 6.0247616768, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4063920975
validation[          , genome  4468] predictions:    9647/  10000 (96.47%), best:    9675/10000 (96.75%), error:      1149.69495, best error:      1073.01453 on epoch:    47, epoch:   73/100, mu: 0.9784117937, learning_rate: 0.0002956192, weight_decay: 0.0000118248

epoch time: 74.4194183350s, input_fired_time: 0.6756972075, output_fired_time: 0.5845832825, propagate_forward_time: 27.7738380432, propagate_backward_time: 42.9960212708, weight_update_time: 0.1075015739, other_time: 2.3892745972
epoch time: 6.2896652222s, input_fired_time: 0.0824425444, output_fired_time: 0.0000000000, propagate_forward_time: 5.7781319618, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4290904999
validation[          , genome  4468] predictions:    9657/  10000 (96.57%), best:    9675/10000 (96.75%), error:      1171.86743, best error:      1073.01453 on epoch:    47, epoch:   74/100, mu: 0.9789912105, learning_rate: 0.0002808383, weight_decay: 0.0000112335

epoch time: 74.8875885010s, input_fired_time: 0.6417338848, output_fired_time: 0.5531510711, propagate_forward_time: 28.3453578949, propagate_backward_time: 43.0544967651, weight_update_time: 0.1104819775, other_time: 2.2928466797
epoch time: 6.2255244255s, input_fired_time: 0.0821615010, output_fired_time: 0.0000000000, propagate_forward_time: 5.6371064186, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5062565804
validation[          , genome  4468] predictions:    9654/  10000 (96.54%), best:    9675/10000 (96.75%), error:      1125.97266, best error:      1073.01453 on epoch:    47, epoch:   75/100, mu: 0.9795416594, learning_rate: 0.0002667964, weight_decay: 0.0000106719

epoch time: 76.2875595093s, input_fired_time: 0.6683814526, output_fired_time: 0.6239884496, propagate_forward_time: 28.7513675690, propagate_backward_time: 43.7793960571, weight_update_time: 0.1249248385, other_time: 2.4644317627
epoch time: 6.2984881401s, input_fired_time: 0.1117265299, output_fired_time: 0.0000000000, propagate_forward_time: 5.7813296318, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4054317474
validation[          , genome  4468] predictions:    9541/  10000 (95.41%), best:    9675/10000 (96.75%), error:      1446.69543, best error:      1073.01453 on epoch:    47, epoch:   76/100, mu: 0.9800645709, learning_rate: 0.0002534565, weight_decay: 0.0000101383

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/mnist_training_50k.bin'
boincified validation filename: '../../projects/csgrid.org_csg/mnist_validation_10k.bin'
boincified testing filename: '../../projects/csgrid.org_csg/mnist_testing_new.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_43_4468.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_43_4468_0_r1691656081_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.980561
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.000240784
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 9.63135e-06
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 77
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 47
read number_validation_images: 10000
read best_validation_predictions: 9675
read best_validation_error: 1073.01
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 4468
read normal distribution: '1 0.791580021381378 -1.03413057327271'
generator_str: '814326777'
read generator: 814326777
read generated_by_map:
1 change_size 1
reading 16 nodes.
reading 29 edges.
number input nodes: 1
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 4923 images.
reading image set with 5742 images.
reading image set with 4958 images.
reading image set with 5131 images.
reading image set with 4842 images.
reading image set with 4421 images.
reading image set with 4918 images.
reading image set with 5265 images.
reading image set with 4851 images.
reading image set with 4949 images.
image_size: 1x28x28 = 784
read 50000 images.
    class    0: 4923
    class    1: 5742
    class    2: 4958
    class    3: 5131
    class    4: 4842
    class    5: 4421
    class    6: 4918
    class    7: 5265
    class    8: 4851
    class    9: 4949
calculating averages and standard deviations for images
average pixel value for channel 0: 0.130530551075935
pixel variance for channel 0: 0.0948646068572998
pixel standard deviation for channel 0: 0.308000981807709
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 980 images.
reading image set with 1135 images.
reading image set with 1032 images.
reading image set with 1010 images.
reading image set with 982 images.
reading image set with 892 images.
reading image set with 958 images.
reading image set with 1028 images.
reading image set with 974 images.
reading image set with 1009 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 980
    class    1: 1135
    class    2: 1032
    class    3: 1010
    class    4: 982
    class    5: 892
    class    6: 958
    class    7: 1028
    class    8: 974
    class    9: 1009
loaded images
starting backpropagation!
epoch time: 81.1447982788086s, input_fired_time: 0.693146049976349, output_fired_time: 0.762359082698822, propagate_forward_time: 30.5255966186523, propagate_backward_time: 46.6928443908691, weight_update_time: 0.112350665032864, other_time: 2.47085189819336
epoch time: 6.17049598693848s, input_fired_time: 0.0960930734872818, output_fired_time: 0, propagate_forward_time: 5.65592336654663, propagate_backward_time: 0, weight_update_time: 0, other_time: 0.418479442596436
validation[          , genome  4468] predictions:    9653/  10000 (96.53%), best:    9675/10000 (96.75%), error:      1140.85254, best error:      1073.01453 on epoch:    47, epoch:   77/100, mu: 0.9805613160, learning_rate: 0.0002407837, weight_decay: 0.0000096314

epoch time: 79.2222366333s, input_fired_time: 0.6531011462, output_fired_time: 0.6291138530, propagate_forward_time: 29.7282543182, propagate_backward_time: 45.7262725830, weight_update_time: 0.1265098155, other_time: 2.4854965210
epoch time: 5.9759573936s, input_fired_time: 0.0808208734, output_fired_time: 0.0000000000, propagate_forward_time: 5.4866628647, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4084734917
validation[          , genome  4468] predictions:    9654/  10000 (96.54%), best:    9675/10000 (96.75%), error:      1125.51416, best error:      1073.01453 on epoch:    47, epoch:   78/100, mu: 0.9810332656, learning_rate: 0.0002287445, weight_decay: 0.0000091498

epoch time: 79.6300125122s, input_fired_time: 0.7313309312, output_fired_time: 0.6813709140, propagate_forward_time: 29.9225082397, propagate_backward_time: 45.7617530823, weight_update_time: 0.1114853621, other_time: 2.5330467224
epoch time: 6.5315895081s, input_fired_time: 0.0876940936, output_fired_time: 0.0000000000, propagate_forward_time: 6.0034356117, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4404597282
validation[          , genome  4468] predictions:    9636/  10000 (96.36%), best:    9675/10000 (96.75%), error:      1192.20996, best error:      1073.01453 on epoch:    47, epoch:   79/100, mu: 0.9814816117, learning_rate: 0.0002173073, weight_decay: 0.0000086923

epoch time: 78.1092071533s, input_fired_time: 0.7014223337, output_fired_time: 0.6873264313, propagate_forward_time: 29.3574409485, propagate_backward_time: 44.8603057861, weight_update_time: 0.1420832127, other_time: 2.5027122498
epoch time: 6.4410133362s, input_fired_time: 0.0864493772, output_fired_time: 0.0000000000, propagate_forward_time: 5.9219117165, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4326524734
validation[          , genome  4468] predictions:    9652/  10000 (96.52%), best:    9675/10000 (96.75%), error:      1162.04321, best error:      1073.01453 on epoch:    47, epoch:   80/100, mu: 0.9819075465, learning_rate: 0.0002064419, weight_decay: 0.0000082577

epoch time: 79.5040588379s, input_fired_time: 0.6947047114, output_fired_time: 0.5762535930, propagate_forward_time: 30.1060733795, propagate_backward_time: 45.7378997803, weight_update_time: 0.1290031970, other_time: 2.3891296387
epoch time: 6.3507213593s, input_fired_time: 0.0996730924, output_fired_time: 0.0000000000, propagate_forward_time: 5.8143167496, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4367313385
validation[          , genome  4468] predictions:    9644/  10000 (96.44%), best:    9675/10000 (96.75%), error:      1166.32739, best error:      1073.01453 on epoch:    47, epoch:   81/100, mu: 0.9823121428, learning_rate: 0.0001961198, weight_decay: 0.0000078448

epoch time: 77.8263473511s, input_fired_time: 0.6729723215, output_fired_time: 0.6006261110, propagate_forward_time: 29.3408355713, propagate_backward_time: 44.6975898743, weight_update_time: 0.1424412876, other_time: 2.5143241882
epoch time: 6.4012575150s, input_fired_time: 0.0837961584, output_fired_time: 0.0000000000, propagate_forward_time: 5.8817024231, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4357590675
validation[          , genome  4468] predictions:    9641/  10000 (96.41%), best:    9675/10000 (96.75%), error:      1179.76831, best error:      1073.01453 on epoch:    47, epoch:   82/100, mu: 0.9826965332, learning_rate: 0.0001863138, weight_decay: 0.0000074526

epoch time: 80.8861923218s, input_fired_time: 0.7542875409, output_fired_time: 0.6865041256, propagate_forward_time: 30.8053550720, propagate_backward_time: 46.1546173096, weight_update_time: 0.1162188873, other_time: 2.4854316711
epoch time: 6.1916933060s, input_fired_time: 0.0830859914, output_fired_time: 0.0000000000, propagate_forward_time: 5.6913514137, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4172558784
validation[          , genome  4468] predictions:    9589/  10000 (95.89%), best:    9675/10000 (96.75%), error:      1332.12927, best error:      1073.01453 on epoch:    47, epoch:   83/100, mu: 0.9830617309, learning_rate: 0.0001769981, weight_decay: 0.0000070799

epoch time: 80.0402297974s, input_fired_time: 0.7308557630, output_fired_time: 0.5741955638, propagate_forward_time: 30.2146339417, propagate_backward_time: 46.0412025452, weight_update_time: 0.1146106496, other_time: 2.4793395996
epoch time: 6.1388850212s, input_fired_time: 0.0830045864, output_fired_time: 0.0000000000, propagate_forward_time: 5.6429920197, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4128885269
validation[          , genome  4468] predictions:    9653/  10000 (96.53%), best:    9675/10000 (96.75%), error:      1106.93469, best error:      1073.01453 on epoch:    47, epoch:   84/100, mu: 0.9834086299, learning_rate: 0.0001681482, weight_decay: 0.0000067259

epoch time: 80.5324096680s, input_fired_time: 0.6795369387, output_fired_time: 0.6257378459, propagate_forward_time: 30.5132617950, propagate_backward_time: 46.2297019958, weight_update_time: 0.1134560779, other_time: 2.4841728210
epoch time: 7.6133551598s, input_fired_time: 0.1115082428, output_fired_time: 0.0000000000, propagate_forward_time: 7.0016508102, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5001959801
validation[          , genome  4468] predictions:    9664/  10000 (96.64%), best:    9675/10000 (96.75%), error:      1122.86597, best error:      1073.01453 on epoch:    47, epoch:   85/100, mu: 0.9837381840, learning_rate: 0.0001597408, weight_decay: 0.0000063896

epoch time: 81.8823852539s, input_fired_time: 0.6905522943, output_fired_time: 0.6559649706, propagate_forward_time: 31.0075225830, propagate_backward_time: 47.0443687439, weight_update_time: 0.1160397008, other_time: 2.4839744568
epoch time: 6.6158776283s, input_fired_time: 0.0785582885, output_fired_time: 0.0000000000, propagate_forward_time: 6.0832476616, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4540715218
validation[          , genome  4468] predictions:    9622/  10000 (96.22%), best:    9675/10000 (96.75%), error:      1231.17834, best error:      1073.01453 on epoch:    47, epoch:   86/100, mu: 0.9840512872, learning_rate: 0.0001517538, weight_decay: 0.0000060702

epoch time: 82.2323074341s, input_fired_time: 0.7944276929, output_fired_time: 0.7281282544, propagate_forward_time: 31.2354488373, propagate_backward_time: 47.0688018799, weight_update_time: 0.1306992918, other_time: 2.4055023193
epoch time: 7.7182898521s, input_fired_time: 0.1149217635, output_fired_time: 0.0000000000, propagate_forward_time: 7.1387982368, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4645700455
validation[          , genome  4468] predictions:    9656/  10000 (96.56%), best:    9675/10000 (96.75%), error:      1125.57715, best error:      1073.01453 on epoch:    47, epoch:   87/100, mu: 0.9843487144, learning_rate: 0.0001441661, weight_decay: 0.0000057666

epoch time: 81.3844223022s, input_fired_time: 0.8168429732, output_fired_time: 0.6557147503, propagate_forward_time: 30.6329669952, propagate_backward_time: 46.8267860413, weight_update_time: 0.1159746200, other_time: 2.4521141052
epoch time: 6.6550474167s, input_fired_time: 0.1435616165, output_fired_time: 0.0000000000, propagate_forward_time: 6.0794434547, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4320421219
validation[          , genome  4468] predictions:    9674/  10000 (96.74%), best:    9674/10000 (96.74%), error:      1068.77869, best error:      1068.77869 on epoch:    88, epoch:   88/100, mu: 0.9846313000, learning_rate: 0.0001369578, weight_decay: 0.0000054783

epoch time: 85.1699447632s, input_fired_time: 0.8258379102, output_fired_time: 0.6132803559, propagate_forward_time: 32.1703681946, propagate_backward_time: 48.8274154663, weight_update_time: 0.1693401337, other_time: 2.7330436707
epoch time: 6.7239503860s, input_fired_time: 0.0877913535, output_fired_time: 0.0000000000, propagate_forward_time: 6.1962270737, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4399318695
validation[          , genome  4468] predictions:    9669/  10000 (96.69%), best:    9674/10000 (96.74%), error:      1105.49475, best error:      1068.77869 on epoch:    88, epoch:   89/100, mu: 0.9848997593, learning_rate: 0.0001301099, weight_decay: 0.0000052044

epoch time: 76.4583053589s, input_fired_time: 0.7041179538, output_fired_time: 0.6138821840, propagate_forward_time: 28.8156795502, propagate_backward_time: 43.8032417297, weight_update_time: 0.1105801463, other_time: 2.5213813782
epoch time: 6.5482711792s, input_fired_time: 0.0842257142, output_fired_time: 0.0000000000, propagate_forward_time: 6.0259537697, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4380917549
validation[          , genome  4468] predictions:    9675/  10000 (96.75%), best:    9674/10000 (96.74%), error:      1100.08911, best error:      1068.77869 on epoch:    88, epoch:   90/100, mu: 0.9851547480, learning_rate: 0.0001236044, weight_decay: 0.0000049442

epoch time: 84.5106048584s, input_fired_time: 0.6487151980, output_fired_time: 0.5941805840, propagate_forward_time: 31.3718605042, propagate_backward_time: 49.4264755249, weight_update_time: 0.1125881746, other_time: 2.4693794250
epoch time: 10.0383901596s, input_fired_time: 0.0906581208, output_fired_time: 0.0000000000, propagate_forward_time: 9.4183139801, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5294179916
validation[          , genome  4468] predictions:    9677/  10000 (96.77%), best:    9674/10000 (96.74%), error:      1075.52441, best error:      1068.77869 on epoch:    88, epoch:   91/100, mu: 0.9853969812, learning_rate: 0.0001174242, weight_decay: 0.0000046970

epoch time: 116.3853836060s, input_fired_time: 0.7318150401, output_fired_time: 0.9551984072, propagate_forward_time: 45.8815803528, propagate_backward_time: 65.8598861694, weight_update_time: 0.1565360576, other_time: 2.9569015503
epoch time: 9.7001895905s, input_fired_time: 0.0905982256, output_fired_time: 0.0000000000, propagate_forward_time: 8.8285989761, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7809925079
validation[          , genome  4468] predictions:    9665/  10000 (96.65%), best:    9674/10000 (96.74%), error:      1090.04504, best error:      1068.77869 on epoch:    88, epoch:   92/100, mu: 0.9856271148, learning_rate: 0.0001115530, weight_decay: 0.0000044621

epoch time: 119.5895538330s, input_fired_time: 1.0135511160, output_fired_time: 0.7204631567, propagate_forward_time: 43.0542793274, propagate_backward_time: 70.9652709961, weight_update_time: 0.1413363367, other_time: 3.8359832764
epoch time: 10.0056419373s, input_fired_time: 0.0898778215, output_fired_time: 0.0000000000, propagate_forward_time: 9.0740318298, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.8417320251
validation[          , genome  4468] predictions:    9634/  10000 (96.34%), best:    9674/10000 (96.74%), error:      1224.06287, best error:      1068.77869 on epoch:    88, epoch:   93/100, mu: 0.9858457446, learning_rate: 0.0001059753, weight_decay: 0.0000042390

epoch time: 117.2705917358s, input_fired_time: 0.8395501375, output_fired_time: 0.7669314742, propagate_forward_time: 44.3244934082, propagate_backward_time: 67.1473693848, weight_update_time: 0.1254867315, other_time: 4.1922454834
epoch time: 9.8768138885s, input_fired_time: 0.0900006890, output_fired_time: 0.0000000000, propagate_forward_time: 9.0689134598, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7178993225
validation[          , genome  4468] predictions:    9660/  10000 (96.60%), best:    9674/10000 (96.74%), error:      1110.67639, best error:      1068.77869 on epoch:    88, epoch:   94/100, mu: 0.9860534668, learning_rate: 0.0001006766, weight_decay: 0.0000040271

epoch time: 119.1210861206s, input_fired_time: 1.4661594629, output_fired_time: 0.9078943729, propagate_forward_time: 46.2925415039, propagate_backward_time: 67.3221511841, weight_update_time: 0.1725149900, other_time: 3.1323394775
epoch time: 10.0635385513s, input_fired_time: 0.0900815800, output_fired_time: 0.0000000000, propagate_forward_time: 9.2247610092, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7486963272
validation[          , genome  4468] predictions:    9634/  10000 (96.34%), best:    9674/10000 (96.74%), error:      1179.41443, best error:      1068.77869 on epoch:    88, epoch:   95/100, mu: 0.9862508178, learning_rate: 0.0000956427, weight_decay: 0.0000038257

epoch time: 117.6262664795s, input_fired_time: 0.7792484760, output_fired_time: 0.6889187098, propagate_forward_time: 45.1199340820, propagate_backward_time: 67.0647811890, weight_update_time: 0.1252880543, other_time: 3.9733810425
epoch time: 9.4229755402s, input_fired_time: 0.0901588872, output_fired_time: 0.0000000000, propagate_forward_time: 8.1323843002, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.2004327774
validation[          , genome  4468] predictions:    9657/  10000 (96.57%), best:    9674/10000 (96.74%), error:      1101.92175, best error:      1068.77869 on epoch:    88, epoch:   96/100, mu: 0.9864382744, learning_rate: 0.0000908606, weight_decay: 0.0000036344

epoch time: 113.2993469238s, input_fired_time: 0.7627719045, output_fired_time: 0.6743607521, propagate_forward_time: 43.4288787842, propagate_backward_time: 65.0529479980, weight_update_time: 0.1566191465, other_time: 3.3803863525
epoch time: 9.6510438919s, input_fired_time: 0.0897969082, output_fired_time: 0.0000000000, propagate_forward_time: 8.7806615829, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7805852890
validation[          , genome  4468] predictions:    9665/  10000 (96.65%), best:    9674/10000 (96.74%), error:      1085.33582, best error:      1068.77869 on epoch:    88, epoch:   97/100, mu: 0.9866163731, learning_rate: 0.0000863176, weight_decay: 0.0000034527

epoch time: 118.0373458862s, input_fired_time: 0.9665023685, output_fired_time: 0.9719447494, propagate_forward_time: 44.7976531982, propagate_backward_time: 68.0605316162, weight_update_time: 0.1410580426, other_time: 3.2407150269
epoch time: 9.7780160904s, input_fired_time: 0.0905638933, output_fired_time: 0.0000000000, propagate_forward_time: 9.2036485672, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4838037491
validation[          , genome  4468] predictions:    9658/  10000 (96.58%), best:    9674/10000 (96.74%), error:      1116.87292, best error:      1068.77869 on epoch:    88, epoch:   98/100, mu: 0.9867855310, learning_rate: 0.0000820017, weight_decay: 0.0000032801

epoch time: 117.0198745728s, input_fired_time: 1.2954896688, output_fired_time: 1.4546179771, propagate_forward_time: 42.9322586060, propagate_backward_time: 67.8136901855, weight_update_time: 0.1253992170, other_time: 3.5238189697
epoch time: 10.0820999146s, input_fired_time: 0.1063179374, output_fired_time: 0.0000000000, propagate_forward_time: 9.4773979187, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4983844757
validation[          , genome  4468] predictions:    9620/  10000 (96.20%), best:    9674/10000 (96.74%), error:      1200.43359, best error:      1068.77869 on epoch:    88, epoch:   99/100, mu: 0.9869462252, learning_rate: 0.0000779016, weight_decay: 0.0000031161

epoch time: 121.2670822144s, input_fired_time: 1.2941360474, output_fired_time: 0.6743178964, propagate_forward_time: 45.5194396973, propagate_backward_time: 69.6953735352, weight_update_time: 0.4071470201, other_time: 4.0838165283
epoch time: 9.9663887024s, input_fired_time: 0.0903985053, output_fired_time: 0.0000000000, propagate_forward_time: 9.3774709702, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4985189438
validation[          , genome  4468] predictions:    9637/  10000 (96.37%), best:    9674/10000 (96.74%), error:      1138.69958, best error:      1068.77869 on epoch:    88, epoch:  100/100, mu: 0.9870989323, learning_rate: 0.0000740065, weight_decay: 0.0000029603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 50.5811500549s, input_fired_time: 0.5145808458, output_fired_time: 0.0000000000, propagate_forward_time: 46.1135597229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 3.9530105591
best training[          , genome  4468] predictions:   49235/  50000 (98.47%), best:    9674/10000 (96.74%), error:      2953.03296, best error:      1068.77869 on epoch:    88, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 9.5848436356s, input_fired_time: 0.1209318340, output_fired_time: 0.0000000000, propagate_forward_time: 8.4938726425, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.9700393677
   testing[          , genome  4468] predictions:    9733/  10000 (97.33%), best:    9674/10000 (96.74%), error:       941.57422, best error:      1068.77869 on epoch:    88, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
backpropagation finished successfully!
00:15:26 (2312): called boinc_finish(0)

</stderr_txt>
]]>