Nameexact_genome_1512568488_44_3678_1
Workunit2589686
Created7 Dec 2017, 6:12:12 UTC
Sent7 Dec 2017, 6:19:22 UTC
Report deadline11 Dec 2017, 22:42:16 UTC
Received7 Dec 2017, 17:37:58 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID55108
Run time2 hours 43 min 16 sec
CPU time2 hours 36 min 38 sec
Validate stateValid
Credit1,170.64
Device peak FLOPS2.57 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size155.15 MB
Peak swap size158.46 MB
Peak disk usage1.75 MB

Stderr output

<core_client_version>7.8.2</core_client_version>
<![CDATA[
<stderr_txt>
ht_update_time: 0.0000000000, other_time: 0.5538821220
validation[          , genome  3678] predictions:    9651/  10000 (96.51%), best:    9651/10000 (96.51%), error:      1260.34326, best error:      1260.34326 on epoch:    14, epoch:   14/100, mu: 0.7510393858, learning_rate: 0.0060959356, weight_decay: 0.0002438374

epoch time: 104.3929443359s, input_fired_time: 1.3535889387, output_fired_time: 1.2376196384, propagate_forward_time: 39.9029312134, propagate_backward_time: 58.3058547974, weight_update_time: 0.1360108703, other_time: 3.5929489136
epoch time: 8.2235412598s, input_fired_time: 0.1508857012, output_fired_time: 0.0000000000, propagate_forward_time: 7.5044584274, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5681972504
validation[          , genome  3678] predictions:    9592/  10000 (95.92%), best:    9651/10000 (96.51%), error:      1322.37280, best error:      1260.34326 on epoch:    14, epoch:   15/100, mu: 0.7629874349, learning_rate: 0.0057911389, weight_decay: 0.0002316456

epoch time: 89.4883728027s, input_fired_time: 1.1724537611, output_fired_time: 1.0847694874, propagate_forward_time: 34.5956115723, propagate_backward_time: 49.5049896240, weight_update_time: 0.1265964806, other_time: 3.1305465698
epoch time: 6.9382662773s, input_fired_time: 0.1434257179, output_fired_time: 0.0000000000, propagate_forward_time: 6.2465124130, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5483279228
validation[          , genome  3678] predictions:    9618/  10000 (96.18%), best:    9618/10000 (96.18%), error:      1259.92444, best error:      1259.92444 on epoch:    16, epoch:   16/100, mu: 0.7743380666, learning_rate: 0.0055015818, weight_decay: 0.0002200633

epoch time: 90.8728103638s, input_fired_time: 1.1730996370, output_fired_time: 1.0943508148, propagate_forward_time: 34.7410697937, propagate_backward_time: 50.7387237549, weight_update_time: 0.1273446530, other_time: 3.1255607605
epoch time: 6.9402880669s, input_fired_time: 0.1427737921, output_fired_time: 0.0000000000, propagate_forward_time: 6.2490258217, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5484886169
validation[          , genome  3678] predictions:    9569/  10000 (95.69%), best:    9618/10000 (96.18%), error:      1487.84802, best error:      1259.92444 on epoch:    16, epoch:   17/100, mu: 0.7851211429, learning_rate: 0.0052265027, weight_decay: 0.0002090601

epoch time: 90.6323776245s, input_fired_time: 1.1976062059, output_fired_time: 1.0944093466, propagate_forward_time: 34.5460853577, propagate_backward_time: 50.6532859802, weight_update_time: 0.1281944513, other_time: 3.1409912109
epoch time: 6.7953820229s, input_fired_time: 0.1418183744, output_fired_time: 0.0000000000, propagate_forward_time: 6.1096863747, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5438771248
validation[          , genome  3678] predictions:    9554/  10000 (95.54%), best:    9618/10000 (96.18%), error:      1480.00500, best error:      1259.92444 on epoch:    16, epoch:   18/100, mu: 0.7953650951, learning_rate: 0.0049651773, weight_decay: 0.0001986071

epoch time: 94.8482437134s, input_fired_time: 1.2318332195, output_fired_time: 1.1551578045, propagate_forward_time: 36.2041206360, propagate_backward_time: 52.9711380005, weight_update_time: 0.1259239465, other_time: 3.2859916687
epoch time: 7.1178650856s, input_fired_time: 0.1429462880, output_fired_time: 0.0000000000, propagate_forward_time: 6.4149913788, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5599274635
validation[          , genome  3678] predictions:    9502/  10000 (95.02%), best:    9618/10000 (96.18%), error:      1635.66223, best error:      1259.92444 on epoch:    16, epoch:   19/100, mu: 0.8050968647, learning_rate: 0.0047169183, weight_decay: 0.0001886768

epoch time: 91.0668869019s, input_fired_time: 1.1869858503, output_fired_time: 1.1002585888, propagate_forward_time: 33.7750015259, propagate_backward_time: 51.8810958862, weight_update_time: 0.1274448186, other_time: 3.1235427856
epoch time: 6.9666852951s, input_fired_time: 0.1424454004, output_fired_time: 0.0000000000, propagate_forward_time: 6.2697649002, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5544748306
validation[          , genome  3678] predictions:    9201/  10000 (92.01%), best:    9618/10000 (96.18%), error:      2596.32251, best error:      1259.92444 on epoch:    16, epoch:   20/100, mu: 0.8143420219, learning_rate: 0.0044810725, weight_decay: 0.0001792429

epoch time: 96.7048721313s, input_fired_time: 1.2520886660, output_fired_time: 1.1448404789, propagate_forward_time: 37.9692573547, propagate_backward_time: 53.0208930969, weight_update_time: 0.1555575579, other_time: 3.3177871704
epoch time: 8.0208244324s, input_fired_time: 0.1467982233, output_fired_time: 0.0000000000, propagate_forward_time: 7.3109116554, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5631146431
validation[          , genome  3678] predictions:    9585/  10000 (95.85%), best:    9618/10000 (96.18%), error:      1386.21765, best error:      1259.92444 on epoch:    16, epoch:   21/100, mu: 0.8231249452, learning_rate: 0.0042570187, weight_decay: 0.0001702808

epoch time: 89.5071334839s, input_fired_time: 1.1702094078, output_fired_time: 1.0843973160, propagate_forward_time: 33.5071716309, propagate_backward_time: 50.6440811157, weight_update_time: 0.1267375946, other_time: 3.1012725830
epoch time: 6.8050603867s, input_fired_time: 0.1428920925, output_fired_time: 0.0000000000, propagate_forward_time: 6.1308584213, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5313100815
validation[          , genome  3678] predictions:    9568/  10000 (95.68%), best:    9618/10000 (96.18%), error:      1437.38928, best error:      1259.92444 on epoch:    16, epoch:   22/100, mu: 0.8314687014, learning_rate: 0.0040441677, weight_decay: 0.0001617667

epoch time: 90.9220199585s, input_fired_time: 1.1754667759, output_fired_time: 1.0944916010, propagate_forward_time: 32.6558532715, propagate_backward_time: 52.8848762512, weight_update_time: 0.1270920932, other_time: 3.1113319397
epoch time: 6.9619917870s, input_fired_time: 0.1449754238, output_fired_time: 0.0000000000, propagate_forward_time: 6.2713465691, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5456700325
validation[          , genome  3678] predictions:    9473/  10000 (94.73%), best:    9618/10000 (96.18%), error:      1698.55872, best error:      1259.92444 on epoch:    16, epoch:   23/100, mu: 0.8393952847, learning_rate: 0.0038419592, weight_decay: 0.0001536784

epoch time: 89.9488830566s, input_fired_time: 1.1770488024, output_fired_time: 1.0952008963, propagate_forward_time: 31.9121551514, propagate_backward_time: 51.6235122681, weight_update_time: 0.1268085539, other_time: 4.1409683228
epoch time: 6.9805226326s, input_fired_time: 0.1429942101, output_fired_time: 0.0000000000, propagate_forward_time: 6.2925786972, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5449495316
validation[          , genome  3678] predictions:    9277/  10000 (92.77%), best:    9618/10000 (96.18%), error:      2370.57568, best error:      1259.92444 on epoch:    16, epoch:   24/100, mu: 0.8469254971, learning_rate: 0.0036498611, weight_decay: 0.0001459945

epoch time: 90.8020935059s, input_fired_time: 1.1878316402, output_fired_time: 1.0934680700, propagate_forward_time: 33.5980873108, propagate_backward_time: 51.8191719055, weight_update_time: 0.1267376393, other_time: 3.1035385132
epoch time: 6.8884606361s, input_fired_time: 0.1422362179, output_fired_time: 0.0000000000, propagate_forward_time: 6.2065958977, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5396285057
validation[          , genome  3678] predictions:    9634/  10000 (96.34%), best:    9634/10000 (96.34%), error:      1257.87952, best error:      1257.87952 on epoch:    25, epoch:   25/100, mu: 0.8540792465, learning_rate: 0.0034673680, weight_decay: 0.0001386948

epoch time: 97.3008651733s, input_fired_time: 1.2856757641, output_fired_time: 1.1508252621, propagate_forward_time: 36.2447814941, propagate_backward_time: 55.3140792847, weight_update_time: 0.1429998279, other_time: 3.3055038452
epoch time: 6.8449239731s, input_fired_time: 0.1434805244, output_fired_time: 0.0000000000, propagate_forward_time: 6.1555795670, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5458641052
validation[          , genome  3678] predictions:    9593/  10000 (95.93%), best:    9634/10000 (96.34%), error:      1440.04077, best error:      1257.87952 on epoch:    25, epoch:   26/100, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 90.7178192139s, input_fired_time: 1.1752262115, output_fired_time: 1.0925643444, propagate_forward_time: 31.6184940338, propagate_backward_time: 52.7174797058, weight_update_time: 1.1356736422, other_time: 4.1140556335
epoch time: 6.9700598717s, input_fired_time: 0.1466467381, output_fired_time: 0.0000000000, propagate_forward_time: 6.2771229744, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5462899208
validation[          , genome  3678] predictions:    6333/  10000 (63.33%), best:    9634/10000 (96.34%), error:     15093.04102, best error:      1257.87952 on epoch:    25, epoch:   27/100, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

resetting weights to those on epoch: 25
epoch time: 96.2729187012s, input_fired_time: 1.2492177486, output_fired_time: 1.1449896097, propagate_forward_time: 35.7199668884, propagate_backward_time: 54.8957939148, weight_update_time: 0.1355164945, other_time: 3.2629547119
epoch time: 6.6938643456s, input_fired_time: 0.1391516477, output_fired_time: 0.0000000000, propagate_forward_time: 6.0190563202, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5356564522
validation[          , genome  3678] predictions:    9596/  10000 (95.96%), best:    9634/10000 (96.34%), error:      1304.00452, best error:      1257.87952 on epoch:    25, epoch:   28/100, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 98.5204391479s, input_fired_time: 1.2907609940, output_fired_time: 1.2248846292, propagate_forward_time: 36.2790794373, propagate_backward_time: 56.3853759766, weight_update_time: 0.1338374317, other_time: 3.3403358459
epoch time: 10.6016340256s, input_fired_time: 0.1997681111, output_fired_time: 0.0000000000, propagate_forward_time: 9.6534481049, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.7484178543
validation[          , genome  3678] predictions:    9611/  10000 (96.11%), best:    9634/10000 (96.34%), error:      1272.61877, best error:      1257.87952 on epoch:    25, epoch:   29/100, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 89.9225158691s, input_fired_time: 1.1702682972, output_fired_time: 1.0925081968, propagate_forward_time: 34.9260482788, propagate_backward_time: 49.6383171082, weight_update_time: 0.1255486906, other_time: 3.0953788757
epoch time: 8.0129222870s, input_fired_time: 0.1452006698, output_fired_time: 0.0000000000, propagate_forward_time: 7.3152899742, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5524315834
validation[          , genome  3678] predictions:    9543/  10000 (95.43%), best:    9634/10000 (96.34%), error:      1431.51050, best error:      1257.87952 on epoch:    25, epoch:   30/100, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

epoch time: 89.3444900513s, input_fired_time: 1.1733922958, output_fired_time: 1.0852196217, propagate_forward_time: 33.5142440796, propagate_backward_time: 50.4685096741, weight_update_time: 0.1256844252, other_time: 3.1031227112
epoch time: 8.2296609879s, input_fired_time: 0.1466320902, output_fired_time: 0.0000000000, propagate_forward_time: 7.5221042633, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5609245300
validation[          , genome  3678] predictions:    9658/  10000 (96.58%), best:    9658/10000 (96.58%), error:      1133.05908, best error:      1133.05908 on epoch:    31, epoch:   31/100, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 97.0859832764s, input_fired_time: 2.1741409302, output_fired_time: 1.2001399994, propagate_forward_time: 35.8915710449, propagate_backward_time: 54.5358543396, weight_update_time: 0.1275275052, other_time: 3.2842750549
epoch time: 7.1283087730s, input_fired_time: 0.1474848837, output_fired_time: 0.0000000000, propagate_forward_time: 6.4189057350, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5619182587
validation[          , genome  3678] predictions:    9626/  10000 (96.26%), best:    9658/10000 (96.58%), error:      1260.82886, best error:      1133.05908 on epoch:    31, epoch:   32/100, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 96.0814666748s, input_fired_time: 1.2446131706, output_fired_time: 1.1574014425, propagate_forward_time: 36.8784523010, propagate_backward_time: 53.5759277344, weight_update_time: 0.1424268335, other_time: 3.2250709534
epoch time: 6.9522132874s, input_fired_time: 0.1436520666, output_fired_time: 0.0000000000, propagate_forward_time: 6.2491254807, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5594358444
validation[          , genome  3678] predictions:    9381/  10000 (93.81%), best:    9658/10000 (96.58%), error:      2008.59802, best error:      1133.05908 on epoch:    31, epoch:   33/100, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 91.0053787231s, input_fired_time: 1.1820957661, output_fired_time: 1.1014740467, propagate_forward_time: 33.7515411377, propagate_backward_time: 51.8422355652, weight_update_time: 0.1273694932, other_time: 3.1280326843
epoch time: 6.8216099739s, input_fired_time: 0.1440150887, output_fired_time: 0.0000000000, propagate_forward_time: 6.1334671974, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5441274643
validation[          , genome  3678] predictions:    9662/  10000 (96.62%), best:    9662/10000 (96.62%), error:      1115.04224, best error:      1115.04224 on epoch:    34, epoch:   34/100, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 89.2656478882s, input_fired_time: 1.1712394953, output_fired_time: 1.0957823992, propagate_forward_time: 34.5385704041, propagate_backward_time: 49.3715324402, weight_update_time: 0.1253736913, other_time: 3.0885238647
epoch time: 7.9353280067s, input_fired_time: 0.1474174410, output_fired_time: 0.0000000000, propagate_forward_time: 7.2295145988, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5583958626
validation[          , genome  3678] predictions:    9524/  10000 (95.24%), best:    9662/10000 (96.62%), error:      1584.60571, best error:      1115.04224 on epoch:    34, epoch:   35/100, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 89.2508773804s, input_fired_time: 1.1730369329, output_fired_time: 1.0862002373, propagate_forward_time: 32.5400314331, propagate_backward_time: 49.3233299255, weight_update_time: 0.1262881011, other_time: 5.1282844543
epoch time: 8.1571197510s, input_fired_time: 0.1505035311, output_fired_time: 0.0000000000, propagate_forward_time: 7.4374151230, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5692014694
validation[          , genome  3678] predictions:    9664/  10000 (96.64%), best:    9662/10000 (96.62%), error:      1118.53540, best error:      1115.04224 on epoch:    34, epoch:   36/100, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 89.6747207642s, input_fired_time: 1.1770552397, output_fired_time: 1.0902888775, propagate_forward_time: 31.6907653809, propagate_backward_time: 51.6178932190, weight_update_time: 1.1352639198, other_time: 4.0987205505
epoch time: 8.3363142014s, input_fired_time: 0.1556475461, output_fired_time: 0.0000000000, propagate_forward_time: 7.6036186218, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5770483017
validation[          , genome  3678] predictions:    9093/  10000 (90.93%), best:    9662/10000 (96.62%), error:      2770.58887, best error:      1115.04224 on epoch:    34, epoch:   37/100, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 100.6307525635s, input_fired_time: 1.3642278910, output_fired_time: 1.1710498333, propagate_forward_time: 36.4076843262, propagate_backward_time: 57.1909599304, weight_update_time: 0.1517371833, other_time: 4.4968299866
epoch time: 6.8310451508s, input_fired_time: 0.1452099383, output_fired_time: 0.0000000000, propagate_forward_time: 6.1385521889, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5472831726
validation[          , genome  3678] predictions:    9681/  10000 (96.81%), best:    9681/10000 (96.81%), error:      1087.78027, best error:      1087.78027 on epoch:    38, epoch:   38/100, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 90.9926834106s, input_fired_time: 1.1774644852, output_fired_time: 1.1012327671, propagate_forward_time: 32.7033081055, propagate_backward_time: 52.9087715149, weight_update_time: 0.1263016313, other_time: 3.1019020081
epoch time: 6.9390282631s, input_fired_time: 0.1448937654, output_fired_time: 0.0000000000, propagate_forward_time: 6.2542057037, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5399289131
validation[          , genome  3678] predictions:    9277/  10000 (92.77%), best:    9681/10000 (96.81%), error:      2344.59570, best error:      1087.78027 on epoch:    38, epoch:   39/100, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 90.8640518188s, input_fired_time: 1.1785103083, output_fired_time: 1.1071108580, propagate_forward_time: 32.7152214050, propagate_backward_time: 52.7370185852, weight_update_time: 0.1276125014, other_time: 3.1261901855
epoch time: 7.0321640968s, input_fired_time: 0.1434359699, output_fired_time: 0.0000000000, propagate_forward_time: 6.3374061584, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5513219833
validation[          , genome  3678] predictions:    9660/  10000 (96.60%), best:    9681/10000 (96.81%), error:      1132.17883, best error:      1087.78027 on epoch:    38, epoch:   40/100, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 116.2809829712s, input_fired_time: 1.5115368366, output_fired_time: 1.3289973736, propagate_forward_time: 41.4003982544, propagate_backward_time: 68.1177368164, weight_update_time: 0.1469129026, other_time: 3.9223175049
epoch time: 9.3208913803s, input_fired_time: 0.1923525631, output_fired_time: 0.0000000000, propagate_forward_time: 8.4475927353, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6809463501
validation[          , genome  3678] predictions:    9698/  10000 (96.98%), best:    9698/10000 (96.98%), error:      1050.06677, best error:      1050.06677 on epoch:    41, epoch:   41/100, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

epoch time: 93.6419906616s, input_fired_time: 1.2414005995, output_fired_time: 1.1391850710, propagate_forward_time: 35.9284934998, propagate_backward_time: 52.1771240234, weight_update_time: 0.1341998577, other_time: 3.1557884216
epoch time: 6.9049239159s, input_fired_time: 0.1456717551, output_fired_time: 0.0000000000, propagate_forward_time: 6.2129445076, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5463075638
validation[          , genome  3678] predictions:    9675/  10000 (96.75%), best:    9698/10000 (96.98%), error:      1107.88367, best error:      1050.06677 on epoch:    41, epoch:   42/100, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 90.7292251587s, input_fired_time: 1.1809645891, output_fired_time: 1.0984516144, propagate_forward_time: 31.7475757599, propagate_backward_time: 52.5797195435, weight_update_time: 0.1265225708, other_time: 4.1225204468
epoch time: 6.9622659683s, input_fired_time: 0.1468431801, output_fired_time: 0.0000000000, propagate_forward_time: 6.2586464882, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5567765236
validation[          , genome  3678] predictions:    9673/  10000 (96.73%), best:    9698/10000 (96.98%), error:      1100.56274, best error:      1050.06677 on epoch:    41, epoch:   43/100, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 90.9127807617s, input_fired_time: 1.1849633455, output_fired_time: 1.0961641073, propagate_forward_time: 31.6182327271, propagate_backward_time: 53.8947944641, weight_update_time: 0.1269267946, other_time: 3.1186256409
epoch time: 6.9981832504s, input_fired_time: 0.1492148638, output_fired_time: 0.0000000000, propagate_forward_time: 6.2974123955, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5515561104
validation[          , genome  3678] predictions:    8910/  10000 (89.10%), best:    9698/10000 (96.98%), error:      3866.21582, best error:      1050.06677 on epoch:    41, epoch:   44/100, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

resetting weights to those on epoch: 41
epoch time: 90.9565353394s, input_fired_time: 1.1825954914, output_fired_time: 1.1012086868, propagate_forward_time: 32.7937507629, propagate_backward_time: 52.7620277405, weight_update_time: 0.1263813227, other_time: 3.1169509888
epoch time: 6.8427972794s, input_fired_time: 0.1449083984, output_fired_time: 0.0000000000, propagate_forward_time: 6.1580605507, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5398283005
validation[          , genome  3678] predictions:    9610/  10000 (96.10%), best:    9698/10000 (96.98%), error:      1269.71790, best error:      1050.06677 on epoch:    41, epoch:   45/100, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 91.0821533203s, input_fired_time: 1.1864753962, output_fired_time: 2.1369831562, propagate_forward_time: 32.7786407471, propagate_backward_time: 51.8873825073, weight_update_time: 0.1271901876, other_time: 3.0926666260
epoch time: 6.9929132462s, input_fired_time: 0.1434105933, output_fired_time: 0.0000000000, propagate_forward_time: 6.3093605042, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5401420593
validation[          , genome  3678] predictions:    9662/  10000 (96.62%), best:    9698/10000 (96.98%), error:      1130.32715, best error:      1050.06677 on epoch:    41, epoch:   46/100, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 104.1039123535s, input_fired_time: 1.3838752508, output_fired_time: 1.2733994722, propagate_forward_time: 37.7196273804, propagate_backward_time: 60.0904350281, weight_update_time: 0.1386168450, other_time: 3.6365776062
epoch time: 7.5605564117s, input_fired_time: 0.1508524865, output_fired_time: 0.0000000000, propagate_forward_time: 6.8452525139, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5644512177
validation[          , genome  3678] predictions:    9702/  10000 (97.02%), best:    9702/10000 (97.02%), error:      1015.38104, best error:      1015.38104 on epoch:    47, epoch:   47/100, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 89.0356979370s, input_fired_time: 1.1786559820, output_fired_time: 1.0922082663, propagate_forward_time: 31.5709323883, propagate_backward_time: 52.1031761169, weight_update_time: 0.1263124943, other_time: 3.0907249451
epoch time: 6.8934874535s, input_fired_time: 0.1436735392, output_fired_time: 0.0000000000, propagate_forward_time: 6.2121748924, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5376391411
validation[          , genome  3678] predictions:    9612/  10000 (96.12%), best:    9702/10000 (97.02%), error:      1361.03955, best error:      1015.38104 on epoch:    47, epoch:   48/100, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 90.5448150635s, input_fired_time: 2.2003989220, output_fired_time: 1.0937423706, propagate_forward_time: 31.6569175720, propagate_backward_time: 52.4817237854, weight_update_time: 0.1258379519, other_time: 3.1120300293
epoch time: 6.7948441505s, input_fired_time: 0.1447936296, output_fired_time: 0.0000000000, propagate_forward_time: 6.1093158722, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5407347679
validation[          , genome  3678] predictions:    9611/  10000 (96.11%), best:    9702/10000 (97.02%), error:      1366.69006, best error:      1015.38104 on epoch:    47, epoch:   49/100, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 90.8886260986s, input_fired_time: 1.1829564571, output_fired_time: 1.0946260691, propagate_forward_time: 33.7817192078, propagate_backward_time: 51.7095718384, weight_update_time: 0.1271104813, other_time: 3.1197547913
epoch time: 6.9203619957s, input_fired_time: 0.1434008181, output_fired_time: 0.0000000000, propagate_forward_time: 6.2319421768, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5450191498
validation[          , genome  3678] predictions:    9559/  10000 (95.59%), best:    9702/10000 (97.02%), error:      1443.10901, best error:      1015.38104 on epoch:    47, epoch:   50/100, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

epoch time: 106.7177886963s, input_fired_time: 1.3723590374, output_fired_time: 1.2768429518, propagate_forward_time: 38.3439445496, propagate_backward_time: 61.1532363892, weight_update_time: 0.1412891895, other_time: 4.5714035034
epoch time: 7.0821290016s, input_fired_time: 0.1479853094, output_fired_time: 0.0000000000, propagate_forward_time: 6.3812193871, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5529241562
validation[          , genome  3678] predictions:    9533/  10000 (95.33%), best:    9702/10000 (97.02%), error:      1498.25159, best error:      1015.38104 on epoch:    47, epoch:   51/100, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488

epoch time: 88.8870773315s, input_fired_time: 1.1727342606, output_fired_time: 1.0780476332, propagate_forward_time: 32.2828178406, propagate_backward_time: 51.2820167542, weight_update_time: 0.1259322912, other_time: 3.0714569092
epoch time: 12.1796550751s, input_fired_time: 0.2148135751, output_fired_time: 0.0000000000, propagate_forward_time: 11.1040267944, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.8608150482
validation[          , genome  3678] predictions:    9700/  10000 (97.00%), best:    9702/10000 (97.02%), error:      1033.87000, best error:      1015.38104 on epoch:    47, epoch:   52/100, mu: 0.9559730291, learning_rate: 0.0008680347, weight_decay: 0.0000347214

epoch time: 97.6102294922s, input_fired_time: 1.2406216860, output_fired_time: 1.2082146406, propagate_forward_time: 36.2263755798, propagate_backward_time: 55.6721534729, weight_update_time: 0.1267963648, other_time: 3.2628631592
epoch time: 6.8857398033s, input_fired_time: 0.1434912086, output_fired_time: 0.0000000000, propagate_forward_time: 6.2004661560, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5417823792
validation[          , genome  3678] predictions:    9561/  10000 (95.61%), best:    9702/10000 (97.02%), error:      1385.30334, best error:      1015.38104 on epoch:    47, epoch:   53/100, mu: 0.9576743841, learning_rate: 0.0008246329, weight_decay: 0.0000329853

epoch time: 105.8741455078s, input_fired_time: 1.3870694637, output_fired_time: 1.2742035389, propagate_forward_time: 38.9184455872, propagate_backward_time: 59.7889709473, weight_update_time: 0.1569402069, other_time: 4.5054626465
epoch time: 7.2630267143s, input_fired_time: 0.1479432583, output_fired_time: 0.0000000000, propagate_forward_time: 6.5639863014, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5510973930
validation[          , genome  3678] predictions:    9626/  10000 (96.26%), best:    9702/10000 (97.02%), error:      1265.80261, best error:      1015.38104 on epoch:    47, epoch:   54/100, mu: 0.9592906833, learning_rate: 0.0007834013, weight_decay: 0.0000313361

epoch time: 95.0850677490s, input_fired_time: 1.2469669580, output_fired_time: 1.1336601973, propagate_forward_time: 33.2536048889, propagate_backward_time: 56.1710205078, weight_update_time: 0.1321102977, other_time: 3.2798194885
epoch time: 6.8439764977s, input_fired_time: 0.1447105110, output_fired_time: 0.0000000000, propagate_forward_time: 6.1576414108, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5416245461
validation[          , genome  3678] predictions:    9682/  10000 (96.82%), best:    9702/10000 (97.02%), error:      1075.71252, best error:      1015.38104 on epoch:    47, epoch:   55/100, mu: 0.9608261585, learning_rate: 0.0007442312, weight_decay: 0.0000297693

epoch time: 90.7929916382s, input_fired_time: 1.1923959255, output_fired_time: 1.0923258066, propagate_forward_time: 31.7108421326, propagate_backward_time: 52.6556282043, weight_update_time: 0.1276739538, other_time: 4.1417999268
epoch time: 6.8788862228s, input_fired_time: 0.1455525458, output_fired_time: 0.0000000000, propagate_forward_time: 6.1867704391, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5465631485
validation[          , genome  3678] predictions:    9673/  10000 (96.73%), best:    9702/10000 (97.02%), error:      1110.30420, best error:      1015.38104 on epoch:    47, epoch:   56/100, mu: 0.9622848630, learning_rate: 0.0007070196, weight_decay: 0.0000282808

epoch time: 99.5087280273s, input_fired_time: 1.3021225929, output_fired_time: 1.1547478437, propagate_forward_time: 39.8985748291, propagate_backward_time: 53.7763404846, weight_update_time: 0.1335958540, other_time: 3.3769416809
epoch time: 6.7785005569s, input_fired_time: 0.1422640979, output_fired_time: 0.0000000000, propagate_forward_time: 6.1017045975, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5345320702
validation[          , genome  3678] predictions:    9647/  10000 (96.47%), best:    9702/10000 (97.02%), error:      1170.01318, best error:      1015.38104 on epoch:    47, epoch:   57/100, mu: 0.9636706114, learning_rate: 0.0006716687, weight_decay: 0.0000268668

epoch time: 90.7223663330s, input_fired_time: 1.1853827238, output_fired_time: 1.0946397781, propagate_forward_time: 33.6167068481, propagate_backward_time: 51.6795539856, weight_update_time: 0.1266393512, other_time: 3.1460838318
epoch time: 6.9322795868s, input_fired_time: 0.1469404548, output_fired_time: 0.0000000000, propagate_forward_time: 6.2362675667, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5490717888
validation[          , genome  3678] predictions:    9655/  10000 (96.55%), best:    9702/10000 (97.02%), error:      1175.75989, best error:      1015.38104 on epoch:    47, epoch:   58/100, mu: 0.9649870992, learning_rate: 0.0006380852, weight_decay: 0.0000255234

epoch time: 90.7641372681s, input_fired_time: 1.1897928715, output_fired_time: 1.0914961100, propagate_forward_time: 33.6175422668, propagate_backward_time: 50.7536697388, weight_update_time: 0.1268901825, other_time: 4.1116294861
epoch time: 6.9677290916s, input_fired_time: 0.1451400220, output_fired_time: 0.0000000000, propagate_forward_time: 6.2732067108, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5493822098
validation[          , genome  3678] predictions:    9672/  10000 (96.72%), best:    9702/10000 (97.02%), error:      1072.81580, best error:      1015.38104 on epoch:    47, epoch:   59/100, mu: 0.9662377238, learning_rate: 0.0006061810, weight_decay: 0.0000242472

epoch time: 90.6743240356s, input_fired_time: 1.1886129379, output_fired_time: 1.0941896439, propagate_forward_time: 33.6755638123, propagate_backward_time: 51.5954818726, weight_update_time: 0.1264346987, other_time: 3.1204719543
epoch time: 6.8758773804s, input_fired_time: 0.1446817368, output_fired_time: 0.0000000000, propagate_forward_time: 6.1857314110, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5454640388
validation[          , genome  3678] predictions:    9694/  10000 (96.94%), best:    9702/10000 (97.02%), error:      1029.46594, best error:      1015.38104 on epoch:    47, epoch:   60/100, mu: 0.9674258232, learning_rate: 0.0005758719, weight_decay: 0.0000230349

epoch time: 103.0279388428s, input_fired_time: 1.3566997051, output_fired_time: 1.2763205767, propagate_forward_time: 38.6119346619, propagate_backward_time: 58.2197341919, weight_update_time: 0.1446051300, other_time: 3.5632514954
epoch time: 8.4233512878s, input_fired_time: 0.1527046114, output_fired_time: 0.0000000000, propagate_forward_time: 7.6999998093, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5706472397
validation[          , genome  3678] predictions:    9694/  10000 (96.94%), best:    9702/10000 (97.02%), error:      1028.77979, best error:      1015.38104 on epoch:    47, epoch:   61/100, mu: 0.9685545564, learning_rate: 0.0005470783, weight_decay: 0.0000218831

epoch time: 95.1843414307s, input_fired_time: 2.2777996063, output_fired_time: 1.1533559561, propagate_forward_time: 34.4825553894, propagate_backward_time: 54.0511207581, weight_update_time: 0.1366439313, other_time: 3.2195053101
epoch time: 8.3399934769s, input_fired_time: 0.1548660845, output_fired_time: 0.0000000000, propagate_forward_time: 7.5858302116, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5992970467
validation[          , genome  3678] predictions:    8860/  10000 (88.60%), best:    9702/10000 (97.02%), error:      4246.87451, best error:      1015.38104 on epoch:    47, epoch:   62/100, mu: 0.9696268439, learning_rate: 0.0005197244, weight_decay: 0.0000207890

resetting weights to those on epoch: 47
epoch time: 109.2599029541s, input_fired_time: 1.4595700502, output_fired_time: 2.3718080521, propagate_forward_time: 40.4205894470, propagate_backward_time: 61.0101013184, weight_update_time: 0.3867993057, other_time: 3.9978332520
epoch time: 8.2745800018s, input_fired_time: 0.1540489495, output_fired_time: 0.0000000000, propagate_forward_time: 7.5440840721, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5764470100
validation[          , genome  3678] predictions:    9732/  10000 (97.32%), best:    9732/10000 (97.32%), error:       938.91113, best error:       938.91113 on epoch:    63, epoch:   63/100, mu: 0.9706454873, learning_rate: 0.0004937382, weight_decay: 0.0000197495

epoch time: 100.3047027588s, input_fired_time: 1.3315740824, output_fired_time: 1.2154343128, propagate_forward_time: 37.5578842163, propagate_backward_time: 56.8766899109, weight_update_time: 0.1463618129, other_time: 3.3231239319
epoch time: 7.0330471992s, input_fired_time: 0.1502097845, output_fired_time: 0.0000000000, propagate_forward_time: 6.3320155144, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5508217812
validation[          , genome  3678] predictions:    9697/  10000 (96.97%), best:    9732/10000 (97.32%), error:      1027.18066, best error:       938.91113 on epoch:    63, epoch:   64/100, mu: 0.9716132283, learning_rate: 0.0004690512, weight_decay: 0.0000187621

epoch time: 90.6403579712s, input_fired_time: 1.1836117506, output_fired_time: 1.0944415331, propagate_forward_time: 33.6423988342, propagate_backward_time: 51.5963134766, weight_update_time: 0.1265817285, other_time: 3.1235923767
epoch time: 6.8643875122s, input_fired_time: 0.1440795809, output_fired_time: 0.0000000000, propagate_forward_time: 6.1814308167, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5388770103
validation[          , genome  3678] predictions:    9650/  10000 (96.50%), best:    9732/10000 (97.32%), error:      1146.93555, best error:       938.91113 on epoch:    63, epoch:   65/100, mu: 0.9725325704, learning_rate: 0.0004455987, weight_decay: 0.0000178240

epoch time: 94.1890411377s, input_fired_time: 1.2436329126, output_fired_time: 1.1624472141, propagate_forward_time: 34.1917266846, propagate_backward_time: 54.4002723694, weight_update_time: 0.1269400120, other_time: 3.1909675598
epoch time: 6.9102196693s, input_fired_time: 0.1445067227, output_fired_time: 0.0000000000, propagate_forward_time: 6.2233552933, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5423574448
validation[          , genome  3678] predictions:    9688/  10000 (96.88%), best:    9732/10000 (97.32%), error:      1069.45825, best error:       938.91113 on epoch:    63, epoch:   66/100, mu: 0.9734059572, learning_rate: 0.0004233187, weight_decay: 0.0000169328

epoch time: 91.0496902466s, input_fired_time: 1.1871439219, output_fired_time: 1.0949707031, propagate_forward_time: 33.8344802856, propagate_backward_time: 50.8143043518, weight_update_time: 0.1277463734, other_time: 4.1187934875
epoch time: 6.9143676758s, input_fired_time: 0.1470455527, output_fired_time: 0.0000000000, propagate_forward_time: 6.2127785683, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5545434952
validation[          , genome  3678] predictions:    9669/  10000 (96.69%), best:    9732/10000 (97.32%), error:      1115.40674, best error:       938.91113 on epoch:    63, epoch:   67/100, mu: 0.9742356539, learning_rate: 0.0004021528, weight_decay: 0.0000160861

epoch time: 91.0491790771s, input_fired_time: 1.1885045767, output_fired_time: 1.0981308222, propagate_forward_time: 33.7202796936, propagate_backward_time: 51.9300193787, weight_update_time: 0.1262777150, other_time: 3.1122436523
epoch time: 6.9785842896s, input_fired_time: 0.1472366750, output_fired_time: 0.0000000000, propagate_forward_time: 6.2580657005, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5732817650
validation[          , genome  3678] predictions:    9681/  10000 (96.81%), best:    9732/10000 (97.32%), error:      1110.32031, best error:       938.91113 on epoch:    63, epoch:   68/100, mu: 0.9750238657, learning_rate: 0.0003820452, weight_decay: 0.0000152818

epoch time: 89.7065811157s, input_fired_time: 1.1793454885, output_fired_time: 1.1034557819, propagate_forward_time: 34.6759071350, propagate_backward_time: 49.6670494080, weight_update_time: 0.1264737099, other_time: 3.0808258057
epoch time: 11.4408674240s, input_fired_time: 0.1996767074, output_fired_time: 0.0000000000, propagate_forward_time: 10.4194774628, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.8217134476
validation[          , genome  3678] predictions:    9562/  10000 (95.62%), best:    9732/10000 (97.32%), error:      1449.58691, best error:       938.91113 on epoch:    63, epoch:   69/100, mu: 0.9757726789, learning_rate: 0.0003629429, weight_decay: 0.0000145177

epoch time: 102.6941680908s, input_fired_time: 1.3328679800, output_fired_time: 1.2799247503, propagate_forward_time: 39.1756172180, propagate_backward_time: 57.4170036316, weight_update_time: 0.1609772146, other_time: 3.4887542725
epoch time: 8.2099876404s, input_fired_time: 0.1522887796, output_fired_time: 0.0000000000, propagate_forward_time: 7.4802875519, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5774116516
validation[          , genome  3678] predictions:    9700/  10000 (97.00%), best:    9732/10000 (97.32%), error:      1042.18640, best error:       938.91113 on epoch:    63, epoch:   70/100, mu: 0.9764840603, learning_rate: 0.0003447958, weight_decay: 0.0000137918

epoch time: 89.8758926392s, input_fired_time: 1.1881788969, output_fired_time: 1.0972863436, propagate_forward_time: 32.7663764954, propagate_backward_time: 51.6842269897, weight_update_time: 0.1344728768, other_time: 3.1398200989
epoch time: 6.8982100487s, input_fired_time: 0.1472952068, output_fired_time: 0.0000000000, propagate_forward_time: 6.2052984238, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5456166267
validation[          , genome  3678] predictions:    9701/  10000 (97.01%), best:    9732/10000 (97.32%), error:      1005.35376, best error:       938.91113 on epoch:    63, epoch:   71/100, mu: 0.9771598577, learning_rate: 0.0003275560, weight_decay: 0.0000131022

epoch time: 91.0174407959s, input_fired_time: 1.2170058489, output_fired_time: 1.0950002670, propagate_forward_time: 35.1813774109, propagate_backward_time: 50.3560180664, weight_update_time: 0.1267260313, other_time: 3.1680412292
epoch time: 9.3330287933s, input_fired_time: 0.1548714936, output_fired_time: 0.0000000000, propagate_forward_time: 8.5459938049, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6321630478
validation[          , genome  3678] predictions:    9650/  10000 (96.50%), best:    9732/10000 (97.32%), error:      1214.10791, best error:       938.91113 on epoch:    63, epoch:   72/100, mu: 0.9778018594, learning_rate: 0.0003111782, weight_decay: 0.0000124471

epoch time: 92.0204544067s, input_fired_time: 1.2219777107, output_fired_time: 1.1306228638, propagate_forward_time: 34.4605178833, propagate_backward_time: 52.0356292725, weight_update_time: 0.1256424934, other_time: 3.1717071533
epoch time: 8.2024602890s, input_fired_time: 0.1521509439, output_fired_time: 0.0000000000, propagate_forward_time: 7.4854040146, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5649051666
validation[          , genome  3678] predictions:    9681/  10000 (96.81%), best:    9732/10000 (97.32%), error:      1080.52527, best error:       938.91113 on epoch:    63, epoch:   73/100, mu: 0.9784117937, learning_rate: 0.0002956192, weight_decay: 0.0000118248

epoch time: 89.3140106201s, input_fired_time: 1.1789904833, output_fired_time: 1.0859500170, propagate_forward_time: 33.5571022034, propagate_backward_time: 50.4185028076, weight_update_time: 0.1250907481, other_time: 3.0734596252
epoch time: 6.9917473793s, input_fired_time: 0.1477199048, output_fired_time: 0.0000000000, propagate_forward_time: 6.2917203903, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5523071289
validation[          , genome  3678] predictions:    9682/  10000 (96.82%), best:    9732/10000 (97.32%), error:      1081.95239, best error:       938.91113 on epoch:    63, epoch:   74/100, mu: 0.9789912105, learning_rate: 0.0002808383, weight_decay: 0.0000112335

epoch time: 96.2761840820s, input_fired_time: 1.3052998781, output_fired_time: 1.1811498404, propagate_forward_time: 34.7328300476, propagate_backward_time: 55.7686004639, weight_update_time: 0.1347880065, other_time: 3.2883033752
epoch time: 8.6849641800s, input_fired_time: 0.1545875818, output_fired_time: 0.0000000000, propagate_forward_time: 7.9232993126, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6070771217
validation[          , genome  3678] predictions:    9655/  10000 (96.55%), best:    9732/10000 (97.32%), error:      1193.22168, best error:       938.91113 on epoch:    63, epoch:   75/100, mu: 0.9795416594, learning_rate: 0.0002667964, weight_decay: 0.0000106719

epoch time: 91.8781051636s, input_fired_time: 1.2108652592, output_fired_time: 1.1310272217, propagate_forward_time: 32.3960952759, propagate_backward_time: 53.9318122864, weight_update_time: 0.1342193633, other_time: 3.2083015442
epoch time: 8.1426362991s, input_fired_time: 0.1503178179, output_fired_time: 0.0000000000, propagate_forward_time: 7.4198341370, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5724844933
validation[          , genome  3678] predictions:    9528/  10000 (95.28%), best:    9732/10000 (97.32%), error:      1488.68567, best error:       938.91113 on epoch:    63, epoch:   76/100, mu: 0.9800645709, learning_rate: 0.0002534565, weight_decay: 0.0000101383

epoch time: 89.3236923218s, input_fired_time: 2.0956165791, output_fired_time: 1.0884702206, propagate_forward_time: 32.5033378601, propagate_backward_time: 50.5530738831, weight_update_time: 0.1261267811, other_time: 3.0831909180
epoch time: 8.0848836899s, input_fired_time: 0.1505728811, output_fired_time: 0.0000000000, propagate_forward_time: 7.3716459274, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5626649857
validation[          , genome  3678] predictions:    9658/  10000 (96.58%), best:    9732/10000 (97.32%), error:      1165.11890, best error:       938.91113 on epoch:    63, epoch:   77/100, mu: 0.9805613160, learning_rate: 0.0002407837, weight_decay: 0.0000096314

epoch time: 89.4886932373s, input_fired_time: 1.1860386133, output_fired_time: 1.0925372839, propagate_forward_time: 34.5479927063, propagate_backward_time: 49.5413780212, weight_update_time: 0.1277561933, other_time: 3.1207504272
epoch time: 7.9744706154s, input_fired_time: 1.1558012962, output_fired_time: 0.0000000000, propagate_forward_time: 6.2662768364, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5523924828
validation[          , genome  3678] predictions:    9633/  10000 (96.33%), best:    9732/10000 (97.32%), error:      1224.99817, best error:       938.91113 on epoch:    63, epoch:   78/100, mu: 0.9810332656, learning_rate: 0.0002287445, weight_decay: 0.0000091498

epoch time: 89.8102493286s, input_fired_time: 1.1894154549, output_fired_time: 2.1085255146, propagate_forward_time: 34.6637191772, propagate_backward_time: 48.7462120056, weight_update_time: 0.1262865812, other_time: 3.1023750305
epoch time: 7.0018949509s, input_fired_time: 0.1484764218, output_fired_time: 0.0000000000, propagate_forward_time: 6.3045158386, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5489025116
validation[          , genome  3678] predictions:    9642/  10000 (96.42%), best:    9732/10000 (97.32%), error:      1176.38904, best error:       938.91113 on epoch:    63, epoch:   79/100, mu: 0.9814816117, learning_rate: 0.0002173073, weight_decay: 0.0000086923

epoch time: 100.8284454346s, input_fired_time: 1.3007735014, output_fired_time: 1.1974372864, propagate_forward_time: 35.5064239502, propagate_backward_time: 58.4685287476, weight_update_time: 0.1381053478, other_time: 4.3552780151
epoch time: 8.2737302780s, input_fired_time: 0.1534342617, output_fired_time: 0.0000000000, propagate_forward_time: 6.5396537781, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5806427002
validation[          , genome  3678] predictions:    9712/  10000 (97.12%), best:    9732/10000 (97.32%), error:      1015.81830, best error:       938.91113 on epoch:    63, epoch:   80/100, mu: 0.9819075465, learning_rate: 0.0002064419, weight_decay: 0.0000082577

epoch time: 105.7541580200s, input_fired_time: 1.4121351242, output_fired_time: 1.2708503008, propagate_forward_time: 39.4981460571, propagate_backward_time: 59.0417137146, weight_update_time: 0.1305540353, other_time: 4.5313148499
epoch time: 8.2408790588s, input_fired_time: 0.1485795081, output_fired_time: 0.0000000000, propagate_forward_time: 7.5341830254, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5581164360
validation[          , genome  3678] predictions:    9661/  10000 (96.61%), best:    9732/10000 (97.32%), error:      1179.76172, best error:       938.91113 on epoch:    63, epoch:   81/100, mu: 0.9823121428, learning_rate: 0.0001961198, weight_decay: 0.0000078448

epoch time: 88.7053146362s, input_fired_time: 1.1781730652, output_fired_time: 1.0828678608, propagate_forward_time: 32.1928443909, propagate_backward_time: 50.1641845703, weight_update_time: 0.1259918958, other_time: 4.0872459412
epoch time: 6.7896718979s, input_fired_time: 0.1453966647, output_fired_time: 0.0000000000, propagate_forward_time: 6.1010789871, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5431962013
validation[          , genome  3678] predictions:    9712/  10000 (97.12%), best:    9732/10000 (97.32%), error:       985.30322, best error:       938.91113 on epoch:    63, epoch:   82/100, mu: 0.9826965332, learning_rate: 0.0001863138, weight_decay: 0.0000074526

epoch time: 90.4093093872s, input_fired_time: 1.1887801886, output_fired_time: 1.0843522549, propagate_forward_time: 31.5519237518, propagate_backward_time: 52.5601081848, weight_update_time: 0.1265182197, other_time: 4.0241432190
epoch time: 6.9175667763s, input_fired_time: 0.1461775452, output_fired_time: 0.0000000000, propagate_forward_time: 6.2311239243, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5402650833
validation[          , genome  3678] predictions:    9672/  10000 (96.72%), best:    9732/10000 (97.32%), error:      1088.20203, best error:       938.91113 on epoch:    63, epoch:   83/100, mu: 0.9830617309, learning_rate: 0.0001769981, weight_decay: 0.0000070799

epoch time: 89.9213333130s, input_fired_time: 1.2013055086, output_fired_time: 1.0840914249, propagate_forward_time: 34.3095397949, propagate_backward_time: 50.2312889099, weight_update_time: 0.1258917004, other_time: 3.0951042175
epoch time: 6.9992213249s, input_fired_time: 0.1493252367, output_fired_time: 0.0000000000, propagate_forward_time: 6.3037190437, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5461769104
validation[          , genome  3678] predictions:    9706/  10000 (97.06%), best:    9732/10000 (97.32%), error:      1013.15411, best error:       938.91113 on epoch:    63, epoch:   84/100, mu: 0.9834086299, learning_rate: 0.0001681482, weight_decay: 0.0000067259

epoch time: 91.0414352417s, input_fired_time: 1.1952142715, output_fired_time: 1.0961850882, propagate_forward_time: 34.6599845886, propagate_backward_time: 50.9665298462, weight_update_time: 0.1273950040, other_time: 3.1235237122
epoch time: 6.8699283600s, input_fired_time: 0.1445414126, output_fired_time: 0.0000000000, propagate_forward_time: 6.1643524170, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5610346794
validation[          , genome  3678] predictions:    9713/  10000 (97.13%), best:    9732/10000 (97.32%), error:       975.49292, best error:       938.91113 on epoch:    63, epoch:   85/100, mu: 0.9837381840, learning_rate: 0.0001597408, weight_decay: 0.0000063896

epoch time: 94.8680343628s, input_fired_time: 1.2283688784, output_fired_time: 1.1443203688, propagate_forward_time: 35.2320213318, propagate_backward_time: 54.0229606628, weight_update_time: 0.1287466735, other_time: 3.2403640747
epoch time: 7.2136073112s, input_fired_time: 0.1493282318, output_fired_time: 0.0000000000, propagate_forward_time: 6.5131607056, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5511183739
validation[          , genome  3678] predictions:    9684/  10000 (96.84%), best:    9732/10000 (97.32%), error:      1083.42969, best error:       938.91113 on epoch:    63, epoch:   86/100, mu: 0.9840512872, learning_rate: 0.0001517538, weight_decay: 0.0000060702

epoch time: 101.7296295166s, input_fired_time: 1.3114115000, output_fired_time: 1.2541027069, propagate_forward_time: 36.5719795227, propagate_backward_time: 59.0221633911, weight_update_time: 0.1617961377, other_time: 3.5699729919
epoch time: 8.0882444382s, input_fired_time: 0.1498960555, output_fired_time: 0.0000000000, propagate_forward_time: 7.3828897476, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5554585457
validation[          , genome  3678] predictions:    9609/  10000 (96.09%), best:    9732/10000 (97.32%), error:      1326.20276, best error:       938.91113 on epoch:    63, epoch:   87/100, mu: 0.9843487144, learning_rate: 0.0001441661, weight_decay: 0.0000057666

epoch time: 93.9076004028s, input_fired_time: 2.2528014183, output_fired_time: 1.1454091072, propagate_forward_time: 33.3737411499, propagate_backward_time: 53.9183044434, weight_update_time: 0.1355112493, other_time: 3.2173461914
epoch time: 9.5050411224s, input_fired_time: 0.1825049222, output_fired_time: 0.0000000000, propagate_forward_time: 8.6954803467, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.6270561218
validation[          , genome  3678] predictions:    9699/  10000 (96.99%), best:    9732/10000 (97.32%), error:      1041.33728, best error:       938.91113 on epoch:    63, epoch:   88/100, mu: 0.9846313000, learning_rate: 0.0001369578, weight_decay: 0.0000054783

epoch time: 93.1863937378s, input_fired_time: 2.2136051655, output_fired_time: 1.1391047239, propagate_forward_time: 32.5437088013, propagate_backward_time: 54.1156768799, weight_update_time: 0.1285502613, other_time: 3.1742935181
epoch time: 7.0387763977s, input_fired_time: 0.1496101618, output_fired_time: 0.0000000000, propagate_forward_time: 6.3386569023, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5505094528
validation[          , genome  3678] predictions:    9707/  10000 (97.07%), best:    9732/10000 (97.32%), error:       996.34918, best error:       938.91113 on epoch:    63, epoch:   89/100, mu: 0.9848997593, learning_rate: 0.0001301099, weight_decay: 0.0000052044

epoch time: 89.3603744507s, input_fired_time: 1.1883875132, output_fired_time: 1.0907517672, propagate_forward_time: 32.5192375183, propagate_backward_time: 51.4856872559, weight_update_time: 0.1256762147, other_time: 3.0763130188
epoch time: 8.0681962967s, input_fired_time: 0.1481348127, output_fired_time: 0.0000000000, propagate_forward_time: 7.3631153107, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5569462776
validation[          , genome  3678] predictions:    9699/  10000 (96.99%), best:    9732/10000 (97.32%), error:      1016.66278, best error:       938.91113 on epoch:    63, epoch:   90/100, mu: 0.9851547480, learning_rate: 0.0001236044, weight_decay: 0.0000049442

epoch time: 99.2669372559s, input_fired_time: 1.2973515987, output_fired_time: 1.2224410772, propagate_forward_time: 38.1661567688, propagate_backward_time: 55.2387428284, weight_update_time: 0.1398129165, other_time: 3.3422470093
epoch time: 6.7712154388s, input_fired_time: 0.1439266801, output_fired_time: 0.0000000000, propagate_forward_time: 6.0942230225, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5330657959
validation[          , genome  3678] predictions:    9670/  10000 (96.70%), best:    9732/10000 (97.32%), error:      1078.14893, best error:       938.91113 on epoch:    63, epoch:   91/100, mu: 0.9853969812, learning_rate: 0.0001174242, weight_decay: 0.0000046970

epoch time: 90.6400070190s, input_fired_time: 1.1921712160, output_fired_time: 1.0900290012, propagate_forward_time: 32.6214561462, propagate_backward_time: 51.5894546509, weight_update_time: 0.1265989542, other_time: 4.1469001770
epoch time: 6.9650688171s, input_fired_time: 0.1455154419, output_fired_time: 0.0000000000, propagate_forward_time: 6.2692365646, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5503168106
validation[          , genome  3678] predictions:    9698/  10000 (96.98%), best:    9732/10000 (97.32%), error:      1033.30762, best error:       938.91113 on epoch:    63, epoch:   92/100, mu: 0.9856271148, learning_rate: 0.0001115530, weight_decay: 0.0000044621

epoch time: 90.8133926392s, input_fired_time: 1.1864577532, output_fired_time: 1.0949872732, propagate_forward_time: 34.6817893982, propagate_backward_time: 50.7567901611, weight_update_time: 0.1263686866, other_time: 3.0933723450
epoch time: 7.0078735352s, input_fired_time: 0.1646088213, output_fired_time: 0.0000000000, propagate_forward_time: 6.2973732948, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5458912849
validation[          , genome  3678] predictions:    9493/  10000 (94.93%), best:    9732/10000 (97.32%), error:      1608.87939, best error:       938.91113 on epoch:    63, epoch:   93/100, mu: 0.9858457446, learning_rate: 0.0001059753, weight_decay: 0.0000042390

epoch time: 90.4770431519s, input_fired_time: 1.1926790476, output_fired_time: 1.0904610157, propagate_forward_time: 34.5342102051, propagate_backward_time: 50.5471343994, weight_update_time: 0.1265875846, other_time: 3.1125564575
epoch time: 7.1906766891s, input_fired_time: 0.1530736238, output_fired_time: 0.0000000000, propagate_forward_time: 6.4706301689, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5669727325
validation[          , genome  3678] predictions:    9714/  10000 (97.14%), best:    9732/10000 (97.32%), error:      1004.40009, best error:       938.91113 on epoch:    63, epoch:   94/100, mu: 0.9860534668, learning_rate: 0.0001006766, weight_decay: 0.0000040271

epoch time: 89.2433319092s, input_fired_time: 1.1860140562, output_fired_time: 1.0890517235, propagate_forward_time: 32.5217628479, propagate_backward_time: 50.4500427246, weight_update_time: 0.1255868673, other_time: 3.9964637756
epoch time: 8.1284122467s, input_fired_time: 0.1502137333, output_fired_time: 0.0000000000, propagate_forward_time: 6.4124670029, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 1.5657315254
validation[          , genome  3678] predictions:    9696/  10000 (96.96%), best:    9732/10000 (97.32%), error:      1014.01013, best error:       938.91113 on epoch:    63, epoch:   95/100, mu: 0.9862508178, learning_rate: 0.0000956427, weight_decay: 0.0000038257

epoch time: 98.5395660400s, input_fired_time: 1.3143732548, output_fired_time: 1.1639206409, propagate_forward_time: 37.6315002441, propagate_backward_time: 55.0844879150, weight_update_time: 0.1283655018, other_time: 3.3452835083
epoch time: 7.0181522369s, input_fired_time: 0.1497294605, output_fired_time: 0.0000000000, propagate_forward_time: 6.3213801384, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5470428467
validation[          , genome  3678] predictions:    9726/  10000 (97.26%), best:    9732/10000 (97.32%), error:       972.66901, best error:       938.91113 on epoch:    63, epoch:   96/100, mu: 0.9864382744, learning_rate: 0.0000908606, weight_decay: 0.0000036344

epoch time: 101.0617599487s, input_fired_time: 1.3194737434, output_fired_time: 1.2177114487, propagate_forward_time: 35.1810607910, propagate_backward_time: 58.8800659180, weight_update_time: 0.1713973880, other_time: 4.4634475708
epoch time: 8.1266937256s, input_fired_time: 0.1510997415, output_fired_time: 0.0000000000, propagate_forward_time: 7.4165186882, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5590753555
validation[          , genome  3678] predictions:    9724/  10000 (97.24%), best:    9732/10000 (97.32%), error:       970.06079, best error:       938.91113 on epoch:    63, epoch:   97/100, mu: 0.9866163731, learning_rate: 0.0000863176, weight_decay: 0.0000034527

epoch time: 90.4195175171s, input_fired_time: 1.2041184902, output_fired_time: 1.1127438545, propagate_forward_time: 31.9544143677, propagate_backward_time: 50.9407882690, weight_update_time: 0.1273137778, other_time: 5.2074508667
epoch time: 6.9604077339s, input_fired_time: 0.1471618563, output_fired_time: 0.0000000000, propagate_forward_time: 6.2702274323, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5430183411
validation[          , genome  3678] predictions:    9691/  10000 (96.91%), best:    9732/10000 (97.32%), error:      1069.70789, best error:       938.91113 on epoch:    63, epoch:   98/100, mu: 0.9867855310, learning_rate: 0.0000820017, weight_decay: 0.0000032801

epoch time: 91.6307373047s, input_fired_time: 1.2311259508, output_fired_time: 1.1259155273, propagate_forward_time: 33.4508094788, propagate_backward_time: 52.6296157837, weight_update_time: 0.1275377274, other_time: 3.1932716370
epoch time: 6.9716453552s, input_fired_time: 0.1470132917, output_fired_time: 0.0000000000, propagate_forward_time: 6.2813582420, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5432739258
validation[          , genome  3678] predictions:    9699/  10000 (96.99%), best:    9732/10000 (97.32%), error:      1007.81531, best error:       938.91113 on epoch:    63, epoch:   99/100, mu: 0.9869462252, learning_rate: 0.0000779016, weight_decay: 0.0000031161

epoch time: 89.4937362671s, input_fired_time: 1.2225278616, output_fired_time: 1.0882399082, propagate_forward_time: 32.5897712708, propagate_backward_time: 50.5037612915, weight_update_time: 0.1258555502, other_time: 4.0894355774
epoch time: 6.9787416458s, input_fired_time: 0.1503661722, output_fired_time: 0.0000000000, propagate_forward_time: 6.2827458382, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5456295013
validation[          , genome  3678] predictions:    9671/  10000 (96.71%), best:    9732/10000 (97.32%), error:      1164.71521, best error:       938.91113 on epoch:    63, epoch:  100/100, mu: 0.9870989323, learning_rate: 0.0000740065, weight_decay: 0.0000029603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 37.1490402222s, input_fired_time: 0.7429523468, output_fired_time: 0.0000000000, propagate_forward_time: 33.4303627014, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.9757270813
best training[          , genome  3678] predictions:   49103/  50000 (98.21%), best:    9732/10000 (97.32%), error:      3234.54126, best error:       938.91113 on epoch:    63, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 8.2145071030s, input_fired_time: 0.1513270140, output_fired_time: 0.0000000000, propagate_forward_time: 7.4972133636, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5659666061
   testing[          , genome  3678] predictions:    9723/  10000 (97.23%), best:    9732/10000 (97.32%), error:       895.38635, best error:       938.91113 on epoch:    63, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
backpropagation finished successfully!
09:31:35 (6532): called boinc_finish(0)

</stderr_txt>
]]>