Nameexact_genome_1512568488_44_7138_0
Workunit2606986
Created8 Dec 2017, 0:03:04 UTC
Sent8 Dec 2017, 0:13:11 UTC
Report deadline14 Dec 2017, 19:25:24 UTC
Received9 Dec 2017, 4:48:53 UTC
Server stateOver
OutcomeSuccess
Client stateDone
Exit status0 (0x0)
Computer ID38872
Run time2 hours 24 min 6 sec
CPU time2 hours 10 min 24 sec
Validate stateValid
Credit1,700.04
Device peak FLOPS4.61 GFLOPS
Application versionEXACT Batch Norm With Scaled FMP CNN Trainer v0.34
Peak working set size157.36 MB
Peak swap size160.32 MB
Peak disk usage1.91 MB

Stderr output

<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
141716
validation[          , genome  7138] predictions:    9689/  10000 (96.89%), best:    9731/10000 (97.31%), error:      1021.36383, best error:       920.42090 on epoch:    18, epoch:   20/100, mu: 0.8143420219, learning_rate: 0.0044810725, weight_decay: 0.0001792429

epoch time: 73.1868133545s, input_fired_time: 1.0823007822, output_fired_time: 1.0042014122, propagate_forward_time: 26.9780158997, propagate_backward_time: 41.9636688232, weight_update_time: 0.0930056572, other_time: 2.1586265564
epoch time: 5.8315067291s, input_fired_time: 0.1319005489, output_fired_time: 0.0000000000, propagate_forward_time: 5.3197851181, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3798208237
validation[          , genome  7138] predictions:    9705/  10000 (97.05%), best:    9731/10000 (97.31%), error:       930.50824, best error:       920.42090 on epoch:    18, epoch:   21/100, mu: 0.8231249452, learning_rate: 0.0042570187, weight_decay: 0.0001702808

epoch time: 75.8690948486s, input_fired_time: 1.1387938261, output_fired_time: 1.0918231010, propagate_forward_time: 27.6244487762, propagate_backward_time: 43.5844688416, weight_update_time: 0.0925056562, other_time: 2.4295616150
epoch time: 6.3150162697s, input_fired_time: 0.1353574693, output_fired_time: 0.0000000000, propagate_forward_time: 5.6665267944, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5131320953
validation[          , genome  7138] predictions:    9651/  10000 (96.51%), best:    9731/10000 (97.31%), error:      1142.30237, best error:       920.42090 on epoch:    18, epoch:   22/100, mu: 0.8314687014, learning_rate: 0.0040441677, weight_decay: 0.0001617667

epoch time: 78.3485717773s, input_fired_time: 1.0820031166, output_fired_time: 1.1058632135, propagate_forward_time: 29.1017227173, propagate_backward_time: 44.8679504395, weight_update_time: 0.0943195298, other_time: 2.1910324097
epoch time: 5.8749942780s, input_fired_time: 0.1424832642, output_fired_time: 0.0000000000, propagate_forward_time: 5.3275604248, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4049506187
validation[          , genome  7138] predictions:    9718/  10000 (97.18%), best:    9731/10000 (97.31%), error:       940.27496, best error:       920.42090 on epoch:    18, epoch:   23/100, mu: 0.8393952847, learning_rate: 0.0038419592, weight_decay: 0.0001536784

epoch time: 76.1260604858s, input_fired_time: 1.1667289734, output_fired_time: 1.0540686846, propagate_forward_time: 27.5397987366, propagate_backward_time: 44.1451797485, weight_update_time: 0.0892505348, other_time: 2.2202873230
epoch time: 7.5440349579s, input_fired_time: 0.1348373145, output_fired_time: 0.0000000000, propagate_forward_time: 7.0193023682, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3898954391
validation[          , genome  7138] predictions:    9613/  10000 (96.13%), best:    9731/10000 (97.31%), error:      1289.85645, best error:       920.42090 on epoch:    18, epoch:   24/100, mu: 0.8469254971, learning_rate: 0.0036498611, weight_decay: 0.0001459945

epoch time: 76.4880447388s, input_fired_time: 1.1427466869, output_fired_time: 1.0418658257, propagate_forward_time: 28.2547988892, propagate_backward_time: 43.7514762878, weight_update_time: 0.0912690163, other_time: 2.2971611023
epoch time: 6.2469701767s, input_fired_time: 0.1925437599, output_fired_time: 0.0000000000, propagate_forward_time: 5.6783871651, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3760390282
validation[          , genome  7138] predictions:    9718/  10000 (97.18%), best:    9731/10000 (97.31%), error:       930.73969, best error:       920.42090 on epoch:    18, epoch:   25/100, mu: 0.8540792465, learning_rate: 0.0034673680, weight_decay: 0.0001386948

epoch time: 77.3752288818s, input_fired_time: 1.1567441225, output_fired_time: 1.2909948826, propagate_forward_time: 29.2210922241, propagate_backward_time: 43.5009117126, weight_update_time: 0.0939245820, other_time: 2.2054862976
epoch time: 7.2707223892s, input_fired_time: 0.1352343261, output_fired_time: 0.0000000000, propagate_forward_time: 6.7288799286, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4066081047
validation[          , genome  7138] predictions:    9622/  10000 (96.22%), best:    9731/10000 (97.31%), error:      1217.71338, best error:       920.42090 on epoch:    18, epoch:   26/100, mu: 0.8608753085, learning_rate: 0.0032939995, weight_decay: 0.0001317600

epoch time: 76.1831359863s, input_fired_time: 1.2063989639, output_fired_time: 1.0366002321, propagate_forward_time: 27.9285240173, propagate_backward_time: 43.7146568298, weight_update_time: 0.1073843613, other_time: 2.2969589233
epoch time: 5.7397174835s, input_fired_time: 0.1261790842, output_fired_time: 0.0000000000, propagate_forward_time: 5.2053551674, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4081830978
validation[          , genome  7138] predictions:    9716/  10000 (97.16%), best:    9731/10000 (97.31%), error:       971.83923, best error:       920.42090 on epoch:    18, epoch:   27/100, mu: 0.8673315644, learning_rate: 0.0031292995, weight_decay: 0.0001251720

epoch time: 79.7123031616s, input_fired_time: 1.1476243734, output_fired_time: 1.0495686531, propagate_forward_time: 29.4297275543, propagate_backward_time: 45.6430320740, weight_update_time: 0.0958493724, other_time: 2.4423561096
epoch time: 6.1724805832s, input_fired_time: 0.1430667043, output_fired_time: 0.0000000000, propagate_forward_time: 5.6095948219, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4198188782
validation[          , genome  7138] predictions:    9719/  10000 (97.19%), best:    9731/10000 (97.31%), error:       937.48456, best error:       920.42090 on epoch:    18, epoch:   28/100, mu: 0.8734650016, learning_rate: 0.0029728345, weight_decay: 0.0001189134

epoch time: 80.1522674561s, input_fired_time: 1.2230372429, output_fired_time: 1.2207849026, propagate_forward_time: 29.0952854156, propagate_backward_time: 46.1576499939, weight_update_time: 0.1003290564, other_time: 2.4555091858
epoch time: 6.2682290077s, input_fired_time: 0.1359002143, output_fired_time: 0.0000000000, propagate_forward_time: 5.7271533012, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4051756859
validation[          , genome  7138] predictions:    9492/  10000 (94.92%), best:    9731/10000 (97.31%), error:      1588.19934, best error:       920.42090 on epoch:    18, epoch:   29/100, mu: 0.8792917728, learning_rate: 0.0028241926, weight_decay: 0.0001129677

epoch time: 80.7884979248s, input_fired_time: 1.2431399822, output_fired_time: 1.1895599365, propagate_forward_time: 29.4647922516, propagate_backward_time: 46.4027748108, weight_update_time: 0.0999425575, other_time: 2.4882316589
epoch time: 6.4917922020s, input_fired_time: 0.1428997964, output_fired_time: 0.0000000000, propagate_forward_time: 5.9165887833, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4323034286
validation[          , genome  7138] predictions:    9664/  10000 (96.64%), best:    9731/10000 (97.31%), error:      1107.29883, best error:       920.42090 on epoch:    18, epoch:   30/100, mu: 0.8848271966, learning_rate: 0.0026829829, weight_decay: 0.0001073193

epoch time: 79.3925323486s, input_fired_time: 1.2670781612, output_fired_time: 1.0826497078, propagate_forward_time: 29.3365001678, propagate_backward_time: 45.3740005493, weight_update_time: 0.0972757563, other_time: 2.3323059082
epoch time: 6.7146458626s, input_fired_time: 0.2129949927, output_fired_time: 0.0000000000, propagate_forward_time: 6.1017870903, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3998637199
validation[          , genome  7138] predictions:    9469/  10000 (94.69%), best:    9731/10000 (97.31%), error:      1735.49048, best error:       920.42090 on epoch:    18, epoch:   31/100, mu: 0.8900858164, learning_rate: 0.0025488338, weight_decay: 0.0001019534

epoch time: 79.8166351318s, input_fired_time: 1.2664294243, output_fired_time: 1.1058194637, propagate_forward_time: 29.1143341064, propagate_backward_time: 46.0051269531, weight_update_time: 0.1137740836, other_time: 2.3249282837
epoch time: 6.5233917236s, input_fired_time: 0.2457905114, output_fired_time: 0.0000000000, propagate_forward_time: 5.8596515656, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4179496765
validation[          , genome  7138] predictions:    9720/  10000 (97.20%), best:    9720/10000 (97.20%), error:       913.07281, best error:       913.07281 on epoch:    32, epoch:   32/100, mu: 0.8950815201, learning_rate: 0.0024213921, weight_decay: 0.0000968557

epoch time: 79.4992294312s, input_fired_time: 1.2607859373, output_fired_time: 1.0841971636, propagate_forward_time: 29.1703510284, propagate_backward_time: 45.4450569153, weight_update_time: 0.0980271846, other_time: 2.5388374329
epoch time: 6.6291246414s, input_fired_time: 0.1542912573, output_fired_time: 0.0000000000, propagate_forward_time: 6.0771183968, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3977150917
validation[          , genome  7138] predictions:    9717/  10000 (97.17%), best:    9720/10000 (97.20%), error:       967.99323, best error:       913.07281 on epoch:    32, epoch:   33/100, mu: 0.8998274207, learning_rate: 0.0023003225, weight_decay: 0.0000920129

epoch time: 80.6148376465s, input_fired_time: 1.1499757767, output_fired_time: 1.0723843575, propagate_forward_time: 29.6433544159, propagate_backward_time: 46.3509635925, weight_update_time: 0.0986565575, other_time: 2.3981513977
epoch time: 7.0924410820s, input_fired_time: 0.1471608728, output_fired_time: 0.0000000000, propagate_forward_time: 6.4705705643, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4747095108
validation[          , genome  7138] predictions:    9760/  10000 (97.60%), best:    9760/10000 (97.60%), error:       847.03644, best error:       847.03644 on epoch:    34, epoch:   34/100, mu: 0.9043360353, learning_rate: 0.0021853063, weight_decay: 0.0000874123

epoch time: 150.2534484863s, input_fired_time: 2.0320703983, output_fired_time: 2.1056158543, propagate_forward_time: 54.4511528015, propagate_backward_time: 87.5927505493, weight_update_time: 0.1698432714, other_time: 4.0718460083
epoch time: 6.7041621208s, input_fired_time: 0.1843522638, output_fired_time: 0.0000000000, propagate_forward_time: 6.1073584557, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4124512672
validation[          , genome  7138] predictions:    9654/  10000 (96.54%), best:    9760/10000 (97.60%), error:      1155.80017, best error:       847.03644 on epoch:    34, epoch:   35/100, mu: 0.9086192250, learning_rate: 0.0020760410, weight_decay: 0.0000830417

epoch time: 81.7011413574s, input_fired_time: 1.1686522961, output_fired_time: 1.0824204683, propagate_forward_time: 30.0473423004, propagate_backward_time: 46.9948959351, weight_update_time: 0.1096236259, other_time: 2.4078292847
epoch time: 6.3313035965s, input_fired_time: 0.1418631822, output_fired_time: 0.0000000000, propagate_forward_time: 5.7997360229, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3897042274
validation[          , genome  7138] predictions:    9590/  10000 (95.90%), best:    9760/10000 (97.60%), error:      1299.35815, best error:       847.03644 on epoch:    34, epoch:   36/100, mu: 0.9126882553, learning_rate: 0.0019722388, weight_decay: 0.0000788896

epoch time: 80.9151763916s, input_fired_time: 1.2103189230, output_fired_time: 1.1921874285, propagate_forward_time: 29.9329452515, propagate_backward_time: 46.1138954163, weight_update_time: 0.1155046746, other_time: 2.4658317566
epoch time: 6.0270662308s, input_fired_time: 0.1360417306, output_fired_time: 0.0000000000, propagate_forward_time: 5.4676237106, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4234008789
validation[          , genome  7138] predictions:    9728/  10000 (97.28%), best:    9760/10000 (97.60%), error:       887.32227, best error:       847.03644 on epoch:    34, epoch:   37/100, mu: 0.9165538549, learning_rate: 0.0018736268, weight_decay: 0.0000749451

epoch time: 79.4498977661s, input_fired_time: 1.2151882648, output_fired_time: 1.1258801222, propagate_forward_time: 29.1971073151, propagate_backward_time: 45.6781005859, weight_update_time: 0.0963567719, other_time: 2.2336273193
epoch time: 6.9345917702s, input_fired_time: 0.1486845165, output_fired_time: 0.0000000000, propagate_forward_time: 6.3003973961, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4855098724
validation[          , genome  7138] predictions:    9746/  10000 (97.46%), best:    9760/10000 (97.60%), error:       889.47443, best error:       847.03644 on epoch:    34, epoch:   38/100, mu: 0.9202261567, learning_rate: 0.0017799455, weight_decay: 0.0000711978

epoch time: 79.3105621338s, input_fired_time: 1.1539564133, output_fired_time: 1.1262692213, propagate_forward_time: 28.9033107758, propagate_backward_time: 45.7139358521, weight_update_time: 0.0985937417, other_time: 2.4130935669
epoch time: 6.2726206779s, input_fired_time: 0.1831282973, output_fired_time: 0.0000000000, propagate_forward_time: 5.6867256165, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4027667046
validation[          , genome  7138] predictions:    9586/  10000 (95.86%), best:    9760/10000 (97.60%), error:      1414.27283, best error:       847.03644 on epoch:    34, epoch:   39/100, mu: 0.9237148762, learning_rate: 0.0016909481, weight_decay: 0.0000676379

epoch time: 80.4060134888s, input_fired_time: 1.1892136335, output_fired_time: 1.1315234900, propagate_forward_time: 29.3346767426, propagate_backward_time: 46.4347305298, weight_update_time: 0.0950649977, other_time: 2.3158645630
epoch time: 6.0830502510s, input_fired_time: 0.1353493929, output_fired_time: 0.0000000000, propagate_forward_time: 5.5292534828, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4184474945
validation[          , genome  7138] predictions:    9671/  10000 (96.71%), best:    9760/10000 (97.60%), error:      1030.56628, best error:       847.03644 on epoch:    34, epoch:   40/100, mu: 0.9270291328, learning_rate: 0.0016064007, weight_decay: 0.0000642560

epoch time: 79.7126998901s, input_fired_time: 1.2308119535, output_fired_time: 1.1023163795, propagate_forward_time: 29.1267433167, propagate_backward_time: 45.6125144958, weight_update_time: 0.2053377628, other_time: 2.6403121948
epoch time: 6.3373904228s, input_fired_time: 0.1376051307, output_fired_time: 0.0000000000, propagate_forward_time: 5.7921724319, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4076128006
validation[          , genome  7138] predictions:    9745/  10000 (97.45%), best:    9745/10000 (97.45%), error:       846.61603, best error:       846.61603 on epoch:    41, epoch:   41/100, mu: 0.9301776886, learning_rate: 0.0015260807, weight_decay: 0.0000610432

arguments:
	'projects/csgrid.org_csg/exact_client_0.34_windows_x86_64.exe'
	'--training_file'
	'training_samples.bin'
	'--validation_file'
	'validation_samples.bin'
	'--testing_file'
	'testing_samples.bin'
	'--genome_file'
	'input_genome.txt'
	'--output_file'
	'output_genome.txt'
	'--checkpoint_file'
	'checkpoint.txt'
converting arguments to vector
boincified training filename: '../../projects/csgrid.org_csg/mnist_training_50k.bin'
boincified validation filename: '../../projects/csgrid.org_csg/mnist_validation_10k.bin'
boincified testing filename: '../../projects/csgrid.org_csg/mnist_testing_new.bin'
boincified genome filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_44_7138.txt'
boincified output filename: '../../projects/csgrid.org_csg/exact_genome_1512568488_44_7138_0_r218175072_0'
boincified checkpoint filename: 'checkpoint.txt'
loading genome
starting from checkpoint file: 'checkpoint.txt'
read CNN_Genome file with version string: 'v0.33'
read exact_id: -1
read genome_id: -1
read initial_mu: 0.5
read mu: 0.933169
read mu_delta: 0.95
read initial_learning_rate: 0.0125
read learning_rate: 0.00144978
read learning_rate_delta: 0.95
read initial_weight_decay: 0.0005
read weight_decay: 5.79911e-05
read weight_decay_delta: 0.95
read batch_size: 50
read epsilon: 1e-07
read alpha: 0.1
read input_dropout_probability: 0
read hidden_dropout_probability: 0
read velocity_reset: 1000
read epoch: 42
read max_epochs: 100
read reset_weights: 0
read padding: 2
read best_epoch: 41
read number_validation_images: 10000
read best_validation_predictions: 9745
read best_validation_error: 846.616
read number_training_images: 50000
read training_predictions: 0
read training_error: 1e+07
read number_test_images: 10000
read test_predictions: 0
read test_error: 1e+07
read generation_id: 7138
read normal distribution: '1 0.492945939302444 0.380815863609314'
generator_str: '575797582'
read generator: 575797582
read generated_by_map:
1 split_node 1
reading 22 nodes.
reading 52 edges.
number input nodes: 1
number softmax nodes: 10
order_size: 50000
parsed input file
loaded genome
loading images
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 4923 images.
reading image set with 5742 images.
reading image set with 4958 images.
reading image set with 5131 images.
reading image set with 4842 images.
reading image set with 4421 images.
reading image set with 4918 images.
reading image set with 5265 images.
reading image set with 4851 images.
reading image set with 4949 images.
image_size: 1x28x28 = 784
read 50000 images.
    class    0: 4923
    class    1: 5742
    class    2: 4958
    class    3: 5131
    class    4: 4842
    class    5: 4421
    class    6: 4918
    class    7: 5265
    class    8: 4851
    class    9: 4949
calculating averages and standard deviations for images
average pixel value for channel 0: 0.130530551075935
pixel variance for channel 0: 0.0948646068572998
pixel standard deviation for channel 0: 0.308000981807709
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
reading image set with 1000 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 1000
    class    1: 1000
    class    2: 1000
    class    3: 1000
    class    4: 1000
    class    5: 1000
    class    6: 1000
    class    7: 1000
    class    8: 1000
    class    9: 1000
number_classes: 10
channels: 1
width: 28
height: 28
reading image set with 980 images.
reading image set with 1135 images.
reading image set with 1032 images.
reading image set with 1010 images.
reading image set with 982 images.
reading image set with 892 images.
reading image set with 958 images.
reading image set with 1028 images.
reading image set with 974 images.
reading image set with 1009 images.
image_size: 1x28x28 = 784
read 10000 images.
    class    0: 980
    class    1: 1135
    class    2: 1032
    class    3: 1010
    class    4: 982
    class    5: 892
    class    6: 958
    class    7: 1028
    class    8: 974
    class    9: 1009
loaded images
starting backpropagation!
epoch time: 79.3749465942383s, input_fired_time: 1.14652752876282, output_fired_time: 1.06668603420258, propagate_forward_time: 29.2943859100342, propagate_backward_time: 45.4417533874512, weight_update_time: 0.104341357946396, other_time: 2.42558670043945
epoch time: 6.78739309310913s, input_fired_time: 0.193812608718872, output_fired_time: 0, propagate_forward_time: 6.14768648147583, propagate_backward_time: 0, weight_update_time: 0, other_time: 0.44589376449585
validation[          , genome  7138] predictions:    9552/  10000 (95.52%), best:    9745/10000 (97.45%), error:      1432.45691, best error:       846.61603 on epoch:    41, epoch:   42/100, mu: 0.9331688285, learning_rate: 0.0014497766, weight_decay: 0.0000579911

epoch time: 79.8846435547s, input_fired_time: 1.1700397730, output_fired_time: 1.1326818466, propagate_forward_time: 29.4947052002, propagate_backward_time: 45.6795806885, weight_update_time: 0.0971124470, other_time: 2.4076385498
epoch time: 6.7273912430s, input_fired_time: 0.1459564865, output_fired_time: 0.0000000000, propagate_forward_time: 6.1328682899, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4485664368
validation[          , genome  7138] predictions:    9622/  10000 (96.22%), best:    9745/10000 (97.45%), error:      1235.30237, best error:       846.61603 on epoch:    41, epoch:   43/100, mu: 0.9360103607, learning_rate: 0.0013772878, weight_decay: 0.0000550915

epoch time: 78.2773208618s, input_fired_time: 1.2243623734, output_fired_time: 1.1142488718, propagate_forward_time: 28.5736331940, propagate_backward_time: 45.0262527466, weight_update_time: 0.0948374793, other_time: 2.3388214111
epoch time: 6.4020886421s, input_fired_time: 0.1433946043, output_fired_time: 0.0000000000, propagate_forward_time: 5.8499321938, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4087619781
validation[          , genome  7138] predictions:    9660/  10000 (96.60%), best:    9745/10000 (97.45%), error:      1142.39038, best error:       846.61603 on epoch:    41, epoch:   44/100, mu: 0.9387098551, learning_rate: 0.0013084234, weight_decay: 0.0000523369

epoch time: 80.4790267944s, input_fired_time: 1.1353297234, output_fired_time: 1.0836318731, propagate_forward_time: 29.2828063965, propagate_backward_time: 46.7081756592, weight_update_time: 0.0929905996, other_time: 2.2690811157
epoch time: 6.7910499573s, input_fired_time: 0.1355390698, output_fired_time: 0.0000000000, propagate_forward_time: 6.2459464073, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4095644951
validation[          , genome  7138] predictions:    9677/  10000 (96.77%), best:    9745/10000 (97.45%), error:      1085.04797, best error:       846.61603 on epoch:    41, epoch:   45/100, mu: 0.9412743449, learning_rate: 0.0012430021, weight_decay: 0.0000497201

epoch time: 80.4836883545s, input_fired_time: 1.1689808369, output_fired_time: 1.0764791965, propagate_forward_time: 29.2656059265, propagate_backward_time: 46.7324752808, weight_update_time: 0.0963531211, other_time: 2.2401466370
epoch time: 6.2361011505s, input_fired_time: 0.1354941279, output_fired_time: 0.0000000000, propagate_forward_time: 5.7275557518, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3730511665
validation[          , genome  7138] predictions:    9679/  10000 (96.79%), best:    9745/10000 (97.45%), error:      1114.05188, best error:       846.61603 on epoch:    41, epoch:   46/100, mu: 0.9437106252, learning_rate: 0.0011808521, weight_decay: 0.0000472341

epoch time: 78.8932037354s, input_fired_time: 1.1244825125, output_fired_time: 1.2798695564, propagate_forward_time: 28.8574409485, propagate_backward_time: 45.3444442749, weight_update_time: 0.0959837139, other_time: 2.2869682312
epoch time: 6.4300980568s, input_fired_time: 0.1416153461, output_fired_time: 0.0000000000, propagate_forward_time: 5.8728747368, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4156079292
validation[          , genome  7138] predictions:    9663/  10000 (96.63%), best:    9745/10000 (97.45%), error:      1067.84888, best error:       846.61603 on epoch:    41, epoch:   47/100, mu: 0.9460250735, learning_rate: 0.0011218095, weight_decay: 0.0000448724

epoch time: 79.7418746948s, input_fired_time: 1.1821533442, output_fired_time: 1.0765239000, propagate_forward_time: 29.4278335571, propagate_backward_time: 45.8702354431, weight_update_time: 0.0935792327, other_time: 2.1851310730
epoch time: 6.4242491722s, input_fired_time: 0.1402512789, output_fired_time: 0.0000000000, propagate_forward_time: 5.8930459023, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3909521103
validation[          , genome  7138] predictions:    9705/  10000 (97.05%), best:    9745/10000 (97.45%), error:       960.43567, best error:       846.61603 on epoch:    41, epoch:   48/100, mu: 0.9482238293, learning_rate: 0.0010657189, weight_decay: 0.0000426288

epoch time: 79.3275146484s, input_fired_time: 1.1571975946, output_fired_time: 1.0418556929, propagate_forward_time: 29.0524597168, propagate_backward_time: 45.9093513489, weight_update_time: 0.1033402607, other_time: 2.1666526794
epoch time: 6.3813710213s, input_fired_time: 0.1430199891, output_fired_time: 0.0000000000, propagate_forward_time: 5.8228697777, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4154810905
validation[          , genome  7138] predictions:    9723/  10000 (97.23%), best:    9745/10000 (97.45%), error:       896.73016, best error:       846.61603 on epoch:    41, epoch:   49/100, mu: 0.9503126144, learning_rate: 0.0010124330, weight_decay: 0.0000404973

epoch time: 80.2598648071s, input_fired_time: 1.1414220333, output_fired_time: 1.0586789846, propagate_forward_time: 29.4364089966, propagate_backward_time: 46.4290733337, weight_update_time: 0.0938772336, other_time: 2.1942863464
epoch time: 6.1813902855s, input_fired_time: 0.1447912008, output_fired_time: 0.0000000000, propagate_forward_time: 5.6432738304, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3933253288
validation[          , genome  7138] predictions:    9616/  10000 (96.16%), best:    9745/10000 (97.45%), error:      1307.89172, best error:       846.61603 on epoch:    41, epoch:   50/100, mu: 0.9522969723, learning_rate: 0.0009618113, weight_decay: 0.0000384725

epoch time: 79.8583221436s, input_fired_time: 1.2478441000, output_fired_time: 1.0753484964, propagate_forward_time: 29.6939048767, propagate_backward_time: 45.4395484924, weight_update_time: 0.0937492028, other_time: 2.4016799927
epoch time: 6.3255739212s, input_fired_time: 0.1354258060, output_fired_time: 0.0000000000, propagate_forward_time: 5.8214507103, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3686976433
validation[          , genome  7138] predictions:    9673/  10000 (96.73%), best:    9745/10000 (97.45%), error:      1094.84619, best error:       846.61603 on epoch:    41, epoch:   51/100, mu: 0.9541821480, learning_rate: 0.0009137207, weight_decay: 0.0000365488

epoch time: 78.0247879028s, input_fired_time: 1.1005668640, output_fired_time: 1.0684789419, propagate_forward_time: 28.5713806152, propagate_backward_time: 45.1290702820, weight_update_time: 0.0951240882, other_time: 2.1552925110
epoch time: 6.1633315086s, input_fired_time: 0.1337127686, output_fired_time: 0.0000000000, propagate_forward_time: 5.6311593056, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3984594345
validation[          , genome  7138] predictions:    9602/  10000 (96.02%), best:    9745/10000 (97.45%), error:      1291.65466, best error:       846.61603 on epoch:    41, epoch:   52/100, mu: 0.9559730291, learning_rate: 0.0008680347, weight_decay: 0.0000347214

epoch time: 78.8053588867s, input_fired_time: 1.1363965273, output_fired_time: 1.0324083567, propagate_forward_time: 29.0167427063, propagate_backward_time: 45.3894233704, weight_update_time: 0.0909244865, other_time: 2.2303848267
epoch time: 6.2693705559s, input_fired_time: 0.1532259583, output_fired_time: 0.0000000000, propagate_forward_time: 5.7306404114, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3855042458
validation[          , genome  7138] predictions:    9674/  10000 (96.74%), best:    9745/10000 (97.45%), error:      1111.94434, best error:       846.61603 on epoch:    41, epoch:   53/100, mu: 0.9576743841, learning_rate: 0.0008246329, weight_decay: 0.0000329853

epoch time: 79.5720520020s, input_fired_time: 1.1347901821, output_fired_time: 1.0320190191, propagate_forward_time: 29.0275497437, propagate_backward_time: 46.2006187439, weight_update_time: 0.0950416401, other_time: 2.1770744324
epoch time: 6.3792238235s, input_fired_time: 0.3097483516, output_fired_time: 0.0000000000, propagate_forward_time: 5.6829385757, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3865370750
validation[          , genome  7138] predictions:    9747/  10000 (97.47%), best:    9747/10000 (97.47%), error:       830.94232, best error:       830.94232 on epoch:    54, epoch:   54/100, mu: 0.9592906833, learning_rate: 0.0007834013, weight_decay: 0.0000313361

epoch time: 79.8667221069s, input_fired_time: 1.1450178623, output_fired_time: 1.2718718052, propagate_forward_time: 29.3790721893, propagate_backward_time: 45.8678092957, weight_update_time: 0.0978760645, other_time: 2.2029457092
epoch time: 6.7650322914s, input_fired_time: 0.1353096664, output_fired_time: 0.0000000000, propagate_forward_time: 6.2162213326, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4135012627
validation[          , genome  7138] predictions:    9718/  10000 (97.18%), best:    9747/10000 (97.47%), error:       915.02924, best error:       830.94232 on epoch:    54, epoch:   55/100, mu: 0.9608261585, learning_rate: 0.0007442312, weight_decay: 0.0000297693

epoch time: 80.5273208618s, input_fired_time: 1.1287747622, output_fired_time: 1.0952054262, propagate_forward_time: 29.6723480225, propagate_backward_time: 46.4389686584, weight_update_time: 0.0915394947, other_time: 2.1920204163
epoch time: 6.5394210815s, input_fired_time: 0.1484216005, output_fired_time: 0.0000000000, propagate_forward_time: 5.9740099907, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4169893265
validation[          , genome  7138] predictions:    9257/  10000 (92.57%), best:    9747/10000 (97.47%), error:      2328.49585, best error:       830.94232 on epoch:    54, epoch:   56/100, mu: 0.9622848630, learning_rate: 0.0007070196, weight_decay: 0.0000282808

resetting weights to those on epoch: 54
epoch time: 80.4666671753s, input_fired_time: 1.1330848932, output_fired_time: 1.0369479656, propagate_forward_time: 30.1296501160, propagate_backward_time: 45.9196815491, weight_update_time: 0.0917006284, other_time: 2.2472991943
epoch time: 6.3868265152s, input_fired_time: 0.1352246255, output_fired_time: 0.0000000000, propagate_forward_time: 5.8828873634, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3687143326
validation[          , genome  7138] predictions:    9743/  10000 (97.43%), best:    9747/10000 (97.47%), error:       843.62286, best error:       830.94232 on epoch:    54, epoch:   57/100, mu: 0.9636706114, learning_rate: 0.0006716687, weight_decay: 0.0000268668

epoch time: 79.3527755737s, input_fired_time: 1.2059694529, output_fired_time: 1.1239457130, propagate_forward_time: 28.9360485077, propagate_backward_time: 45.7902107239, weight_update_time: 0.0913397968, other_time: 2.2965965271
epoch time: 6.2687540054s, input_fired_time: 0.1392110437, output_fired_time: 0.0000000000, propagate_forward_time: 5.7350440025, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3944988251
validation[          , genome  7138] predictions:    9655/  10000 (96.55%), best:    9747/10000 (97.47%), error:      1187.93042, best error:       830.94232 on epoch:    54, epoch:   58/100, mu: 0.9649870992, learning_rate: 0.0006380852, weight_decay: 0.0000255234

epoch time: 79.3083572388s, input_fired_time: 1.1098158360, output_fired_time: 1.1085618734, propagate_forward_time: 29.6768093109, propagate_backward_time: 45.2305908203, weight_update_time: 0.0943468809, other_time: 2.1825790405
epoch time: 6.5698823929s, input_fired_time: 0.1321638077, output_fired_time: 0.0000000000, propagate_forward_time: 5.8980450630, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.5396733284
validation[          , genome  7138] predictions:    9700/  10000 (97.00%), best:    9747/10000 (97.47%), error:       963.56879, best error:       830.94232 on epoch:    54, epoch:   59/100, mu: 0.9662377238, learning_rate: 0.0006061810, weight_decay: 0.0000242472

epoch time: 78.7772827148s, input_fired_time: 1.1105511189, output_fired_time: 1.1310861111, propagate_forward_time: 29.3818626404, propagate_backward_time: 45.0143394470, weight_update_time: 0.0918509439, other_time: 2.1394424438
epoch time: 6.2269129753s, input_fired_time: 0.1366151869, output_fired_time: 0.0000000000, propagate_forward_time: 5.5905289650, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4997687340
validation[          , genome  7138] predictions:    9740/  10000 (97.40%), best:    9740/10000 (97.40%), error:       822.70245, best error:       822.70245 on epoch:    60, epoch:   60/100, mu: 0.9674258232, learning_rate: 0.0005758719, weight_decay: 0.0000230349

epoch time: 79.4604644775s, input_fired_time: 1.1694276333, output_fired_time: 1.0605442524, propagate_forward_time: 29.1198501587, propagate_backward_time: 45.6842422485, weight_update_time: 0.1001595333, other_time: 2.4263992310
epoch time: 6.1243720055s, input_fired_time: 0.1388406008, output_fired_time: 0.0000000000, propagate_forward_time: 5.5678424835, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4176888466
validation[          , genome  7138] predictions:    9728/  10000 (97.28%), best:    9740/10000 (97.40%), error:       911.80609, best error:       822.70245 on epoch:    60, epoch:   61/100, mu: 0.9685545564, learning_rate: 0.0005470783, weight_decay: 0.0000218831

epoch time: 79.3239364624s, input_fired_time: 1.2856140137, output_fired_time: 1.0442856550, propagate_forward_time: 28.9680900574, propagate_backward_time: 45.8591690063, weight_update_time: 0.0978071094, other_time: 2.1667747498
epoch time: 6.1802430153s, input_fired_time: 0.1439913213, output_fired_time: 0.0000000000, propagate_forward_time: 5.6486058235, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3876457214
validation[          , genome  7138] predictions:    9748/  10000 (97.48%), best:    9740/10000 (97.40%), error:       849.19800, best error:       822.70245 on epoch:    60, epoch:   62/100, mu: 0.9696268439, learning_rate: 0.0005197244, weight_decay: 0.0000207890

epoch time: 79.9814605713s, input_fired_time: 1.2972561121, output_fired_time: 1.0247174501, propagate_forward_time: 29.5445804596, propagate_backward_time: 45.8942184448, weight_update_time: 0.0951949954, other_time: 2.2206878662
epoch time: 6.5165314674s, input_fired_time: 0.1434325874, output_fired_time: 0.0000000000, propagate_forward_time: 5.9631814957, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4099173546
validation[          , genome  7138] predictions:    9621/  10000 (96.21%), best:    9740/10000 (97.40%), error:      1195.32410, best error:       822.70245 on epoch:    60, epoch:   63/100, mu: 0.9706454873, learning_rate: 0.0004937382, weight_decay: 0.0000197495

epoch time: 79.0675277710s, input_fired_time: 1.1504832506, output_fired_time: 1.0603793859, propagate_forward_time: 29.1236038208, propagate_backward_time: 45.4636192322, weight_update_time: 0.0945711285, other_time: 2.2694435120
epoch time: 6.5702409744s, input_fired_time: 0.1384804398, output_fired_time: 0.0000000000, propagate_forward_time: 6.0655641556, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3661961555
validation[          , genome  7138] predictions:    9727/  10000 (97.27%), best:    9740/10000 (97.40%), error:       902.00171, best error:       822.70245 on epoch:    60, epoch:   64/100, mu: 0.9716132283, learning_rate: 0.0004690512, weight_decay: 0.0000187621

epoch time: 77.0448150635s, input_fired_time: 1.1151837111, output_fired_time: 1.0066357851, propagate_forward_time: 28.9992656708, propagate_backward_time: 43.8269233704, weight_update_time: 0.1047215015, other_time: 2.0968055725
epoch time: 6.5481433868s, input_fired_time: 0.1383894086, output_fired_time: 0.0000000000, propagate_forward_time: 6.0165524483, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3932013512
validation[          , genome  7138] predictions:    9710/  10000 (97.10%), best:    9740/10000 (97.40%), error:       937.42065, best error:       822.70245 on epoch:    60, epoch:   65/100, mu: 0.9725325704, learning_rate: 0.0004455987, weight_decay: 0.0000178240

epoch time: 76.7038574219s, input_fired_time: 1.0756287575, output_fired_time: 1.1403590441, propagate_forward_time: 27.9251098633, propagate_backward_time: 44.5187454224, weight_update_time: 0.0882313773, other_time: 2.0440139771
epoch time: 5.8210287094s, input_fired_time: 0.1350490749, output_fired_time: 0.0000000000, propagate_forward_time: 5.3251738548, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3608059883
validation[          , genome  7138] predictions:    9756/  10000 (97.56%), best:    9756/10000 (97.56%), error:       795.33893, best error:       795.33893 on epoch:    66, epoch:   66/100, mu: 0.9734059572, learning_rate: 0.0004233187, weight_decay: 0.0000169328

epoch time: 70.9373474121s, input_fired_time: 1.0628478527, output_fired_time: 0.9263110757, propagate_forward_time: 25.7606182098, propagate_backward_time: 41.2430343628, weight_update_time: 0.0846652240, other_time: 1.9445343018
epoch time: 6.2598738670s, input_fired_time: 0.1224725097, output_fired_time: 0.0000000000, propagate_forward_time: 5.7797780037, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3576235771
validation[          , genome  7138] predictions:    9601/  10000 (96.01%), best:    9756/10000 (97.56%), error:      1309.24561, best error:       795.33893 on epoch:    66, epoch:   67/100, mu: 0.9742356539, learning_rate: 0.0004021528, weight_decay: 0.0000160861

epoch time: 75.4410705566s, input_fired_time: 1.0882887840, output_fired_time: 0.9580251575, propagate_forward_time: 27.7089996338, propagate_backward_time: 43.5292167664, weight_update_time: 0.0865784511, other_time: 2.1565437317
epoch time: 5.3699107170s, input_fired_time: 0.1216955185, output_fired_time: 0.0000000000, propagate_forward_time: 4.8814086914, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3668065071
validation[          , genome  7138] predictions:    9618/  10000 (96.18%), best:    9756/10000 (97.56%), error:      1243.77087, best error:       795.33893 on epoch:    66, epoch:   68/100, mu: 0.9750238657, learning_rate: 0.0003820452, weight_decay: 0.0000152818

epoch time: 70.3625411987s, input_fired_time: 1.0432183743, output_fired_time: 0.9287100434, propagate_forward_time: 25.5046005249, propagate_backward_time: 40.8982086182, weight_update_time: 0.0833959132, other_time: 1.9878005981
epoch time: 6.1872930527s, input_fired_time: 0.1309819818, output_fired_time: 0.0000000000, propagate_forward_time: 5.6220917702, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4342193604
validation[          , genome  7138] predictions:    9725/  10000 (97.25%), best:    9756/10000 (97.56%), error:       952.16473, best error:       795.33893 on epoch:    66, epoch:   69/100, mu: 0.9757726789, learning_rate: 0.0003629429, weight_decay: 0.0000145177

epoch time: 71.0175094604s, input_fired_time: 1.0367137194, output_fired_time: 0.9353555441, propagate_forward_time: 26.1119365692, propagate_backward_time: 40.8947715759, weight_update_time: 0.0837915540, other_time: 2.0387306213
epoch time: 4.9968705177s, input_fired_time: 0.1124313176, output_fired_time: 0.0000000000, propagate_forward_time: 4.5002021790, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3842368126
validation[          , genome  7138] predictions:    9686/  10000 (96.86%), best:    9756/10000 (97.56%), error:      1061.89600, best error:       795.33893 on epoch:    66, epoch:   70/100, mu: 0.9764840603, learning_rate: 0.0003447958, weight_decay: 0.0000137918

epoch time: 76.9592056274s, input_fired_time: 1.1576006413, output_fired_time: 1.0483378172, propagate_forward_time: 29.0947418213, propagate_backward_time: 43.6268577576, weight_update_time: 0.0884822160, other_time: 2.0316658020
epoch time: 5.3818206787s, input_fired_time: 0.1559724957, output_fired_time: 0.0000000000, propagate_forward_time: 4.8764266968, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3494215012
validation[          , genome  7138] predictions:    9703/  10000 (97.03%), best:    9756/10000 (97.56%), error:       987.53699, best error:       795.33893 on epoch:    66, epoch:   71/100, mu: 0.9771598577, learning_rate: 0.0003275560, weight_decay: 0.0000131022

epoch time: 74.5298233032s, input_fired_time: 1.1263792515, output_fired_time: 0.9525909424, propagate_forward_time: 27.3741302490, propagate_backward_time: 43.0588722229, weight_update_time: 0.0860474110, other_time: 2.0178489685
epoch time: 5.5154614449s, input_fired_time: 0.1272353977, output_fired_time: 0.0000000000, propagate_forward_time: 5.0443782806, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3438477516
validation[          , genome  7138] predictions:    9729/  10000 (97.29%), best:    9756/10000 (97.56%), error:       924.47644, best error:       795.33893 on epoch:    66, epoch:   72/100, mu: 0.9778018594, learning_rate: 0.0003111782, weight_decay: 0.0000124471

epoch time: 67.7480545044s, input_fired_time: 1.0531815290, output_fired_time: 1.0167728662, propagate_forward_time: 24.2811603546, propagate_backward_time: 39.4285926819, weight_update_time: 0.0831217393, other_time: 1.9683494568
epoch time: 6.8583040237s, input_fired_time: 0.1361240000, output_fired_time: 0.0000000000, propagate_forward_time: 6.3500671387, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3721127510
validation[          , genome  7138] predictions:    9744/  10000 (97.44%), best:    9756/10000 (97.56%), error:       898.51581, best error:       795.33893 on epoch:    66, epoch:   73/100, mu: 0.9784117937, learning_rate: 0.0002956192, weight_decay: 0.0000118248

epoch time: 79.1476058960s, input_fired_time: 1.0751739740, output_fired_time: 1.0103625059, propagate_forward_time: 28.9503555298, propagate_backward_time: 45.9067649841, weight_update_time: 0.0905412361, other_time: 2.2049522400
epoch time: 5.9804582596s, input_fired_time: 0.1318033487, output_fired_time: 0.0000000000, propagate_forward_time: 5.4746370316, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3740177155
validation[          , genome  7138] predictions:    9633/  10000 (96.33%), best:    9756/10000 (97.56%), error:      1201.38635, best error:       795.33893 on epoch:    66, epoch:   74/100, mu: 0.9789912105, learning_rate: 0.0002808383, weight_decay: 0.0000112335

epoch time: 76.1697921753s, input_fired_time: 1.0565930605, output_fired_time: 1.1532045603, propagate_forward_time: 28.0932865143, propagate_backward_time: 43.8165702820, weight_update_time: 0.0912562683, other_time: 2.0501365662
epoch time: 5.7686381340s, input_fired_time: 0.1260450631, output_fired_time: 0.0000000000, propagate_forward_time: 5.1697516441, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4728412628
validation[          , genome  7138] predictions:    9751/  10000 (97.51%), best:    9756/10000 (97.56%), error:       856.08685, best error:       795.33893 on epoch:    66, epoch:   75/100, mu: 0.9795416594, learning_rate: 0.0002667964, weight_decay: 0.0000106719

epoch time: 78.6605758667s, input_fired_time: 1.1809502840, output_fired_time: 1.3051064014, propagate_forward_time: 29.1942806244, propagate_backward_time: 44.6865768433, weight_update_time: 0.0897697508, other_time: 2.2936553955
epoch time: 6.2693414688s, input_fired_time: 0.1320041716, output_fired_time: 0.0000000000, propagate_forward_time: 5.6678767204, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4694604874
validation[          , genome  7138] predictions:    9675/  10000 (96.75%), best:    9756/10000 (97.56%), error:      1108.13525, best error:       795.33893 on epoch:    66, epoch:   76/100, mu: 0.9800645709, learning_rate: 0.0002534565, weight_decay: 0.0000101383

epoch time: 71.0881958008s, input_fired_time: 1.2703362703, output_fired_time: 0.9450045228, propagate_forward_time: 25.5452632904, propagate_backward_time: 41.3767318726, weight_update_time: 0.0826061442, other_time: 1.9508514404
epoch time: 5.4373884201s, input_fired_time: 0.1175115481, output_fired_time: 0.0000000000, propagate_forward_time: 4.9823021889, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3375744820
validation[          , genome  7138] predictions:    9587/  10000 (95.87%), best:    9756/10000 (97.56%), error:      1374.61523, best error:       795.33893 on epoch:    66, epoch:   77/100, mu: 0.9805613160, learning_rate: 0.0002407837, weight_decay: 0.0000096314

epoch time: 76.3526306152s, input_fired_time: 1.1402306557, output_fired_time: 0.9935758114, propagate_forward_time: 28.3267478943, propagate_backward_time: 43.6796722412, weight_update_time: 0.0867690966, other_time: 2.2124061584
epoch time: 5.8353896141s, input_fired_time: 0.1310611665, output_fired_time: 0.0000000000, propagate_forward_time: 5.3289651871, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3753633499
validation[          , genome  7138] predictions:    9651/  10000 (96.51%), best:    9756/10000 (97.56%), error:      1188.41870, best error:       795.33893 on epoch:    66, epoch:   78/100, mu: 0.9810332656, learning_rate: 0.0002287445, weight_decay: 0.0000091498

epoch time: 73.9536514282s, input_fired_time: 1.0630050898, output_fired_time: 0.9497522116, propagate_forward_time: 26.9894409180, propagate_backward_time: 42.9634284973, weight_update_time: 0.0863851607, other_time: 1.9880256653
epoch time: 5.4724135399s, input_fired_time: 0.1200920418, output_fired_time: 0.0000000000, propagate_forward_time: 5.0005841255, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3517374992
validation[          , genome  7138] predictions:    9639/  10000 (96.39%), best:    9756/10000 (97.56%), error:      1211.16943, best error:       795.33893 on epoch:    66, epoch:   79/100, mu: 0.9814816117, learning_rate: 0.0002173073, weight_decay: 0.0000086923

epoch time: 79.0738677979s, input_fired_time: 1.0652067661, output_fired_time: 1.0008490086, propagate_forward_time: 29.0792598724, propagate_backward_time: 45.7185668945, weight_update_time: 0.0891626328, other_time: 2.2099838257
epoch time: 5.9594025612s, input_fired_time: 0.1306494772, output_fired_time: 0.0000000000, propagate_forward_time: 5.4637231827, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3650298119
validation[          , genome  7138] predictions:    9551/  10000 (95.51%), best:    9756/10000 (97.56%), error:      1464.56616, best error:       795.33893 on epoch:    66, epoch:   80/100, mu: 0.9819075465, learning_rate: 0.0002064419, weight_decay: 0.0000082577

epoch time: 76.5174713135s, input_fired_time: 1.0622617006, output_fired_time: 0.9825146198, propagate_forward_time: 27.9933624268, propagate_backward_time: 44.2415885925, weight_update_time: 0.2417124659, other_time: 2.2377433777
epoch time: 6.2236876488s, input_fired_time: 0.1274585277, output_fired_time: 0.0000000000, propagate_forward_time: 5.7382802963, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3579487801
validation[          , genome  7138] predictions:    9598/  10000 (95.98%), best:    9756/10000 (97.56%), error:      1294.49280, best error:       795.33893 on epoch:    66, epoch:   81/100, mu: 0.9823121428, learning_rate: 0.0001961198, weight_decay: 0.0000078448

epoch time: 70.7862091064s, input_fired_time: 1.0262238979, output_fired_time: 0.9197141528, propagate_forward_time: 25.8570785522, propagate_backward_time: 41.0419654846, weight_update_time: 0.0839807168, other_time: 1.9412269592
epoch time: 5.6006226540s, input_fired_time: 0.1192252189, output_fired_time: 0.0000000000, propagate_forward_time: 5.0040302277, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4773674011
validation[          , genome  7138] predictions:    9607/  10000 (96.07%), best:    9756/10000 (97.56%), error:      1332.17615, best error:       795.33893 on epoch:    66, epoch:   82/100, mu: 0.9826965332, learning_rate: 0.0001863138, weight_decay: 0.0000074526

epoch time: 74.7366104126s, input_fired_time: 1.0509929657, output_fired_time: 0.9480744004, propagate_forward_time: 26.8901290894, propagate_backward_time: 43.6056404114, weight_update_time: 0.0866751000, other_time: 2.2417716980
epoch time: 6.5871925354s, input_fired_time: 0.1320210546, output_fired_time: 0.0000000000, propagate_forward_time: 6.0575132370, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3976583481
validation[          , genome  7138] predictions:    9613/  10000 (96.13%), best:    9756/10000 (97.56%), error:      1294.91760, best error:       795.33893 on epoch:    66, epoch:   83/100, mu: 0.9830617309, learning_rate: 0.0001769981, weight_decay: 0.0000070799

epoch time: 77.5018768311s, input_fired_time: 1.1098475456, output_fired_time: 1.1415697336, propagate_forward_time: 28.6590709686, propagate_backward_time: 44.3922195435, weight_update_time: 0.0863679573, other_time: 2.1991653442
epoch time: 6.0647964478s, input_fired_time: 0.1287538558, output_fired_time: 0.0000000000, propagate_forward_time: 5.5766487122, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3593940735
validation[          , genome  7138] predictions:    9568/  10000 (95.68%), best:    9756/10000 (97.56%), error:      1455.47339, best error:       795.33893 on epoch:    66, epoch:   84/100, mu: 0.9834086299, learning_rate: 0.0001681482, weight_decay: 0.0000067259

epoch time: 74.9323120117s, input_fired_time: 1.0202336311, output_fired_time: 0.9553964138, propagate_forward_time: 27.4216194153, propagate_backward_time: 43.3853874207, weight_update_time: 0.1631262004, other_time: 2.1496734619
epoch time: 6.2878656387s, input_fired_time: 0.1245458499, output_fired_time: 0.0000000000, propagate_forward_time: 5.7686486244, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3946709633
validation[          , genome  7138] predictions:    9628/  10000 (96.28%), best:    9756/10000 (97.56%), error:      1227.71521, best error:       795.33893 on epoch:    66, epoch:   85/100, mu: 0.9837381840, learning_rate: 0.0001597408, weight_decay: 0.0000063896

epoch time: 79.6188812256s, input_fired_time: 1.1738328934, output_fired_time: 1.0507088900, propagate_forward_time: 29.2708721161, propagate_backward_time: 45.8059616089, weight_update_time: 0.0884517506, other_time: 2.3174972534
epoch time: 6.6846461296s, input_fired_time: 0.1369326115, output_fired_time: 0.0000000000, propagate_forward_time: 6.0862102509, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4615030289
validation[          , genome  7138] predictions:    9657/  10000 (96.57%), best:    9756/10000 (97.56%), error:      1156.51599, best error:       795.33893 on epoch:    66, epoch:   86/100, mu: 0.9840512872, learning_rate: 0.0001517538, weight_decay: 0.0000060702

epoch time: 74.0741500854s, input_fired_time: 1.1103488207, output_fired_time: 0.9956347346, propagate_forward_time: 27.1848545074, propagate_backward_time: 42.7247200012, weight_update_time: 0.0882005692, other_time: 2.0585899353
epoch time: 6.2602586746s, input_fired_time: 0.1334964633, output_fired_time: 0.0000000000, propagate_forward_time: 5.7458982468, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3808641434
validation[          , genome  7138] predictions:    9651/  10000 (96.51%), best:    9756/10000 (97.56%), error:      1161.40161, best error:       795.33893 on epoch:    66, epoch:   87/100, mu: 0.9843487144, learning_rate: 0.0001441661, weight_decay: 0.0000057666

epoch time: 75.1160812378s, input_fired_time: 1.0423101187, output_fired_time: 1.2699439526, propagate_forward_time: 27.7233028412, propagate_backward_time: 42.9474105835, weight_update_time: 0.1021226719, other_time: 2.1331100464
epoch time: 6.2522020340s, input_fired_time: 0.1260453910, output_fired_time: 0.0000000000, propagate_forward_time: 5.7092103958, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4169464111
validation[          , genome  7138] predictions:    9594/  10000 (95.94%), best:    9756/10000 (97.56%), error:      1362.56848, best error:       795.33893 on epoch:    66, epoch:   88/100, mu: 0.9846313000, learning_rate: 0.0001369578, weight_decay: 0.0000054783

epoch time: 75.1920547485s, input_fired_time: 1.0656144619, output_fired_time: 0.9588400722, propagate_forward_time: 27.5929985046, propagate_backward_time: 43.5307006836, weight_update_time: 0.0879536346, other_time: 2.0439033508
epoch time: 6.3927636147s, input_fired_time: 0.1323604435, output_fired_time: 0.0000000000, propagate_forward_time: 5.8757166862, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3846864700
validation[          , genome  7138] predictions:    9499/  10000 (94.99%), best:    9756/10000 (97.56%), error:      1660.33740, best error:       795.33893 on epoch:    66, epoch:   89/100, mu: 0.9848997593, learning_rate: 0.0001301099, weight_decay: 0.0000052044

epoch time: 70.3736267090s, input_fired_time: 1.0310364962, output_fired_time: 0.9288680553, propagate_forward_time: 25.8407840729, propagate_backward_time: 40.5208396912, weight_update_time: 0.0818947181, other_time: 2.0520973206
epoch time: 5.0585956573s, input_fired_time: 0.1176726669, output_fired_time: 0.0000000000, propagate_forward_time: 4.6081247330, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3327984810
validation[          , genome  7138] predictions:    9564/  10000 (95.64%), best:    9756/10000 (97.56%), error:      1402.56213, best error:       795.33893 on epoch:    66, epoch:   90/100, mu: 0.9851547480, learning_rate: 0.0001236044, weight_decay: 0.0000049442

epoch time: 76.5939407349s, input_fired_time: 1.1159405708, output_fired_time: 1.0392285585, propagate_forward_time: 28.1591033936, propagate_backward_time: 44.1934967041, weight_update_time: 0.1527259648, other_time: 2.0861663818
epoch time: 6.4040479660s, input_fired_time: 0.1360336244, output_fired_time: 0.0000000000, propagate_forward_time: 5.8909001350, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3771142960
validation[          , genome  7138] predictions:    9638/  10000 (96.38%), best:    9756/10000 (97.56%), error:      1203.42651, best error:       795.33893 on epoch:    66, epoch:   91/100, mu: 0.9853969812, learning_rate: 0.0001174242, weight_decay: 0.0000046970

epoch time: 76.3366165161s, input_fired_time: 1.0609396696, output_fired_time: 1.0227726698, propagate_forward_time: 28.9108467102, propagate_backward_time: 43.1542739868, weight_update_time: 0.0862272680, other_time: 2.1877861023
epoch time: 5.2401957512s, input_fired_time: 0.1314792782, output_fired_time: 0.0000000000, propagate_forward_time: 4.7526974678, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3560190201
validation[          , genome  7138] predictions:    9648/  10000 (96.48%), best:    9756/10000 (97.56%), error:      1209.17358, best error:       795.33893 on epoch:    66, epoch:   92/100, mu: 0.9856271148, learning_rate: 0.0001115530, weight_decay: 0.0000044621

epoch time: 79.8859710693s, input_fired_time: 1.2098510265, output_fired_time: 1.0066366196, propagate_forward_time: 29.2773036957, propagate_backward_time: 46.1933670044, weight_update_time: 0.0921231434, other_time: 2.1988067627
epoch time: 5.9754948616s, input_fired_time: 0.1325965375, output_fired_time: 0.0000000000, propagate_forward_time: 5.4049549103, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4379434586
validation[          , genome  7138] predictions:    9648/  10000 (96.48%), best:    9756/10000 (97.56%), error:      1182.96680, best error:       795.33893 on epoch:    66, epoch:   93/100, mu: 0.9858457446, learning_rate: 0.0001059753, weight_decay: 0.0000042390

epoch time: 77.1808166504s, input_fired_time: 1.1335725784, output_fired_time: 1.0280463696, propagate_forward_time: 28.1641521454, propagate_backward_time: 44.6646156311, weight_update_time: 0.1219879314, other_time: 2.1904258728
epoch time: 6.6116614342s, input_fired_time: 0.1326568127, output_fired_time: 0.0000000000, propagate_forward_time: 6.0912146568, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3877902031
validation[          , genome  7138] predictions:    9668/  10000 (96.68%), best:    9756/10000 (97.56%), error:      1097.90479, best error:       795.33893 on epoch:    66, epoch:   94/100, mu: 0.9860534668, learning_rate: 0.0001006766, weight_decay: 0.0000040271

epoch time: 78.0017166138s, input_fired_time: 1.1224436760, output_fired_time: 0.9976291656, propagate_forward_time: 28.7426967621, propagate_backward_time: 44.8057365417, weight_update_time: 0.0907211304, other_time: 2.3332099915
epoch time: 6.2184572220s, input_fired_time: 0.1327471882, output_fired_time: 0.0000000000, propagate_forward_time: 5.6862564087, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3994536400
validation[          , genome  7138] predictions:    9668/  10000 (96.68%), best:    9756/10000 (97.56%), error:      1109.53479, best error:       795.33893 on epoch:    66, epoch:   95/100, mu: 0.9862508178, learning_rate: 0.0000956427, weight_decay: 0.0000038257

epoch time: 79.1300888062s, input_fired_time: 1.0829935074, output_fired_time: 1.2355929613, propagate_forward_time: 28.4576168060, propagate_backward_time: 45.8718643188, weight_update_time: 0.0949822590, other_time: 2.4820175171
epoch time: 5.1633100510s, input_fired_time: 0.1180816665, output_fired_time: 0.0000000000, propagate_forward_time: 4.6689233780, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3763051033
validation[          , genome  7138] predictions:    9673/  10000 (96.73%), best:    9756/10000 (97.56%), error:      1107.25525, best error:       795.33893 on epoch:    66, epoch:   96/100, mu: 0.9864382744, learning_rate: 0.0000908606, weight_decay: 0.0000036344

epoch time: 75.7530517578s, input_fired_time: 1.0744872093, output_fired_time: 1.0002828836, propagate_forward_time: 27.5078926086, propagate_backward_time: 43.7978019714, weight_update_time: 0.0925759077, other_time: 2.3725891113
epoch time: 6.6545667648s, input_fired_time: 0.2540436983, output_fired_time: 0.0000000000, propagate_forward_time: 5.9139809608, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4865422249
validation[          , genome  7138] predictions:    9677/  10000 (96.77%), best:    9756/10000 (97.56%), error:      1060.37305, best error:       795.33893 on epoch:    66, epoch:   97/100, mu: 0.9866163731, learning_rate: 0.0000863176, weight_decay: 0.0000034527

epoch time: 75.5137176514s, input_fired_time: 1.0941776037, output_fired_time: 1.4544414282, propagate_forward_time: 27.4094696045, propagate_backward_time: 43.2979888916, weight_update_time: 0.0898305476, other_time: 2.2576370239
epoch time: 5.7837786674s, input_fired_time: 0.1193254963, output_fired_time: 0.0000000000, propagate_forward_time: 5.2971782684, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3672747612
validation[          , genome  7138] predictions:    9670/  10000 (96.70%), best:    9756/10000 (97.56%), error:      1091.12231, best error:       795.33893 on epoch:    66, epoch:   98/100, mu: 0.9867855310, learning_rate: 0.0000820017, weight_decay: 0.0000032801

epoch time: 77.7806091309s, input_fired_time: 1.0988347530, output_fired_time: 1.1083997488, propagate_forward_time: 28.8155040741, propagate_backward_time: 44.4390144348, weight_update_time: 0.0960897058, other_time: 2.3188591003
epoch time: 6.2987151146s, input_fired_time: 0.1278617829, output_fired_time: 0.0000000000, propagate_forward_time: 5.7855544090, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3852987289
validation[          , genome  7138] predictions:    9593/  10000 (95.93%), best:    9756/10000 (97.56%), error:      1335.61841, best error:       795.33893 on epoch:    66, epoch:   99/100, mu: 0.9869462252, learning_rate: 0.0000779016, weight_decay: 0.0000031161

epoch time: 74.5225982666s, input_fired_time: 1.0363110304, output_fired_time: 0.9856764078, propagate_forward_time: 27.0351314545, propagate_backward_time: 43.2651290894, weight_update_time: 0.0908526853, other_time: 2.2003479004
epoch time: 6.4559035301s, input_fired_time: 0.1328459680, output_fired_time: 0.0000000000, propagate_forward_time: 5.9342837334, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.3887739182
validation[          , genome  7138] predictions:    9647/  10000 (96.47%), best:    9756/10000 (97.56%), error:      1182.45093, best error:       795.33893 on epoch:    66, epoch:  100/100, mu: 0.9870989323, learning_rate: 0.0000740065, weight_decay: 0.0000029603

evaluating best weights on full training data.
evaluting training set with running mean/variance:
epoch time: 30.3961067200s, input_fired_time: 0.7149429917, output_fired_time: 0.0000000000, propagate_forward_time: 27.5942039490, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 2.0869598389
best training[          , genome  7138] predictions:   49418/  50000 (98.84%), best:    9756/10000 (97.56%), error:      2110.45459, best error:       795.33893 on epoch:    66, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
evaluating best weights on test data.
evaluting test set with running mean/variance:
epoch time: 6.7177534103s, input_fired_time: 0.1390414238, output_fired_time: 0.0000000000, propagate_forward_time: 6.1431064606, propagate_backward_time: 0.0000000000, weight_update_time: 0.0000000000, other_time: 0.4356055260
   testing[          , genome  7138] predictions:    9771/  10000 (97.71%), best:    9756/10000 (97.56%), error:       720.01038, best error:       795.33893 on epoch:    66, epoch:  101/100, mu: 0.9872440100, learning_rate: 0.0000703062, weight_decay: 0.0000028122
backpropagation finished successfully!
19:47:45 (7296): called boinc_finish(0)

</stderr_txt>
]]>