Advanced search

Message boards : News : [wildlife] new app: EXACT MNIST BATCH

Author Message
Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 6960 - Posted: 27 Apr 2017, 15:43:51 UTC

Well, that went quicker than I thought. The new application is released with new workunits (and they seem to be crunching without issue on my systems). This new version of the application changes the memory layout a bit which I hope will fix some of the issues we've seen with the "monster" workunits -- my hunch is that the larger workunits cause caching issues on some processors resulting in the 10x slower runtimes.

The new version of the application uses batch stochastic gradient descent, instead of stochastic gradient descent, which should keep memory accesses closer to each other and make caching better. It also has the benefit of requiring less epochs to train, so the new WUs should run a fair bit quicker in that regard as well.

I've updated the calculation for credit on these workunits which I believe should scale better to larger workunits. This means the deadlines should also be calculated more accurately. Let me know how the credit rate is (especially as the WUs get larger) and if the deadlines are okay so I can tweak things if need be.

Hoping this makes things run smoother! Let me know if you run into any issues.

Sergey Kovalchuk
Send message
Joined: 22 Jan 16
Posts: 4
Combined Credit: 748,848
DNA@Home: 14,685
SubsetSum@Home: 25,316
Wildlife@Home: 708,846
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 36

        
Message 6972 - Posted: 28 Apr 2017, 13:54:08 UTC

It seems that credit for new applications do not count to any subproject and do not affect any badge

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 6973 - Posted: 28 Apr 2017, 14:25:37 UTC - in response to Message 6972.

It seems that credit for new applications do not count to any subproject and do not affect any badge


Thanks! This should be fixed now.

Sergey Kovalchuk
Send message
Joined: 22 Jan 16
Posts: 4
Combined Credit: 748,848
DNA@Home: 14,685
SubsetSum@Home: 25,316
Wildlife@Home: 708,846
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 36

        
Message 6996 - Posted: 1 May 2017, 10:57:45 UTC - in response to Message 6973.

It seems that credit for new applications do not count to any subproject and do not affect any badge


Thanks! This should be fixed now.

Again, now for Batch 2.0

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 7001 - Posted: 1 May 2017, 22:13:20 UTC - in response to Message 6996.

It seems that credit for new applications do not count to any subproject and do not affect any badge


Thanks! This should be fixed now.

Again, now for Batch 2.0


Should be fixed now.

Profile Skivelitis2
Avatar
Send message
Joined: 16 May 15
Posts: 60
Combined Credit: 2,601,836
DNA@Home: 19,068
SubsetSum@Home: 575,552
Wildlife@Home: 2,007,215
Wildlife@Home Watched: 9,158s
Wildlife@Home Events: 4
Climate Tweets: 383
Images Observed: 52

            
Message 7003 - Posted: 1 May 2017, 23:36:11 UTC - in response to Message 7001.

It seems that credit for new applications do not count to any subproject and do not affect any badge


Thanks! This should be fixed now.

Again, now for Batch 2.0


Should be fixed now.

Will this somehow be retroactive? If not, don't sweat it, but my totals are as follows:

Total: 970597
DNA: 19068
SSS: 575552
Wildlife: 348971 (short 27006)

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 7004 - Posted: 2 May 2017, 0:04:09 UTC - in response to Message 7003.

It seems that credit for new applications do not count to any subproject and do not affect any badge


Thanks! This should be fixed now.

Again, now for Batch 2.0


Should be fixed now.

Will this somehow be retroactive? If not, don't sweat it, but my totals are as follows:

Total: 970597
DNA: 19068
SSS: 575552
Wildlife: 348971 (short 27006)


I should be able to fix that. Will try and get it done tonight.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 7005 - Posted: 2 May 2017, 3:04:11 UTC - in response to Message 7003.

It seems that credit for new applications do not count to any subproject and do not affect any badge


Thanks! This should be fixed now.

Again, now for Batch 2.0


Should be fixed now.

Will this somehow be retroactive? If not, don't sweat it, but my totals are as follows:

Total: 970597
DNA: 19068
SSS: 575552
Wildlife: 348971 (short 27006)


Should now be updated. Might take a little bit to propagate through the webpages.

Profile Skivelitis2
Avatar
Send message
Joined: 16 May 15
Posts: 60
Combined Credit: 2,601,836
DNA@Home: 19,068
SubsetSum@Home: 575,552
Wildlife@Home: 2,007,215
Wildlife@Home Watched: 9,158s
Wildlife@Home Events: 4
Climate Tweets: 383
Images Observed: 52

            
Message 7006 - Posted: 2 May 2017, 3:09:02 UTC

Fixed on my account page. Thanks Travis!

Peppernrino
Avatar
Send message
Joined: 20 Mar 17
Posts: 9
Combined Credit: 55,666,629
DNA@Home: 0
SubsetSum@Home: 0
Wildlife@Home: 55,666,629
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 48
Images Observed: 0

    
Message 7008 - Posted: 2 May 2017, 14:24:52 UTC - in response to Message 7006.

when i began citizen science grid, i was getting bsod's until i reduced amount of work units to 4 at a time. figured it was just RAM-heavy.

i started getting bsod's yesterday... i narrowed it down to citizen science grid. i think it might be the new work units. the error was: KMODE_EXCEPTION_NOT_HANDLED

Peppernrino
Avatar
Send message
Joined: 20 Mar 17
Posts: 9
Combined Credit: 55,666,629
DNA@Home: 0
SubsetSum@Home: 0
Wildlife@Home: 55,666,629
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 48
Images Observed: 0

    
Message 7009 - Posted: 2 May 2017, 16:28:55 UTC - in response to Message 7008.

sorry. for the record:

Windows 8.1 Pro WMC x64
AMD FX-8350 no overclock
16GB RAM @ 1866

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 7010 - Posted: 2 May 2017, 17:05:14 UTC - in response to Message 7009.

sorry. for the record:

Windows 8.1 Pro WMC x64
AMD FX-8350 no overclock
16GB RAM @ 1866


Just doing some googling it looks like you might have a corrupted driver:

http://windowsreport.com/kmode-exception-windows-10/

Peppernrino
Avatar
Send message
Joined: 20 Mar 17
Posts: 9
Combined Credit: 55,666,629
DNA@Home: 0
SubsetSum@Home: 0
Wildlife@Home: 55,666,629
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 48
Images Observed: 0

    
Message 7012 - Posted: 2 May 2017, 19:33:19 UTC - in response to Message 7010.

indeed, that was my first thought as well. i thought it was amicable numbers, so i reinstalled gpu driver and turned down the kernel mode, but the problem persisted. it's been good since i turned off csg. i could try to turn it on again if you like... full disclosure: i've heard of google before. :P

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 7013 - Posted: 2 May 2017, 20:21:18 UTC - in response to Message 7012.

indeed, that was my first thought as well. i thought it was amicable numbers, so i reinstalled gpu driver and turned down the kernel mode, but the problem persisted. it's been good since i turned off csg. i could try to turn it on again if you like... full disclosure: i've heard of google before. :P


lol. :) maybe I'm just used to dealing with students who forget they can google things.

I'll do some digging and see if it's something on my end. Does the BSOD only happen when you're over a certain number of workunits at a time?

Peppernrino
Avatar
Send message
Joined: 20 Mar 17
Posts: 9
Combined Credit: 55,666,629
DNA@Home: 0
SubsetSum@Home: 0
Wildlife@Home: 55,666,629
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 48
Images Observed: 0

    
Message 7018 - Posted: 2 May 2017, 21:13:03 UTC - in response to Message 7013.

no worries. i understand the g-check. :P

it seems to occur randomly-ish after about 10 minutes. might even just be when in tandem with amicable numbers, as they are pretty heavy on the gpu. like a mixture of both, maybe?

Peppernrino
Avatar
Send message
Joined: 20 Mar 17
Posts: 9
Combined Credit: 55,666,629
DNA@Home: 0
SubsetSum@Home: 0
Wildlife@Home: 55,666,629
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 48
Images Observed: 0

    
Message 7079 - Posted: 26 May 2017, 15:16:29 UTC - in response to Message 7018.

update: i installed even newer GPU drivers and while doing the settings, i ended up finding out that my monitor was dummy switched to shut off every 15 minutes... i'm thinking that Amicable Numbers wasn't able to start the GPU in such a high kernel mode while sleeping... but still only occured in tandem with CSG tasks. so weird. i will post this to their forums shortly as well.

i also increased the RAM and page/swap file availability to 95%, and turned BOINC to "Run always" instead of "Run based on preferences", even though my preferences are set to 100%.

the RAM usage is also reduced quite a bit on the Norm 0.30 tasks. i am now running 8 work units happily. feels very stable and smooth. :D

Peppernrino
Avatar
Send message
Joined: 20 Mar 17
Posts: 9
Combined Credit: 55,666,629
DNA@Home: 0
SubsetSum@Home: 0
Wildlife@Home: 55,666,629
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 48
Images Observed: 0

    
Message 7206 - Posted: 15 Jul 2017, 16:52:40 UTC

i went into the BIOS and disabled some dummy settings that i think crunchers should generally know, but aren't talked about in enough places. So here goes:

turn off c1e
turn off c6 state
turn off turbo
turn off cool n quiet
turn off apm

and there was another timing thing that i turned off, i'll figure it out after a restart and post here.

anyway, i've got everything back to full blast 100% and have had NO errors since.

Sergey Kovalchuk
Send message
Joined: 22 Jan 16
Posts: 4
Combined Credit: 748,848
DNA@Home: 14,685
SubsetSum@Home: 25,316
Wildlife@Home: 708,846
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 0
Images Observed: 36

        
Message 7239 - Posted: 25 Jul 2017, 8:43:14 UTC - in response to Message 7001.

It seems that credit for new applications do not count to any subproject and do not affect any badge

Thanks! This should be fixed now.

Again, now for Batch 2.0

Should be fixed now.

And again I'm with the same problem
Summation for subprojects is less than the total for exactly the number of credits for a new application

Is it possible to add a subproject to the template for new applications?

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1795
Combined Credit: 2,265,607
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 1,622,832
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 21
Images Observed: 710

              
Message 7242 - Posted: 25 Jul 2017, 18:38:37 UTC - in response to Message 7239.

It seems that credit for new applications do not count to any subproject and do not affect any badge

Thanks! This should be fixed now.

Again, now for Batch 2.0

Should be fixed now.

And again I'm with the same problem
Summation for subprojects is less than the total for exactly the number of credits for a new application

Is it possible to add a subproject to the template for new applications?


This should be fixed now.


Post to thread

Message boards : News : [wildlife] new app: EXACT MNIST BATCH