Advanced search

Message boards : Science : Validation Process

Author Message
Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5271 - Posted: 25 May 2015, 3:15:45 UTC

Travis,

Could you please describe the validation process / methodology?

I imagine it is similar to Wildlife videos.

S.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 23
Images Observed: 774

              
Message 5277 - Posted: 25 May 2015, 13:48:56 UTC - in response to Message 5271.

Travis,

Could you please describe the validation process / methodology?

I imagine it is similar to Wildlife videos.

S.


Hi Steve,

it will be quite similar to the wildlife videos, but given the the fact that having people agree exactly on the tweets might not happen super frequently there'll be some tweaks to it. I'll make a post once we have everything up and running.

Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5283 - Posted: 26 May 2015, 4:54:43 UTC - in response to Message 5277.

One thing I appreciate is being able to check on the definitions as I'm classifying.

It will be even better once I can see how my classifications turned out. For example, I just classified a post as political because it mentioned a Welsh Labour party delegate making a speech (via hash tags). I'm not sure everyone would pick up on that so it will be interesting to see what happens.

The sooner we get to that discussion, the better I think.

Werinbert
Send message
Joined: 24 Aug 14
Posts: 17
Combined Credit: 1,000,279
DNA@Home: 300,319
SubsetSum@Home: 699,960
Wildlife@Home: 0
Wildlife@Home Watched: 25,839s
Wildlife@Home Events: 20
Climate Tweets: 6
Images Observed: 0

        
Message 5284 - Posted: 26 May 2015, 5:47:32 UTC - in response to Message 5283.

I agree with Steve Hawker*, it would help to find out if we are reviewing using the same mental criteria as other reviewers. I expect the first large batch to be all over the place, but getting better as we go along and figure out the CSG trends in tweet classification.

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 23
Images Observed: 774

              
Message 5289 - Posted: 26 May 2015, 11:27:59 UTC - in response to Message 5284.
Last modified: 26 May 2015, 11:28:51 UTC

I agree with Steve Hawker*, it would help to find out if we are reviewing using the same mental criteria as other reviewers. I expect the first large batch to be all over the place, but getting better as we go along and figure out the CSG trends in tweet classification.


Yes, this certainly is going to take some refinement. Especially since a lot of the criteria for classification are rather subjective.

I'm thinking the validation process will be similar to the old validation process for wildlife@home, when we had yes/no/unsure for different classifications.

So what I'm thinking now is a process something like this:

If 2+ classifications agree, then flag those as valid. For this, positive and extremely positive would match, along with negative and extremely negative. Don't need another set of eyes to look at the tweet in this case.

If all the classifications match, except one has unknown and the other doesn't then this is a partial match. This tweet will need to be looked at again (up to some max views, probably 4-5). Unknowns will get partial credit (maybe .75 of a tweet) if other viewers had an exact match without unknowns. If unknowns are the exact match then those get full credit.

Definitely open to discussion. Basically I want to encourage people to not use unknown if possible, but if it really is unknown to use it without penalty. Validation gets a bit tricky when there are so many possible options and "i don't know" is also a possibility.

Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5299 - Posted: 27 May 2015, 2:40:48 UTC - in response to Message 5289.

Definitely open to discussion. Basically I want to encourage people to not use unknown if possible, but if it really is unknown to use it without penalty. Validation gets a bit tricky when there are so many possible options and "i don't know" is also a possibility.


This could get problematic without a definition of "unknown" when "inconclusive" is also on the table. In the absence of specific examples, I'd argue that if an attitude is unknown, it is also inconclusive. Ergo you don't need to choose between unknown and inconclusive or "can't tell"

Would also help a bit if Extreme and Weather weren't so similar. I mean, extreme temperatures go under Weather and weather like tornados goes under Extreme.

Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5457 - Posted: 11 Jun 2015, 23:04:18 UTC - in response to Message 5299.

Travis,

Do you have some idea when validation will start?

Also, might be nice to include some stats like:

Total Tweets
Total Tweets Validated
Total Tweets Classified (aka Pending)

and stuff like that...

lindseymwingate
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 5 Mar 15
Posts: 20
Combined Credit: 0
DNA@Home: 0
SubsetSum@Home: 0
Wildlife@Home: 0
Wildlife@Home Watched: 0s
Wildlife@Home Events: 0
Climate Tweets: 35
Images Observed: 0

  
Message 5484 - Posted: 15 Jun 2015, 14:42:53 UTC - in response to Message 5457.

We are currently discussing a stats page :) Good suggestion.
____________

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 23
Images Observed: 774

              
Message 5485 - Posted: 15 Jun 2015, 16:57:24 UTC - in response to Message 5457.

Travis,

Do you have some idea when validation will start?

Also, might be nice to include some stats like:

Total Tweets
Total Tweets Validated
Total Tweets Classified (aka Pending)

and stuff like that...


Hi Steve,

Unfortunately not until I get back from my vacation. ETA first week in July.

--Travis

Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5498 - Posted: 16 Jun 2015, 15:05:47 UTC - in response to Message 5485.

Travis,

Do you have some idea when validation will start?

Also, might be nice to include some stats like:

Total Tweets
Total Tweets Validated
Total Tweets Classified (aka Pending)

and stuff like that...


Hi Steve,

Unfortunately not until I get back from my vacation. ETA first week in July.

--Travis


Didn't you just have a vacation in Iceland?

:)

Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5726 - Posted: 24 Jul 2015, 19:34:42 UTC - in response to Message 5485.

Travis,

Do you have some idea when validation will start?

Also, might be nice to include some stats like:

Total Tweets
Total Tweets Validated
Total Tweets Classified (aka Pending)

and stuff like that...


Hi Steve,

Unfortunately not until I get back from my vacation. ETA first week in July.

--Travis


Any update? Inquiring minds and all that...

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 23
Images Observed: 774

              
Message 5729 - Posted: 24 Jul 2015, 21:16:06 UTC - in response to Message 5726.

Travis,

Do you have some idea when validation will start?

Also, might be nice to include some stats like:

Total Tweets
Total Tweets Validated
Total Tweets Classified (aka Pending)

and stuff like that...


Hi Steve,

Unfortunately not until I get back from my vacation. ETA first week in July.

--Travis


Any update? Inquiring minds and all that...


Just made a news post. Should be testing tonight and hopefully have the tweet validator up and going by the end of the weekend. It's on the top of my to-do list!

Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5734 - Posted: 24 Jul 2015, 22:26:32 UTC - in response to Message 5729.

Travis,

Do you have some idea when validation will start?

Also, might be nice to include some stats like:

Total Tweets
Total Tweets Validated
Total Tweets Classified (aka Pending)

and stuff like that...


Hi Steve,

Unfortunately not until I get back from my vacation. ETA first week in July.

--Travis


Any update? Inquiring minds and all that...


Just made a news post. Should be testing tonight and hopefully have the tweet validator up and going by the end of the weekend. It's on the top of my to-do list!


Terrific!

Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5829 - Posted: 30 Aug 2015, 18:25:50 UTC - in response to Message 5729.

Any progress with the validator?

Travis Desell
Volunteer moderator
Project administrator
Project developer
Project scientist
Send message
Joined: 16 Jan 12
Posts: 1813
Combined Credit: 23,514,257
DNA@Home: 293,563
SubsetSum@Home: 349,212
Wildlife@Home: 22,871,482
Wildlife@Home Watched: 212,926s
Wildlife@Home Events: 51
Climate Tweets: 23
Images Observed: 774

              
Message 5830 - Posted: 31 Aug 2015, 9:12:25 UTC - in response to Message 5829.

Any progress with the validator?


Hi Steve,

We should have something going next week once I'm back from Munich and have time to meet with Lindsey about this. She's working on this as an undergraduate research project so I want to work on it with her so she get an idea of what's going on, in terms of validation, etc. It might slow things down slightly but it's a valuable learning experience for her, and let us get things done quicker in the long run.

Profile Skivelitis2
Avatar
Send message
Joined: 16 May 15
Posts: 60
Combined Credit: 11,066,329
DNA@Home: 19,068
SubsetSum@Home: 575,552
Wildlife@Home: 10,471,708
Wildlife@Home Watched: 9,158s
Wildlife@Home Events: 4
Climate Tweets: 391
Images Observed: 52

            
Message 5916 - Posted: 28 Sep 2015, 0:14:01 UTC - in response to Message 5484.

We are currently discussing a stats page :) Good suggestion.

Could a column be included indicating percentage of tweets valid? This may invoke a thought process rather than users speeding through to increase their total tweets and rolling the dice on validation.

Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5919 - Posted: 29 Sep 2015, 5:13:22 UTC - in response to Message 5916.

We are currently discussing a stats page :) Good suggestion.

Could a column be included indicating percentage of tweets valid? This may invoke a thought process rather than users speeding through to increase their total tweets and rolling the dice on validation.


We used to have a %accuracy on videos but having synch an indication is more a source of frustration than inspiration.

What is preferred is to give users better instructions on how to be accurate and better feedback when they are not.

In videos the instructions have been improved massively over time and you have the opportunity to review your classifications and change them if you want or need. You can see how other people classified the same video and you can learn from them too.

I fully expect this to be the case as Tweet Classifying moves forward.

But of course, first: a validation process...

Profile JumpinJohnny
Avatar
Send message
Joined: 24 Sep 13
Posts: 237
Combined Credit: 10,275,610
DNA@Home: 192,548
SubsetSum@Home: 201,740
Wildlife@Home: 9,881,323
Wildlife@Home Watched: 55,997,833s
Wildlife@Home Events: 15,584
Climate Tweets: 336
Images Observed: 351

              
Message 5970 - Posted: 4 Nov 2015, 21:44:18 UTC

OK ... So I did not expect to be "That Guy" who questions the tweet validation because it is so subjective. I figured not to worry about it at all and never discuss it, BUT...
I was waiting to see how it would work out before continuing on with classifying more tweets and had stopped at 1238. The top 5 "tweeties" are over 1000.
Only 2 of those have any tweets validated. 3 of us that classified over 1000 tweets have not a single validated tweet. The odds that this would reflect reality is not good at all.
Or am I missing something?
Does this mean that I can go through another 1238 of these things and still not get a single validation??? I thought I was being pretty reasonable in my assesment of the tweets. Can't imagine where I went wrong.

Profile Steve Hawker*
Send message
Joined: 8 Apr 13
Posts: 134
Combined Credit: 829,896
DNA@Home: 11,932
SubsetSum@Home: 299,708
Wildlife@Home: 518,257
Wildlife@Home Watched: 5,541,577s
Wildlife@Home Events: 2,169
Climate Tweets: 8,659
Images Observed: 55

              
Message 5971 - Posted: 4 Nov 2015, 23:14:25 UTC - in response to Message 5970.

OK ... So I did not expect to be "That Guy" who questions the tweet validation because it is so subjective. I figured not to worry about it at all and never discuss it, BUT...
I was waiting to see how it would work out before continuing on with classifying more tweets and had stopped at 1238. The top 5 "tweeties" are over 1000.
Only 2 of those have any tweets validated. 3 of us that classified over 1000 tweets have not a single validated tweet. The odds that this would reflect reality is not good at all.
Or am I missing something?
Does this mean that I can go through another 1238 of these things and still not get a single validation??? I thought I was being pretty reasonable in my assesment of the tweets. Can't imagine where I went wrong.


Hear, hear!

Exactly what he said.


Post to thread

Message boards : Science : Validation Process