Repository navigation
Classify importance of Data (Characters, Houses,...) by Centrality measure #103
Description
Hi there,
as mentioned around and in other teams, it would be extremely useful to have a measure of importance of a character, house, and everything that can be scraped from the wiki and has a unique link.
This can be done by creating a graph of in- and outgoing links via https://www.npmjs.com/package/ngraph.centrality and then storing in separate values the in-, out-, degree and betweenness centralities. Just be careful: the centralities are graph dependent. Thus you need to think wether you want to have a mega-graph or separate graphs for characters, houses,... Both are correct, it's a matter of choice.
Also: think about the package handles multiple links from A -> B
Last but not least, some other teams have already been working on these things, so @Rostlab/js_cs_sose_2016_students please raise your voice. For now we have been wrongly advising this as page rank but that's just one flavor of centrality measure.
EDIT: Centrality is: https://en.wikipedia.org/wiki/Centrality
Activity
boriside commentedon Mar 20, 2016
Is this a call for help/collaboration with another team or it is an issue for Project_A?
kajo404 commentedon Mar 20, 2016
I did something . I took the data from Guy's wiki scraper and generated some jsons from it. I then matched this with the character list provided by and ordered by score.
The code can be found here . Its not great but it works. Maybe this helps someone...
sacdallago commentedon Mar 21, 2016
It's a call for collab if someone has done something about it. But I think I described quite extensively how you could implement this, regardless of other's people work. So :D you know... :D
AlexBeischl commentedon Mar 21, 2016
Well, we have just gathered some images for the characters, we created paths for and which are most important/popular. Those images can be found here:
https://github.com/Rostlab/JS16_ProjectC_Group10/tree/develop/mockup/img/persons
sacdallago commentedon Mar 21, 2016
@AlexBeischl that's still not really what I had in mind :) but again, a start:
@kordianbruck @adiolis can you assign someone except you two or @togiberlin to take care of this one? Assign as in, this issue is assigned.
kordianbruck commentedon Mar 22, 2016
Yea, I don't think this is in the project scope of A nor can be done till the 25. I vouch to move this issue to 'someday'
gyachdav commentedon Mar 22, 2016
I actually think it is within the scope of A, but since this requirement showed up late in the game we can defer it to the next version.
In the mean time, please integrate the data from here https://rostlab.org/~gyachdav/awoiaf/Data/pageRank/allchars.tgz
it's not perfect but at least gives a measure of which character is more referenced than others. I believe the range is [1-300](unknown to popular) . All you want to do here is, read the first item in the array from which you pick up the page_name to identify the character and assign that character the "score" value.
We really need this popularity measure in, to be able to sort the characters by the most important ones. otherwise we run into a case where we show bunch of negligible characters on the character portal.
42 remaining items
gyachdav commentedon Apr 2, 2016
okay guys let's solve this mess with the page rank
We currently have characters with page rank in the [0-1] range (normalized)
e.g.
https://api.got.show/api/characters/Gregor "pageRank":0.84
other characters have page rank in the [1-300] range
e.g.
https://api.got.show/api/characters/Qyburn "pageRank":161
This means that not all characters were updated with the normalized page rank.
@sacdallago says that this is because some characters are missing on @Hack3l maybe so.
What we need to do now is:
for each character in the characters collection
Once this procedure is done please discuss with @sacdallago how to migrate the data to the got.show mongo instance.
I need a volunteer to see this through from start to finish and hammer this procedure into the filler code so we wont have any mistakes the next time we refill the db.
Who will take care of this? @kordianbruck @adiolis @boriside @togiberlin
sacdallago commentedon Apr 2, 2016
Well, I guess returning to your old page rank is not much hassle. I just need to
npm run updatePageRanks --update=charactersI GUESS?!But having @Hack3l list extended to include also the characters that are missing now is more interesting to me!
Then someone would just need to replace the old file in the repo with the new one, I pull and run
npm run updatePageRanks --update=characters --file=data/normalizedPageRanks.jsonP.S.: See @kordianbruck , I am fast enough 💃
gyachdav commentedon Apr 2, 2016
will i have it done by the time I have my cereal tomorrow morning? :D
On Apr 2, 2016, at 5:20 PM, Christian Dallago notifications@github.com wrote:
sacdallago commentedon Apr 2, 2016
That sounds familiar! @Hack3l ?
sacdallago commentedon Apr 2, 2016
@gyachdav I am reverting the pageranks to the original ones.
@Hack3l please look at https://github.com/Rostlab/JS16_ProjectA/blob/master/data/normalizedPageRanks.json and https://github.com/Rostlab/JS16_ProjectA/blob/master/data/pageRanks.json and assign values to the ones present in the second file but missing in the first.
I counted and we are talking 2234 vs 2020, so a total of 214 missing!
Hack3l commentedon Apr 3, 2016
Now it contains all 2234 values
If u want to normalize values on updating the database it's just
(value - min)/(max - min)normalizedRanks.json.txt
sacdallago commentedon Apr 3, 2016
Thx guys, I'll do that soon
EDIT: done.
gyachdav commentedon Apr 3, 2016
hi @adiolis,
@sacdallago just refilled the database with the original pageranks (range [1-300])
What is the procedure he needs to run in order to half the page rank of characters with no image?
I remember you already implemented this just dont remember what is the command.
Thanks!
#103
gyachdav commentedon Apr 4, 2016
Please don't use the normalize values list. It is incomplete and using it messed up our data. All that is left to do here is just pull the changes @adiolis made in the code and then run the pageRank filler again.
sacdallago commentedon Apr 4, 2016
Oh.. I forgot? Mmhh well, I'll do it again just to be sure. You commented those three linea out @adiolis right?
EDIT Done ✔️