Resolved Export Thread feature doesn't work for large threads.

Status
Not open for further replies.

Ben

Bingo! Dino DNA.
Badministrator
Executive
Moderator
GWF Sponsor
Administrator
GW Elder
Messages
4,034
I'm dropping this here rather than keep bugging Crystal in PM, so that it can be tracked more easily.

The new Export thread feature that was enabled, does not seem to cooperate with very large threads. I think the cut-off is somewhere between 509 posts and 586. I have some examples below.

1693203360652.png

509 post thread - https://gwforums.com/threads/fq-wolf-vol-2-night-0-day-1.551/ (Export)
586 post thread - https://gwforums.com/threads/vts-mafia-night-2-day-2.640/ (Export)
779 post thread - https://gwforums.com/threads/gwf-wolf-zells-game-5-night-1-day-1.675/ (Export)
944 post thread - https://gwforums.com/threads/gwf-wolf-zells-version3-night-1-day-1.482/ (Export)

I'm not sure if this is a setting or a memory issue for the plugin, but I'm going to guess latter since the cut-off is above 500 posts.

Not urgent. đź‘Ť
 
Solution
Sit Down Hello GIF by Pudgy Penguins

Ben

Bingo! Dino DNA.
Badministrator
Executive
Moderator
GWF Sponsor
Administrator
GW Elder
Messages
4,034
Stop using AI to win wolf games, Ben.
I legit thought you were talking about Amaloo for a second...

You're right though that I was thinking about wolf. Not trying to use AI to win, but being able to plug data into some graphs and such. I was considering running one and thought nerds might dig a post-game breakdown.

You can keep pming me, I don't mind in the least, but I'm a bit unsure why it's not working. I will contact the developer ASAP :)
Thank you!
 

Ben

Bingo! Dino DNA.
Badministrator
Executive
Moderator
GWF Sponsor
Administrator
GW Elder
Messages
4,034
@Ben -- can you outline a bit more about what you're looking to pull from the export? To help myself and the Developer, Andy, better understand :)
What I'm after is the actual post data, timestamp, username, message, etc. Ideal format would be a CSV or something similar.

This add-on allows you to get a condensed thread view but it's still a webpage intended for printing to PDF, etc. I can try to be sneaky about it and pull the page source from that Export Thread page, which does contain all of the detail, I just then have to go through stripping out all of the HTML markup and manipulating it into something else, probably an XML file.

I'd imagine I can do that with a Python script or similar, but it's a lot of extra steps and of course doesn't work on large threads.

I think it's doable the way I'm thinking with this, but if there was a way to actually export threads that'd be far better.
 

Crystal

Formerly Apollo
Administrator
GW Elder
Messages
9,273
What I'm after is the actual post data, timestamp, username, message, etc. Ideal format would be a CSV or something similar.

This add-on allows you to get a condensed thread view but it's still a webpage intended for printing to PDF, etc. I can try to be sneaky about it and pull the page source from that Export Thread page, which does contain all of the detail, I just then have to go through stripping out all of the HTML markup and manipulating it into something else, probably an XML file.

I'd imagine I can do that with a Python script or similar, but it's a lot of extra steps and of course doesn't work on large threads.

I think it's doable the way I'm thinking with this, but if there was a way to actually export threads that'd be far better.
Thanks, I've pointed the developer here :)
 

Ben

Bingo! Dino DNA.
Badministrator
Executive
Moderator
GWF Sponsor
Administrator
GW Elder
Messages
4,034
OK so just to clarify again the main reason I posted the read was because I noticed it not working on the larger threads, not because I specifically expected the add-on to get changed for my needs. And by no means do I mean to imply that it's not a great add-on, it does what it's supposed to well and looks great.

As I said earlier I've tried to just grab the HTML source code from this add-ons output, thinking I might be able to condense it down to an XML format or something. So, things like the avatars being included in the add-on, and the default quote block being used, add just that much more HTML into the source code to where, for the stupid idea I have, it complicates it. I could potentially go through and wildcard all of the values in each block of code I'm trying to condense to just the data.

I did, while typing this, realize I can more easily just CTRL+A - COPY everything, and dump it into Notepad++ to start with, then format it to CSV from that. I've almost got that working, I just need to figure to figure out escaping characters in the post content... Anyway!

Unless the dev is a chill bro and wants to come up with a way to export TIMESTAMP; USERNAME; POST_NUMBER; POST in a CSV format in one click, I'm not looking to change anything about the current plugin.

Just looking to get it working on all threads.
 

Ben

Bingo! Dino DNA.
Badministrator
Executive
Moderator
GWF Sponsor
Administrator
GW Elder
Messages
4,034
I was able to get some usable output with the second (simpler) method I thought of.

The only issue with this is that the timestamp only shows the date, not the time, until you mouse-over.

1693288082059.png

If I had the full timestamp there without having to hover, I've more or less got the data pull part solved in some capacity.

The bigger issue is the add-on not working at all for large threads.

In so far as what those analytics are, I don't even really know yet, until I have the data and can see what my options are.

In regards to running a wolf game, I'd probably want to track posts made by each player at what time, isolate if it was a vote/action post, whether they changed vote, etc.

The what I'm trying to do with it isn't that important. I'm just trying to get those four fields as plain text in a CSV file.

Code:
TIMESTAMP;USERNAME;POST_NUMBER;POST
Jul 31, 2023;Fire Queen;1;RULES\n    DAY PHASES will last at least 24 hours.\n    during DAY PHASES, discuss amongst yourselves who you think the wolves are. cast your vote by putting the member name in bold in your post. you can post discussion as much as you like, just make sure your vote is clearly labeled whenever it happens. i will periodically tally votes and post updates throughout the day phase.\n    during each DAY PHASE, you may change your vote a total of ONCE. when changing your vote, please indicate who you are changing your vote from and please still bold the name of who you are changing your vote to\n    ACTIVITY RULES\: you will only be removed from the game if you fail to vote in either 3 consecutive day phases or 5 day phases total. you may vote to abstain, just make sure it is bolded like any other regular vote\n    at the end of each DAY PHASE, there will be an EXECUTION. whoever gets the most votes will be EXECUTED. in the event of a tie, the TYCOON will PM me to make the final decision. whoever kills the TYCOON gains this ability, unless the TYCOON is lynched, in which case ties will be resolved using the means in my previous game (extension of day phase, forced vote changes, and coin flip if it's still tied)\n    after the EXECUTION we will enter the NIGHT PHASE. the thread will go quiet as everyone lays down their heads to sleep for the night. the wolves will discuss who they want to target and one of them will send me a message. whenever that choice has been made (and whenever i am awake to reflect it), the next DAY PHASE will start with the discovery of the next body. any other role with a night ability will message me as well should they wish to use their ability, otherwise they will indicate that they are skipping.\n    REGARDING ROLE CLAIMING\: do not share correspondence with me with any other player. do not explicitly claim a role (or deny that a role is yours by name). accept that vaguely claiming a role puts a target on your back. be sneaky about it.
Jul 31, 2023;Fire Queen;2;numbers have been assigned, roles have been sent out, i'll see you all here at noon eastern time tomorrow. we'll see who survives this first night...\nEDIT\: have made a quick and minor change with regards to the Polycule, please consult the first post\nEDIT2\: i woke up a little late today so i'm allowing a couple extra hours for people to get their night actions in since i was a little late in answering some questions
Aug 1, 2023;Fire Queen;3;Dawn shines upon the colonial town of Good Wisdom, and the town loon has been raving about wolves for weeks. "Mark my words," he said, "when next the full moon glimmers in the night sky, death will come for our town. Death!!!" And then he left and the property values immediately skyrocketed, bringing in a host of new settlers with secret professions and lycanthropic tendencies.\nThe morning after that full moon, however, there was no blood to be found. Surely the loon must have been wrong, then? No, something different happened. And for the people of Good Wisdom, it was now time to lob accusations at each other with absolutely nothing to go on, which was a good thing because it just so happened to be their favorite pastime.\nNIGHT 0 HAS ENDED. DAY 1 HAS BEGUN.\nhere is everyone's numbers:\nScreenshot 2023-07-31 221834.png\nyou may play as normal now. one other thing since i know i'm going to get questions about it: there are three possible scenarios here.\n    The Wolves and Doctor targeted the same person, who will have scars they can voluntarily show.\n    The Wolves targeted the Somnambulist and therefore failed to register a kill.\n    The Wolves chose not to kill anyone during the first night phase.\nEDIT TO CLARIFY\: if you have scars proving you were saved by the Doctor, you would have received a message from me confirming that. should anyone claim to have scars, i will post to verify whether they are authentic.\nanyway have fun gang, this phase will end at approximately 1 PM EDT on Wednesday
Aug 1, 2023;VashTheStampede;4;Hi
Aug 1, 2023;TD;5;Episode 4 Hello GIF by The Simpsons
Aug 1, 2023;VashTheStampede;6;No role claiming
Aug 1, 2023;TD;7;Sooooo.\nAnyone got some scars?
Aug 1, 2023;VashTheStampede;8;I have a crescent moon shaped scar on my forehead.\nBetween that and the glasses and the brown bowl cut hair growing up, I really wanted to sue Rowling for the slight switch to a lightning bolt.
Aug 1, 2023;TD;9;VashTheStampede said:\nNo role claiming\nI'm tempted to vote you for overthinking, but APPARENTLY I'd only be able to switch it once after that.
Aug 1, 2023;canadaguy;10;VashTheStampede said:\nNo role claiming\nI think the rule doesn't prevent role ACCUSATIONS
Aug 1, 2023;shortkut;11;Voting @Ants!
Aug 1, 2023;Olga;12;I love the polycule :3
Aug 1, 2023;canadaguy;13;Olga said:\nI love the polycule :3\nMy brain reads it as polycube, and I picture some sort of 4+ dimensional geometry
Aug 1, 2023;TD;14;Oh okay.\nSo we are jumping right into voting - I'm in.\nVote Tubby\nbecause only monsters put pineapple on pizza.
Aug 1, 2023;VashTheStampede;15;canadaguy said:\nMy brain reads it as polycube, and I picture some sort of 4+ dimensional geometry\nThe Tesseract of Love
Aug 1, 2023;Local Hero;16;canadaguy said:\nMy brain reads it as polycube, and I picture some sort of 4+ dimensional geometry\nThe Tesseract of Love
Aug 1, 2023;shortkut;17;Local Hero said:\nMe too 🥰🥰🥰🥰\nMe three! However my brain keeps reading it molecule
Aug 1, 2023;VashTheStampede;18;shortkut said:\nMe three! However my brain keeps reading it molecule\nThe chemical bond of love
Aug 1, 2023;Zell 17;19;Not killing anyone as a wolf is such a Coleman move I'm stunned to read the list

(I need to work out a solution for quote blocks still.)
 

Mark

Dumbass Progenitor
Administrator
GW Elder
Messages
6,219
I've passed this on :)

I didn’t touch this thread at first because you had it handled, but after reading more… I think I can simplify what Ben’s saying. Remember when vB first introduced the “archive” feature and it allowed us to strip down threadbit and postbit data into what was essentially a clean Wordpad file? Hyperlinks were converted to full URL’s, images and ads were stripped out, and the only thing that remained were basic tables to separate content and basic formatting for ease of review or RSS feeding later on in its lifespan. That essentially sounds like what he’s looking for… but it doesn’t appear XenForo has that option for default since those kinda features have become useless with all of the integration elsewhere.
 

Crystal

Formerly Apollo
Administrator
GW Elder
Messages
9,273
I didn’t touch this thread at first because you had it handled, but after reading more… I think I can simplify what Ben’s saying. Remember when vB first introduced the “archive” feature and it allowed us to strip down threadbit and postbit data into what was essentially a clean Wordpad file? Hyperlinks were converted to full URL’s, images and ads were stripped out, and the only thing that remained were basic tables to separate content and basic formatting for ease of review or RSS feeding later on in its lifespan. That essentially sounds like what he’s looking for… but it doesn’t appear XenForo has that option for default since those kinda features have become useless with all of the integration elsewhere.
I'll pass this on, too :)

Yeah, the archive is essentially what the export thread add-on is for, it just doesn't seem to work above 500 posts.
 
  • Like
Reactions: Ben

Ben

Bingo! Dino DNA.
Badministrator
Executive
Moderator
GWF Sponsor
Administrator
GW Elder
Messages
4,034
I didn’t touch this thread at first because you had it handled, but after reading more… I think I can simplify what Ben’s saying. Remember when vB first introduced the “archive” feature and it allowed us to strip down threadbit and postbit data into what was essentially a clean Wordpad file? Hyperlinks were converted to full URL’s, images and ads were stripped out, and the only thing that remained were basic tables to separate content and basic formatting for ease of review or RSS feeding later on in its lifespan. That essentially sounds like what he’s looking for… but it doesn’t appear XenForo has that option for default since those kinda features have become useless with all of the integration elsewhere.
Yeah, this is accurate. I don't recall any vb feature like this, but my hope was for this kind of bare bones output. This add-on still dresses the page up to look nice for printing. I really just want the text and little else.

But I did devise a process to rip that plain text out.

The only thing I'd ask to *change* with this add-on is having the full timestamp displayed, not just "Aug 29, 2023", etc. If I could get that as "Aug 29, 2023 11:25:43 AM" (maybe even without the second) I'd be set.

@Ben Would it be simpler to show the data you want by exporting a smaller thread and then showing what it looks like when you try a larger thread?
In a post above, I gave a sample of a smaller number of posts I converted into the CSV format I need. That's essentially the output I'm looking for, regardless of thread size.

In the first post, I did include some examples of threads where the export works and those that don't. For those that don't, it's just a blank page.

I'll pass this on, too :)

Yeah, the archive is essentially what the export thread add-on is for, it just doesn't seem to work above 500 posts.
This is still the primary thing I'm hoping can be solved. Scraping the data out somehow is workable, but if it doesn't work on large threads then I'm stuck.

Though, I'm not sure whether it could handle more posts if it were super bare bones. 🤷🏻‍♂️
 

Crystal

Formerly Apollo
Administrator
GW Elder
Messages
9,273
@Ben -- I just found what we might be looking for. Can you select the style "XenForo Lite" from the list (in the footer you'll see a paint brush that says GWForums, click it) and then see if that might be what we're looking for here?
 

Mark

Dumbass Progenitor
Administrator
GW Elder
Messages
6,219
Yeah, this is accurate. I don't recall any vb feature like this, but my hope was for this kind of bare bones output. This add-on still dresses the page up to look nice for printing. I really just want the text and little else.

It wasn’t so much of a feature, so that doesn’t surprise me, we never utilized it on any of our forums we worked on outside of GW, and GW never really had a use for it because of its scale. If I recall, it was even more stripped down than the “show printable version” feature, which existed going back as far as vB2.x something or another.
 

Ben

Bingo! Dino DNA.
Badministrator
Executive
Moderator
GWF Sponsor
Administrator
GW Elder
Messages
4,034
@Ben -- I just found what we might be looking for. Can you select the style "XenForo Lite" from the list (in the footer you'll see a paint brush that says GWForums, click it) and then see if that might be what we're looking for here?
Finally checked this. Unfortunately that style doesn't help much more than the current Export Thread feature, as there's still at ton of HTML markup in the source.

I did notice that the timestamps change based on the age of the posts, though. If it's today/yesterday, they do include the time portion but not the month/day/year. Likewise if it's past yesterday, it'll show the month/day/year but not the time portion.

I'm thinking this might be a forum-wide setting for the date format, so if updating it would cause more complications I'd understand, but if you can force it to show the full date every time that'd be sweet.

1693358183933.png


1693358673561.png
1693358698797.png
 

Mark

Dumbass Progenitor
Administrator
GW Elder
Messages
6,219
Finally checked this. Unfortunately that style doesn't help much more than the current Export Thread feature, as there's still at ton of HTML markup in the source.

I did notice that the timestamps change based on the age of the posts, though. If it's today/yesterday, they do include the time portion but not the month/day/year. Likewise if it's past yesterday, it'll show the month/day/year but not the time portion.

I'm thinking this might be a forum-wide setting for the date format, so if updating it would cause more complications I'd understand, but if you can force it to show the full date every time that'd be sweet.

View attachment 2598


View attachment 2599
View attachment 2600

I’m not seeing anything about a setting for changing the date format forum-wide in the panel, and that’s going as far as broadening the search for “date” and going through all the results. I’m sure it’s a template edit somewhere, though…
 
I legit thought you were talking about Amaloo for a second...

You're right though that I was thinking about wolf. Not trying to use AI to win, but being able to plug data into some graphs and such. I was considering running one and thought nerds might dig a post-game breakdown.


Thank you!
I am a nerd that wants this.
 

Mark

Dumbass Progenitor
Administrator
GW Elder
Messages
6,219
We're going to need to create a script that will query the database and pull the relevant data from a specific thread id.

By “we’re” she means “her” and by “create a script” she means “go THROUGH it figuring it out like a fucking pro”.
 
Status
Not open for further replies.
Back
Top Bottom