Is it possible to download the wiki to have it locally?

Mattson · #1 02-25-2026, 06:58 PM

The p99 wiki is soooo laggy. I just got back to the game and it takes forever to load pages if they dont 502 first.

I know you can download the whole of wikipedia and its only 98Gb... I'm pretty sure the p99 website has to be a fraction of that. I browse it on my laptop when I'm out at the lake house where there is no internet.

So yeah... is there a link I can download or should I just get to scraping?

loramin · #2 02-25-2026, 07:53 PM

There is no link, because if it existed it would constantly have to be rebuilt every time anyone updates their Magelo (or anything else).

As for scraping, there might be a tool out there that can do it, but personally I'd just use AI to do it with custom code.

However, if you do it, one detail is critically important: only scrape at off hours (eg 1am PST) and "rate limit" your scraper to only fetch a page a minute. If you don't do that you're effectively attacking the wiki and preventing anyone else from using it ... and I wouldn't be surprised if Rogean has code to blacklist IPs that do that sort of thing.

Mortdecai99 · #3 02-25-2026, 08:47 PM

Quote:

Originally Posted by loramin [You must be logged in to view images. Log in or Register.]

If you don't do that you're effectively attacking the wiki and preventing anyone else from using it ... and I wouldn't be surprised if Rogean has code to blacklist IPs that do that sort of thing.

Isn't that exactly what's happening to the wiki now? Obviously there is no code to prevent it if so. [You must be logged in to view images. Log in or Register.]

loramin · #4 02-25-2026, 09:40 PM

Right, but whoever is spamming the wiki right now is probably not a player here; it's some loser hawking penis pills and other garbage.

Presumably you don't want to be like them because A) you don't want to ruin the wiki for others, and B) unlike the spammer, if Rogean does blacklist your IP you'll actually care (because you won't be able to play anymore).

To be clear, I have no idea what protective measures Rogean does or doesn't have in place ... but again, even if B) isn't a concern, I'd hope A) is.

Swish · #5 02-25-2026, 11:07 PM

Use the wayback machine if its being slow

http://web.archive.org

Hannibal · #6 02-27-2026, 03:50 PM

Quote:

Originally Posted by Swish [You must be logged in to view images. Log in or Register.]

Use the wayback machine if its being slow

http://web.archive.org

I have been here since 2012. I have used the wayback machine when asking for old forum post info etc. And yet, I never thought to use it for our own wiki when it takes a nosedive? Dammit... Thank you Swish!

Mattson · #7 Yesterday, 04:57 PM

Quote:

Originally Posted by loramin [You must be logged in to view images. Log in or Register.]

Right, but whoever is spamming the wiki right now is probably not a player here; it's some loser hawking penis pills and other garbage.

Presumably you don't want to be like them because A) you don't want to ruin the wiki for others, and B) unlike the spammer, if Rogean does blacklist your IP you'll actually care (because you won't be able to play anymore).

To be clear, I have no idea what protective measures Rogean does or doesn't have in place ... but again, even if B) isn't a concern, I'd hope A) is.

Every time someone with pigparse considers a mob it sends a request for the relevant p99 page.

Some people just spam the consider button. Just remember if you have pig parse and you hit the C button you're making it worse for everyone.

Its worse on raiding days because all those raiders are hitting consider and 99% of the time they're not even using the mob info window so they don't care that they're killing the wiki.

loramin · #8 Yesterday, 05:52 PM

Quote:

Originally Posted by Mattson [You must be logged in to view images. Log in or Register.]

Every time someone with pigparse considers a mob it sends a request for the relevant p99 page.

Some people just spam the consider button. Just remember if you have pig parse and you hit the C button you're making it worse for everyone.

Its worse on raiding days because all those raiders are hitting consider and 99% of the time they're not even using the mob info window so they don't care that they're killing the wiki.

I would imagine/hope the coder behind it has heard of caching ... in which case, you're mistaken.

EDIT: Oh, it's a tunnel quest site. If it literally has a "update from wiki" button then yes, hitting it hits the wiki ... but why would anyone spam that button? Wiki prices don't even change daily on most items, let alone hourly or sooner?

Regardless, there's a world of difference between hitting one page, even a few times ... and hitting every page in a short period. How is that not obvious to you?

Jimjam · #9 Yesterday, 06:15 PM

The beauty of a wiki is it is a product of player contributions and edits, so you can simply create your own p99 wiki to browse when offline.

bcbrown · #10 Yesterday, 07:39 PM

Quote:

Originally Posted by loramin [You must be logged in to view images. Log in or Register.]

I would imagine/hope the coder behind it has heard of caching ... in which case, you're mistaken.

EDIT: Oh, it's a tunnel quest site. If it literally has a "update from wiki" button then yes, hitting it hits the wiki ... but why would anyone spam that button? Wiki prices don't even change daily on most items, let alone hourly or sooner?

Regardless, there's a world of difference between hitting one page, even a few times ... and hitting every page in a short period. How is that not obvious to you?

Pretty sure it's you who's mistaken. I don't think you understand what PigParse is. It's not just a tunnelquest website, it's also a log-reading utility that does DPS parsing and mapping, and I guess loading all items the wiki lists as dropping from a mob every time you /con it.

Here's the ConHandler hitting the the WikiApi:
https://github.com/smasherprog/EqToo...Handler.cs#L46
Here's the WikiApi directly hitting the wiki:
https://github.com/smasherprog/EqToo...ikiApi.cs#L107

Sure don't see any caching happening.

If it's hitting the p99 wiki every time any player using pigparse /cons a mob, that's a lot more than "one page, even a few times". And sure, I'm aware of the difference between that and an non-rate-limited multithreaded crawler, but unless you have some special insight into wiki traffic from being an administrator, neither of us knows whether the wiki slowness is caused by crawlers or something else. In contrast, the issue pointed out by Mattson is a real source of a nontrivial amount of unnecessary traffic. So why bother responding dissmissively if you're not even going to put in the effort to understand what the issue is?

It seems like the maintainer is aware that this is... not a best practice, and I see some (unfinished) work today to begin to implement caching:
https://github.com/smasherprog/EqToo...0ae286a691d333