Searchable portal for the Everquest Archives

[Izmael]

Exporting the data would be great though.

Think of the future, 20, 40 years from now, when we might be gone or simply have lost interest. New generation of elves would be so grateful if that data was preserved.

If exporting is an option, I'll donate the resources to make it happen.

[Dolalin]

Agreed I would like to do it.

Let me see how it could be done and I'll return here with an update.

[Dolalin]

It looks like they've recently added a way to do this:

https://docs.microsoft.com/en-us/sam...restore-index/

[Dolalin]

I've open sourced the search UI, PRs with contributions and improvements are welcome.

https://github.com/dbsanfte/eqarchives-searchui

It's based off a Cognitive Search UI template from Azure, I added some custom UI tweaks to fit it better to the data and to fix some UI bugs.

[Izmael]

I think there's a misunderstanding.

I was under the impression that you built a mirror / archive of all those sites. If I understand correctly now, you have not - you made a searchable index of those sites, but the original content is actually mirrored nowhere.

Do I understand correctly?

[Dolalin]

Nope.

The captured content is all archived on Github here, mostly under /websites/:

https://github.com/dbsanfte/eq-archives

What I've done is synced it to an Azure storage account and run cognitive services search indexer against the contents, to create a searchable index, then published a web-frontend for that searchable index.

So all the file captures are already saved somewhere public, ie in that Github repo. Just the indexing (thus far accomplished) is locked up in Azure.

[Izmael]

Got it.

Using Github for that sounds like a pretty smart move.

I don't always agree with your EQ-related opinions but have to admit that you're doing God's work here.

Going to set up a periodic clone job so if Github is blown up by a nuclear warhead, our precious elf-info is preserved.

[Dolalin]

Originally Posted by Izmael [You must be logged in to view images. Log in or Register.]

Got it.

Using Github for that sounds like a pretty smart move.

I don't always agree with your EQ-related opinions but have to admit that you're doing God's work here.

Going to set up a periodic clone job so if Github is blown up by a nuclear warhead, our precious elf-info is preserved.

Funnily enough my laptop is currently chugging away converting it to use Git LFS because the repo has grown so big that GitHub will no longer let me push to it.

[Scalem]

ELI5 I dumb you use big words.

[Izmael]

FYI Github LFS has a 1 gigabyte limitation (overall, not per file), then you have to buy more storage.