Lemmy

12511 readers

47 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to [email protected].

founded 4 years ago

MODERATORS

[email protected]

Is Lemmy search-engine unfriendly? (vlemmy.net)

submitted 1 year ago by [email protected] to c/[email protected]

26 comments fedilink hide all child comments

Any post and community could be accessed through a theoretically limitless amount of instances, which also means a theoretically limitless amount of URLs.

Will this hinder Lemmy from ever coming into the mainstream? If I type any topic in Google, I will get a reddit thread that deals with that. Can something like that ever happen for Lemmy?

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 16 points 1 year ago (9 children)

I'm doing tests in the next couple days. But I'm trying to build a search engine specifically for Lemmy.

It should in theory work similar-ish to Google / Bing.
You can filter by instance, community or author.
it only indexes Lemmy posts and it won't keep duplicates.
It'll also open any link you find in your instance.
You'll be able to self host it and point it to any instance you want as well.

I'm hoping I can open it to the public in a week or so.

[–] [email protected] 2 points 1 year ago (3 children)

Cool! How does it technically work? Does it fetch all titles (and maybe the body and comments) via the api from each instance or do you set up your own private instance and tap into the instance database?

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (2 children)

I'm using the public API to grab every post / comment and then I essentially replace the content with only the unique words. Then when you go to search it just looks for any post or comment, in my database, that has the words you typed in. Finally I sort based on the number of upvotes.

Right now it only craws a specific instance that you point it to. But as long as that instance is federated it /should/ get everything. But eventually I plan on using that instance's list of federated instances to scan everything and lighten the load on any one particular instance.

Edit: I thought about tapping into the existing database but the existing database is more geared towards serving content but not necessarily searching. The database that I'm building you can search but I drop so much of the original data that using it for content is worthless.

[–] [email protected] 1 points 1 year ago (1 children)

Now I'm curious what your stack is? Are you using an elastic database? Have you considered possibly using something like Azure Cognitive Search (hosted Elastic with AI/ML functions to add some NLP to your data/queries)? Bing uses it as part of their backend.

[–] [email protected] 3 points 1 year ago

HTML + JavaScript frontend. Rust backend with a postgres database.

It'll be open sourced once I can get the MVP ready.

load more comments (5 replies)