overview for slashzero

As of the time of this posting, lemmy.ml is throwing nginx 500 errors about 1 out of 10 page refreshes for me in the past hour. It is always fast to respond, so it isn't a timeout... in c/[email protected]

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago)

I’ve seen similar behavior (fail fast,) when there are outbound networking issues from any of the service containers. In one instance, lemmy backend was not able to make any outbound connections to any external services. It wild fail in milliseconds and show a timeout error in the logs.

One hypothesis here could be there are sporadic network issues on the lemmy.ml servers. 🤷‍♂️

Do your part, try not to lurk! in c/[email protected]

[–] [email protected] 0 points 1 year ago* (last edited 1 year ago) (2 children)

Aww man, no auto-play of gifs? OK...

Have an instance running, now what? in c/[email protected]

[–] [email protected] 1 points 1 year ago

This was the key that led to to the fix. Thank you!

Have an instance running, now what? in c/[email protected]

[–] [email protected] 1 points 1 year ago

Its federating to lemmy.ca now. : https://lemmy.ca/c/[email protected]

I tried search for https://lemmy.ca/c/[email protected] for giggles, and that also had the exact timeout error.

Have an instance running, now what? in c/[email protected]

[–] [email protected] 1 points 1 year ago

I have no idea what that means, either. I checked all the admin sections, and nothing pending anywhere on my end.

Have an instance running, now what? in c/[email protected]

[–] [email protected] 1 points 1 year ago

That's a great idea. I will try some other communities.

Have an instance running, now what? in c/[email protected]

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (2 children)

Yes, I have a few comments in the world of warcraft community on lemmy.ml. When I search for that using the shortcode: [email protected] , I get an immediate response with no results.

Server side, there is a timeout error in the log, but that is timing out in less than a second.

I wonder if instances need to be allowed to interact with lemmy.ml?

Have an instance running, now what? in c/[email protected]

[–] [email protected] 1 points 1 year ago

It's possible I have something misconfigured... but what I don't know!

Have an instance running, now what? in c/[email protected]

[–] [email protected] 2 points 1 year ago

Thank you for this. I will look through it now to see what I can learn.

Have an instance running, now what? in c/[email protected]

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (2 children)

[email protected]

Searched for that 👆 and checked the logs... timeout issue...

ERROR HTTP request{http.method=GET http.scheme="http" http.host=hakbox.social http.target=/api/v3/ws otel.kind="server" request_id=882fc7cd-bb3f-4df7-b6d4-a9bece907745 http.status_code=101 otel.status_code="OK"}: lemmy_server::api_routes_websocket: couldnt_find_object: Request error: error sending request for url (https://links.dartboard.social/.well-known/webfinger?resource=acct:[email protected]): operation timed out 0: lemmy_apub::fetcher::search::search_query_to_object_id at crates/apub/src/fetcher/search.rs:17 1: lemmy_apub::api::resolve_object::perform with self=ResolveObject { q: "[[email protected]](/c/[email protected])", auth: Some(Sensitive) } at crates/apub/src/api/resolve_object.rs:21 2: lemmy_server::root_span_builder::HTTP request with http.method=GET http.scheme="http" http.host=hakbox.social http.target=/api/v3/ws otel.kind="server" request_id=882fc7cd-bb3f-4df7-b6d4-a9bece907745 http.status_code=101 otel.status_code="OK" at src/root_span_builder.rs:16

Have an instance running, now what? in c/[email protected]

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (6 children)

Thanks! I've been searching for known communities I have posts in, but not getting any hits. Using the full shorthand, like: [[email protected]](/c/[email protected]) there are no results.

Have an instance running, now what? in c/[email protected]

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

No, it is not there.

9

Have an instance running, now what? (lemmy.ml)

submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]

38 comments fedilink

Well, I have an instance running. At startup it outputs federation enabled, host is redacted, but I don't see anything when looking at all. I can't search for other instance communities. I thought I would be able to use my local instance to browse other instance/communities and post there from my instance. Is that not how this works? Did I miss a step setting it up?

Some more details: Any search I do results in a timeout error. This is from the lemmy logs.

In all my years of software development and testing, I've never seen an HTTP status code 101... 101: switching protocols

UPDATE: it's fixed!

Thanks to @[email protected] who pointed me in the right direction, and thank you everyone who contributed to this post in the comments with suggestions and support!

Here are the steps:

add a new network interface

docker network create lemmyexternal

connect the lemmy container (lemmy_lemmy_1, unless you renamed it)

docker network connect NETWORK_ID CONTAINER_ID

(you can get the network and container IDs with docker network ls and docker container ls)

modify the docker-compose.yml to add the new network, and link it to the lemmy service

networks:
  # communication to web and clients
  lemmyexternalproxy:
  lemmyexternal:
  # communication between lemmy services
  lemmyinternal:
    driver: bridge
    internal: true

services:
  lemmy:
    image: dessalines/lemmy:0.17.4-rc.1
    hostname: lemmy
    networks:
      - lemmyinternal
      - lemmyexternal

save, and restart

docker-compose restart

8

Help setting up lemmy instance (lemmy.ml)

submitted 1 year ago* (last edited 1 year ago) by [email protected] to c/[email protected]

15 comments fedilink

Hi! I decided to experiment with my own instance. Went the docker route, and got everything working, except mail.

I setup postfix, and I can send mail to yahoo from the cli, however neither localhost:25 or actual.fqdm:25 work from the running lemmy_lemmy_1 container. The error is:

FQDM: email_send_failed: Connection error: Connection refused (os error 111)
localhost:25 email_send_failed: Connection error: Address not available (os error 99)

Has anyone seen this? Feels like a connectivity issue between the running container and host.

This is not letting me save an email address in my user settings.

I guess the intent is for people to use an external smtp service, rather than set up their own?

UPDATE:

tldr; I got mail working with postfix running locally.

I tried several "free" smtp relay services, and consistently got timeout errors.
Out of frustration, I re-enabled my local postfix server and reconfigured Lemmy back to that.
Tailing the postfix logs, I noticed it was rejecting the connection from the Lemmy instance with a relay not allowed error, which had an "internal IP of 192.x" (from the lemmyinternal network).
So, I added that IP /32 to "mynetworks" on the host in /etc/postfix/main.cf, restarted postfix, and my instance can now send out emails. Yay!

Thanks for the suggestions everyone. I'll leave this post here in case anyone else runs into this problem.