/coomer/ - coomer.party

Nope, It stopped working yesterday

They changed the API for the posts, so they will need to update the code to grab the media. The new API sucks balls, because now you cant get the server for the files directly from the posts API, you have to go into each post individually, which makes things 50 x slower to scrape. As a positive since the move to coomer.st my download speeds have increased quite significantly. Its not abnormal to get hours @ 1-4mbps, whereas previously i was struggling to get much over 200-500 kbps (average constant download speed, including waiting for reconnections since the server still gives lots of errors). But that could just be a product of all of these other downloaders being offline freeing up the bandwidth. I just updated my own scraper and with the change to the API and the rate limits for scraping the API, means a large creator can take like 30 mins to just scrape all of the media urls. Im sure theres ways to beat the rate limit and just async it to death, but it's an unfortunate change. Alternative is to just test every server at download time until you get the right one, that would leave scraping times as per normal. Not sure why they made this change, its going to mean theres way more API hits than before which cant be a good thing.

>>6441
My ghetto custom scraper ignores the API and goes post by post anyway, I figured it was a safer way to not trigger any bot alarms even though its slower. I'm having major issues with speeds/disconnects for most profiles and now revisions aren't loading either.

Version 1.30.4 is working now. Woot!

>>6441
>The new API sucks balls, because now you cant get the server for the files directly from the posts API, you have to go into each post individually, which makes things 50 x slower to scrape.
This is not true, but it's not so obvious from first glance. You should be able to see that a given page for an artist does a fetch on a file called "posts". That will contain the json content you're looking for. But you need to ensure that the request for the posts file has all the right headers if doing it programmatically to avoid a 403. For simplicity of satisfying the requirements I recommend automating a webview that will navigate to each page and intercept the relevant "response received" event for the posts file.

the posts API only returns the path for the media file, not the server (eg. https://n2.coomer.st)s. Can you still donwload the files without knowing the server? the full url for a media file is server / data / path. The server use to be in the posts json, so you could grab the media of 50 posts at a time. But now its missing, and can only find it if hit the individual post api. I use python to scrape, not a coder by any means.

>>6455
If you have this:

{
"name": "0hqe5044pdke1lintpxnw_source.mp4",
"path": "/cd/22/cd229bcb22a356d9c112b21fc763e86b9209d4995d4ad8ff06a906bddfa09c82.mp4"
}

Then you can do this:

url = "http://" + coomerHost + "/data" + jsonFilePath + "?f=" + jsonFileName"

Or:

http://coomer.st/data/cd/22/cd229bcb22a356d9c112b21fc763e86b9209d4995d4ad8ff06a906bddfa09c82.mp4?f=0hqe5044pdke1lintpxnw_source.mp4

admins stay fucking everything that works instead of focusing on the shit that doesn't work.

api now returns 403 on auth on pretty much every user despite authenticating with session cookie. sometimes the posts endpoint goes through, but only if you've loaded that user in your local browser

icing on the cake is that posts endpoint doesn't paginate lmao, these people are retards. coomer death inc