- The tracker was stopped because all existing warriors got their IP addresses banned when using the previous crawl script. A lot of bad data ("banned" error pages etc.) also got archived upstream on the staging server. So until a new script was formed and proven itself and the bad data was pruned from the tracker, the tracker was paused so as to not cause any more problems.
- The tracker has been resumed.
- Working on Frontend (Warrior Queue) and new crawlers.
- Also trying to get more IPs to circumvent bans.
- Reverse engineering the private APIs to get valid data.
- We don't have time to hand-pick blogs.
- Please submit the link to ArchiveTeam's NSFW Tumblr form, but we can't guarantee it'll be saved.
- If there's a blog you absolutely want to save, see "How do I do my own personal backup of a tumblr blog?" below.
- Python (cross-platform):
- tumblr_backup.py can make a local backup of posts (XML default), video, audio and images. Uses APIv2
- Windows:
- TumblThree can archive an entire blog by feeding it an URL, including asks, text posts and reblogs to XML format and can download all images. Downloadable here. Windows only until the dev implements mono support.
- Web-based exporters/importers:
- Wordpress.com importer can migrate your entire Tumblr blog to your own Wordpress.com subdomains. This is an easy and cheap solution to let your blog live. Wordpress.com allows mature content but you can host Wordpress yourself to be in total control, because it's free and open source software.
- Tumblr export: you can allegedly get all your data in a single ZIP file, but it may or may not be an open format usable elsewhere. Migrating to Mastodon is possible thanks to several independent and nsfw-friendly instances supported by donations but you may need to import the exported data yourself until an automatica importer is developed.
- Note that the data produced by these programs is incompatible with Archive Team's format and thus cannot be used by the official project. These are usable for your own backups only.
- Only if you believe their data is at risk.
- Starting 2018-12-17.
- According to some reports, the deletion has already begun.
- Every warrior using the official crawl script got their IP address permanently banned sometime around saturday.
- The bans were evidently done by hand and most likely relied on the User-Agent string as well as the excessive access patterns the script caused.
- Everyone got banned, regardless of how much or little they had crawled, and regardless of whether they used a residential or a datacenter IP.
- Yes, use the official warrior from #tumbledown on EFnet (IRC).
- Just ask where to find it there.
- If it's from an uncompleted warrior job, delete it.
- If not, come to #tumbledown on EFnet (IRC).
- Login- and API-based crawling has been reported to still work regardless of the previous IP ban, but API-based crawling is not compatible with the Archive Team's objective. You can still do it for yourself, though.
- Apart from that, if you're banned, only getting a new IP address somehow.
- If you are on a residential connection, restarting your router might assign you a new IP address from your ISP's pool.
- IP addresses from different warriors around the world.
- Every (HTML) crawler will get rate limited quite quickly, so the number of different connections is much more valuable than huge bandwidth if you cannot spread it over multiple IP addresses.
- Data integrity is a very high priority for the Archive Team so use of VPNs with the official crawler is discouraged.
- You will be banned from the tracker if we are aware of you using a VPN.
NoNO. Open proxies are absolutely the worst offender in shitting on data in transit or just straight up serving you malware. Data integrity is a very high priority for the Archive Team so don't use open proxies with the official project.
- Not yet. If this becomes available, this page will be updated.
- They're running their own test software, that might produce invalid data.
- Also they're not crawling without limits, Tumblr bans everyone at some point.
- Coordinated low volume of crawling likely makes Tumblr a lot less trigger-happy.
Other mass crawlers, in use at the moment:
- https://github.com/terorie/tumblr-panic
Outputs blogs' internal data and media (images, videos and audio)
However:
- This is not compatible with the Archive Team's format.
- You can't view the output in the browser yet.
- A second script is added later to output HTML.
A small tumblr-monitor.sh script can give you something more human-readable and specific than your typical top or atop.
For more advanced monitoring, some ArchiveTeam members have shared their orchestration solutions, for instance Prometheus with Terraform on DigitalOcean; see the Warrior wiki page for more information.
- There's only three endpoins with IPv6: www.tumblr.com, api.tumblr.com and api-http2.tumblr.com.
- api-http2.tumblr.com can be used for API-based crawling and includes all metadata, but this is not compatible with Archive Team's format.
- To summarize, the format Archive Team uses (full HTML dumps) cannot be reached via IPv6.
- IP bans aren't enforced on CDN URLs.
- Primarily because warriors in the EU were receiving a GDPR opt-in page. Using the Googlebot UA avoided this.
- The bans are IP bans, changing the user agent neither helps us avoid them nor lets us bypass them.
- Tumblr treats all bots equally.
- Most likely not needed, if Tumblr were banning by UA, it'd happen a lot sooner.
- You can submit PRs though if you want.
- Note: Don't rotate UAs within the same crawl session/machine/IP address, that is highly suspicious behavior and might get your session flagged.
- Absolutely, but their IP addresses could be banned already, so it's a gamble.
- Some of the biggest players that were hit in the first ban wave were using DigitalOcean.
- AWS with elastic IPs seems mostly unexplored so far, also smaller ISPs might fare better than the big ones.
- Please share your experiences on #tumbledown on EFnet.