@jgoerzen @ajroach42 with a possible pit stop on a shared lan server that untars it long enough to index into elasticsearch or whatever along with a pointer to the tarball on lan storage for caching etc. yeah, this is more than doable.
Mastodon is the best way to keep up with what's happening.
Follow anyone across the fediverse and see it all in chronological order. No algorithms, ads, or clickbait in sight.