Sermet Pekin writes:
I built an open-source project that discovers blogs through recursive network exploration–basically PageRank for the blogosphere. Your blogroll was the main seed source.
It recursively discovers new blogs by following citations and parsing RSS feeds, mapping out how blogs link to each other. Starting from a curated seed list (I used your blogroll recommendations), it can scale to hundreds or thousands of blogs depending on exploration depth. It supports different exploration strategies—breadth-first to explore widely across communities, depth-first to dive deep into niches, or mixed approaches.
I thought you might find it interesting:
– Helps surface quality blogs without relying on social media algorithms
– Your blogroll made excellent seed data—the blogs were well-curated and interconnected
Fun!
Clicked on the link. Explored the repository a little. Didn’t find an answer to my question: what is the definition of a “blog”? That is, how does the link analysis differentiate between a blog and not a blog?
Good question! The tool uses a few filters working together:
The main filter is RSS/Atom feed detection – if a site has a valid feed, that’s the strongest signal it’s meant for syndication. Beyond that, I filter by domain (allowing .com, .org, .io, .edu, etc. but blocking social media like Twitter, Facebook, GitHub, Wikipedia, arXiv).
I also look for common blog platforms in the URL (Substack, WordPress, Blogspot, Medium, Ghost) or path patterns like “/blog/” or “/posts/”. Personal domains like name.com or organization.org typically pass through too.
The basic logic: if it has an RSS feed + passes domain checks + isn’t on the blocklist = it’s treated as a blog. This works pretty well for catching traditional blogs, academic sites with feeds, and personal websites while filtering out social platforms and code repos.
The full logic is in config.py (BLOG_INDICATORS, SKIP_DOMAINS, ALLOWED_EXTENSIONS) and discover.py (is_likely_blog function) if you want to see the details.
Thanks.
Nice! However, one big problem here is that “link to each other” tends to promote backscratching cliques, which isn’t the same as “quality”. And people who are doing good work, but aren’t popular, tend to suffer.
Been exploring the repository. Don’t think this is a page rank algorithm. Think this is simply a recursive function that looks for outgoing and ingoing links to “blogs” (chosen from a set of domains). A page rank algorithm would 1) be a full rank adjacency matrix and all entries would necessarily sum to 1 2) the comparison of the links would be to a stochastic adjacency matrix; that is, what is the likelihood that these links are greater than would be found through a random process of linking?
You’re completely right – I should have been more precise with the terminology.
It’s a recursive link crawler that builds a directed graph, not the actual PageRank algorithm. I was using “PageRank” loosely to mean “network-based discovery” (like how Google’s crawler works), but that’s misleading. It doesn’t implement the eigenvector computation or create a stochastic adjacency matrix or rank anything.
What it actually does:
– Starts with seed blogs
– Extracts links from their posts
– Validates discovered blogs (RSS check, domain filtering)
– Recursively explores them
– Builds a graph of the citation network
The output is just the discovered network (nodes + edges). You could feed that into PageRank or other centrality algorithms, but the tool itself doesn’t do any ranking.
Better descriptions would be “recursive blog network discovery” or “citation graph crawler” – thanks for the correction!
…did you use an LLM to write the comment?
You’re absolutely right, and I’ve been thinking about this limitation. The tool basically maps the visible network – blogs that cite each other – which definitely favors connected communities over isolated voices.
A few things worth noting:
The tool doesn’t rank blogs or claim discovered blogs are “better” – it just maps who’s linking to whom. It’s agnostic to whether those links represent quality or just mutual citation. The seed quality matters a lot though – starting with Andrew’s blogroll helped because his curation already filters for thoughtful writing.
You’re right that great isolated bloggers won’t surface here. I see this more as complementary to other discovery methods (search, social recommendations, manual curation) rather than a replacement. The real value is visualizing the blogosphere’s social graph and discovering blogs within connected communities that might not show up in algorithmic feeds.
But yeah, the echo chamber problem is real – if you seed it with low-quality blogs, you’ll just discover their network instead.
I was just thinking about this a few days ago. The rapid deployment of AI technologies in all applications, social media search engines is driving me mad. Many erroneous results if you have a keen eye. I thought, “Can we just go back to Page-Rank? I keep trying to turn them off, but they keep coming back, similar to the startup application issues with Zoom. AI technologies are like Zombies. There’s also AI interviews, now, too. They’re a disaster. You can’t effectively communicate your CV, and the AI is inaccurately giving you summaries. Moreover, it’s a legal liability. Some things I do are proprietary or have signed an NDA, and I’m sometimes required to explain what I’ve done at a previous position. If it’s with a senior manager, it’s done in confidentiality, whereas here, they’re recording it and I have no idea who they’re distributing it to.
Totally agree! That frustration is exactly what motivated this project – I wanted to discover blogs through actual citations (blogrolls, post links) rather than opaque algorithms.
RSS feels like the perfect middle ground: publishers control what they syndicate, readers control what they subscribe to, it’s chronological with no manipulation, and it’s completely decentralized.
The weird thing is this infrastructure (RSS, blogrolls, web directories) has been around for 20+ years but got buried by social media. Hoping projects like this can help revive some of those decentralized discovery mechanisms.
I put up a demo page here for anyone to see what it looks like without running the code: https://sermetpekin.github.io/rss-discovery-engine/
This is just a small snapshot from a short run. It can be run locally for longer to build a larger network.
This is quite a cool project, though the “style” of both the code – what it’s printing to stdout (since I git cloned and tried it out, against my better judgment) – and the responses from Sermet – feel like LLMs had a heavy hand here. Which is… okay, I guess. But also a bit disappointing/alienating.
I will say, I am starting to really miss the idiosyncratic, human touch of both crafted code and crafted comments…
About the usage of LLM in code:
– Although I have been coding for 35 years now and I am proficient in many of the programming languages out there (I will not be naming here I have proof of them on my github so long before LLMs emerged.) Life is too short not to use LLMs for minimum viable products or even later while documenting or polishing code. Life is too short not to get some help while creating a Minimum Viable Product.
About writing in comments :
– While writing however since it is my second language – which although pretty good for a second language of a person – still not close to perfect unless I do some warm up before (I am better in terms where I am in procedure of writing a thesis or studying for an exam in English etc. ) it feels fare to me to get some help while polishing a draft for written text not for my ideas. In addition I check every word after being polished. -To be honest I check my text less when I do not use LLM to polish – Here we go, this is my crafted comment with no touch from any LLM and which goes to my genuine samples repository :)
LLMs are really good in shortening the text since as humans at least in my case It requires a lot of time to come up with a well crafted short and concise text. In addition while using LLM to polish my writing I feel like I am learning how to be more concise while writing. This example is not a good cause it is here to make my case stronger to be a good excuse to use LLM :)
Another reason to use LLM is while reading Andrew’s blog, since he is perfect at writing concisely I feel pressure to write a better text and in this blog platform unfortunately there is no edit button which would be really helpful in cases where we make a mistake or typo. Polishing with LLM gives the guarantee of zero typo or repetition of words, formatting etc.
Here we go. This is my real voice. Untouched by LLM or fine tuning, however I had to use my keyboard instead of pen and paper. By the way I have another project which collects authors’ writing samples and use them while polishing his drafts that will hopefully do better job in helping writers to protect their own voice.
I think these years will be a huge transition years that we may sometimes feel like the ones who complained when we started using keyboard. They said probably similar things keyboards are killing human touch of hand-writing etc. but lets have a look how far we have come!
Thank you Anonymous for checking out the code and for you honest comment. Now it was a good chance for me to defend my case about use of LLMs and here is my human touched and crafted comment which may have some repetitions.
By the way, I will say, I am starting to miss comments from non-Anonymous people which in general has some more human touch and empathy for the work done.
Sermet:
I’m glad that you can use all available tools in your coding. For your comments here, I’ll just say that what you wrote in your own voice just above is just fine. When I write in French, Spanish, or Dutch, I try my best and I look up a lot of words in the dictionary or Google translate or whatever, but I don’t feel comfortable having the machine write for me. I like to know what I’m saying and not to have this fuzzy barrier between my thoughts and what the world sees. What this means is that when I write emails to Dutch-speaking people I only use a few words of that language. When I wrote an entire article in Dutch, I asked a colleague to check it, and she had many corrections to offer. I guess there is a way you can ask the chatbot not to rewrite your email but instead to mark it up so you can make the corrections yourself, and I’d feel more comfortable doing it that way than just having it emit an entirely rewritten note. I say all this recognizing that many people will find my stance ridiculous, comparable to insisting on walking up and down the nine flights of stairs to my apartment rather than taking the elevator that’s right there.
Andrew:
Thank you for your helpful advice and kind words. I will be using the stairs more often (although sometimes it feels like life is too short to take the stairs—but it is good for health I guess :)
I will be using my “life is too short” phrase in better places now. Such as: Life is too short to be a Frequentist, since for some events we just do not have enough data. :)