OpenGoogle: a collaborative search engine
We all know and use google, of course. The big central repository of knowledge, the Institute that crawls the web day after day collecting information and links. Ever thought about the influence google exercises on the internet, or even bigger: on society? As the most popular search engine, they have the power to decide what you are going to see if you enter a search query. Especially know that they have gone commercial for real, selling stocks and all, the danger is more than real that google gives us a reality that is largely determined by capital: who transfers funds to google to end up high in the search rankings, which stock holders use their influence to jimmy the search results?
One of the things about google that bites so much is that we simply don’t know. It is a closed source operation, you can’t peek behind the scenes. This is why it’d be nice to have something I’d like to call OpenGoogle for now: a search engine with the potential of google but whose content is build collaborativelly by contributors world wide.
Let me explain. What I envision is something like setiathome, where people world wide contribute cpu power, but not only that, they also contribute some storage space and a bit of bandwidth. Perhaps we have specialised clients, some that only do the crawling and others that exclusively handle queries with yet others coordinating everything. But maybe a universal client, that does all these things together is more convenient.
Two keywords, however, are important: decentrality and redundancy. Obviously, we can’t have a central storage with all data and query handling concentrated in one place. Whats more, we need redundancy. We can’t have that because one of the nodes is (temporarily) unavailable, the whole system breaks down.
So far, this idea is in the pre-brainstorm phase, something rolling around the back of my head. Together with Erik Borra, I hope to make the plans a slight bit more concrete. We’ll have some reading up to do on grid computing, multi-agent systems and distributed applications of course. Who knows, perhaps at whatthehack (the international hackers event in The Netherlands) we can have a kick off meeting.
