Our purpose is rather simple. We want to make the internet as open as possible. Currently only a select few corporations have a complete and useful index of the web. Our goal is to change that fact by crawling the web and releasing as much information about its structure and content as possible. We plan on doing this in a manner that will cover our costs (selling our index) and releasing it for free for the benefit of all webmasters. Obviously, this goal has many potential legal, financial, ethical and technical problems. So while we can't promise specific results, we can promise to work hard, share our results, and help make the internet a better and more open space.

Our figure shows the performance of a server when subject to parallel load. This kind of load is often generated in a so-called "Distributed denial of service attack".

Apache dies at about 4,000 parallel sessions. Yaws is still functioning at over 80,000 parallel connections.

In our benchmarks we were only able to push 35 MB/s on small instances. So the actual requests per seconds were dependent on the object size we were pushing. The limit was always ~35 MB/s. Our typical HTML pages were around 50 to 70 KB, so we couldn’t reach the desired requests per second as our instance was at its bandwidth limit.

Usually when one instance hits its resource limits you load balance multiple ones. HAProxy is a fine example for a very robust TCP/HTTP load balancer. The problem is though, that it will not increase your bandwidth as all your traffic has to go through this one HAProxy instance. So even when you load balance multiple instances, each one is capable of pushing ~35 MB/s (—> ~350 MB/s with 10 small instances), the bottleneck will still be at ~35 MB/s (aka the load balancer).

Jackson’s findings showed that outside of flu season, the baseline risk of death among people who did not get vaccinated was approximately 60 percent higher than among those who did, lending support to the hypothesis that on average, healthy people chose to get the vaccine, while the “frail elderly” didn’t or couldn’t. In fact, the healthy-user effect explained the entire benefit that other researchers were attributing to flu vaccine, suggesting that the vaccine itself might not reduce mortality at all.

The Current State of Web Privacy, Data Collection, and Information Sharing

[...] some tools that use your Last.fm data to make it more personal and insightful.

LastGraph lets you explore your last.fm listening history. Stick your username in the box above (or someone else's, if you feel like snooping), and hit enter.

The most under-appreciated search engine on the Internet.

We are pleased to announce the release of our first research prototype of a social dynamic analysis tool for Wikipedia called WikiDashboard.

The idea is that if we provide social transparency and enable attribution of work to individual workers in Wikipedia, then this will eventually result in increased credibility and trust in the page content, and therefore higher levels of trust in Wikipedia.

This demo features the Wikipedia, with the text colored according to trust. Text on white background is fully-trusted text; the more orange the background, the lower the value of trust. Type a page title in the search box on the left to see the page trust-colored.