Awesome! Suggestions:

Do a reverse lookup on the site's IP and add matchers for the hostnames that show up.

Mention if they're round robining their DNS.

Add nmap OS fingerprinting.

Do a traceroute and log the IP of the closest router (final hop) to the site and add matching for that.

Add a wiki interface and build a crunchbase like app.

Add archiving of data and monitoring over time (as netcraft did in their original app).

nmap OS fingerprinting seems a bit aggressive, but reverse-DNS checking seems like a great way to check which hosting platform a site uses.

nmap is too aggressive. It's a prelude to actual hacking attempts and labeled by IDS systems as such. Don't use it for this or you may end up in legal trouble.

Agreed. UnderTheSite makes a specific point of only considering information that would be returned when a user's browser hits a website. It doesn't scan or probe ports / urls.

Thanks for the great feedback!

Awww, I wish I didn't have to leave SHDH before the lightning talks started!

I wrote a Ruby program to do something similar to what you're doing: https://github.com/jpf/domain-profiler - If you ever start profiling sites using information from places other than what the server returns, perhaps what I've done can help inspire you?

Is there a working example of domain profiler please?

Thanks, I'll check it out!

Cool site! I'm having a lot of fun. However, I was confused by this flow:

First: http://drktd.com/8Jgn

After clicking the 'Add It!' link: http://drktd.com/8Iaf

The wording on the previous page had me expecting a page for manually adding technologies to my site's stack a la Bagcheck.com.

Ah! This is a feature that I plan to add. Do you want to select technologies from a list, or write free-form text about your technology stack?

I'd like to suggest avoiding manual additions of technology for as long as possible. Focus on adding more ways to match specific technologies. After all, a site could always advertise more technologies in server headers or meta generator tags.

What if we propose an extension to humans.txt called technologies.txt where a site can self-describe their stack?

A mechanism for sites themselves to advertise their stack seems like a great idea (though I'd prefer it not occur via fixed URLs like humans.txt or robots.txt, but via headers or meta tags). I'd just suggest not allowing arbitrary additions to a site's stack without any way to verify them.

Follow @underthesite for updates as I implement all of the great suggestions that you guys have made here.

Would love to see if it could detect more scripting languages and frameworks than PHP.

It incorrectly stated that my site uses cold fusion.

What's the site?

I posted it buddy, I liked your demo a lot!

Thanks :)