Here is a way of detecting search engine (Google, live…) crawlers,spiders etc..

[code:c#]
            System.Web.HttpBrowserCapabilities clientBrowserCaps = Request.Browser;
            if (((System.Web.Configuration.HttpCapabilitiesBase)clientBrowserCaps).Crawler)
            {
                Response.Write ( "Browser is a search engine.");
            }
            else
            {
               Response.Write ("Browser is not a search engine.");
            } 

//Here is another approach..

            if (Request.ServerVariables["HTTP_USER_AGENT"].Contains("Googlebot"))
            {
                //log Google bot visit
            }
            else if (Request.ServerVariables["HTTP_USER_AGENT"].Contains("msnbot"))
            {
                //log MSN bot visit
            }
            else if (Request.ServerVariables["HTTP_USER_AGENT"].Contains("Yahoo"))
            {
                //log yahoo bot visit
            }
            else
            {
                //similarly you can check for other search engines
            }

[/code]

Web Site Project is deployed with source code to the server and all compilation takes place at runtime.

Web Application Projects, the code behind classes are compiled to dll. That dll is deployed and at runtime, the compiled code in the dll and the markup is combined to create a class which is used by the server to render output.

Update:

Good post by anthony on this

post by Stephen on this topic