Here is a short description of  the minimal steps necessary in order to use Ajax in your EPiServer CMS 5 project.
Note that in this case the Ajax controls where used in a custom property with no WebServices involved, i.e I haven't tested any possible issues related to the FriendlyUrlProvider when using Ajax control in templates.

Web.config

Add the following to the httpHandlers section in web.config:

<add verb="GET,HEAD" path="ScriptResource.axd"
    type="System.Web.Handlers.ScriptResourceHandler, 
     System.Web.Extensions, Version=1.0.61025.0, Culture=neutral,     
         PublicKeyToken=31bf3856ad364e35" validate="false"/>

References

Add a reference to the System.Web.Extensions.dll (version 1.0.61025.0).

WebForm.aspx

Register the System.Web.Extenstions assembly in the top of your page.

<%@ Register Assembly="System.Web.Extensions, 
     Version=1.0.61025.0, Culture=neutral, 
         PublicKeyToken=31bf3856ad364e35"
    Namespace="System.Web.UI" TagPrefix="Ajax" %>

Add the ScriptManager right after the form tag:

<form id="form1" runat="server">        
    <Ajax:ScriptManager ID="ScriptManager1" runat="server" />

If you then download the ASP.NET AJAX Control Toolkit, add a reference to the AjaxControlToolkit.dll, you can easily use controls listed and demoed here.

More advanced use of Ajax

If you want to use any of the controls that runs webservice calls, such as the AutoComplete control you'll need the complete configuration setup, read more on configuration here.

And finally, if you're running IIS 5, this FAQ will save your day.