Here is an example to validate Fileupload control. This validation will check user to upload files of particular type/extension.

Client side validation:

<asp:FileUpload ID="UploadFile" runat="server" />
<asp:Button Text="Save" ID="btnSave" runat="server" OnClientClick="javascript:return ValidateUpload();" />     
 <script type="text/javascript" language="javascript">
    function ValidateUpload()
            {           
                var Upload_file = document.getElementById('<%= UploadFile.ClientID %>');                
                var myfile = Upload_file.value;               
                if(myfile.indexOf("doc")>0)
                {
                 alert('Valid Format');
                }
                else
                {
                 alert('Invalid Format');
                }            
            }
 </script>
Validating with RegularExpression validator:

<asp:FileUpload ID="UploadFile" runat="server" />
<asp:RegularExpressionValidator ID="RegularExpressionValidator1" runat="server" ErrorMessage="Only Doc files are allowed" 
ControlToValidate="UploadFile" ValidationExpression="^(([a-zA-Z]:)|(\\{2}\w+)\$?)(\\(\w[\w].*))+(.doc|.DOC)$">
</asp:RegularExpressionValidator>

In this example, I checked  ".doc" file type but you can use this example for any file type. Just replace ".doc" with file type you would  like to check( "Xml", "wmv"...).