Communicating with FAST ESP from a .NET application through SSL

For a custom application we’re building, we decided to use SSL to communicate between our .NET application and the FAST query server. Setting up SSL to enable secure communication between a .NET application and a FAST query server is fairly simple but the FAST documentation doesn’t exactly explain it a straightforward manner. These were the steps I took to enable and use SSL.

The first step is to request a new certificate.You can use the openssl executable found in c:\esp\bin to request the new certificate. Open up a command prompt and change the working directory to c:\esp\bin. Enter the following command:

openssl req –new –out client.pem –days 365 –config c:\esp\etc\openssl.cnf

The above command creates a new certificate and should be fairly easy to understand. The request file will be saved as client.pem and the certificate generated (later) will be valid for 365 days. The configuration file specified by the –config parameter is the file that specifies the fields that you will be prompted for as identification to be included in the certificate request. When you run the command, you will be asked to enter and verify a passphrase. You will then be asked for the fields specified in the config file (note that you can just leave the ‘extra’ attributes being asked for blank). The command also outputs the private key to a file called privkey.pem. The exact output of the command is below:

C:\Users\btubalinal>openssl req -new -out client.pem -days 365 -config c:\esp\etc\
Loading 'screen' into random state - done
Generating a 1024 bit RSA private key
writing new private key to 'privkey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

The next step is to create a PKCS12 file from the .pem file. Again, you can use the openssl executable to create this:

openssl pkcs12 -export -out client.p12 -in client.pem -inkey privkey.pem

This exports out the PKCS12 file to a file called client.p12. It uses the client.pem file (which is the certificate) and the privkey.pem file (which is the private key) and merges them into the client.p12 file. The client.p12 file needs to be imported into your personal certificate store on whatever machine you’ll use to access the FAST query server. So copy the p12 file to your machine and double click the file to start the certificate import wizard. The wizard will ask you for the password/pass phrase you gave when creating the original client.pem certificate. The rest of the wizard is straightforward.

The last file we need to generate is a .cer file. This file is what is required to be used by your .NET application to issue the queries to the query server. Here is the command to create the .cer file:

openssl x509 -in client.pem -out client.cer -outform DER

This command basically converts the client.pem certificate (which is in PEM format) to a certificate that is in DER format and saves it to the client.cer file. Copy this cer file to a location on your machine where your app has access to.

The following is a code snippet that shows how the .cer file is used:

   1: using System;
   2: using Com.FastSearch.Esp.Search;
   3: using Com.FastSearch.Esp.Search.View;
   4: using Com.FastSearch.Esp.Search.Result;
   5: using Com.FastSearch.Esp.Search.Query;
   6: using Com.FastSearch.Esp.Search.Http;
   7: using System.Collections.Specialized;
   9: namespace FASTSampleSSL
  10: {
  11:     class Program
  12:     {
  13:         static void Main(string[] args)
  14:         {
  15:             string fastserver = "";
  16:             string certfilepath = @"c:\client.cer";
  18:             NameValueCollection searchFactoryConfiguration = new NameValueCollection();
  19:             searchFactoryConfiguration.Add("Com.FastSearch.Esp.Search.Http.QRServers", fastserver);
  21:             ISearchFactory searchFactory = SearchFactory.NewInstance(searchFactoryConfiguration);
  23:             string fastserverurl = "https://" + fastserver;
  24:             HttpSearchEngine searchEngine = searchFactory.GetSearchEngine(new Uri(fastserverurl));
  25:             ISearchView view = searchEngine.GetView("mypublishedview", fastserverurl, false, false, certfilepath);
  27:             IQuery query = new Query("string(\"sharepoint\", mode=simpleall)");
  28:             IQueryResult result = view.Search(query);
  30:             //PROCESS THE RESULTS
  31:         }
  32:     }
  33: }

Most of the code should be easy enough to understand. The cert file is used in the call to get the view from the instance of the HttpSearchEngine. The FAST ESP API will take care of reading that certificate file and using it to communicate securely to the FAST query server.

Querying an SPFieldMultiLineText field for an exact line match with LINQ

Today, I had to come up with a way to return a value from one column in a SharePoint list based on finding an exact line match of a user’s query terms in an SPFieldMultiLineText field in the same list. To clarify, here’s an example of how the data looked like in the SharePoint list (this is just sample data):

Term Synonyms
SharePoint SharePoint 2010
SharePoint 2007
Microsoft Office SharePoint Server
Windows SharePoint Services
SharePoint Foundation
SharePoint Server
FAST Search for Internet Sites
FAST Search for SharePoint

In the UI provided, if a user looked for the term ‘Microsoft Office SharePoint Server’, I needed to return back the ‘SharePoint’ term (first item, first column in the table).

There were three options I could think of to accomplish this:

  1. CAML query.I ruled this out fairly quickly. Unfortunately, I had to do an exact match on the term that the user entered. So, for instance, if the user entered the term ‘Windows SharePoint Services’, then the first term (SharePoint) would be returned. However, if the user only entered the term ‘Windows’, the ‘SharePoint’ item shouldn’t be returned because it was not an exact match. With CAML, there was really no way for me to specify an exact match per line. I couldn’t use <Eq> because that would necessitate the entire Synonyms field to match what the user entered. I also couldn’t use <Contains> as that would result in false positives. None of the other comparison operators seemed appropriate either.
  2. For-each loop. Of course, i could have iterated through the items one by one and examine the Synonyms field. For each item in the list, I’d have to examine the Synonyms line-by-line to find the exact match. This was originally what I planned on doing, however, it felt inelegant and potentially inefficient, especially because this list is going to contain many items.
  3. LINQ. I didn’t know exactly how I could use LINQ to tackle this but I always felt like I could. So this is the path I attempted. Obviously, it worked or I wouldn’t be writing this blog post :).

Below is the code I wrote to query the field for an exact line match:


 1: public string GetTerm(string userSearchTerm)
 2: {
 3:     using (SPSite site = new SPSite("http://mysite"))
 4:     {
 5:         using (SPWeb web = site.OpenWeb("myweb"))
 6:         {
 7:             SPListItemCollection items = web.Lists["mylist"].Items;
 9:             string[] delimiters = new string[] { "\r\n" };
 11:             var term = (from e in items.OfType<SPListItem>()
 12:                         where ((string)e["Term"]).Equals(userSearchTerm, StringComparison.InvariantCultureIgnoreCase) 
 13:                         || ((string)e["Synonyms"]).ToLower().Split(delimiters, StringSplitOptions.RemoveEmptyEntries).Contains(userSearchTerm.ToLower())
 14:                         select e).FirstOrDefault();
 16:             if (term != null)
 17:                 return (string)term["Term"];
 18:             else
 19:                 return null;
 20:         }
 21:     }
 22: }

The code isn’t terribly difficult but a few lines probably warrant some explanation. The first thing to understand is how the values are stored in an SPFieldMultiLineText field. Each line is in the field is split using a carriage return/line-feed combination (\r\n). So if I wanted the value of the second item in the list, the value would be represented as FAST ESP\r\nFAST Search for Internet Sites\r\nFAST Search for SharePoint\r\nFSIS. Line 9 sets up the pattern that I’m going to be looking for to treat each line as a separate item.

Lines 11-14 is the LINQ code. In line 11, I first need to convert the items object into a generic, IEnumerable<T> representation of the collection. Even though the SPListItemCollection class implements IEnumerable and LINQ is supposed to work with IEnumerable objects, LINQ didn’t seem to work directly with the SPListItemCollection. It’s almost like I needed to give LINQ a hint what type of items the collection held.

Line 12 checks for an exact match of the user’s query terms to the term stored in the SharePoint list itself. For example, if the user searches for FAST Search, then the second item in the list better be returned.

Line 13 is where I handle the Synonyms multiline field. The first thing I do is convert the entire string in the field to lowercase, then I split the value of that into an array using the ‘\r\n’ combination as the delimiter. I then just need to call the Contains() method to see if the term the user entered is in the array and return the SharePoint list item if it is or null if it isn’t (Line 14 – call to FirstOrDefault()).

That’s pretty much it. Not hard stuff but I think a pretty useful technique.