Setting up SqlCe Merge Replication with ISA Server in the middle.

Wow! I had to venture into the 'cave' and solved this problem - talk about a character building experience!

I'm currently building a Mobile BizTalk RFID 1.1 solution for TechEd08 that runs on a PPC with a Kenetics CFUHF Reader.

*** Early Screen Shot *** :)

So in building out this application the details always bring unforeseen challenges to light:

1) The application houses all the BizTalk RFID pieces (providers, device proxies etc) so registration, and starting/stopping providers/device discovery and applying properties to the device needs to be all taken care of.

2) I built an RFID Mobile Provider for the Kenetics device - I worked with their support engineers solidly for a week to build what I needed. I took a trip down memory lane and have had enough pinvoking to last till Christmas.

3) The app also manages a several local SQLCe databases - one for my app, the others for the operation of BizTalk RFID Mobile locally on the device (mainly for it's OOTB store/forward mechanism).

After weighing up several options in this solution and how to get data to/from the device reliably I decided to go with SqlCe Merge Replication as we needed to push/pull data from several tables and schema changes.

4) Which leads me onto one of the most little known items......

How do I setup SqlCe Merge replication? it's a mine field, change something here and boom over there.

The picture

Phase 1:

Forget ISA for the moment. If you can, aim to get replication running in a local environment first (e.g. Local LAN on same network, through VPNs etc)

Getting the SQL bits Setup
Ok - the pieces to the initial puzzle.....

1. Sql Server Side
   1. Sql Server and it's additional Sql Mobile Replication Bits - download from here.
   2. IIS to expose a replication 'end point' where the remote devices will connect to and replication will take place through. IIS can be separate out onto a different machine.
   3. As in my case, somewhere that the 'snapshot' DB information will live to merge down to the devices. Mine was a UNC share - SQL created this after I completed the Publication wizard.
   4. Installation -You want the SQL Server Compact 3.5 Server Tools installed on BOTH the IIS AND SQL Machines (if these are one and the same, then you only need it once)
      The server tools has two main components - one being the bits that drive IIS and the other being a wizard that configures the exposed virtual directory and sets security onto it.
      If IIS and SQL are on separate machines, the easiest way to go is:
      - get SQL to publish the snapshot to a UNC share e.g. \\sqlserver\data
      - On the IIS box, run the Configure Web and Synchronization Wizard (installed with the above server tools) and a later screen will ask you where this data is coming from - simply point to the UNC share.
2. Mobile Device Side
   1. The equivalent SQL Mobile Replication tools need to be installed (above and beyond just normal SqlCe database components install) - SQL Server Compact 3.5 for Windows Mobile
      *** NOTE: make sure that the bits on both the Mobile + Servers all match ***
3. Server Side Security - For this let's work backwards, from the publication through to the exposed endpoint.
   1. Publication Security - this is set through the Publication Access List within SQL Mgmnt Studio
      The group in question is the ExhibitorsGroup
      
      Create a publication within the SQL Management Studio
      
      
      (Publication General Properties)
      
      
      (Snapshot Properties - note the file location)
      
      
      (FTP Snapshot + Internet - I've just used Internet and no IIS server name as this is configured in the Mobile Wizard)
      
      
      (Publication access list - I've blanked out sensitive info, but you can see the BETDEV\ExhibitorGroup being manually addded to the list)
      The rest of the publication settings are defaults - for me anyway.
      
   2. Let's go to the UNC share - = C:\Public\Exhibitor.SqlCE.FileShare
      This is the UNC share that IIS repl component will connect to at the back end.
      Note: the BETDEV\ExhibitorsGroup obviously needs r/w access to this folder.
      
      
   3. Let's run the 'Configure Web and Synchronization Wizard' to configure the IIS component.
      (you'll find it off the tools menu after you've installed the Mobile Server Tools from the links above)
      Note: one of the interesting things I found here is that after running the wizard, I normally go a tweak a few things in IIS - directory browsing etc. As a rule of thumb, if you want to change something with the Virtual Directory that is created at the end of this wizard, re-run the wizard to do it!!! :)
      
      
      Press next if prompted with the welcome screen note my options here - SQL Mobile and press Next.Cool
      
      
      Select the site and Create a Virtual Directory (I'm re-running the wizard so I'm going to select Configure Existing). Press Next.
      
      
      I created an alias of SqlCERepl directory and accepted a sub-directory under the SqlMobile dir.
      (you can change this, but looking around the forums it was a source of grief - I could do without :) )
      
      
      Here - I selected HTTP and not HTTPS access to the VirtualDirectory (and SQL Service agent).
      I did this as if you remember the diagram at the top of this post, ISA will serve as the HTTPS endpoint and will fwd the request via HTTP to our IIS/SQL box.
      IF you do want to change from HTTP to HTTPS or visa versa - re-run this wizard. Save you about 4 hrs of head banging.
      Click Next when ready.
      
      
      On this page - I selected 'Authentication required' and not anonymous. This has something to do with the data that I'm replicating as I'm using a Filter based on 'UserName'. So in my case, the username that the devices connect with will be my differentiator (I looked into using something like 'deviceID' but didn't get too far)
      Click Next.
      
      
      Select the type of authentication to be made against IIS - I selected NTLM (basic is fine also - but you need to be mindful that we're using HTTP at this point)
      Quick note on Security: So far, we've got 2 areas that need authentication.
      1) the IIS virtual directory and 2) accessing the actual SQL Publication in the UNC share and SQL Publisher Access List.
      
      So if the two machines are separated (IIS + Sql), NTLM will no transverse these machines (known as the 'double-hop' problem) so I'm assuming Basic or Kerberos is the safer bet.
      Click Next when ready.
      
      
      On the Directory Access Screen note the presence of the ExhibitorsGroup and also this publication is accessing the UNC Share we created earlier.
      Next to continue.
      
      
      UNC path specified - here you can see how this could be pointing to this SQL Share sitting on another machine as in the 2 machine hosted scenario.
      Click Next and Finish to see something like:
      
      
      You're virtual directory is now configured.
      To test your configuration so far go to:
      http://<server>/sqlcerepl/sqlcesa35.dll?diag - diagnostics screen to get something like:
      You should be prompted to login - enter account details that have access.
      
      This is our fallback screen - next we will configure the ISA component and come back to our test screen to make sure.
      You're done - here. :)
4. Configure ISA Server
   ISA server will be the bridge between our public SSL access and our internal IIS/SQL Server. We would effectively like ISA to simply route the request and pass it through without to much tampering with our good packets.
   
   ISA Server is on IP address: IP:Y_Internal
   The Internal Server here is : 10.1.0.191
   The public Interface on the ISA Server is for our purpose known as IP:X_Public
   and it's FQDN is : demo.micks.org (in otherwords - this is the public DNS name that will point to the public interface of your ISA box)
   
   NOTE: Make sure you have your SSL cert ready - I created an inhouse cert from a standalone cert server.
   You need at least a 'Server Authentication' Certificate to apply within ISA.
   (I'll show you a little trick in the mobile app to get round the fact that the certificate is from a non-trusted Cert. Authority by default)
   The friendly name on the cert should be - 'demo.micks.org' (without the quotes)
   All this keeps SSL happy.
   
   
   1. Create a publishing rule in ISA 2006 that will effectively route all requests coming to the public interface to our internal IIS/SQL Server.
      
   2. Fire up the ISA MMC and create a New Web Server Publishing Rule - I've called this sample rule, "Public to Internal IIS/SQL Repl"
      
      
      Click Next when done.
      
   3. Rule Action - set to Allow
      
      Next
      
   4. Publishing Type=Single Web
      
      Next
      
   5. Server Connection Security - SSL.This means that SSL is going to be used over the public network.
      
      Next
      
   6. On the Internal Publishing Details - I tend to hardcode the IP address in, just to reduce any ambiguity.
      Note the IP address - internally acessible only. 10.x.x.x
      
      Next
      
   7. Further settings on the Internal Publishing Details
      NOTE: the option of fwding the original client host headers to the internal IIS/SQL (I found a variety of incomplete  HTTP Header details errors attempting to sync if I cleared this checkbox)
      
      We also can restrict the access on this rule by specifying the path of /SqlCeRepl/* (this is obviously the Virtual Directory created earlier)
      
      Next
      
   8. Fill in your public DNS name - don't worry that the wizard screen is showing http://demo.micks.org and NOT https://demo.micks.org
      
      Next
      
   9. Create a listener (if you need to ) as follows:
      (I've modified the screen shot slightly - from my listener)
      Note the ports: 8443 that SSL requests is coming on. You can use 443 if you prefer, I had other things on those ports)
      Also - I setup NO Authentication and replication works. You *could* try setting up Basic Authentication here and using Delegated Authentication (ISA server will login to the IIS/SQL box on your behalf with the inputted security credentials).
      
      I've also supplied the Certificate here as well (add your cert to the machine store ahead of time)
      
      A way to test if your auth is going to work - fire up your browser and try http://<server>/sqlcerepl/sqlcesa35.dll?diag
      
      You should be prompted for login details ONLY ONCE. If you need to supply them twice and then you see the diagnostic page, your mobile application replication will fail :-(. Once and once only.
      
      Next.
      
   10. Authentication Delegation- we want the client to auth. directly against the backend (routed through ISA of course :) )
       
       Next.
       
   11. User Sets - because we don't have authentication here, ISA can't determine users, so All Users is our only option.
       
       Next.
       
   12. What a glorious site....almost done......
       
       Click Finish to complete the wizard.
       
   13. Right click on the rule just created and select Properties - we need to change the Link Translation to OFF
        
       This was the major source of my grief - I kept getting 'HTTP Headers malformed...' ERROR:28035 when trying to sync from the Device - yay!
       
       I was fortunate to be able to contact a friend of mine Darren Shaffer (Mobile MVP) that explained what was required to be sent back/forth in the headers during the conversation - big thanks there Darren!
       
   14. You should be able to browse to https://demo.micks.org/sqlcerepl/sqlcesa35.dll?diag - it should WORK :)
       If not - resolve before moving on. (you may get IE grumbling about the Certificate being invalid if it's an inhouse cert)
5. Configure the MOBILE replication piece!!!
   1. Make sure you have installed the SQL CE 3.5 Core + Repl CABs at least.
   2. On the mobile device, I tend to have routines to Add and Remove DB Subscriptions as I found that if any publication changes on SQL Server happened - e.g. a field was modified, or a table added/removed from the Publication, then Merge Repl would fail even though it previously was working.
      
      Easier to Remove the Subscription on the local SQLCE db, and then add it again.
      
      Note: InternetUrl = https://demo.micks.org
      Username + pass must be a user that has access to all the bits we configured above. In my case, someone who is a member of the ExhibitorsGroup.
      
      The code looks like this:
      

       1:  public void AddReplAndSync()

       2:  {

       3:  //using System.Data.SqlServerCe;

       4:  bool bAddRepl = false;

       5:  try

       6:  {

       7:  if (DoDBLookup("SELECT
      count(*) as cRow FROM __sysMergeSubscriptions WHERE Subscriber='ExhibitorSubscription'", "cRow")
      != "1")

       8:  {

       9:  bAddRepl = true;

       10:  }

       11:  }

       12:  catch 

       13:  {

       14:  bAddRepl = true;

       15:  }

       16:  

       17:  SqlCeReplication repl = new SqlCeReplication();

       18:  repl.InternetUrl = AppSettings.Settings.ReplServer
      + "sqlcesa35.dll";

       19:  repl.InternetLogin = AppSettings.Settings.ReplUser;

       20:  repl.InternetPassword = "XXXXXX";

       21:  

       22:  repl.Publisher = AppSettings.Settings.ReplPublisher;

       23:  repl.PublisherDatabase = AppSettings.Settings.ReplPubDB;

       24:  repl.PublisherSecurityMode = SecurityType.NTAuthentication;

       25:  repl.Publication = AppSettings.Settings.ReplPubName;

       26:  repl.Subscriber = AppSettings.Settings.ReplSubName;

       27:  repl.SubscriberConnectionString = string.Format("DATA
      SOURCE='{0}'", ESDAL.GetDBPath());

       28:  

       29:  try

       30:  {

       31:  if (bAddRepl)

       32:  repl.AddSubscription(AddOption.ExistingDatabase);

       33:  CloseAllDBConnections();

       34:  repl.Synchronize();

       35:  }

       36:  catch (SqlCeException
      e)

       37:  {

       38:  MessageBox.Show(e.ToString() + e.NativeError.ToString());

       39:  }

       40:  

       41:  }

       42:  

       43:  public void ReplRemove()

       44:  {

       45:  CloseAllDBConnections();

       46:  SqlCeReplication repl = new SqlCeReplication();

       47:  repl.SubscriberConnectionString = string.Format("DATA
      SOURCE='{0}'", ESDAL.GetDBPath());

       48:  repl.InternetUrl = AppSettings.Settings.ReplServer
      + "sqlcesa35.dll";

       49:  repl.InternetLogin = AppSettings.Settings.ReplUser;

       50:  repl.InternetPassword = "XXXXXX";

       51:  repl.Publisher = AppSettings.Settings.ReplPublisher;

       52:  repl.PublisherDatabase = AppSettings.Settings.ReplPubDB;

       53:  repl.PublisherSecurityMode = SecurityType.NTAuthentication;

       54:  repl.Publication = AppSettings.Settings.ReplPubName;

       55:  repl.Subscriber = AppSettings.Settings.ReplSubName;

       56:  try

       57:  {

       58:  CloseAllDBConnections();

       59:  repl.DropSubscription(DropOption.LeaveDatabase);

       60:  }

       61:  catch (SqlCeException
      e)

       62:  {

       63:  MessageBox.Show(e.ToString() + e.NativeError.ToString());

       64:  }

       65:  }

       66:  

       67:  private void CloseAllDBConnections()

       68:  {

       69:  if ((_dbCon != null)
      && (_dbCon.State != ConnectionState.Closed))

       70:  {

       71:  _dbCon.Dispose();

       72:  _dbCon = null;

       73:  GC.Collect();

       74:  }

       75:  

       76:  }

Trick to deal with Inhouse generated certificates -
Within your mobile app we create a class that essentially returns True when asked 'Is this Cert. valid?'

Somewhere upon starting up your app - e.g. Form_Load - insert LINE#1 below.

LINE#3 onwards describes the class 'MyCustomSSLPolicy'

 1: System.Net.ServicePointManager.CertificatePolicy
= new MyCustomSSLPolicy();

 2: ......

 3: using System;

 4: using System.Collections.Generic;

 5: using System.Text;

 6: using System.Net;

 7: using System.Security.Cryptography.X509Certificates;

 8:  

 9: namespace MicksDemos.Utilities

 10: {

 11:  public class MyCustomSSLPolicy
: ICertificatePolicy

 12:  {

 13:  public bool CheckValidationResult(ServicePoint
srvPoint,

 14:  X509Certificate certificate, WebRequest request, int certificateProblem)

 15:  {

 16:  return true;

 17:  }

 18:  }

 19: }

Closing note:

Hope you find this useful - I've done this a few times now and am amazed with the lack of info around this especially through ISA.

If you get any errors around "Can't contact SQL Reconciler..." etc errors - GENERALLY try and rebuild the snapshop server side, then try syncing again.

Nighty night!