Configuring mutual SSL authentication between ProxySG and BCAAA agent

Hopefully coming soon to an official KB article. Till then, enjoy the following article. Excuse the formatting, I originally wrote this in word, and it didn’t convert to HTML quite so nicely.

The objective of this KB is to configure mutual authentication between the ProxySG and BCAAA server. Each of these would verify the others’ identity using ssl certificates. If either node presents a wrong hostname / selfsigned / expired certificate, they will refuse the connection.

The following procedure allows for the most secure options, i.e. both nodes verify each other’s identity and the user running BCAAA is a non administrative user. The connection between BCAAA and ProxySG will be SSL encrypted

Assumptions : Correctly configured Active Directory + DNS + CA using Microsoft

1. Setup the correct PKI (assuming Microsoft 2003 CA):

  • Create a new certificate template to mark private keys as exportable, to be able to export certificate if BCAAA is moved to another machine
        • Domain controller > mmc > add snap in : certificate templates
        • Right click on “Web Server” and select “duplicate template”
        • Name it something like “Web Server – Exportable Keys”
        • Select the “request handling” tab and change the minimum key size and enable the “Allow Private key to be exported” option:

        • Start > Administrative Tools > Certificate Authority > Certificate templates
        • Right click > new > certificate template to issue

        • Select the newly created certificate template
  • Generate certificate for bcaaa agent.
        • Via internet explorer, go to http://server_ip/CertSrv (case sensitive).
        • Enter in domain admin credentials
        • Select “Request a certificate”
        • Select “Advanced certificate request”
        • Select “Create and submit a request to this CA”
        • As certificate template select “Web Server – Exportable keys”, and fill in the required information . Please note, for conformity, as a “Name” please enter the DNS hostname of the BCAAA server. In this example this will be bcaaa_kb.davidv.local

        • Submit the request, and install the certificate.
        • Open an MMC and add the snap in for “certificates – my user account”. You should see the newly created certificate in the “personal” store:

        • Open the certificate, and under the “Details” tab, click the “copy to file” button. Make sure the option to export the keys is enabled, and save the file. Make a note of the password you gave the file and delete the certificate from your store.
        • Transfer the file to the BCAAA server.

2. Generate the certificate for the proxySG.

  • Configuration > SSL > Keyrings > Create
  • Give an appropriate name, and create a new 2048 bit keyring. Apply the settings

  • Edit the newly created certificate and create a new “Certificate signing request”. Fill in the appropriate info and make sure the “common name” is set to the fqdn of the proxy. In this case it is proxysg.davidv.local.

  • Close the window and apply the settings
  • Again, edit the keyring and copy / paste the CSR to a txt file, including the —BEGIN and —END lines
  • Via internet explorer, go to http://server_ip/CertSrv (case sensitive).
  • Enter in domain admin credentials
  • Select “Request a certificate”
  • Select “Advanced certificate request”
  • Select “Submit a certificate request by using a base64….”
  • Copy / paste the certificate request to the site, and select the web server certificate template.
  • Submit the request and download the certificate as base64 encoded
  • Back on the bluecoat, edit the keyring and import the certificate by copy/pasting the certificate. Apply the change
  • Ensure that you import any intermediate and root certificates into the proxy in SSL > CA certificates

3. Setup the BCAAA server:

  • Create a user in active directory to control the BCAAA service. In this example we’ll use “BCAAAuser”
  • Allow the BCAAA user access to the service certificate store (full control to registry + keys). If this is not done, you will see error messages in bcaaa debug log similar to:

AcquireCredentialsHandle failed: 0x8009030d; status=-2146893043:0x8009030d: The credentials supplied to the package were not recognized

  • From the Run prompt, launch the regedit program to give the BCAAA user full access to the following key and its children:


  • Install the BCAAA server (keeping the defaults if needed, we will change these later). Ensure the BCAAA services is stopped
  • Open services.msc and  ensure the BCAAA service is running as the newly created domain user
  • As administrator, open an mmc  > add / remove snap in > certificates > service account > local computer > BCAAA
  • In the personal store, right click and select “All tasks > import”, and import the pfx file created previously

  • Open the BCAAA configuration file from C:\Program Files\Blue Coat Systems\BCAAA > “bcaaa.ini”
  • Scroll down to the [SSL] section and change the following values to:





  • Note that the “CertificateSubject” must match the name of the certificate we just imported. It is also a good idea to enable bcaaa debug logging temporarily to make sure everything works. Add the following line to the end of the ini file:


  • Start the BCAAA service and check the bcaaa.log file. You should see the following logging indicating the BCAAA service found the right certificate:

Desired subject is bcaaa_kb.davidv.local

Certificate store open.

CAASA:Certificate for b

CAASA:Certificate count 1

Found desired certificate

4. Configuring the ProxySG

  • Under SSL > Device Profiles, create a new profile. Give it a suitable name and select the previously created keyring. Change the CCL to “<ALL CA certificates>” (or to a suitable CCL that contains the proper root/intermediate certificate). Enable the “verify peer” option

  • Create an IWA realm, and edit the “IWA servers” tab to tick the option “enable ssl”.
  • As a “SSL device profile”, select the device profile created previously



  • Open the VPM, and create a web access layer.

  • Create a new rule, and as a source select “new > user”

  • Choose the correct IWA realm and select the browse button:

This should show a lost of users in the domain



During troubleshooting, set event log on the proxySG to verbose + check debug logs on BCAAA server



BCAAA debug log error:

An established connection was aborted by the software in your host machine.

SG event logs:

“CFSSL DOMAIN VALIDATION ERROR: Domain name mismatch:Name in certificate: bcaaa_kb.davidv.local, name in URL:”

“Login[tester]ERR:Cannot verify authentication agent server certificate; reason: Certificate subject does not match expected name”


Change the IWA  configuration to use the BCAAA hostname not IP. Make sure the hostname is resolvable!


BCAAA debug log error:

The credentials supplied to the package were not recognized


grant access to BCAAA user for folder:

C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

Imp : allow access to all files in the folder (when allowing permissions >replace permissions on all child objects)

4 thoughts on “Configuring mutual SSL authentication between ProxySG and BCAAA agent

  1. Hello David,

    Do you happen to have any article or pointers which regards to configuring ssl authentication between Bluecoat SG and Microsoft AD using Ldap. i used step 2 and 4 from this article but the SG is not able to browse the AD, not connecting. Is there a configuration I might be missing with regards the AD I’m quite confident about the SG part but not the AD. Hope you can help and thanks for sharing.


    1. Hey Bola

      The main two articles i’ve used in the past for LDAPS are the following:
      “How to enable LDAP over SSL with a third-party certification authority” (

      “How to troubleshoot LDAP over SSL connection problems” ( In this article the “Enable Schannel logging” is especially useful for gathering information.

      I’d recommend you first setup normal, unencrypted LDAP and see if that works as expected. Then move over to LDAPS, (dont forget to change the port to 636 from 389) and start looking at the above two articles to help.

      Dont forget to put the event logging level to verbose on the bluecoat and keep an eye on the ProxySG event log for any clues.


      1. David, thanks a million, I now have this working your notes helped greatly. Keep up the good work in your knowledge share.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.