The Mystery of the STARTTLS command in the CDO library – Or: SmtpUseSsl Explained

by Philipp Stiefel, originally published 2024-01-12

last revision on 2024-01-12

Image genereated with Canva

I wrote a detailed text on how to send emails from VBA code using the CDO library. As this text is somewhat popular, I get quite a bit of feedback on this text. A particular hotspot of comments is the topic of opportunistic encryption using the STARTTLS SMTP extension command.

Before we dive deeper into the capabilities of the CDO library (Microsoft CDO for Windows 2000 Library, sometimes also called: CDOSYS) and those discussions, I would like to establish some basic knowledge about how sending email can be encrypted.

Encryption of Email Communication between Client and Mail Server

There a two commonly used mechanisms to encrypt the communication between the client and the mail server when sending an email.

SMTPS

The first option is SMTPS (Simple Mail Transfer Protocol Secure). This works by using a connection to the mail server, which is already encrypted with SSL/TLS at the TCP/IP transport level. The TCP/IP stack of the operating system already negotiates encryption when the connection is established. Communication with the mail server using the SMTP protocol does not need to be altered from standard SMTP, because it is only happening encapsulated in an encrypted TCP/IP connection.

By convention, SMTPS is usually offered by mail servers on port 465. However, technically it can be used on any available TCP port.

SMTPS connections work on an all-or-nothing basis. Either the encrypted connection can be established at the TCP/IP level, which then allows further communication via SMTP, or the initial connection fails and is terminated.

SMTPS works very much like the HTTPS protocol, which supports encrypted communication with a web server and is currently the most widely used communication protocol on the internet.

Opportunistic TLS - STARTTLS

The second option is opportunistic TLS encryption for SMTP. This is based on the STARTTLS extension to the SMTP protocol as specified in RFC 3207. The client initiates an unencrypted connection to the mail server. After the basic, clear text SMTP connection is established, the client issues the STARTTLS command and thus starts the negotiation process between client and server to elevate their communication to TLS encryption on application level. – Application level means that email client and the mail server applications encrypt their communication, while the outer TCP/IP connection is not encrypted.

As the initial connection is a basic SMTP connection, there will be no encryption related problem establishing this initial connection. Only after they are connected, the server will announce that it is supporting the STARTTLS command. The client can then choose to send the STARTTLS command or not. Likewise, the server can choose whether it wants to continue its communication with a client not sending the STARTTLS command. - Hence the name “opportunistic TLS”. – Of course, both parties can be configured individually to enforce encryption and thus terminate any connection where the counterparty does not cooperate to establish mandatory encryption.

By convention, OpportunisticTLS/STARTTLS is usually offered by mail servers on port 587. Servers are often, but not always, configured to demand mandatory TLS encryption on this port. Additionally, some servers also support STARTTLS on the default SMTP port 25, but don’t enforce its use on that port. – Technically it can be used on any available TCP port, optionally or mandatory.

Official Encryption Support in CDO

Now, back to our main topic, the support for encryption in the CDO library.

The Configuration object of the Message object from the CDO library supports an option named http://schemas.microsoft.com/cdo/configuration/smtpusessl (the option name is also available via the constant CdoConfiguration.cdoSMTPUseSSL). The documentation of smtpusessl simply states:

“A boolean value indicating whether Secure Sockets Layer (SSL) should be used when posting messages over the network using the SMTP protocol.”

After I explained the two different options for TLS encryption (TLS is the successor to SSL, and uses the same principle), it should be obvious that having just one single option for switching SSL/TLS encryption on or off, is a somewhat limited choice.

Exactly this single option and its sparse documentation is the root of the comments and discussions that lead to this text.

The Quest for the Truth About STARTTLS with CDO

If you use smtpusessl=True when connecting to a mail server providing SMTPS services on port 465, it is a very simple situation. Due to the all-or-nothing approach of the TLS encryption of the TCP/IP connection, this will work right away, or it won’t if there are configuration issues outside of the CDO library.

The situation is fundamentally different with opportunistic encryption by STARTTLS, which isn’t even explicitly mentioned in the documentation.

The Elusive Sendtls Configuration Option

When writing my text on VBA and CDO, I researched that there should be an undocumented option sendtls (http://schemas.microsoft.com/cdo/configuration/sendtls), which will configure CDO to send the STARTTLS command and initiate opportunistic TLS encryption. – There are dozens, if not hundreds, of sources on the internet mentioning sendtls. Many of those sources suggest it as the solution to the STARTTLS problem in CDO. – So, surely, this must be the solution, mustn’t it?

Well, not so much…

I let myself be fooled into taking the internet’s suggestions at face value and adding this to my text without in-depth verification.

Soon thereafter questions started coming in from readers who tried it sendtls but couldn’t get it to work for them. This made me research this option in more depth. – I should have done this right away.

It quickly turned out that sendtls would also not work for me. Upon closer inspection, many sources on the web reported problems with getting it to work and a few even stated that this option is a hoax and does not exist. But still many people on the web maintained that it is the working solution to the problem.

So, it wasn’t working for many people including myself. Why? Were we doing something wrong?

I tried a multitude of different configurations for different mail accounts and tried to use sendtls in all those scenarios. To no avail. It appeared that sendtls had absolutely no effect in any of those scenarios.

Looking at the CDO library in the VBA Object Browser did not reveal any further clue. All the documented options are listed as members of the CdoConfiguration module. But sendtls cannot be found even when the Show Hidden Members option is enabled. This is another clue that this undocumented option does not exist. Unfortunately, it is just a clue, no definitive result.

To really get to the bottom of the matter, I resorted to an unusual approach. I opened the cdosys.dll in a hex editor and searched for the names of the configuration options. I wasn’t sure whether this approach would lead to anything. The option names could have been externalized to a resource file, or anywhere else where the hex editor could find them.

This worry was unfounded. I was able to quickly identify all of the documented configuration options hardcoded in the DLL file. - But the string “sendtls” did not exist in the whole of the DLL file.

This on its own would still be no definitive proof. But think about this: A software development team at Microsoft, working on a component like CDO, which is included in every Windows installation on this planet, would neatly document their configuration options and organize them in a dedicated module for the convenience of the developers using the library. – Then they later add a new option, which is much more important in modern email communication than many of the other options, but they implement this new option entirely different than the existing options and go to great lengths to hide it in a way that no one can find it? Seriously? This is completely absurd!

The only logical conclusion is: The sendtls option does not exist!

I added this information to my main text on CDO and also stated the following, indirect conclusion there: “The CDO Library does not support opportunistic encryption with STARTTLS.”

We’ll see about that in a minute …

Before, let me offer an explanation for the numerous “success stories” about sendtls: These reports on the internet about having successfully used the non-existent option, can only having been written by people using this to send via servers that supported but not enforced the encryption via STARTTLS. In this scenario, people would set sendtls=true, which does absolutely nothing, but as the mail server did not enforce encryption there was no error and the emails were sent. – People believed they were using opportunistic encryption, but actually their emails were sent through an unencrypted connection.

To STARTTLS or not to STARTTLS

Everyone lived happily ever after…

… until an email arrived in December 2023 from a reader who didn’t believe my statement about STARTTLS not being supported by CDO. And he had not only lack of belief, but also a TCP trace that actually showed CDO connecting to a mail server on port 25 and sending the STARTTLS command.

At first, I thought of potential confusion and various mistakes he could have made. Luckily, he was persistent and convinced me that the above trace is actually showing what he claimed it was. So, I embarked on another journey of analysis.

I used CDO to send emails with various configurations via a mail server I operate myself for one of my domains. So, I was able to take the mail server configuration into account as well. – Isladogs’ Email Tester for CDO, was a huge help with this process.

After some experimenting, I was able to reliably reproduce the reported behavior. When CDO is configured to send via port 25 with SmtpUseSsl=True, CDO would indeed consistently issue a STARTTLS command. But using the very same configuration on the client and on the server and just changing the port to 587, which is the standard port for opportunistic TLS, equally consistently failed with an Error 0x80040213 “The transport failed to connect to the server.”.

It all made no sense to me. So, I resorted to Windows port forwarding on the mail server to redirect connections on port 25 to the port 587 and vice versa. This yielded interesting results:  If the CDO client connects to port 25, even though it is internally redirected to port 587, it was also able to send the STARTTLS command on port 587. Likewise, it failed to send STARTTLS, when it connected to port 587 from the outside, but was redirected to internal port 25. – This proved that the server configuration was not a relevant factor in this mystery.

A bad suspicion began to dawn on me. It all looked more and more like the CDO DLL contains hardcoded logic to behave differently when connecting to port 25 as opposed to other ports.

I then set up some more listeners for opportunistic TLS on the mail server on different ports. CDO exhibited the same behavior on all those different ports as it did on port 587. The connection was not successful on any of those ports. – This supported my theory of port 25 being treated specially.

(I did a lot more tests with different configurations, but they are not as relevant to the immediate question. So, I will not describe them here in detail but will only draw on their results in my conclusion.)

Next, I even tried to gather additional clues on CDO’s inner workings by decompiling the CDO DLL to assembly using Hex Ray’s IDA Free dissembler. – Unfortunately, my very limited knowledge of the Assembly programming language prevented me from gaining any relevant insights.

I was already convinced that CDO exhibits port specific behavior. The remaining open question was whether this behavior was hardcoded into the DLL or configurable by means of a configuration file or the Windows Registry. To answer this question, I monitored an Access application sending emails via CDO with SysInternals’ Process Monitor. – This tool allows to monitor all network, file system, and registry access from any running process on a Windows computer. –  The results were unpleasant but clear: There was absolutely no Registry or file system access for the whole time from CDO being loaded to an email being sent, that could read any relevant configuration options for CDO’s behavior.

With these results, I concluded my research and drew the now fairly obvious conclusion.

Conclusion

With all the above research and analysis, I believe I can draw pretty accurate conclusions about CDO’s behavior, even though none of this is officially documented anywhere.

CDO’s behavior in handling sending emails over an encrypted connection, is hardcoded into the DLL file and cannot be changed by any means of configuration. Neither directly via the Configuration object, nor indirectly via modifying the Windows Registry or any configuration file.

Behavior with SmtpUseSsl =false in the CDO configuration

If smtpusessl=false, CDO will never try to use any encryption. If the server expects encryption, sending emails will fail; either immediately or after expiry of the connection timeout for servers expecting SMTPS.

If the server does not enforce encryption, emails will be sent, but through an unencrypted connection.

Behavior with SmtpUseSsl=true in the CDO configuration

If smtpusessl=true, the situation is much less straightforward. When beginning to send an email, CDO will evaluate the target port used for the connection.

Special Behavior on Port 25

If CDO is configured for sending with smtpserverport=25, it will not attempt to establish an SMTPS connection on TCP/IP transport level. If the server is configured for SMPTS connections on port 25 (this is against sane and accepted practice for mail server configuration!), CDO will neither connect nor fail immediately, but will wait until the connection timeout expires and then fail with a timeout error.

If the server is configured to support STARTTLS on port 25, either optionally or mandatory, CDO will initially connect unencrypted and then initiate the encryption of the connection by sending the STARTTLS command. – This works as opportunistic TLS is intended to.

If the server does not support STARTTLS, the connection will fail(!) and an Error “0x80040213 – The transport failed to connect to the server.” will be raised. – Strictly speaking, the error number and message are incorrect, as CDO actually connected to the server successfully but didn’t like that the server did not accept STARTTLS.

Behavior for all other ports

I did not test *all* other ports, but only 465, 587, and half a dozen random port numbers between 33 and 9999. The behavior was consistent on all these ports. Therefore, I conclude that it will be the same for all ports except 25.

CDO will try to connect to the server using encryption on the transport level (SMTPS). If it fails to establish this initial encrypted connection, it will abort sending the email and will fail with Error “0x80040213 – The transport failed to connect to the server.”.

Conclusion Summary

When enabling the SmtpUseSsl option, CDO will try to send emails on port 25 using STARTTLS. On all other ports it will try to use SMPTS transport level encryption.

If SmtpUseSsl=True, CDO will never send an email over an unencrypted connection!

Share this article: