Email and chat still constitute the majority of electronic communication on the Internet. The standardisation and acceptance of protocols such as SMTP, IMAP, POP3, XMPP, and IRC has allowed to deploy servers for email and chat in a decentralised and interoperable fashion. These protocols can be secured by providing encryption with TLS-directly or via the STARTTLS extension. X.509 PKIs and ad hoc methods can be leveraged to authenticate communication peers. However, secure configuration is not straight-forward and many combinations of encryption and authentication mechanisms lead to insecure deployments and potentially compromise of data in transit. In this paper, we present the largest study to date that investigates the security of our email and chat infrastructures. We used active Internet-wide scans to determine the amount of secure service deployments, and employed passive monitoring to investigate to which degree user agents actually choose secure mechanisms for their communication. We addressed both client-to-server interactions as well as server-to-server forwarding. Apart from the authentication and encryption mechanisms that the investigated protocols offer on the transport layer, we also investigated the methods for client authentication in use on the application layer. Our findings shed light on an insofar unexplored area of the Internet. Our results, in a nutshell, are a mix of both positive and negative findings. While large providers offer good security for their users, most of our communication is poorly secured in transit, with weaknesses in the cryptographic setup and especially in the choice of authentication mechanisms. We present a list of actionable changes to improve the situation.