Encrypting for Apple's Secure Enclave

Encryption, once you have a safe and well-implemented algorithm, is all about the keys. Lose control of your keys, and it’s “Game over, man!” What if we could put our keys somewhere completely out of reach, where even their owner can’t get to them? Yibikeys and HSMs can provide that security, but they’re external devices. However, recent iOS devices and MacBook Pros have something just as good: the Secure Enclave (SE).

Fortunately, using the Secure Enclave for encryption is super well documented. No, wait, it’s hardly documented at all. Trail of Bits encountered this with Tidas, and I recently plunged into the same abyss.

Some demonstration libraries are available on GitHub, but they’re all self-contained. I wanted to generate a message somewhere else, and then decrypt it in the SE. But nothing I found addressed cross-system interoperability.

This led to a long and frustrating detective story. I looked for official docs. Pored over Apple header files and source code. Investigated other example projects, read ECIES papers, and ran down several blind alleys. And of course, I made plenty of stupid mistakes along the way. But I’ll cut to the chase: I eventually figured it out, though it doesn’t quite match the formal specifications. And now, I’ll show exactly how it all works.

Theory

Let me back up a little and explain exactly what I’m trying to achieve. Current MacBook Pros and iOS devices (those with Touch ID and, as I’d forgotten, Face ID) include a Secure Enclave. The Secure Enclave is a separate computer, used for high-security features like TouchID. It comes with its own encrypted firmware, memory, and storage, and hardware-based encryption. Programs talk to the SE through a “mailbox” system, rather than a direct connection. The application places data and commands in a specific memory location, then asks the SE to execute the command. The SE then returns results in the same way.

One feature added in iOS 9, and macOS 10.13, is the ability to store keys and perform cryptography entirely within the Secure Enclave. The application asks the SE to create a public/private keypair. The SE returns the public key (which should then be stored somewhere safe), but it holds onto the private key. Then it can ask “Here, sign this message” and the SE will grab the private key, sign the message, and return the result. Or “Here, decrypt this,” and it’ll decrypt the message using the private key, and return the plaintext. The application itself never has direct access to the private key, so the key should be very secure.

Algorithm Details

Large quantities of data are usually encrypted using a symmetric key. Symmetric algorithms are fast, efficient, and can handled by dedicated hardware. But it uses the same key for encryption and decryption, which means it the Secure Enclave can’t store it. Asymmetric encryption solves this problem, but with a cost: It’s much slower. So in practice, most systems making use of public key encryption use a hybrid approach.

According to Apple’s documentation, the algorithm used for Symmetric Encryption with the Secure Enclave is called:

which refers to an ECIES standard algorithm. The details are a little arcane, but that’s exactly why we’re here. Let’s take this ugly string one part at a time:

  • ECIES: Elliptic Curve Integrated Encryption System - an open standard that defines exactly how to do what we’re about to do

  • Cofactor: Include the elliptic curve’s “cofactor” when completing the Diffie-Hellman key agreement process

  • X963SHA256: Use the ANSI x9.63 key derivation function (KDF), with SHA-256 as an underlying hash function

  • AESGCM: For the final symmetric encryption, use AES in Galois Counter Mode (GCM), a form of authenticated encryption

The actual process that takes place is what took some digging to understand. It’s slightly different from the EICES standards, and even from Apple’s published code. Simplified, it works like this:

  • Create a brand-new, “ephemeral” public/private keypair. We’ll use it for only this message and then throw it away.

  • Create a unique shared secret for the message, using the ephemeral private key, and the recipient’s public key. The Elliptic-Curve Diffie-Hellman Key Agreement Process (ECDH) generates the secret.

  • In some circumstances, this shared secret can leak information about the private keys. The x9.63 KDF prevents that, reducing the risk of an attacker decrypting the message. Additionally, the ephemeral public key is used as “Shared Information” for this process.

  • The final key then encrypts the message, and generates an AES-GCM authentication tag.

  • The ephemeral public key, the ciphertext, and the GCM tag are all concatenated, and returned as the final encrypted message.

To decrypt the message, the recipient must:

  • Extract the ephemeral public key from the front of the message, and send it to the Secure Enclave.

  • Using the ephemeral key, and the recipient’s private key, the SE performs the same ECDH process. In the end, it should generate the same shared secret as the sender.

  • The SE then applies the x9.63 KDF to generate the symmetric key

  • Using that final symmetric key, the message can be decrypted.

Because the SE communicates in small chunks of data, it could take a while to decrypt a large message. So, that last bit is probably handled by the application processor, and not the SE. That is, the SE would return the final key, and application code decrypts the message. (I didn’t dig enough into this to know for certain, but it seems a reasonable assumption).

As mentioned before, this Implementation var slightly from the expected standard:

  • AES-GCM for encryption instead of XOR

  • Relies on GCM tag instead of separate authentication algorithms

  • Reorders final message to {pubkey} + {ct} + {tag} (instead of key, tag, ct)

Yet, it’s hard to completely fault Apple for straying from the vast array of choices in ECIES. In A Survey of the Elliptic Curve Integrated Encryption Scheme, the authors conclude:

So, it seems unlikely that we’ll ever see any broadly interoperable ECIES implementations. Given that, cherry-picking the best components to create a similar system isn’t much of a stretch. A later paper from the same authors reaches similar conclusions.

Technical Details

The curve generated by the SE in the demo application is a “prime256v1” curve, also known as “SecP256R1”. By reviewing keychain/SecKey.h, I was able to learn some of the deeper technical details. One such detail is the fact that this algorithm I’m spending so much time discussing is now considered “legacy” and shouldn’t be used for new code:

This comment also indicates that the AES-GCM uses the “static public key data” (the recipient’s public key) as additional authentication data (AAD). But it doesn’t work if you include that. So the docs are wrong, or the implementation is broken…or both.

For the preferred “VariableIV” algorithm, the KDF generates the IV, as well as the key. Instead of deriving 16 bytes, the KDF returns 32 bytes: The first 16 are the 128-bit AES key, the following 16 are the IV. For a curve that’s larger than 256 bits, you’d derive 48 bytes – 32 bytes of key material, then 16 bytes of IV. This VariableIV algorithm also does not appear to use AAD.

The KDF itself is actually less of a black box than I’d initially thought. It’s simply a hash of three concatenated values: sha-256( { shared_key } + counter + { shared_info} ):

  • The shared_key is the result of the ECDH function.
  • Shared_info is the ephemeral public key data
  • The counter is a four-byte, big-endian number, starting at ‘0000 0001’
  • The counter increments by one for every block that the KDF produces

Since SHA-256 produces 32-byte outputs, we only need to run through the KDF once. If we have to produce a 2nd block (bytes 33-64), then the counter changes to ‘0000 0002’, and a new hash is generated. And so on.

Example Code

None of this would’ve been possible if I hadn’t found some library to help me with the various primitives. The x9.63 KDF is pretty simple, but I needed to find the ECDH function, and AES-GCM. The last isn’t as widely available in popular python cryptography libraries.

I’m making use of the “hazmat” primitives available in the pyca/cryptography library. I also used PyCryptodome to verify the GCM output. For fun, I replicated the KDF function myself using an off-the-shelf SHA-256 routine (not shown here). So the only black box left (to me, anyway) is the ECDH key agreement phase, but that’s standard too. Plus, the script works with the SE demo code, so I’m confident that everything is working fine.

Again, I’m using this macOS demo application, from GitHub. To use my test script:

  • Start up the demo application, and click on Encryption
  • Then click “Encrypt” to fill the lower box with a ciphertext. We don’t need it, but we do need the public key.
  • Copy the public key displayed in the interface
  • Replace the “bob_pem” value in the script with the public key
  • Run the script, and copy the result into the demo app, overwriting the original ciphertext. (You may need to backspace once to get rid of any trailing newlines).
  • Click “Decrypt”, and if necessary, authenticate via TouchID
  • Gaze in wonder at the properly decrypted message

Here’s the bare minimum of the test script. A full, commented version can be found in this Gist on Github.

上面的例子，介绍了 Python 下编解码 Apple's Secure Enclave 算法的逻辑。那么如何在 Java 下实现相同的逻辑呢？可以参考 iOS-compatible ECIES implementation in Java 中的代码实现(可以点击这里下载代码拷贝)。需要注意的是，工程介绍里面声明实现的是 SecKeyAlgorithmECIESEncryptionCofactorVariableIV* 算法，而我们使用的是  SecKeyAlgorithmECIESEncryptionCofactor* 算法。我们只需要在代码中，加解密的地方传入16位全零的数组，即可兼容两者。

如下：

Changes for Variable IV, Other Curves

I mentioned before that this algorithm has been deprecated by Apple. The recommended algorithm is now “kSecKeyAlgorithmECIESEncryptionCofactorVariableIVX963SHA256AESGCM.” To support this algorithm, which derives a unique IV instead of using all zeroes, we only need minor changes:

This rough demo script isn’t set up to handle curves other than Prime256v1. But since the Secure Enclave only supports 256-bit curves, it’s not really an issue. We’d need to inspect the recipient’s public key, and use the same curve for the ephemeral keys. Then, if the curve is larger than 256 bits, we need to derive a larger symmetric key. In the above snippet, change the length from 32 to 48. The encryption key comes from the first 32 bytes, while the IV is the last 16.

Conclusion

So why is all this cool? Because we can be confident that nobody can read our data without our device. Granted, this presumes that the Secure Enclave is, in fact, Secure. We haven’t yet seen any (public) breaks of SE security, but that doesn’t mean it’s impossible. But since it’s so crucial to iOS and now MacBook security, I expect it’s likely pretty good.

What would be even better is if some decrypted items could remain in the Secure Enclave. Consider a system with many layers of encryption keys. The iOS Data Protection hierarchy is a good example of such a system. If you decrypt a group key with the Secure Enclave, then the key gets returned to the application. An attacker may be able to extract that key from the process' memory space. But if that key never leaves the SE, then there’s no risk of it getting leaked through application memory.

Hopefully features like this are in the works for future versions of iOS and macOS. In the meantime, it’s still a very powerful tool, and definitely worth investigating.

References

参考链接

iOS Keychain: using Secure Enclave-stored keys

One of the great hardware features of iPhone is Secure Enclave — a special hardware element designed to protect user’s sensitive data, including biometric data like fingerprints when device has a Touch ID sensor and face scans in case of Face ID. Secure Enclave ensures that this kind of data is safe even if a hacker gets access to device RAM or disk storage — the thing is that this data never gets to RAM and is never processed by OS or by programs in user space. OS or a program can only interact with Secure Enclave hardware using a predefined set of commands which doesn’t allow access to raw data.

Besides biometric data, Secure Enclave can store cryptographic keys. Such keys are generated inside this hardware element and never “leave” it — there’s no way to get them (unless you break somehow the hardware protection — as for now, there were no reports, suggesting that this is possible).

Currently, only Elliptic-curve cryptography keys can be stored in Secure Enclave. This is an asymmetric cryptography approach so we can talk about public and private keys here. The private key is generated and stored in Secure Enclave. The corresponding public key is available for export and can be transmitted to a communication counterparty or used for encryption locally.

Generating and fetching a key

To generate a Secure Enclave-stored key we can use a SecKeyCreateRandomKey call with special attributes. Here’s how this can be done:

The key type is set to kSecAttrKeyTypeEC and its size is indicated as 256 bits — only this kind of keys can be stored in Secure Enclave presently.

We can additionally use biometry to protect the key. This means that in order to use it, user will be required to go through Touch ID or Face ID authentication (see my previous post about biometry-protected keychain entries). In the fragment of code above we have a requiresBiometry parameter which being set to true enables such protection. This is done by setting the .biometryCurrentSet flag when we create the SecAccessControl instance that we later use to create the key.

Note that at the end of this procedure we’re getting a SecKey instance that we later can use in our program. This, however, doesn't mean that the private key we just created is loaded into RAM. Our SecKey is just a handle object allowing access to the key stored in Secure Enclave.

When we have our key in keychain, we can always load it using a function like this:

We’re getting the key calling the SecItemCopyMatching function and using the same “application tag” value (the kSecAttrApplicationTag field) that we indicated at the key creation stage.

Encryption and decryption algorithm

Now let’s look how we can use the key that we just created. Before we proceed we have to decide what encryption algorithm we’re going to use and check that our current device/OS supports it.

In the official documentation Apple suggests using the following algorithm: eciesEncryptionCofactorX963SHA256AESGCM. This long name basically means that we are usign Elliptic Curve Integrated Encryption Scheme. It refers to the ANSI X9.63 standard. A key derivation function defined in that standard (with SHA-256 hash) is used to generate an encryption key that’s finally used to encrypt the data using the AES-GCM algorithm. The “cofactor” term in the name of the algorithm tells us that some additional measures are taken for protection from certain types of attacks (we won’t delve into details here).

If we take a look at the corresponding constant kSecKeyAlgorithmECIESEncryptionCofactorX963SHA256AESGCMdefined in SecKey.h (see here for example) then we can see that this algorithm is considered “legacy” and the recommended one is kSecKeyAlgorithmECIESEncryptionCofactorVariableIVX963SHA256AESGCMinstead (in Swift it’s eciesEncryptionCofactorVariableIVX963SHA256AESGCM). We’ll use this recommendation in our example. The only difference between these two algorithms is that in the old algorithm an all-zero initialization vector was used to perform AES-GCM encryption. In the newer algorithm the IV is generated together with shared encryption key during the ECDH procedure (see below).

ECIES and ECDH overview

All this sounds a bit scary so let’s first look at the way ECIES works in general. As we’re talking about cryptography, of course we’ll be talking about Alice and Bob. Suppose Bob wants to send an encrypted message to Alice. The following diagram describes how Alice and Bob can in a safe way agree on a shared key that later can be used for encryption by both of them. Here we omit all the technical and mathematical details to make it as simple as possible.

Elliptic-curve Diffie–Hellman (ECDH) key agreement protocol.

Here’s what happens on this diagram:

  • First Bob gets Alice’s public key.
  • Then Bob generates a random value. This random value is also referred as an “ephemeral private key”. “Ephemeral” means that this key will be used only once — for this encryption session only.
  • After that Bob generates a public key for his ephemeral key and sends that to Alice
  • Now Bob generates a symmetrical key that can be used for encryption. For this he uses Alice’s public key and his own ephemeral private key.
  • On the other side Alice can reconstruct the same key using her private key and Bob’s ephemeral public key

So both sides have the same key and Bob can successfully encrypt his message and send it to Alice and she won’t have any problems decrypting it.

This is all nice but how is this related to our case with a key stored in Secure Enclave? We don’t have any network communications in our basic scenario. We just want to encrypt some data. Who is playing the role of Alice and who is playing the role of Bob? We’ll try to answer these questions in the next section.

How does it work in Apple’s crypto-APIs?

Sadly, Apple documentation is quite far from being full and detailed when it comes to this topic. So it takes a bit of guess work and reverse engineering to fully understand what exactly happens when you use these APIs. I can recommend this great post by David Schuetz for detailed description of this matter. It will be especially useful if you’re thinking about some encryption data interoperability (with your backend or other systems).

Here we will focus mostly on the practical side of the problem omiting some low level details. We have already seen how to generate a private key that’s stored in Secure Enclave (makeAndStoreKey method described above). Now let’s look how we can obtain the corresponding public key and how we can check that a given encryption algorithm is supported by the system.

If we have a key returned by the makeAndStoreKey method, then getting the corresponding public key is as easy as calling SecKeyCopyPublicKey and passing the original key as parameter.

Once we have our keys we can check if the algorithm we want to use is supported by the system. Here’s a fragment of code where we get our public key and check that we can encrypt data with it using the eciesEncryptionCofactorVariableIVX963SHA256AESGCM algorithm:

As far as I know it will be supported on all iOS 10+ devices with Secure Enclave, but anyway it won’t hurt you to have such checks in your code.

As you can see SecKeyIsAlgorithmSupported has three parameters: key, operation and algorithm. And all the three parameters are important. You can have different keys, and for each key type different algorithms and operations can be supported. If you experiment a little bit with the code above, you’ll see that if you change the operation to .decrypt, the method will return false. At the same time, if you try it with our private key, then you’ll get true for decryption and false for encryption.

With this information let’s look at our ECIES scheme again. We know that our private key is stored in Secure Enclave, so in our scenario it is Alice who owns (and uses for decryption) the private key, and it is Bob who uses Alice’s public key to encrypt the message.

But what about the ephemeral key? It turns out that it’s generated every time some data is encrypted. And Bob puts the ephemeral public key together with the ciphertext he generates. Then Alice can pick it up, reconstruct the shared key and decrypt the data.

In Apple’s crypto API SecKeyCreateEncryptedData is used to encrypt data and SecKeyCreateDecryptedData is used for decryption. When Bob will call SecKeyCreateEncryptedData, this function will return a CFData object containing the ephemeral public key (generated and used during this particular call) and the actual ciphertext. Now Alice can call SecKeyCreateDecryptedData to decrypt this data. All she needs is her key in Secure Enclave and the ephemeral public key that comes with the data from Bob.

A few more words on the ciphertext generated by SecKeyCreateEncryptedData: since it’s generated by AES-GCM, besides the encypted data itself, it contains an authentication tag (a piece of information used to verify the integrity of the data). At the same time initialization vector (IV) is not put into the output because it’s generated together with the shared key. It means that both parties (Alice and Bob) can reconstruct not only the shared key used for encryption but also the IV used to initialize the AES-GCM procedure (see ECIES/ECDH scheme).

Here’s a short example demonstrating how to call the encryption and decryption functions. For encryption:

And for decryption:

Note that we call SecKeyCreateDecryptedData on a background thread because if our key is biometry-protected, then this call will trigger Touch ID or Face ID authentication UI and won’t return until user authenticates or cancels the authentication. We don’t want to block the main thread while this function waits for user authentication.

Signing and verifying signature

Keys that are stored in Secure Enclave can be used not only for data encryption but also for digital signature. To sign a message you need your private key (stored in Secure Enclave). To verify the signature another party needs the corresponding public key. Of course we suppose that the public key is handed to the other party in a secure manner to avoid man-in-the-middle attack.

For digital signature we will be using the following algorithm: ecdsaSignatureMessageX962SHA256. This means that an ANSI X9.62-compliant version of ECDSA (Elliptic Curve Digital Signature Algorithm) is used and SHA256 is used as a cryptographic hash function. A SHA256 hash value is calculated for the original message and all the other steps of ECDSA are applied to it. By the way, if you already have that SHA256 value calculated, you can use the ecdsaSignatureDigestX962SHA256 algorithm which does exactly the same taking a SHA256 hash value as its input instead of the original message.

This is an example where we show how to calculate digital signature using our Secure Enclave-stored key:

First we check that our digital signature algorithm is supported and then the digital signature value is calculated by the SecKeyCreateSignature function. We call it on a background thread to prevent the main thread from blocking in case our key is protected by biometry.

You can call the sign function presented above in the following ways:

Note that these two calls will give you exactly the same result.

Now, let’s look how we can verify the signature that we just calculated:

Here we also check that the algorithm is supported and then call SecKeyVerifySignature to verify the signatue. This function takes as its arguments the original message and the digital signature that we previously calculated. The function returns true if the signature is valid.

I hope that this article will help you to make your iOS app more secure by applying some of the features coming with secure hardware element that’s included in iOS devices (Secure Enclave).

Full code samples for this article can be found here:

algrid/keychain-sample

If you want to store small bits of generic data in iOS keychain (not EC encryption keys as described here) and protect them with a password or biometry (Touch ID/Face ID), you can take a look at my other posts: Password-protected entries in iOS keychain and Biometry-protected entries in iOS keychain.

参考链接

家庭搭建OPENCONNECT VPN(ubuntu 22.04)

家里运行了一些服务，希望在外面也可以方便的访问它们。

旧办法是通过路由器端口映射到内网端口，但这种方式已经很难满足我的场景，因为例如hadoop服务会开放很多web页面端口，实在难以逐一映射。

于是我选择了在家里自建VPN服务器，然后只把VPN服务的端口映射到路由器，这样只要在外面连接VPN即可像在家里一样任意访问内部网络了。

搭建Openconnect

openconnect是开源VPN项目，包含了服务端和客户端（PC/MAC/Android），是商业产品思科cisco anyconnect的开源版本。

为什么选择它呢？因为它能够从服务端自定义下发路由规则到客户端，这样就可以控制客户端哪些流量需要送往VPN隧道、哪些流量不需要经过VPN隧道，实现类似于同时兼顾访问家庭内网和公司内网的目的。

搭建方式参考官方手册：https://gitlab.com/openconnect/recipes，我以debian为例进行安装配置，非常方便：

这样就安装完成openconnect服务端了。

配置Ocserv

接下来配置VPN服务端，分为2大部分：

  • 自签一下SSL证书，因为这个VPN协议需要SSL加密。（自签没关系，只要客户端登录时选择信任证书即可）
  • 配置ocserv.conf配置文件，主要是定义VPN如何认证用户、VPN网段、还有更高级的路由表下发。
SSL证书

首先搞证书，按官方教程来即可：https://gitlab.com/openconnect/recipes/-/blob/master/ocserv-configuration-basic.md#certificate-management-self-signed

第一步：

第二步（原样敲入即可，信息都不重要）：

第三步（原样敲入即可，信息都不重要）：

第四步（生成CA证书）：

第五步（用CA签VPN证书）：

把VPN证书拷到ocserv配置目录下：

Ocserv.Conf

先修改登录方式，采用用户密码进行登录：

创建密码文件

创建用户

删除用户(如果需要的话），此处只是留下删除用户的命令，不是要求必须执行，否则刚刚创建的用户就被删除了：

然后VPN监听端口有需要可以改一下：

然后就是VPN网段的配置了，我家里是192.168.1.x网段，我们要让VPN使用一个不同的网段，因为VPN server要给每个连接过来的VPN client分配一个VPN网段的IP，如果这个IP和192.168.1.x网段的某个IP冲突就有问题了，所以一定是不同网段的，同样道理也不能和客户端所在的局域网段冲突。

所以我配置了192.168.50.x的一个冷门网段，足够分配254个VPN client，为什么不是255个呢？因为VPN服务端会先占1个IP。

暂时我们配置这些，现在我们启动ocserv来介绍一下VPN是如何工作的。

分析Ocserv工作原理

大家自行在路由器映射VPN端口到公网，然后下载openconnect gui客户端（https://openconnect.github.io/openconnect-gui/），然后连接这样的地址：

https://公网IP:路由器映射端口

即可连接到家里内网的VPN服务端，输入任意正确的linux账号密码即可登录。

连接成功后，在客户端打开命令行查看路由表（我是windows电脑，使用route print命令），会发现2条本就存在的路由记录：

这是因为windows电脑也在家里，所以本身就被局域网下发了默认网关路由到192.168.2.1路由器、以及192.168.2.x网段的路由记录，这台电脑的自身IP是192.168.2.103。

但是连接VPN后，则多了2条VPN网段的记录：

又来了一条优先级更高（2）的默认网关路由，指向了192.168.50.1，也就是VPN server的IP地址，同时也多了一条192.168.50.x 这个VPN网段的路由记录，此时电脑上也出现了一张虚拟的网卡，其IP就是192.168.50.250，这是VPN server给分配的，由VPN client负责虚拟出本地网卡。

此时发往192.168.50.x网段的流量会直接从VPN网卡送出，发往其他IP的流量也会命中192.168.8.1的默认网关（同样是从VPN网卡送出），而不是电脑所在局域网原本下发的192.168.2.1的默认网关。

所以VPN网卡已经劫持了所有外出流量，经过VPN虚拟网卡封包为物理IP送往VPN server，再由VPN server拆包后得到真实IP地址，继续进行转发。

此时VPN server也多了一条该客户端的路由记录：

也就是192.168.50.250这个vpn client的流量全部送给vpns0虚拟网卡，也就是说回给该vpn client的包也需要经过vpn server的虚拟网卡封包送走。

vpn基本就是这个原理。

这点原理还不够

不要以为理解上面就万事大吉了，当VPN client请求baidu.com时，baidu的IP地址会命中默认网关发往192.168.50.1默认网关，经过封包送到VPN server的监听端口。

然后VPN server会把封包拆开，linux协议栈看到真实目标IP是baidu.com，那么显然该IP地址不是本机，需要forward给百度，此时需要linux的netfilters进行流量forward才能再次送给192.168.2.1家庭物理网关到达百度，这一步需要我们在服务端开启ipv4 forward特性：

sudo vim /etc/sysctl.conf

net.ipv4.ip_forward=1
net.ipv6.ip_forward=1

然后执行sysctl -p生效。

配置防火墙的IP Masquerading

请使用以下命令检查服务器的主网卡接口

例如，可以检查到网卡接口名字是eth0，则执行

这里命令-A表示添加到nat表的POSTROUTING链。这样就可以把VPN网络连接到Internet，并且把你的网络对外隐藏起来。这样Internet只能看到你的VPN服务器IP，但是不能看到你的VPN客户端IP，类似于你家中路由器隐藏起家庭网络。

现在可以检查一下NAT表的POSTROUTING链，可以看到目标是anywhere的源为anywhere的都执行MASQUERADE。

显示输出：

为了持久化iptables规则（默认重启后就没了），所以我们需要安装一下工具：

sudo apt install iptables-persistent

然后保存一次：

sudo netfilter-persistent save

最后

更高级的特性就是配置ocserv.conf的路由下发，下发一些路由规则到VPNclient ，这样可以控制哪些网段的流量不要发给VPN隧道，可以兼顾客户端本地局域网和家庭局域网两路一起访问的需求，但是这个要求客户端本地局域网段、VPN网段、家庭局域网段三者不能冲突。

大家可以试一下ocserv.conf中的route和no-route配置项，比如我希望VPN client不要让公司网段走VPN，那么就可以加一条这样的配置（假设公司网段是172.26.201.x）：

no-route = 172.26.201.0/255.255.255.0

这样VPN client侧的虚拟网卡就会判断目标IP是这个网段的就不发往VPN server了，而是走本地其他路由规则寻址。

如果你家里的网络是路由器/旁路由FQ或者绿色dns的话，那么可以令vpn server的默认网关指向fq路由器，同时让vpn client使用你家里的绿色dns，这样就可以让vpn客户端透明fq了。

比如我家里旁路由上有绿色DNS，所以我可以令ocserv.conf下发它的IP地址作为客户端使用的DNS服务器地址（家里的192.168.2.201运行着绿色DNS，然后vpn client可以通过隧道访问到它）：

在ocserv.conf中指定dns服务器地址，下发给vpn client：

dns = 192.168.2.201

这样vpn client就会请求192.168.2.201进行域名解析，得到无污染IP。

理解上述原理需要对路由表、netfilters有清晰认识，否则可能比较困难，希望对你有点帮助。

智能分流

https://github.com/don-johnny/anyconnect-routes

国内IP不进行代理，参考如下开源项目：

https://github.com/CNMan/ocserv-cn-no-route

遇到的问题

1. 对于 ubuntu 22.04 系统，目前（2023.09.01) 系统自带的 ocserv 1.1.3 会在启动之后崩溃，出现如下日志：

解决此问题的方法就是安装 ocserv 1.1.6以及以后的版本，可以从源代码编译安装。这里有个简单的解决方法，就是从 ubuntu 23.04 的安装源中下载已经编译好的安装包，直接进行安装，如下：

2. 对于腾讯云购买的海外 轻量应用服务器 需要在管理控制台配置防火墙规则，放开端口访问，否则无法进行正常的通讯访问。

参考链接

UWB中TOF测距法的公式推导

UWB中TOF测距法的公式推导

UWB常用测距方法有两种：飞行时间测距法（TOF）和到达时间差法（TDOA）。这里说一下TOF。

TOF

飞行时间法（Time of Flight，TOF）是一种双向测距技术，它通过测量UWB信号在基站与标签之间往返的飞行时间来计算距离。根据数学关系，一点到已知点的距离为常数，那么这点一定在以已知点为圆心，以该常数为半径的圆上。有两个已知点，就有两个交点。以三个已知点和距离作三个圆，他们交于同一个点，该点就是标签的位置。

TOF定位方式需要基站和标签往返通信，因此就造成了TOF功耗大大提高，续航时间相对较短.

TOF又分为两种：单边双向测距和双边双向测距。

单边双向测距

单边双向测距(Single-sided Two-way Ranging: SS-TWR)是对单个往返消息时间上的简单测量，设备A主动发送数据到设备B，设备B返回数据响应设备A。如下图所示：

单边双向测距的流程是这样的：设备A（Device A）主动发送（TX）数据，同时记录发送时间戳，设备B（Device B）接收到之后记录接收时间戳；延时 $T_{reply}$ 之后，设备B发送数据，同时记录发送时间戳，设备A接收数据，同时记录接收时间戳。

所以可以拿到两个时间差数据，设备A的时间差 $T_{round}$ 和设备B的时间差 $T_{reply}$ ，最终得到无线信号的飞行时间 $T_{prop}$ 如下：

$ T_{prop} = \frac{1}{2}(T_{round}-T_{reply}) $

两个差值时间都是基于本地的时钟计算得到的，本地时钟误差可以抵消，但是不同设备之间会存在微小的时钟偏移，假设设备A和B的时钟偏移分别为 $e_A$ 和 $e_B$ ，则飞行时间测量值为：

$ \hat{T}_{prop} = \frac{1}{2}[T_{round}(1+e_A)-T_{reply}(1+e_B)] $

于是测距误差如下：

$ Error = \hat{T}_{prop} - T_{prop} = \frac{1}{2}(T_{round}\cdot e_A-T_{reply}\cdot e_B) = \frac{1}{2}T_{reply}(e_A-e_B) + T_{prop}\cdot e_A $

因为 $ T_{reply} >> T_{prop} $ , 所以可以忽略后一项，得到

$ Error = \hat{T}_{prop} - T_{prop} \approx \frac{1}{2}T_{reply}(e_A-e_B) $

由此可以看出，随着 $T_{reply} $ 和时钟偏移的增加，会增加飞行时间的误差，从而使得测距不准确。因此单边双向测距（SS-TWR）并不常用，但对于特定的应用，如果对于精度要求不是很高，但是需要更短的测距时间可以采用。注意 $ T_{reply} $ 不仅仅是设备B接收到发送的时间，也包括装载数据和发送数据耗费的时间（UWB除了支持定位之外，也可以传输数据，标准可以装载128字节，扩展模式可以装载1024字节数据）。

双边双向测距

双边双向测距（Double-sided Two-way Ranging）是单边双向测距的一种扩展测距方法，记录了两个往返的时间戳，最后得到飞行时间，虽然增加了响应的时间，但会降低测距误差。

双边双向测距分为两次测距，设备A主动发起第一次测距消息，设备B响应，当设备A收到数据之后，再返回数据，最终可以得到如下四个时间差：$ T_{round1} $ 、$ T_{reply1} $ 、$ T_{round2} $ 、$ T_{reply2} $ ，如下图所示：

双边双向测距飞行时间计算方法：

由单边双向测距方法可得

$ T_{prop} = \frac{1}{2}(T_{round1}-T_{reply1}) $

$ T_{prop} = \frac{1}{2}(T_{round2}-T_{reply2}) $

所以

$ \begin{flalign}T_{round1} \times T_{round2} = (2T_{prop}+T_{reply1})(2T_{prop}+T_{reply2}) = 4T_{prop}^2+2T_{prop}(T_{reply1}+T_{reply2})+T_{reply1}T_{reply2} \end{flalign}$

$ \begin{flalign} T_{round1} \times T_{round2} - T_{reply1}T_{reply2} = 4T_{prop}^2+2T_{prop}(T_{reply1}+T_{reply2}) \\ = T_{prop}(4T_{prop}+2T_{reply1}+2T_{reply2}) \\ = T_{prop}(T_{round1} + T_{round2} + T_{reply1} + T_{reply2}) \end{flalign}$

于是可以得到如下计算 $ T_{prop} $ 的公式：

$ T_{prop} = \frac{T_{round1} \times T_{round2} - T_{reply1} \times T_{reply2}}{T_{round1} + T_{round2} + T_{reply1} + T_{reply2}} $

以上测距的机制是非对称的测距方法，因为他们对于响应时间不要求是相同的。下面分析双边双向测距飞行时间的误差：

$ \begin{flalign} \hat{T}_{prop} = \frac{T_{round1}(1+e_A) \times T_{round2}(1+e_B) - T_{reply1}(1+e_B) \times T_{reply2}(1+e_A)}{T_{round1}(1+e_A) + T_{round2}(1+e_B) + T_{reply1}(1+e_B) + T_{reply2}(1+e_A)} \\ = \frac{(4T_{prop}^2+2T_{prop}(T_{reply1}+T_{reply2}))(1+e_A)(1+e_B)} {4T_{prop}+2(T_{reply1}+T_{reply2})+(2T_{prop}+T_{reply1}+T_{reply2})(e_A+e_B)}\\ =\frac{2(1+e_A)(1+e_B)}{(1+e_A)+(1+e_B)}T_{prop} \end{flalign}$

于是

$ T_{prop} = \frac{(1+e_A)+(1+e_B)}{2(1+e_A)(1+e_B)}\hat{T}_{prop} $

$ \begin{flalign} Error = \hat{T}_{prop} - T_{prop} = \left(1-\frac{(1+e_A)+(1+e_B)}{2(1+e_A)(1+e_B)}\right)\hat{T}_{prop} \\ = \frac{e_A+e_B+2e_A e_B}{2(1+e_A)(1+e_B)}\hat{T}_{prop} \end{flalign} $

因为 $ e_A <<1 $，$ e_B<<1$，略去高次项，可得

$ Error \approx \frac{e_A+e_B}{2}\hat{T}_{prop} $

由此可以看出，误差仅与钟漂和飞行时间有关。

假设一个使用场景：使用20ppm的晶体，UWB的工作距离范围为300m，则无线信号空中飞行时间大概为$1 \mu s$，误差为$ 20 \times 10^{-6} \times 1 \times 10^{-6} = 20 \times 10^{-12} = 20ps $时钟误差是在ps级别的，换算为距离之后仅为6mm。

注意：响应时间是不需要相等的，也就是 $ T_{reply1} $ 不一定要等于 $ T_{reply2} $ ，这样对于MCU系统的处理带来了很多便利。

若双边双向测距方法响应时间对称，也就是 $ T_{reply1} $ 和 $ T_{reply2} $ 相等，飞行时间计算方法如下：

$ T_{prop} = \frac{1}{4}(T_{round1}-T_{reply1}+T_{round2}-T_{reply2}) $

这种方法比较简单，只是需要一些时间戳做加减法，但其难点在于，怎么保证 $T_{reply1} $ 和 $ T_{reply2} $ 是相等的。

此种方法的误差分析如下：

$ \begin{flalign} \hat{T}_{prop} = \frac{1}{4}\left[T_{round1}(1+e_A)-T_{reply1}(1+e_B)+T_{round2}(1+e_B)-T_{reply2}(1+e_A)\right] \end{flalign}$

$ \begin{flalign} Error = \hat{T}_{prop} - T_{prop} = \frac{1}{4}\left[(T_{round1}-T_{reply2})e_A +(T_{round2}-T_{reply1})e_B\right] \\ =\frac{1}{4}\left[2(e_A+e_B)T_{prop} +(e_A-e_B)(T_{reply1}-T_{reply2})\right] \end{flalign} $

因为 $ T_{reply1}-T_{reply2} >> T_{prop} $ ，可忽略$ T_{prop} $项，从而得到

$ Error \approx \frac{1}{4}(e_A-e_B)(T_{reply1}-T_{reply2})$

可见此误差与响应时间差成正比。

参考链接

使用SSH Key访问服务器

1. 配置

生成的 SSH Key 默认存储在 ~/.ssh 目录下，可以使用 ls ~/.ssh 查看之前是否已经生成过 SSH Key，如果提示 No such file or directory 可以使用如下指令用于生成 SSH Key：

接着会询问保存文件的位置以及是否要设定 passphrase，如果设定了 passsphrase 那么每次使用该 SSH Key 的时候都需要输入这个 passsphrase，可以根据自己对安全性的需求设定，空白表示不设定。

如果采用默认的设置，那么会在 ~/.ssh 路径下生成两个 Key，一个私钥 id_rsa ，另一个公钥 id_rsa.pub，私钥需要好好保管。

2. 使用 ssh-copy-id

如果操作系统中有 ssh-copy-id，那么可以直接使用以下命令设置：

  • username：连接服务器的用户名
  • server：服务器的域名或者 ip 地址
  • ~/.ssh/id_rsa.pub：默认的公钥地址，如果修改过 SSH Key 存储地址，请填写对应地址

3. 复制公钥到服务器的 authorized_keys 中

上述命令用于在远程服务器上创建 ~/.ssh 文件夹并用本机的公钥创建服务器上的 ~/.ssh/authorized_keys 文件，接着设置权限：

4. 验证

经过上述的配置之后，可以再次进行 SSH 连接验证配置是否生效：

如果不需要输入密码就能够连接服务器，说明设置生效。

参考链接

使用 SSH Key 访问服务器

ubuntu 22.04 WARNING: [pool ***] server reached max_children, consider raising it

最近 WordPress 上传图片的时候，偶尔会发生失败，报错 503。以前比较少，最近越来越频繁，于是跟踪了一下服务器的日志，看到如下报错信息：

查看 PHP 的日志，却找不到任何有用的日志，只看到如下警告：

起初是百思不得其解，按理说应该有相关报错才对。

后来仔细思考了一下，才想明白，原来是并发数超过最大限制了，默认的并发数量是 5，这个绝对是不够的，并发超过限制是没有任何报错信息的，日志只会给出一个警告。

详细的解决方案如下：

If you encounter such an error in the php logs, then it is time to modify max_children. The php logs are usually located in /var/log/,  and in my case they are in /var/log/php7.4-fpm.log

Search for the file where the max_children variable is located, in my case the file is www.conf and is located in the folder: /etc/php/7.4/fpm/pool.d/

Change the value pm.max_children = 8 to a larger one, 16 for example

save the .conf file and restart the web server:

or only the php-fpm service as follows:

Eventually you can check these logs, who knows, you may not even know you have such a problem!

** As I noticed, for HestiaCP (VestaCP) users max_children can be found in both the /etc/php/7.4/fpm/pool.d/www.conf file and the file /usr/local/hestia/php/etc/php-fpm.conf

** Attention, you must also have enough memory for this, otherwise the results are not as expected!

Check the memory used by each php-fpm service-child with the following command:

Then, to see the memory required for these processes, we need to add the rss parameter which is “Resident Set Size” or the physical memory. Note: Modify php-fpm7.4 to suit your version of PHP:

These numbers are in Kbytes , so we calculate an average of the displayed values and multiply by max_children, and we find approximately the memory needed only for php, but don’t forget, the server also needs memory for Apache, Nginx, mysql … maybe Varnish, HAproxy, Redish, etc. .. so, check these values carefully!

参考链接

Solved! WARNING: [pool ***] server reached max_children, consider raising it

ubuntu 22.04 Warning: apt-key is deprecated

在尝试安装 nvidia-docker 的时候（ubuntu 22.04系统Docker和Nvidia-docker的安装、测试，及运行GUI应用），执行如下命令增加 nvidia 源：

系统发出如下警告：

经过分析，是脚本中的如下部分导致：

解决方法为修改为使用如下脚本：

完整修改后的脚本如下：

参考链接

Warning: apt-key is deprecated (SOLVED)

ubuntu 22.04系统Docker和Nvidia-docker的安装、测试，及运行GUI应用

快速搭建所需开发环境

Docker文档：https://docs.docker.com/，Docker安装指南： Install Docker Engine on Ubuntu

Dokcer安装

Docker测试

其他Docker命令：

Tips：Docker中一般Crtl+C退出，传送门：停止、删除所有的 docker 容器和镜像。

Nvidia-docker安装

查看nvidia版本

参考链接：官网 installation guide
Github：NVIDIA/nvidia-docker

测试

Docker 容器 GUI

若遇到X Error时，添加参数：--ipc=host 或 --env="QT_X11_NO_MITSHM=1"，参考链接：
How to fix X Error: BadAccess, BadDrawable, BadShmSeg while running graphical application using Docker?
Docker: gazebo: cannot connect to X server
若遇到 libGL error: No matching fbConfigs or visuals found libGL error... ，参考链接：
使用docker时出现libGL error: No matching fbConfigs or visuals found libGL error: failed to load driver...
已成功测试上述链接中的 pull image 方式
使用nvidia-smi查看nvidia driver和cuda版本，根据 nvidia/cudagl ，选择合适的TAG

创建新的长期镜像：

如果遇到如下报错：

上述报错目前没有找到很好的解决方法，应该是某个安装包意外修改了系统配置，导致出现问题，重装系统可以顺利解决此问题。

参考链接