By adopting cloud computing, us developers place trust in service providers that our code and data is not manipulated once deployed - but how can we make sure?

The rise of blockchain finance has shown that we are short one thing on the internet: trust. Although most of the time when we run code on cloud services (PaaS and “lower”) nothing happens, the incentives for attacking systems are growing. Stealing tokens, ransomware, or political interests promise great rewards - and injecting malicious code is becoming a more lucrative avenue (e.g. SolarWinds).

So … is there a way to encrypt memory content and even CPU instructions while making sure this encryption actually happened? Yes! Thanks to Intel’s (proprietary) Software Guard Extensions (or SGX). In this blog post we’ll cover the setup - and how you can use Rust to run code inside this secure enclave.

TL;DR

Back in November I gave a talk about this topic at a (remote) Rust meetup:

For those who like to read, please step into the walled garden:

Creating a true walled garden

Intel’s SGX technology has been available in Intel CPUs since late 2015 (Skylake) with the goal of protecting memory and instructions from the operating system (OS), hypervisor, BIOS, and remote attacks. Consequently, neither the OS nor a cloud service provider can see what or how a program is running. In addition to that, the enclave (i.e. the secure execution environment) provides a signature that can be verified using another enclave - or Intel’s attestation service API. This way, a developer can make sure that whatever is the result of this encrypted processing actually was created using a secure enclave.

In summary, this is what the enclave does (from SGX 101):

• Enclave memory cannot be read or written from outside the enclave regardless of the current privilege level and CPU mode.
• Production enclaves cannot be debugged by software or hardware debuggers.
• The enclave environment cannot be entered through classic function calls, jumps, register manipulation, or stack manipulation. The only way to call an enclave function is through a new instruction that performs several protection checks.
• Enclave memory is encrypted using industry-standard encryption algorithms with replay protection. Tapping the memory or connecting the DRAM modules to another system will yield only encrypted data.
• The memory encryption key randomly changes every power cycle. The key is stored within the CPU and is not accessible.
• Data isolated within enclaves can only be accessed by code that shares the enclave.

In a sentence, SGX is a technology that uses specialized CPU instructions and a portion of system memory to create an encrypted execution environment for individual function calls or entire programs; secured by a hardware key that is derived from an Intel master key. Thus - all trust is relayed to Intel, which has proven reasonable so far. If you want to dive deeper, check out these other resources.

All of that seems really practical - so who’s actually using SGX in production?

Real world examples

There are a few real-world users, most notably 1Password which uses Rust to store and retrieve your password from the enclave. Additionally, Azure Confidential Compute is a SGX-based confidential compute service that mentions the popular messenger Signal, MobileCoin, and Fortranix as customers.

Where the light doesn’t shine

Obviously, SGX is not a cure-all and has been breached in various ways in the past. Some of those breaches were fixed, wheras some of the threats can only be mitigated using other means. Additionally, SGX isn’t the only technology for running encryped workloads on CPUs - although it’s maybe the most advanced. The generic term for those technologies is Trusted Execution Environmend or TEE and AMD, as well as ARM have their own flavors.

Trusted execution in Rust

Intel’s SDK developer handbook for SGX is a whopping 400 pages and uses - unsurprisingly - C++. However, the people over at Baidu X-Labs have built a Rust SDK on top which makes everything a lot simpler. Having seen the public announcement of the Rust SDK at a RustFest, I had no idea what I was witnessing. Only years later (in 2020) I looked for the SDK only to find out it was now in Apache’s hands: https://github.com/apache/incubator-teaclave-sgx-sdk.

Setting everything up

Setting up SGX for development is a bit of a chore. To get started, you’ll need:

1. Linux (ideally Ubuntu, RHEL, Fedora, or CentOS)
2. An SGX compatible CPU (emulated SGX may work too)
3. Install SGX Drivers
4. Docker/any container engine to simplify compilation
5. The SGX SDK & its samplecode

First, follow Intel’s guide on checking your CPU for SGX support. If your CPU does support SGX, you have to enable it in the BIOS - check your mainboard manual for that.

Once the hardware questions are out of the way, install a driver. DCAP is used primarily for data center type applications (i.e. when you are running on Azure), so you can most likely ignore it. Most likely, you’ll need the regular driver.

After installing the driver, you should see a block device /dev/isgx (or /dev/sgx/provision and /dev/sgx/enclave with DCAP). With that available, check out the Docker setup. In essence, be sure to clone the git repository for the Rust SDK and then use:

$ docker run -ti --rm -v /path/to/sdk:/root/sgx \
             --device /dev/isgx \
             baiduxlab/sgx-rust
root@913e6a00c8d8:~#

Inside the container, make sure that aesm is running and its libraries are available every time you start it:

root@406ed34d2728:~# export LD_LIBRARY_PATH="/opt/intel/sgx-aesm-service/aesm"
root@406ed34d2728:~# /opt/intel/sgx-aesm-service/aesm/aesm_service 
...

Befor we jump into compiling Rust code, let’s also look at what Rust version we will be running:

root@406ed34d2728:~# rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /root/.rustup

nightly-2020-10-25-x86_64-unknown-linux-gnu (overridden by '/root/sgx/rust-toolchain')
rustc 1.49.0-nightly (ffa2e7ae8 2020-10-24)

While 1.49 nightly is obviously old, it’s the currently supported version by the SDK. Some patches for memory allocation make updating to a newer version very hard, but is typically done within several months….

If all of the above worked as expected, and you have a shell open inside the container, let’s compile and run some sample code.

Running some code

Inside the container, you should be able to find the SDK (and it’s samplecode) in /root/sgx. Let’s check out the hello-rust example in /root/sgx/samplecode.

Each SGX Rust project has two parts:

• Trusted code in enclave
• Untrusted code in app

Let’s run tree quickly to go over the directory tree. The basic setup is two Rust projects that are built with a root Makefile and ultimately the enclave (trusted) code is linked to the app binary via build.rs:

root@406ed34d2728:~/sgx/samplecode/hello-rust# tree -L 3
.
|-- Makefile                  # No Cargo.toml at the root because it's not an actual workspace
|-- app
|   |-- Cargo.toml
|   |-- build.rs              # The build file links up the app and enclave projects
|   `-- src
|       `-- main.rs           # Untrusted source code
|-- bin
|-- enclave
|   |-- Cargo.toml            # Enclave-only dependencies go here. no_std ideally...
|   |-- Enclave.config.xml    # Some config like memory size and thread counts
|   |-- Enclave.edl           # Enclave definition file, where you define the interfaces for e(nclave) calls
|   |-- Enclave.lds
|   |-- Enclave_private.pem
|   |-- Makefile
|   |-- Xargo.toml            # not actually used
|   |-- src
|   |   `-- lib.rs            # Trusted source code
|   `-- x86_64-unknown-linux-sgx.json
`-- lib
    `-- readme.txt

The trusted code is built using a Makefile which results in a dynamic library (.so) that will be linked by the untrusted code. There are two major use cases:

1. Building the app
2. Running tests (in a future post)

By running the Makefile, it creates a library (the trusted enclave stuff) and a binary in the bin directory and both together are your application:

$ sudo docker run -ti -v (pwd):/root/sgx --device /dev/isgx baiduxlab/sgx-rust bash
root@406ed34d2728:~# cd sgx/samplecode/hello-rust
root@406ed34d2728:~/sgx/samplecode/hello-rust# ls
Makefile  app  enclave  lib
root@406ed34d2728:~/sgx/samplecode/hello-rust# make
/opt/sgxsdk/bin/x64/sgx_edger8r --trusted enclave/Enclave.edl --search-path /opt/sgxsdk/include --search-path ../../edl --trusted-dir enclave
/opt/sgxsdk/bin/x64/sgx_edger8r --untrusted enclave/Enclave.edl --search-path /opt/sgxsdk/include --search-path ../../edl --untrusted-dir app
GEN  =>  enclave/Enclave_t.c enclave/Enclave_t.h app/Enclave_u.c app/Enclave_u.h
CC   <=  enclave/Enclave_t.c
ar rcsD lib/libEnclave_u.a app/Enclave_u.o
    Updating git repository `https://github.com/apache/teaclave-sgx-sdk.git`
    Updating crates.io index
  Downloaded libc v0.2.94
  Downloaded 1 crate (511.7 KB) in 0.62s
   Compiling libc v0.2.94
   Compiling sgx_types v1.1.3 (/root/sgx/sgx_types)
   Compiling app v1.0.0 (/root/sgx/samplecode/hello-rust/app)
   Compiling sgx_urts v1.1.3 (/root/sgx/sgx_urts)
    Finished release [optimized] target(s) in 1m 03s
Cargo  =>  bin/app
mkdir -p bin
cp ./app/target/release/app ./bin
make -C ./enclave/
make[1]: Entering directory '/root/sgx/samplecode/hello-rust/enclave'
cargo build --release
 ...
cp ./target/release/libhelloworldsampleenclave.a ../lib/libenclave.a
make[1]: Leaving directory '/root/sgx/samplecode/hello-rust/enclave'
CC   <=  enclave/Enclave_t.c
LINK =>  enclave/enclave.so
mkdir -p bin
<!-- Please refer to User's Guide for the explanation of each field -->
<EnclaveConfiguration>
    <ProdID>0</ProdID>
    <ISVSVN>0</ISVSVN>
    <StackMaxSize>0x40000</StackMaxSize>
    <HeapMaxSize>0x100000</HeapMaxSize>
    <TCSNum>1</TCSNum>
    <TCSPolicy>1</TCSPolicy>
    <DisableDebug>0</DisableDebug>
    <MiscSelect>0</MiscSelect>
    <MiscMask>0xFFFFFFFF</MiscMask>
</EnclaveConfiguration>
tcs_num 1, tcs_max_num 1, tcs_min_pool 1
The required memory is 1736704B.
The required memory is 0x1a8000, 1696 KB.
Succeed.
SIGN =>  bin/enclave.signed.so

The Makefile also runs a cargo release build on the app project creating a runnable binary of the program. This binary is stored alongside the enclave library in the bin/ subfolder (they need to be in the same place) and when running the ./app binary …

root@1ad4435d0e32:~/sgx/samplecode/hello-rust# cd bin
root@1ad4435d0e32:~/sgx/samplecode/hello-rust/bin# ls
app  enclave.signed.so

… the program shows the expected output!

root@1ad4435d0e32:~/sgx/samplecode/hello-rust/bin# ./app
[+] Init Enclave Successful 2!
This is a normal world string passed into Enclave!
This is a in-Enclave Rust string!
[+] say_something success...

⚠️ A major source of errors (for me at least): something in the default setup requires the working directory to be the bin directory and to start the app binary from there. Additionally, the aesm (from above) has to be run beforehand in the same session - otherwise the output will be a generic “SGX not found” error in both cases…

This is it

To recap, this post has shown four things:

1. A brief overview over SGX
2. Directory layout of Rust apps for SGX
3. Environment setup
4. building (and running) SGX apps

You can take a plunge into Intel’s developer zone, or SGX101 to learn more; but a future post here will show how and where to write Rust code for SGX. 🦀