Kubernetes on Oracle Cloud...for free

Oracle Cloud Infrastructure (OCI) may not spring to mind if you are thinking of hosting in the cloud, and it definitely won't come to mind if you are thinking about free hosting. However you might be surprised to learn that OCI has a very generous free tier that can be used to run a fully functioning Kubernetes cluster in the cloud for absolutely nothing. The Always Free tier includes the following resources:

• 4 OCPUs for A1.Flex instances
• 24GB of memory
• 200GB of block volumes
• A Flex load balancer with 10 Mbps traffic allowance

Full list of Always Free resources here: https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm

In this post I'll show you how to deploy a Kubernetes cluster in OCI using these free resources, and get started running workloads on it. And it won't cost you a penny*.

*assuming you don't make any mistakes!

What you will get

• An OCI account in your datacentre location of choice
• An Oracle Kubernetes Engine (OKE) control plane
• Four worker nodes, each with 1 OCPU (ARM64 compatible), 6GB of memory and a ~50GB boot volume
• A bill of $0 per month

What you should be aware of

One caveat is that you will need to provide a credit card for billing, and you will have to convert your account from a Free Trial account to a Pay As You Go (PAYG) account for this to work. Stay within the Always Free limits and you won't be charged, but you still need to provide payment details.

A technical caveat is that the instance nodes are based on Ampere A1 ARM CPUs, which means they are not x86 compatible. For most workloads this shouldn't be a problem, but you may come across some applications that do not support ARM64. If you are planning to develop your own applications to run on this cluster, you will of course have to build your containers with ARM64 support.

Sign up for a Trial Account

Head to https://signup.cloud.oracle.com/and fill in your details. Note that the "Country/territory" field does not define which datacentre your OCI assets will be created, that comes later - at this time it's just so they know what currency to bill you in.

Once you've filled in these details and verified your email address, then you'll be able to specify your Cloud Account Name and Home Region. You'll probably want to specify your customer type as "Individual". Note the current limitation around A1 Compute availability in some regions - this matters to us as these are the instance types we will be using:

Next you'll have to fill in your contact details including address and phone number, and you'll also have to provide credit card details - unfortunately this bit isn't optional. There is a "test charge" of about 80p ($1) that is charged to your card, but it should be immediately refunded.

Once the process is completed, you should have access to your brand new OCI Free Tier account. The free tier gives you access to lots of services with a $300/£250 budget to spend in your first 30 days. Feel free to try some of these out, but know that to use OKE you must convert your account to Pay As You Go - and when you do so, you will lose any entitlement to the remaining credit. So if you start to spend these credits, make sure you delete any associated services before converting to PAYG just to be sure that you don't get any unexpected charges.

Sign in to OCI and get yourself familiar with the platform and the resources available before jumping to the next step.

Convert your trial to PAYG

Converting to PAYG is pretty straightforward. Within the OCI navigation go to "Billing and Cost Management", and then under "Billing" go to "Upgrade and Manage Payment". There you can select the PAYG or Corporate options - you want to pick PAYG. Once you do so, additional services become available to you, including Oracle Kubernetes Engine (OKE).

Create an OKE cluster

Now we are ready to create our cluster. In the main menu, search for "Kubernetes Clusters (OKE)" and click that option. You should be taken to the main 'Clusters' page where you can create your new cluster:

Click the "Create cluster" button to start a new OKE cluster. You'll be asked whether you want to use "Quick create" or "Custom create". Choose "Quick":

Next we begin to define our cluster. Specify:

• Name - a name for your cluster (e.g. cluster1)
• Compartment - the compartment within your OCI account where it will be defined (you should only have one option here)
• Kubernetes version - the K8S version you want. Unless you definitely need a specific version, just go for the latest - 1.26.2 at the time of writing

• Private or Public API - the API is how you interact with K8S and tell it to run your workloads. A private API endpoint is potentially more secure, but you will need to connect into your OCI account via a jump box to access it. A public endpoint is potentially less secure but much easier to work with from your desktop. With those caveats in mind, for this guide we will select "Public endpoint" and receive a public IPv4 address where we interact with our cluster.
• Managed or Virtual Nodes - we want managed nodes so we have dedicated compute instances for our cluster.
• Private or public workers - this option allows us to specify whether our worker nodes have public IPs. There is no need for any worker nodes to have these, we will use other techniques (Load balancers, ingresses and Cloudflare tunnels) to expose our services to the world, so choose "Private".

• Node shape - you must choose the "VM.Standard.A1.Flex" option to stay within the Always Free criteria. This shape refers to the Ampere A1 instances that OCI specifies within Always Free - any other shape will probably incur charges.
• Number of OCPUs - for our 4-node cluster, specify 1 OCPU (effectively a 1-core CPU)
• Amount of Memory - for our 4-node cluster, specify 6GB of RAM.

• Image - our only option here is Oracle Linux, so just choose the latest version (Image build 2023.06 30-0 at time of writing).
• Node count - choose 4 for a 4-node cluster.
• Advanced options - the basic options are enough for us, but the Advanced options allow us to specify different Boot volume sizes (default 46.6GB which fits in the 200GB Always Free limit for a 4-node cluster), volume encryption with custom keys, Image verification, and SSH keys for authentication to worker (which we shouldn't need). Peruse these but stick to the defaults unless you know you need something different.

Having configured all these settings, click next to move to the review screen. Here, you will see an option to select creating a "Basic" cluster. Make sure you select this option. Basic clusters are free, Enhanced clusters are subject to a daily charge of about $2/£1.50:

An enhanced cluster in OKE has more integrations that can be used, and it also has a more intelligent way of handling updates to the worker nodes (e.g. to K8S version 1.27 when its available for OKE). But you can get by with the basic option if you're prepared for some annoying aspects of the cluster upgrade process -  more on this in a future post.

Finally, select "Create cluster" and wait for your new services to be spun up. After 10-15 minutes your cluster should be created with your 4 nodes spun up and ready for workloads:

Click on Node Pools, then "pool1", and you should see your 4 nodes listed by IP address:

Our cluster is now running, so let's get connected to it.

Connect to your cluster

Most Kubernetes clusters provide you with a "kubeconfig" file which contains information about the cluster such as name and IP address, as well as a client authentication certificate you can use to connect to the API. OKE clusters are no different, but somewhat annoyingly Oracle require you to connect via the Oracle Command-Line Interface. Without this installed on your client machine, you cannot interact with the cluster.

Installation instructions for OCI on various operating systems can be found here:

• Windows/Mac/Linux: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm
• Docker: https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/clicontainer.htm

Personally I like the Docker option with an alias set so that I can run the oci command natively from my terminal.

Once you have installed the CLI, then you need to authenticate. Open your CLI and enter the command:

oci session authenticate

You'll be asked for the region your tenancy resides in - enter the corresponding number into the CLI, and you will be given a URL to open in your browser through which you can authenticate with your credentials. Once you do so, your CLI should be authenticated and you can issue commands.

Finally, obtain the kubeconfig file from the OCI portal by going into your newly created cluster details and clicking the "Access Cluster" button.

You will see instructions for the Cloud Shell and Local Access. Choose Local Access and follow the instructions to download the kubeconfig file via the OCI CLI and setting the KUBECONFIG environment variable so that kubectl can use it. The commands should look something like this:

oci ce cluster create-kubeconfig --cluster-id ocid1.cluster.oc1.uk-london-1.<clusterid> --file $HOME/.kube/ociconfig --region uk-london-1 --token-version 2.0.0  --kube-endpoint PUBLIC_ENDPOINT

export KUBECONFIG=$HOME/.kube/ociconfig

Once you've done this, then you should be able to use kubectl to interact with your cluster:

Success!

Is it really free?

The short answer is yes. As long as your are careful to stay within the limits of the Always Free resources you should not see any charges. The longer answer is that it's possible to get some charges that might be avoidable. Let me explain.

Several new versions of Kubernetes have been released since I originally built the cluster, so I've needed to run upgrades. There is no in-place upgrade of worker nodes, you simply one-click upgrade your control plane to the latest, one-click upgrade your worker node pool to the same. To upgrade the workers you effectively just delete each node and OKE automatically provisions a replacement, which will have the new version of K8S. Repeat the process for all 4 nodes and eventually you have a set of new worker nodes all on the latest version.

The problem I've encountered is that sometimes the boot volumes of worker nodes sometimes fail to get the "Always Free" tag applied, which means there is a charge for the storage. This is typically on the order of a few cents per day so not exactly bank-breaking, but it is an unexpected charge. The fix is fairly simple, just delete the node again and OKE will automatically recreate it, and this time the "Always Free" tag is properly applied.

That being said all my upgrade experience was using a "Basic" OKE cluster, the new "Enhanced" clusters have a more automated upgrade experience. When the next version of K8S is available in OKE I will document the process of upgrading a basic cluster and add a post to this series.

As evidence of the costs, here is a report of my bills for the last two years:

Notice that for most months there is no invoice. The months where there is a charge are when I hit the upgrade billing issue and didn't catch it for a day or two.

In Conclusion...

Now you should have all you need to go ahead and build your own cluster on OCI. In future posts I'll explain how to add some storage into the cluster (using Longhorn) and various other fun shenanigans but for now you can take a closer look at your new toy and run some stateless workloads at a minimum.

I've been running small workloads on an OKE cluster for about 2 years now, and it's been relatively trouble free. Given the performance constraints of single-threaded worker nodes and the small load balancer available, my experience has been good and the service has performed well. Reliability has also been good, I've had no Oracle-imposed downtime since I built this cluster, and any loss of service has been down to my own mistakes.

Aside from the minor upgrade/billing problem, I don't have anything too negative to say about OCI in general or OKE in particular. I've even been surprised how few sales calls/emails I get, I just have occasional emails from an Oracle customer success rep who offers to set up some facetime. All-in-all it's been a surprisingly painless experience - not something you can often say about Oracle!

I will extend this series by writing guides about how to get more use out of your free OCI resource. I intend to cover:

• How to expose services to the Internet using load balancers, ingresses, and tunnels
• How to use Longhorn for in-cluster storage and File Storage for backups
• How to use OCI Vault for secrets management
• How to upgrade a cluster

Plus anything else I can think of. Stay tuned!

The Good

• Generous free tier
• True cloud-based Kubernetes cluster
• Reliable and performant (in my experience)

The Bad

• The need to install the Oracle CLI client to use basic tools like kubectl
• ARM64 machines can be a bit limiting in terms of which workloads you can run
• Storage limitations can restrict you, particularly if you need to run stateful workloads and you don't have any other storage options
• Oracle Linux would not be my first choice for worker OS - but that's your only option

The Ugly

• Unexpected bills, particularly when upgrading cluster worker nodes. Not huge amounts, but annoying nevertheless
• It's Oracle, so who knows when the rug will be pulled from under you!