A little while ago, in a previous role, an engineer posted a question in Slack about using AWS KMS to encrypt data. They’re a bright and thoughtful person, and their product manager had set a requirement to ensure the data was encrypted at rest at the data level. I was really pleased to see this being discussed as a non-functional requirement, especially given the sensitivity of the data. This kind of thinking is crucial when considering defence-in-depth.

However, when it came to implementation, the engineer found that AWS KMS didn’t seem to be working as expected—encryption and decryption were taking longer than anticipated. After a quick refresher on envelope encryption with AWS Encryption SDK Guide, I joined the discussion. It struck me that this exact question had come up in previous roles too. To ensure I—and others—have something to refer back to in the future, I decided to document it properly.

What is AWS KMS and the AWS Encryption SDK?

Straight from the horse’s mouth - AWS KMS is a AWS managed service that makes it easy for you to create and control the encryption keys that are used to encrypt your data. The AWS KMS you create are protected by FIPS-140-3 Security Level 3 hardware security modules (HSM), which is the cryptographic standard approved by the US Government.

When encrypting data, it’s essential to secure the encryption key used to generate the ciphertext. If that key is encrypted, the key protecting it must also be safeguarded, and so on. Eventually, this chain leads back to the root key, which never leaves AWS KMS. This is critical because if the root key were ever compromised, all other encryption keys derived from it would be compromised as well.

Lastly, we should quickly talk about KMS key types, of which there are three:

• Customer Managed Keys - This key type is the de facto choice for customers that want full control of the usage and lifecycle policy. This means that the customer is responsible for setting rotation, deletion and regional location of keys. Auditing/logging is available for CMK’s via AWS CloudTrail or Event Data Store. You pay a monthly fee for the existence of keys (pro-rated hourly), and also the customer is charged for key usage.

• AWS Managed Keys - This key type is a KMS key that exists in your account, but can only be used under certain circumstances. Specifically, it can only be used in the context of the AWS service you’re operating in and it can only be used by principals within the account that the key exists. AWS managed keys are a legacy key type that is no longer being created for new AWS services as of 2021. You get the same auditing/logging capabilities as customer managed keys, but a slight difference in that there is no monthly fee; but the caller is charged for API usage on these keys. The main key difference between AWS managed keys and customer managed keys, is that AWS Managed keys manage rotation, deletion and regional location etc. This significantly reduces the overhead of managing keys, so you can just concentrate on encrypting the data and forget about the management of the root keys.

• AWS Owned Keys - We mentioned the discontionuation of AWS Managed Keys, because the new key on the block (sorry I couldn’t resist!) is AWS Owned Keys. So, what do they do? An AWS owned key is a KMS key that is in an account managed by the AWS service, so the service operators have the ability to manage its lifecycle and usage permissions. By using AWS owned keys, AWS services can transparently encrypt your data and allow for easy cross-account or cross-region sharing of data without you needing to worry about key permissions. This is an incredibly important feature of using AWS Owned Keys, which are effectively AWS Managed Keys but with reduced blast radiuses. Exclusively controlled and only viewable by the AWS service that encrypts your data, and you also lose auditing/logging capabilities, for now. AWS service manages rotation, deletion, and Regional location, exactly as AWS Managed Keys do. You pay no charges for using AWS Owned keys which is by far one of the main advantages of using them!

The last thing we need to quickly discuss in this section is what the AWS Encryption SDK is, and envelope encryption. Straight from the horse’s mouth again, the AWS Encryption SDK is a client-side encryption library designed to make it easy for everyone to encrypt and decrypt data using industry standards and best practices. It enables you to focus on the core functionality of your application, rather than on how to best encrypt and decrypt your data. The AWS Encryption SDK is provided free of charge under the Apache 2.0 license. This is a nice move by AWS, but it is within their best interest to give their customers access to such functionality, under the shared responsibility model.

The security of your encrypted data depends in part on protecting the data key that can decrypt it. One accepted best practice for protecting the data key is to encrypt it. To do this, you need another encryption key, known as a key-encryption key or wrapping key. The practice of using a wrapping key to encrypt data keys is known as envelope encryption. See the images below for a visual explanation on enevelope encryption:

Multiple wrapping keys can be used to encrypt the same data key, which adds some really good fault tolerance/disaster recovery controls:

Ok, this wraps up all that I wanted to cover about KMS and Enevelope Encryption for now. The AWS documentation is fantastic for further reading if you really want to know more. Now we will run through an example in Python, and call it a day.

Lets play with Envelope Encryption

First of all, we need to import a bunch providers and helpers from the aws cryptographic material providers sdk.

import boto3
from aws_cryptographic_material_providers.mpl import AwsCryptographicMaterialProviders
from aws_cryptographic_material_providers.mpl.config import MaterialProvidersConfig
from aws_cryptographic_material_providers.mpl.models import CreateAwsKmsKeyringInput
from aws_cryptographic_material_providers.mpl.references import IKeyring
from typing import Dict 

import aws_encryption_sdk
from aws_encryption_sdk import CommitmentPolicy

Next, we can create a global variable just purely as an example, with a byte string value. Lets also create the SDK+KMS clients and the encryption context:

EXAMPLE_DATA: bytes = b"Hello KMS Learners"


def encrypt_and_decrypt_with_keyring(kms_key_id: str):
    client = aws_encryption_sdk.EncryptionSDKClient(
        commitment_policy=CommitmentPolicy.REQUIRE_ENCRYPT_REQUIRE_DECRYPT
    )

    kms_client = boto3.client('kms', region_name="us-west-2")

    encryption_context: Dict[str, str] = {
        "encryption": "example context",
        "is not": "secret",
        "but adds": "useful metadata",
        "that can help you": "be confident that",
        "the data you are handling": "is what you think it is",
    }

We need to create the KMS keyring next:

material_provider: AwsCryptographicMaterialProviders = AwsCryptographicMaterialProviders(
        config=MaterialProvidersConfig()
    )

    keyring_input: CreateAwsKmsKeyringInput = CreateAwsKmsKeyringInput(
        kms_key_id=kms_key_id,
        kms_client=kms_client
    )

    kms_keyring: IKeyring = material_provider.create_aws_kms_keyring(
        input=keyring_input
    )

Next up we want to encrypt our example data, and do a quick assertion to confirm its now ciphertext (encrypted message):

ciphertext, _ = client.encrypt(
        source=EXAMPLE_DATA,
        keyring=kms_keyring,
        encryption_context=encryption_context
    )

    assert ciphertext != EXAMPLE_DATA, \
        "Ciphertext and plaintext data are the same. Invalid encryption"

And lastly, do a quick decryption test:

plaintext_bytes, _ = client.decrypt(
        source=ciphertext,
        keyring=kms_keyring,
        # Provide the encryption context that was supplied to the encrypt method
        encryption_context=encryption_context,
    )

    assert plaintext_bytes == EXAMPLE_DATA, \
        "Decrypted plaintext should be identical to the original plaintext. Invalid decryption"

This playground example is just demonstrating a single key keyring, have a look at discovery_multi_keyring for multi key keyrings.

Letssss Goooo

Thank you for reading this post and hopefully you found some value in it! Some quick takeaway points:

• We learnt what KMS and the AWS Encryption SDK is, as well as Envelope Encryption.

• We learnt about the KMS key types, and a little bit of information on each type.

• You can have multiple wrapping keys for one data key, which is a really nice fault tolerance feature in the event of a disaster.

• Do you still need to think about things like salt for additional hashing of ciphertext? Yes, I highly recommend adding these additional layers of defence in. Remember, your security is only as good as the layers of controls you put in-place to protect your data.

• I do recommend thinking about using Customer Managed Keys over AWS Managed/Owned keys, and also thinking about multi-region and multiple wrapping keys. Think about how to reduce blast radiuses in the event of a major security incident.

Thanks all, and see you next time!

tech, devops, aws, security, software_engineering

Introduction

You know the score by now and I won’t delve into a full explanation or analogy of observability engineering again (for a refresher, please check out the first post in the series), but let’s quickly recap. Observability engineering involves measuring the internal states of a system or application by examining its outputs. It’s straightforward – no secret sauce, and no hocus pocus! In this series, we’ll cover important topics within observability engineering that should benefit both newcomers and seasoned engineers alike.

I enjoy having my opinions challenged and changed through healthy discussion.

Pillar Three - Tracing

What is tracing?

In modern distributed systems, particularly those built on microservices or serverless architectures, different services often need to interact with each other to fulfil a single user request. This interconnectedness makes it incredibly challenging to identify performance bottlenecks, diagnose issues, and analyse overall system behaviour. The difficulty is amplified when these services span multiple domains and are managed by different teams.

Consider a simple example of an online bookstore. Zod, the senior engineer on the orders team, notices that requests are timing out. His team is responsible for the orders microservice, which interacts with the inventory microservice to check the stock of an item during checkout. After receiving the stock information, the orders service places an order by sending another request to the inventory service. The inventory service then contacts the logistics service to get an estimated delivery time. However, unbeknownst to Zod and his team, the logistics service recently made some changes to their backend queries and inadvertently missed some crucial index updates in their NoSQL database. As a result, queries to the logistics service’s database are taking much longer than usual, ultimately causing the orders service to time out because it can’t provide the user with a delivery ETA.

I know, I know—it’s clear I’ve never worked in the orders or logistics domains before, but you get the point. In distributed systems supported by multiple teams, often across different domains, pinpointing the root cause of an issue like this would be nearly impossible without tracing. In my opinion, of the three main pillars of observability, tracing is the most crucial.

The anatomy of traces and spans

To keep it very simple, a trace consists of a series of interconnected spans. Each span represents an individual operation or activity within a specific service or component, ie. a database query like SELECT product_description, product_stock_count FROM Inventory WHERE product_id="xxxxx". The crucial piece of information about tracing and the passing of requests is that when a request enters a service or component, the trace context is propagated along with the request. This usually involves injecting trace. headers (including the trace_id) into the request, allowing downstream services to participate in the same trace.

Right, I am going to do my obligatory thing in this series and mention OpenTelemetry again. Deal with it! But seriously, the easiest and most standardised way of getting started is to pick an OpenTelemetry APM and off you go. Most modern vendors (ie. Datadog, Sumologic, Honeycomb) will provide exporters and well tested documentation.

Let’s quickly cover the make up of spans, the core component of a trace. A span represents an operation (or a unit of work) in a trace. A span could be a database query, or an in-process function call, or even a remote procedure call (RPC). A span has all of these things:

• A span name (operation name).

• A parent span.

• A span kind.

• A start and end time.

• A status that reports whether operation succeeded or failed.

• A set of key-value attributes describing the operation.

• A timeline of events.

• A list of links to other spans.

• A span context that propagates trace ID and other data between different services.

A trace is a tree of spans that shows the path that a request makes through an app. The root span is the first span in a trace. An example:

Now, time for some sage advice. All OpenTelemetry backends use span names and some attributes to group similar spans together. To group spans properly, I highly recommend giving them short and concise names. You should aim to have less than 1000 unique span names, for performance reasons. Let’s look at some good and bad span names:

Good

Bad

Lastly on spans, every span must have a kind/type, and it must be one of these values:

• server for server operations, for example, HTTP server handler.

• client for client operations, for example, HTTP client requests.

• producer for message producers, for example, a Kafka producer.

• consumer for message consumers and async functions, for example, a Kafka consumer.

• internal for internal operations.

…also, spans must have a status code of one of the following values:

• ok - success.

• error - failure.

• unset - the default value which allows the backends to assign the status.

Some additional features of traces and spans

Attributes

If you wish to record contextual information, you can annotate spans with attributes that carry information specific to the operation. Let’s give a basic example such as a HTTP endpoint, which may have attributes like http.method = GET and http.route = /orders/:id.

You have the freedom to name attributes as you want, but for common operations you should use the OpenTel semantic attributes convention.

Events

You have the option to annotate spans with events that have a start time and an arbitary number of attributes. The main difference between events and spans is that events don’t have an end time (and therefore no duration).

Events can usually represent exceptions, errors, logs, and messages (such as in RPC), but you can also create custom events if you so wish. For example, you may have a telemetry wrapper that your engineeing team uses to annotate spans in a standardised way, including sending your logs via tracing as well.

The observant of you may remember back in my first post in this series, you will note that the first pillar of observability is on Logs. So, if you can send event-logs via spans in a trace, why do you need logs as a seperate pillar? Well, the simple answer is, you probably don’t. However, not all systems architecture is created equally, and through acquisitions or divergent paths in your tech choices, you may still want to collect logs from different components. So, if your tech stack is fairly simple, and you just instrument your code to send traces to your backend/observability platform of choice, then you can likely just stick with this approach.

Context

Context is an important feature of spans. The span context carries information about the span as it propagates through different components and services in the tree.

The trace/span context is a request-scoped data object such as:

• Trace ID. A globally unique identifier that represents the entire trace or query. All spans within a trace have the same trace ID.

• Span ID. A unique identifier for the specific span within a trace. Each span within a trace has a different span ID.

• Trace flags. Flags that indicate various properties of the trace, such as whether it’s sampled or not. Sampling refers to the process of determining which spans should be recorded and reported to the observability backend.

• Trace State. An optional field that contains additional vendor or application-specific data related to the trace.

The span context is incredibly important for maintaining the continuity and correlation of spans within a distributed system. It allows different services and components to associate their spans with the correct trace and provides true end-to-end visibility into the flow of requests or transactions. The span context is typically propagated using headers or metadata of the communication protocols between services, similar to how baggage data is propagated, which we will cover in a minute. This is to make sure that when a service receives a request, it can extract the span context, and associate the ingress span with the correct trace.

You can use data from the context for spans correlation or sampling. For example, you can use the trace_id to know which spans belong to which traces, which is obviously incredibly important during troubleshooting or sampling!

Lastly on context, read up on context propagation from the OpenTelemetry docs here. There is a section on supported serialisation and deserialisation protocols on that page as well, which is helpful!

Baggage

We all come with baggage, and fortunately so does tracing! Baggage allows you to propgate custom key:value pairs (attributes) from one service to another. The example on the OpenTelemetry documentation is fantastic, so I won’t give another - just have a read.

What should we instrument then and how?

You really do not need to instrument every operation in your code to get the most out of tracing - it would be very time consuming and it’s not really necessary, or even valuable for your observability practices. Consider prioritising these operations:

• Network operations, for example, HTTP requests or RPC calls.

• Filesystem operations, for example, reading/writing to files.

• Database queries which combine network and filesystem operations.

• Errors and logs, for example, using structured logging, which from my first post in this series.

Ok, so now a quick example in Golang before the wrap up!

Step 1. Let’s instrument the following example function where we are inserting a new order:

func insertOrder(ctx context.Context, order *Order) error {
  if _, err := db.NewInsert().Model(order).Exec(ctx); err != nil {
    return err
  }
  return nil
}

Step 2. Let’s wrap the operation with a span:

import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("app_or_package_name")

func insertOrder(ctx context.Context, order *Order) error {
  ctx, span := tracer.Start(ctx, "insert-order")
  defer span.End()

  if _, err := db.NewInsert().Model(order).Exec(ctx); err != nil {
    return err
  }
  return nil
}

Step 3. Let’s record errors and set a status code:

import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("app_or_package_name")

func insertOrder(ctx context.Context, order *Order) error {
  ctx, span := tracer.Start(ctx, "insert-order")
  defer span.End()

  if _, err := db.NewInsert().Model(order).Exec(ctx); err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
    return err
  }
  return nil
}

Step 4. We should also record some contextual information with attributes:

import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("app_or_package_name")

func insertOrder(ctx context.Context, order *Order) error {
  ctx, span := tracer.Start(ctx, "insert-order")
  defer span.End()

  if _, err := db.NewInsert().Model(order).Exec(ctx); err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
    return err
  }

  if span.IsRecording() {
        span.SetAttributes(
            attribute.Int64("endorder.id", order.ID),
            attribute.String("endorder.description", order.Description),
        )
    }

  return nil
}

Let’s wrap it up

I hope you have found this blog post helpful, and if anything, you have some takeaway pointers to use in your observability practices. Let’s do a quick a list of the most important takeaways from this post:

• Use OpenTelemetry for your Tracing! You will not regret it one bit and most repetable vendors support OpenTel standards, which is fantastic!

• You probably won’t need logs as well as traces! As mentioned in the events section above, you can send event-logs in your span payloads, which means you won’t need to send log’s seperately. This is why I will always recommend collaborating inside your engineering team to come up with solid observability patterns, and even wrapper libraries to help with consistency across all of your services and systems.

• Context is really important when your request paths traverse many different services and components within your distributed architecture. Always populate the context and remember to check out the supported standards that OpenTel offers!

• Only instrument key components! We covered a database insert in our example, which is considered a key component to the service/system.

• Use Semantic Conventions! When you add instrumentation to your code, it is important to follow semantic conventions. This means using standardised attribute names, span names, and span tags as all defined within the OpenTel specifications. Doing so ensures consistency and interoperability across different instrumentation libraries and backends.

Thanks again for reading and I hope you’re looking forward to the next blog post in this series, which will be about centralised observability platforms (also labelled as backends).

Introduction

I won’t delve into a full explanation or analogy of observability engineering again (for a refresher, please refer to the first post in the series); however, let’s quickly recap. Observability engineering involves measuring the internal states of a system or application by examining its outputs. It’s straightforward – no secret sauce, and no hocus pocus! In this series, we’ll cover essential topics within observability engineering that should benefit both newcomers and seasoned engineers alike.

As always, my strong opinions are based on extensive experience, but I hold them lightly. I take great pride in my ability to change my views when presented with new information or different experiences.

Pillar Two - Metrics

What are metrics?

First, what are metrics? Metrics are simply the numeric representation of data measured over intervals of time. Using my pilot analogy from the first post, a commercial pilot needs real-time metrics to make informed decisions throughout a flight. This data is crucial for the pilot’s situational awareness. If it were in log format, the pilot wouldn’t have time to sift through detailed logs to get the necessary information, which could be disastrous!

For the aeroplane manufacturer, having historical numeric data points is essential for hypothesis-based investigations. For instance, if a new engine version is introduced, manufacturers might hypothesise that these engines run significantly hotter than previous models. By analysing historical data points, they can mark when the new engines were installed and validate or refute their hypothesis. Metrics might not explain why the engines run hotter, but they provide a clear picture of when and what.

The incredible value of metrics

My first piece of advice is, if anyone ever tells you that metrics are unnecessary in modern observability engineering, take a deep breath and firmly disagree. While it might not warrant drastic actions (unless they’ve committed equally heinous acts like stealing your lunch from the office fridge), it’s essential to remind them of the immense value numerical data points provide for single-pane-of-glass troubleshooting. Metrics offer an additional data dimension, giving you immediate, data-based insights.

For instance, imagine a service or application reporting numerous errors early on a Monday morning at 9am. Your service might emit a metric through a framework like python_gc_objects_uncollectable_total, and your cloud provider might collect a memory_utilised metric. These numerical data points are unique to metrics and cannot be easily derived from other observability pillars.

Now, consider Bob, the senior engineer, strolling into the office at 10am with his frappamochachino iced coffee and a croissant from his favourite bakery. Bob recalls a Python version bump on Friday afternoon, “but it had been tested and approved,” he mentions, mid-bite. What if Bob wasn’t there to provide that critical piece of information? No worries – you’re a super observability engineer who thought ahead and emitted a metric indicating the Python version of your service, such as python_version. Brilliant!

With this metric, you can easily correlate the version bump with the rise in uncollectable garbage collection objects due to increased request throughput on Monday morning. The team can then roll back the Python version, averting any major issues. Disaster mitigated.

I acknowledge this is a basic example, and something you might also catch in your logs, like SystemOutOfMemory exceptions. However, not all systems or services operate at the same scale. Engineers shouldn’t have to hunt for a needle in a haystack. So, always strive for clarity and precision in your observability practices.

Get collecting

Great stuff! We have covered how important metrics are, but what is the anatomy of a modern metric?

First of all, my strong opinion here is that you should pick OpenTelemetry to instrument and collect your metrics. Trust me and save a significant amount of time by making that choice early on. The OpenTel standards for metrics collection are supported by most major centralised observability platforms. We will cover centralised observability platforms in a future blog post.

The OpenTel metrics data model structure can be found here. OpenTelemetry has three models:

• The Event model, in which you instrument metrics as the software engineer

• The Stream model, which OpenTelemetry uses for transport

• The Timeseries model, which OpenTelemetry uses for storage

The OpenTelemetry metrics are constructed by using the global MeterProvider to create a Meter, and associating it with one or more instruments. An instrument is a specific type of metric (e.g., a counter, gauge, histogram) that you use to collect data about a particular aspect of your service or application’s behaviour. You capture measurements by creating instruments that are comprised of:

• A unique name, for example, http.proxy.request.duration

• An instrument type, for example, Histogram

• An optional unit of measure, for example, milliseconds or bytes

• An optional description for the instrument

A single instrument can produce multiple timeseries. A timeseries is a metric model with a unique set of attributes. For example, let’s say you have a Kubernetes cluster; each host in the cluster has a separate timeseries for the same metric name.

It’s very important to mention additive instruments at this stage. Additive or summable instruments produce timeseries that, when added up together, produce another meaningful and accurate timeseries. Additive instruments that measure non-decreasing numbers are also called monotonic. For example, http.server.requests is an additive timeseries because it can be summed from multiple hosts to get the actual total number of requests from your service if you load balance requests, which you should be doing, of course! There are also synchronous instruments, which are invoked together with the operations they are measuring. For example, +1 to a counter when a request is fired off for your service or application. Lastly, there are also asynchronous instruments, which periodically invoke a callback function to collect measurements. Asynchronous instruments are also known as observers, and observers can be used to periodically measure things like system memory or CPU usage.

When to use what type and some examples!

Some simple guidance on when to use what:

1. If you need to measure request_latency or maybe request_size, pick a Histogram

2. If you need to measure things like processed_requests, errors, received_bytes, disk_reads, pick a Counter if the value is monotonic. Otherwise, use UpDownCounter as your instrument type

3. If you need to measure things like cpu_time, memory_usage_bytes, memory_utilisation_percentage, if the value is additive/summable and if the value is monotonic, use CounterObserver; otherwise, use UpDownCounterObserver. Lastly, if the value is NOT additive/summable, use the GaugeObserver type.

Lets make this fun by doing a few examples! I’ve picked golang in the following examples, but there are well supported SDK’s/API’s in most popular programming languages. We want to measure the number of requests our startup app sends to social media API’s, so we should pick a Counter, and increment whenever a request is sent:

import "go.opentelemetry.io/otel/metric"

socialMediaAPIRequestCounter, _ := meter.Int64Counter(
	"some.prefix.api.requests",
	metric.WithDescription("Number of sent Social Media API requests"),
)
// Your secret sauce code does things here
socialMediaAPIRequestCounter.Add(ctx, 1)

simple example, but you can see we are incrementing the counter each time our secret sauce code runs. Lets do another very quick example with a Histogram:

import "go.opentelemetry.io/otel/metric"

opHistogram, _ := meter.Int64Histogram(
	"some.prefix.process.image.duration",
	metric.WithDescription("Duration of image enhancement"),
)

t1 := time.Now()
op(ctx)
dur := time.Since(t1)

opHistogram.Record(ctx, dur.Microseconds())

That example is showing the time it takes to do an image enchancement in our CSI: New York image enchancement app.

Lets cover one last example of cache hit rates, because we are using CounterObserver which if we remember correctly is used to instrument monotonic numbers that are additive/summable:

import "go.opentelemetry.io/otel/metric"

counter, _ := meter.Int64ObservableCounter("some.prefix.request.cache")

// Arbitrary key/value labels.
hits := []attribute.KeyValue{attribute.String("type", "hits")}
misses := []attribute.KeyValue{attribute.String("type", "misses")}
errors := []attribute.KeyValue{attribute.String("type", "errors")}

if _, err := meter.RegisterCallback(
	func(ctx context.Context, o metric.Observer) error {
		stats := cache.Stats()

		o.ObserveInt64(counter, stats.Hits, metric.WithAttributes(hits...))
		o.ObserveInt64(counter, stats.Misses, metric.WithAttributes(misses...))
		o.ObserveInt64(counter, stats.Errors, metric.WithAttributes(errors...))

		return nil
	},
	counter,
); err != nil {
	panic(err)
}

We should also note that CounterObserver is an asynchronous instrument, and that means it will periodically invoke a callback function to collect measurements. They were very basic examples that I have provided, but hopefully in the future I can link a repository with some better and more fleshed out real-world examples.

The last step is to choose a backend to send your timeseries metrics to, but as mentioned previously we will cover centralised observability platforms in a later post.

Wrap Up Part Deux

A recap of the advice and guidance I have given in this post:

• Metrics are incredibly useful, and don’t let anyone tell you otherwise. I can share countless stories on the importance of having dashboarded data points during late-night troubleshooting sessions while on-call.

• Choose OpenTelemetry to model, transport, and store your metrics. You can’t go wrong by making this choice early on.

• Learn the basics of the anatomy of an OpenTel metric, and familiarise yourself with the instrument types to help you make informed decisions on which instruments to pick for your requirements.

Thank you so much for reading, and as always, I hope this post was helpful! Keep an eye out for my next post on Tracing and APM!

Introduction

So, what exactly is observability engineering? Simply put, it’s the ability to monitor, measure, and understand the state of a system or application through its external outputs.

Let’s use an analogy to illustrate. Consider a commercial aircraft. An aeronautical engineer relies on data from the aircraft to make informed maintenance decisions, such as monitoring the oil pressure of the jet engines. Pilots, on the other hand, need real-time metrics like altitude and cabin pressure to ensure safe flights. All this data is displayed on a dashboard, providing a comprehensive view of the aircraft’s performance. Manufacturers also analyse this data to plan improvements. To put it in perspective, a single commercial aircraft can generate up to 20 terabytes of data per engine, per hour of flight – a staggering amount of information!

In the same way, observability is crucial in modern software engineering. Mastering it means you can confidently answer how your system or application is performing without constantly checking if it’s still running. Instead, you’ll have a robust alerting and incident response platform, giving you peace of mind. After all, you want to make sure you know about an issue before your customers do!

As for my background, I’ve been in the observability field for years, working with network devices, server hardware, monolithic applications, micro-services, and event-driven architecture. I’ve designed and implemented platforms capable of handling tens of thousands of data points per second. My experience spans creating “observability in a box” solutions, offering platform-as-a-product services that software engineering teams can easily integrate with. I bring a wealth of knowledge and strong opinions on the subject.

The Three Pillars of O11y - Logs

Warning, there be strong opionions here.

What are these three pillars of observability then? Great question, and this isn’t a conclusion and term that I have coined, this was created by super smart engineers in the observability space. I am only going to be covering logs in this post, because during the draft it was getting quite long and I want to make this a fairly digestable series.

Pillar One - Logs/Event-Logs

Wait a minute, why Logs and/or Event-Logs? Whats the difference? Dont panic, we will get into that in a minute or two. Essentially, logs are human-readable flat text files that are used by engineers to capture useful data about their systems/services. Log messages occur when a developer deems it important to tell the system or application owner that something happened that they should probably know about. For example, your service could be dropping requests, and you should probably know why sooner rather than later.

Let’s look at a good example of a log file:

[2021-02-23T13:26:23.505892 #22473] INFO -- : [6459ffe1-ea53-4044-aaa3-bf902868f730] Started GET "/" for ::1 at 2021-02-23 13:26:23 -0800

Source: The Path from Logs to Traces, by Alex Vondrak

The example log line starts with a timestamp and a PID (Process ID) [2021-02-23T13:26:23.505892 #22473], which is incredibly important for time-series investigation. If the logging entity has incorrectly configured time, then this data is effectively useless. Precision timing is crucial in modern software engineering. Please accept that as my first piece of sage advice - make sure you write timezone aware code, and the systems you host on are all connected to accurate centralised time protocol servers.

Next up we have the logging level, INFO in the example log file. Logging level basically means the “importance” of the log message to the system owner or operator. The logging level should ALWAYS be set at the application/service/runtime vars config stage, and the reason for that is that if you have to change it out in the wild for investigative purposes, ie. change to DEBUG level, you don’t want to have to trawl through code in the early hours of the morning and update everywhere where the logging level is set.

Next up we have something called a Universally unique identifier (uuid) v4 6459ffe1-ea53-4044-aaa3-bf902868f730, which is basically a randomly generated id to represent the request ID. This request ID is important, to help chain events/units of work together for a given request.

We have a GET request next, which is one of the HTTP verbs. It is useful to know the HTTP method used when making a request to a service or web app.

We then see a request path /. In this case, this is the root path of the application/service. We then see ::1 which is the host network address, an IPv6 localhost address.

Finally, we have the request start time with the timezone UTC offset 2021-02-23 13:26:23 -0800. This is also important to know when the request hit your service or web app, so that you can accurately piece together the events up until a given point.

This was just purely a random example of how an application or service might log a message. There are many other examples out there where a lot of the formatting choices are relatively sensible out-of-the-box. I give another piece of advice to always find a good logging library for your chosen programming language, unless the standard library logger is really good already. You shouldn’t want to reinvent the wheel for something like logging, just pick the most popular and easy to use open source solution. If you wan’t to make modifications, then fork it, and create a new package, but always inner source it for your engineering team to encourage internal contributions. For example, the software engineers in your org might want to add standardised fields, or even masking of data before it is sent out on the wire.

Now, some of the observant amongst you may have remembered that we were going to discuss the difference between logs and event-logs. Don’t Worry, I haven’t forgotten! I also don’t want to cover this topic in too much detail because otherwise the post will bloat quite significantly, but I will link you to some excellent views from the superstar of Observability Engineering, Charity Majors. Event-logs are just better than logs. Thats right, to the annals of history with you logs! “But before we stand on that hill with you, Mark, can you tell us what the difference is, please?” Alright, I hear you. Event-logs are structured logs, and they follow a standardised format (JSON), which makes it trivial to parse and query for most centralised observability platforms. Let’s take the log example from before and put that into an event-log:

{
  "name": "request",
  "timestamp": "2021-02-23T13:26:23.505892",
  "pid": 22743,
  "level": "info",
  "request_id": "6459ffe1-ea53-4044-aaa3-bf902868f730",
  "request.method": "GET",
  "request.path": "/",
  "request.ip": "::1",
  "request.start_at": "2021-02-23 13:26:23 -0800"
}

Source: The Path from Logs to Traces, by Alex Vondrak

I think you’ll agree it’s even easier to read in it’s raw format. Lastly on event-logs - always emit a single event per request per service that it hits. Use the Open Telemetry conventions and ALWAYS fire off an event before the request errors or exits the service, otherwise you have no breadcrumbs for your investigation! Also, if you have distributed systems, include a trace_id to pass onto other services in the stream.

Wrap Up

Ohh, that isn’t the last piece of advice regarding logs actually. My last piece of advice is to just avoid putting Personally Identifiable Information (PII) into your logs. You don’t need it. If you have an anonymized user id in your data model, you do not need to log PII. Your security/compliance team will keep their hair, and you will also be way happier and content. But lets say you must absolutely include the event payloads from publisher event buses in your event-logs, then, please please think about your event schemas. I will give a very basic example, but hopefully this highlights the point:

{
  "name": "request",
  "timestamp": "2021-02-23T13:26:23.505892",
  "pid": 22743,
  "level": "info",
  "request_id": "6459ffe1-ea53-4044-aaa3-bf902868f730",
  "request.method": "GET",
  "request.path": "/",
  "request.ip": "::1",
  "request.start_at": "2021-02-23 13:26:23 -0800"
  "request.payload": {
    "event": {
      "user.data" : {
        "id": "d6279b68-3460-4799-b26d-ea87a865f7fc"
        "private" {
          "full_name": "Joe Bloggs"
          "email": "joe.bloggs@example.com",
          "address": "1 Mount Olympus"
        }
      }
    }
  }
}

With the field request.payload, and the full field path of request.payload.event.user.data.private I now know everything under that path is private user data, and I can filter/mask that at ingestion time. This is why engineering standardisation is so important in modern engineering orgs. You should agree with your fellow engineering community how to design your schemas to avoid problems later down the line, where mitigating PII data leaking into your observability platforms will be very difficult, and will force you down paths you will not want to go. I am talking reducing data retention periods to 30 days of warm data kinda craziness.

Let me provide a quick summary of all the sage advice and tips we have covered in this post:

• Be smart, be cool, and write timezone aware code that is hosted on infrastructure that have accurate time servers to hand.

• Always make sure you centralise your logging level settings, at the config or runtime variables stage.

• Always pick a good logging package or module for your chosen programming language. Open source solutions are perfect, and if you must customise it, then inner source so that you and your colleagues can modify as desired.

• Event-logs are structured logs and are just way better than bog standard logs. If you had to see some of the regex parsing patterns I have had to create over the years to get logs indexable in observability platforms, you will understand why I stand on that hill.

• Always emit a single event per request to your service before exiting or erroring, and stream over the wire to your observability platform of choice, using Open Telemetry Standards.

• Lastly, do not put PII in your logs. If you absolutely must, work with your fellow engineers to standardise your logging structure, and or your event schemas if you run Event Driven Architecture.

I hope I didn’t miss anything important about logging in the world of observability, but if you feel I did, please tweet me and we can chat (my handle is linked on the left). Thanks for reading!