I want to use this blog post to cover everyone’s least favourite subject, documentation, but hopefully in a helpful enough way that you can solve the problem of documentation in your tech org for good. Ohh, and the title of this blog post is a tongue-in-cheek borrow of the quote from Shakespeare’s Richard III, and if I remember correctly, used for the Horseback Riding technology breakthrough in Sid Meier’s Civilization 5 game, narrated by Leonard Nimoy.

Documentation Types

So, what exactly do we mean by documentation? There are various types of documentation, including Request for Comments (RFCs), Architecture Decision Records (ADRs), Runbooks, and product support documentation. I won’t be covering more product-focused documentation like Product Requirements Documents (PRDs), or even engineering team-type documentation like team canvases, team social contracts, or team interfaces, etc. I will keep it more technically focused for now.

Let’s get stuck in.

Request for Comments (RFCs)

Request for Comments (RFCs) have been around since the early days of tech, and were created by Steve Crocker whilst working on the development of ARPANET. Steve published the first RFC in 1969, so I think you’ll agree, this has been around for a while and probably serves some usefulness. The Internet Engineering Task Force (IETF) have published many RFC’s since, covering protocols such DNS, DHCP etc. RFCs are meant to address real problems, with a supplied proposal in the format of a standard. That proposal should then undergo discussion with comments from your colleagues/peers/leadership. Everyone who has a question or a piece of feedback should be welcome to chime in. I have found over the years that healthy tech teams regularly participate in community discourse and support each other as we learn and grow together.

Should I use AI to create an RFC document? No. No, you shouldn’t. RFC’s should carry your style, your opinions, and your individuality. If you use AI to create an RFC, then you’re not opening up a healthy opportunity for the rest of the team to give you valuable feedback — and that’s the point.

So, what should an RFC look like? The format below is merely a guide based on things I have picked up and contributed to over the years:

RFC - <insert title>

—----------------------------------------------------------------------------------------------------------------------------

ABSTRACT

This “abstract” section should provide a synopsis of the problem so that reviewers/stakeholders can quickly determine whether their feedback or opinions might be relevant to this RFC.

PROBLEM STATEMENT

This “problem statement” section should include a detailed description of the problem and an explanation of why it’s essential to present a proposal for it.

WHY NOW (Optional)

This “why now” section can include further details of why the defined problem should be tackled now. This might be a detailed business or technical case.

PROPOSAL

This “proposal” section should contain a single detailed proposal to solve the defined problem. If you have more than one proposal, it probably means you weren’t ready to write an RFC yet. You need to make sure you have done the due diligence and dismissed other potential proposal candidates.

SCOPE (Optional)

This “scope” section can include a scope of systems or processes that this proposal is intended for. For example, you may have a tagging standard for AWS, but your team also operates in GCP, but you only want to standardise AWS resource tagging for now.

APPENDIX

This “appendix” section should contain any relevant resources or supporting information that is relevant to the described problem, and provided proposal. Ie. reading materials from SaaS or Cloud vendors.

A simple example to help show the value of an RFC:

RFC - AWS Account+Root Email Naming Convention

—----------------------------------------------------------------------------------------------------------------------------

ABSTRACT

We will soon be creating an entirely new AWS org, and creating multiple accounts within that org. We do not currently have an account naming convention, and this is causing inconsistencies, which is making a bad developer experience. We should also define a root account email structure process to create a seamless experience when managing new AWS accounts and storing the never-to-be-used root account credentials in our password vault.

PROBLEM STATEMENT

Right now we have naming inconsistencies across new AWS accounts, existing AWS accounts, acquisitions etc. This creates a confusing environment for engineers with access to multiple accounts. Furthermore, the admin/root email addresses are inconsistent, which only adds to the confusion.

WHY NOW (Optional)

If we define this standard now, we can use it in our automation and manual AWS account creation process to avoid adding further inconsistencies in our new AWS environments.

PROPOSAL

AWS Account Names Convention

acme-<product_or_initiative>-<optional_workload>-<env>

The regex pattern:

^(acme)-{1}([a-z]+)-?([a-z\-]+)?-?-{1}(dev|staging|prod|sandbox)$

Matching examples:

acme-example-inventory-dev

acme-example-admin-dashboard-dev

acme-data-migration-sandbox

acme-security-tooling-mgmt-prod

AWS Account Root Email Address Convention

We are proposing using a single Google Workspace email group to maintain and manage our root account email addresses.

This email group will only be accessible by Platform Engineering.

aws-admin+<product_or_initiative>-<optional_workload>-<env>@acme.net

The regex pattern:

^(aws-admin)\+{1}([a-z]+)-?([a-z\-]+)?-?-{1}(dev|staging|prod|sandbox)(@{1}acme\.net)$

Matching examples:

aws-admin+data-migration-sandbox@acme.net

aws-admin+example-admin-dashboard-prod@acme.net

aws-admin+example-inventory-prod@acme.net

SCOPE

The scope is only for AWS accounts and root account email aliases.

APPENDIX

https://docs.aws.amazon.com/organizations/latest/userguide/best-practices_member-acct.html#bp_member-acct_define-acct

And that’s it! A simple example to show the art of the possible.

RFCs should be maintained in a version-control system, such as Notion, Confluence, or a git repo (though sometimes you may have non-technical reviewers who might struggle with git). You should have the RFC template as an RFC itself, in case you want to add new sections or change the template in the future.

Technical or non-technical stakeholders can create RFCs if they have a standard that they would like to define. Standards are critical in the world of tech, because the alternative is infinite combinations and permutations of implementations that become extremely difficult to change later down the line. Let’s use the example that you have some compliance work on the horizon, where the controls dictate that all databases should be encrypted at rest. If you didn’t implement a standard early on, you may have production databases that aren’t encrypted at rest. This becomes a nightmare to fix now because there was no standard to follow earlier.

In the new fangled world of AI there is even the opportunity to take an RFC, and create a pipeline where you build codified compliance policies to enforce your standard, using tools like OPA, Cloud Custodian and Prowler to provide guardrails to your product engineering pipelines. I am working on another blog post to cover this exact topic in the near future, with examples!

Ok, so I hope you understand RFCs and how they will help you define standards in your tech org, and encourage healthy discourse from your lovely engineers.

Architecture Decision Records (ADRs)

Architecture decision records (ADRs) are an ideal tool for documenting system and service decisions. They capture all relevant concerns and consequences, and each entry undergoes full peer review before being added to the log.

Where RFCs describe holistic, tech-org-wide standards, ADRs are for a specific service/system. ADRs should be stored as close to your project code repositories as possible. Creating a docs folder in your repo, with an ADR subfolder, is an excellent standard to follow (but you can write an RFC for that, remember!)

So, what should an ADR look like? Again, this is merely an example based on experience over the years:

ADR-XXX: Title

Status: Draft, In Discussion, Decided, Implemented, Abandoned, Superseded

Authors:

Decision Date:

Context / Problem

// Define the problem and explain the context driving the decision. What was the need? Don’t write war and peace; keep it short and succinct, linking all relevant materials.

Decision & Rationale

// Explain your decision in one sentence and then explain your rationale in bullet point format

(Expected or Known) Consequences

// Explicitly list down the consequences. Also, what about the consequences of consequences (second-order thinking)?

Considered Options (and why they haven’t been chosen?)

// List down all the other options you considered with their pros/and cons. For each, explain with one sentence why you didn’t pick them.

Appendix / References

// Add additional materials that can give more context or are relevant to the decision.

Update Log:

// After the ADR moves out of Draft status, log changes here. A good analogy for this log is having them as meaningful commit messages.

Example Log:

2025-10-01: Moved the ADR to In Discussion

2025-10-10: Added an extra considered option: XYZ

2025-10-12: Abandoned the ADR. The problem is invalid; no solution is needed.

Let me provide an example to show how this looks out in the wild:

ADR-001: Adopt Static Site Generator for Company Blog

Status: Implemented

Authors: Sarah Chen (Lead Developer), Marcus Thompson (Content Lead)

Decision Date: 15 March 2025

Context / Problem

Our current WordPress blog requires constant security patches, plugin updates, and dedicated hosting infrastructure costing £200/month. The marketing team spends roughly 3 hours per week dealing with performance issues and spam. We need a more maintainable solution that reduces operational overhead whilst improving site performance for our growing audience.

Related: Performance audit report, Security incident log Q4 2024

Decision & Rationale

We will migrate our company blog to Astro, a static site generator, hosted on Cloudflare Pages.

Performance: Static sites deliver pre-rendered HTML, eliminating database queries and dramatically improving page load times (estimated 3x faster)

Security: No server-side code execution means virtually no attack surface for common exploits

Cost reduction: Hosting costs drop from £200/month to approximately £15/month, a 92% saving

Developer experience: Component-based architecture allows us to use React components where needed whilst maintaining excellent performance

Content workflow: Integration with our existing GitHub workflow means content can be version-controlled and reviewed via pull requests

Maintenance burden: Eliminates plugin updates, database maintenance, and reduces security patching to essentially zero

(Expected or Known) Consequences

Positive:

Significantly faster page loads improve SEO rankings and user experience

Reduced hosting costs free up budget for content creation tools

Version-controlled content prevents accidental deletions and provides full audit history

Development team can focus on features rather than maintenance

Negative:

Content editors must learn Markdown and basic Git workflows (estimated 2-day learning curve)

Dynamic features (comments, search) require third-party integrations rather than plugins

Migration effort estimated at 40 developer hours plus 20 hours for content team training

Second-order consequences:

Improved performance may increase organic traffic, requiring us to scale content production capacity

Version control workflow might slow down urgent content updates initially until team adjusts

Success could lead to requests to migrate other properties (documentation, knowledge base) to similar architecture

Considered Options (and why they haven’t been chosen?)

Next.js with ISR (Incremental Static Regeneration)

Pros: Familiar React ecosystem, flexible hybrid rendering, excellent developer experience

Cons: More complex infrastructure, higher hosting costs than pure static, overkill for our content-focused use case

Why not chosen: We don’t need the complexity of a full React framework when our blog is primarily static content

Hugo

Pros: Extremely fast build times, mature ecosystem, zero JavaScript by default

Cons: Go templating is less familiar to our team, limited component reusability, steeper learning curve

Why not chosen: Team familiarity with JavaScript ecosystem outweighs Hugo’s raw speed advantages

Continue with WordPress using managed hosting

Pros: Zero migration effort, familiar to content team, extensive plugin ecosystem

Cons: Doesn’t address core performance or cost issues, simply shifts maintenance burden to vendor

Why not chosen: Fails to solve the fundamental problems we’re experiencing with dynamic CMS architecture

Ghost CMS

Pros: Modern Node.js-based CMS, better performance than WordPress, cleaner interface

Cons: Still requires database and server infrastructure, doesn’t eliminate maintenance burden, subscription costs

Why not chosen: Keeps us in the dynamic CMS paradigm that’s causing our current issues

Appendix / References

Astro documentation

Migration plan and timeline

Content team training materials

Cloudflare Pages pricing

Lighthouse performance comparison

Update Log

2025-02-15: Moved the ADR to In Discussion

2025-02-22: Added Hugo as a considered option after team feedback

2025-03-01: Updated cost analysis with actual Cloudflare Pages pricing

2025-03-08: ADR approved; status changed to Decided

2025-03-15: Migration completed successfully; status changed to Implemented

2025-03-20: Added note about second-order consequence regarding scaling content production

The benefits of ADRs are not just for now; they are instrumental in the future as well. For example, if you have a new joiner on your team who is curious about how the team reached past decisions and would like more information, they can read the ADRs and logs to see the sequence of decisions. They could even point an AI agent at it to get a decision-tree output. This will allow them to paint a clear mental picture and get up to speed in record time.

Should you use AI to create your ADR documents? Maybe. Partially. At the bare minimum, you should make sure you have thoroughly reviewed the document to ensure it meets its intended purpose — a log of why and what.

As you can see, ADRs are extremely valuable and definitely worth introducing into your tech org if you haven’t already.

Runbooks

In the fast-paced world of IT operations and DevOps, having clear, actionable documentation can mean the difference between a swift resolution and hours of troubleshooting chaos. Enter the runbook: a detailed, step-by-step guide that documents routine operational procedures and responses to common incidents. Think of it as a recipe book for your infrastructure, where each recipe outlines precisely how to handle specific situations, from routine maintenance tasks to urgent production incidents.

Runbooks serve as the institutional knowledge of your organisation, capturing the expertise of your most experienced team members and making it accessible to everyone. They’re particularly valuable during high-pressure situations when stress levels are high, and memories might fail. By documenting procedures in advance, teams can respond consistently and efficiently, regardless of who’s on call or what time an incident occurs. This consistency not only reduces mean time to resolution (MTTR) but also helps prevent minor issues from escalating into major outages.

The beauty of a well-crafted runbook lies in its ability to democratise knowledge across your organisation. Junior engineers can confidently handle tasks that might otherwise require senior expertise, whilst experienced team members can focus on more complex challenges rather than repeatedly explaining routine procedures. Moreover, runbooks serve as living documents that evolve with your infrastructure, capturing lessons learnt from past incidents and continuously improving your operational resilience. Whether you’re managing cloud infrastructure, maintaining legacy systems, or orchestrating complex deployments, runbooks are an essential tool for building reliable, maintainable systems.

Let’s look at a template example for runbooks:

## Runbook Template

```markdown

# [Service/System Name] - [Procedure/Incident Type]

## Metadata

- **Last Updated:** [Date]

- **Owner:** [Team/Individual Name]

- **Severity Level:** [Critical/High/Medium/Low]

- **Estimated Time:** [Expected duration]

- **Required Access:** [List of systems, credentials, or permissions needed]

## Overview

[Brief description of what this runbook covers and when it should be used]

## Prerequisites

- [Required access level or credentials]

- [Necessary tools or software]

- [Environmental conditions or system states]

## Impact Assessment

**Affected Services:** [List of impacted services]

**User Impact:** [Description of how users are affected]

**Business Impact:** [Revenue, SLA, or other business implications]

## Detection and Diagnosis

### Symptoms

- [Observable symptom 1]

- [Observable symptom 2]

- [Observable symptom 3]

### Monitoring Alerts

- [Alert name/type that triggers this runbook]

- [Dashboard or monitoring tool to check]

### Diagnostic Steps

1. [Step to verify the issue]

```

[Command or action]

```

**Expected Output:** [What you should see]

**If Different:** [What it might indicate]

2. [Next diagnostic step]

## Resolution Steps

### Step 1: [Action Name]

**Description:** [What this step accomplishes]

```

[Command or detailed instructions]

```

**Verification:** [How to confirm this step succeeded]

**Rollback:** [How to undo this step if needed]

### Step 2: [Action Name]

**Description:** [What this step accomplishes]

```

[Command or detailed instructions]

```

**Verification:** [How to confirm this step succeeded]

**Rollback:** [How to undo this step if needed]

[Continue for all steps...]

## Verification and Testing

1. [Check to confirm resolution]

2. [Test to ensure service is healthy]

3. [Monitor for [X] minutes to ensure stability]

## Communication Plan

- **Stakeholders to Notify:** [List of people/teams]

- **Communication Channels:** [Slack channel, email list, status page]

- **Status Update Template:** [Brief template for updates]

## Escalation Path

**If resolution fails after [X] attempts or [Y] minutes:**

1. Escalate to: [Next level contact/team]

2. Contact Method: [Phone/Pager/Slack]

3. Additional Context to Provide: [Specific information needed]

## Post-Incident Actions

- [ ] Update incident ticket with resolution details

- [ ] Document any deviations from this runbook

- [ ] Schedule post-mortem if severity warrants

- [ ] Update monitoring or alerting if needed

- [ ] Record incident in knowledge base

## Related Documentation

- [Link to architecture diagram]

- [Link to related runbooks]

- [Link to system documentation]

## Revision History

| Date | Author | Changes |

|------|--------|---------|

| [Date] | [Name] | [Description of changes] |

## Notes and Gotchas

- [Important caveats or common mistakes]

- [Known issues or limitations]

- [Tips from past experiences]

I won’t provide an example in this case because this blog post would otherwise become too long, but please reach out if you have any questions or would like to discuss examples.

Should you use AI to create runbook documents? Yes. Yes, you should. However, you absolutely must review the runbook and game it with your teammates. Runbooks should be Agentic-friendly as well, because you will likely be using AI-automation to run them.

You should make sure there are housekeeping rules stating that any changes in and around systems/services documented in runbooks must be updated.

Runbooks can be incredibly useful for teams working within a DevSecOps culture, where supporting a system/service requires a defined process. Runbooks can be in your documentation centre, in your git repository for the relevant system/service, or both.

Product support documentation

Product support documentation is the supporting documentation for your products. The why, what and how. This documentation type is for internal consumption only. You will likely have public-facing product support documentation in the form of user guides, workshops, etc. The product support documentation I am speaking of in this post is what you and your tech team use to understand a service/system and how it adds to a value stream.

I will not give a template for this documentation type, because this should be a standard that you or your team define, using an RFC.

Should you use AI to create product support documentation? Yes. Yes, you should. This is exactly the sort of thing AI is meant for. The product support documentation should also be Agentic-friendly, so you can retrieve summaries of what different products do within the value stream and how they all piece together.

Product support documentation should sit within a given system/service code repository, and it should be continuously synced to your documentation centre. The reason for keeping it in code is so that when your engineers are working on delivery, they also have a checklist item to remind their agent fleet to update any product support documentation with the applied changes.

Conclusion

To Summarise:

I really hope this post is helpful to the community. Documentation is not a particularly exhilarating subject, but an absolute necessity in the tech world.

Without documentation, we would not have the internet; we would not have the very devices you’re sitting down to read this post on — in fact, we wouldn’t have modern civilisation.

I will be sharing some agentic skills in the near future to help create runbooks, architectural decision records, etc.

If you have any comments or questions, please reach out to me.

A little while ago, in a previous role, an engineer posted a question in Slack about using AWS KMS to encrypt data. They’re a bright and thoughtful person, and their product manager had set a requirement to ensure the data was encrypted at rest at the data level. I was really pleased to see this being discussed as a non-functional requirement, especially given the sensitivity of the data. This kind of thinking is crucial when considering defence-in-depth.

However, when it came to implementation, the engineer found that AWS KMS didn’t seem to be working as expected—encryption and decryption were taking longer than anticipated. After a quick refresher on envelope encryption with AWS Encryption SDK Guide, I joined the discussion. It struck me that this exact question had come up in previous roles too. To ensure I—and others—have something to refer back to in the future, I decided to document it properly.

What is AWS KMS and the AWS Encryption SDK?

Straight from the horse’s mouth - AWS KMS is a AWS managed service that makes it easy for you to create and control the encryption keys that are used to encrypt your data. The AWS KMS you create are protected by FIPS-140-3 Security Level 3 hardware security modules (HSM), which is the cryptographic standard approved by the US Government.

When encrypting data, it’s essential to secure the encryption key used to generate the ciphertext. If that key is encrypted, the key protecting it must also be safeguarded, and so on. Eventually, this chain leads back to the root key, which never leaves AWS KMS. This is critical because if the root key were ever compromised, all other encryption keys derived from it would be compromised as well.

Lastly, we should quickly talk about KMS key types, of which there are three:

• Customer Managed Keys - This key type is the de facto choice for customers that want full control of the usage and lifecycle policy. This means that the customer is responsible for setting rotation, deletion and regional location of keys. Auditing/logging is available for CMK’s via AWS CloudTrail or Event Data Store. You pay a monthly fee for the existence of keys (pro-rated hourly), and also the customer is charged for key usage.

• AWS Managed Keys - This key type is a KMS key that exists in your account, but can only be used under certain circumstances. Specifically, it can only be used in the context of the AWS service you’re operating in and it can only be used by principals within the account that the key exists. AWS managed keys are a legacy key type that is no longer being created for new AWS services as of 2021. You get the same auditing/logging capabilities as customer managed keys, but a slight difference in that there is no monthly fee; but the caller is charged for API usage on these keys. The main key difference between AWS managed keys and customer managed keys, is that AWS Managed keys manage rotation, deletion and regional location etc. This significantly reduces the overhead of managing keys, so you can just concentrate on encrypting the data and forget about the management of the root keys.

• AWS Owned Keys - We mentioned the discontionuation of AWS Managed Keys, because the new key on the block (sorry I couldn’t resist!) is AWS Owned Keys. So, what do they do? An AWS owned key is a KMS key that is in an account managed by the AWS service, so the service operators have the ability to manage its lifecycle and usage permissions. By using AWS owned keys, AWS services can transparently encrypt your data and allow for easy cross-account or cross-region sharing of data without you needing to worry about key permissions. This is an incredibly important feature of using AWS Owned Keys, which are effectively AWS Managed Keys but with reduced blast radiuses. Exclusively controlled and only viewable by the AWS service that encrypts your data, and you also lose auditing/logging capabilities, for now. AWS service manages rotation, deletion, and Regional location, exactly as AWS Managed Keys do. You pay no charges for using AWS Owned keys which is by far one of the main advantages of using them!

The last thing we need to quickly discuss in this section is what the AWS Encryption SDK is, and envelope encryption. Straight from the horse’s mouth again, the AWS Encryption SDK is a client-side encryption library designed to make it easy for everyone to encrypt and decrypt data using industry standards and best practices. It enables you to focus on the core functionality of your application, rather than on how to best encrypt and decrypt your data. The AWS Encryption SDK is provided free of charge under the Apache 2.0 license. This is a nice move by AWS, but it is within their best interest to give their customers access to such functionality, under the shared responsibility model.

The security of your encrypted data depends in part on protecting the data key that can decrypt it. One accepted best practice for protecting the data key is to encrypt it. To do this, you need another encryption key, known as a key-encryption key or wrapping key. The practice of using a wrapping key to encrypt data keys is known as envelope encryption. See the images below for a visual explanation on enevelope encryption:

Multiple wrapping keys can be used to encrypt the same data key, which adds some really good fault tolerance/disaster recovery controls:

Ok, this wraps up all that I wanted to cover about KMS and Enevelope Encryption for now. The AWS documentation is fantastic for further reading if you really want to know more. Now we will run through an example in Python, and call it a day.

Lets play with Envelope Encryption

First of all, we need to import a bunch providers and helpers from the aws cryptographic material providers sdk.

import boto3
from aws_cryptographic_material_providers.mpl import AwsCryptographicMaterialProviders
from aws_cryptographic_material_providers.mpl.config import MaterialProvidersConfig
from aws_cryptographic_material_providers.mpl.models import CreateAwsKmsKeyringInput
from aws_cryptographic_material_providers.mpl.references import IKeyring
from typing import Dict 

import aws_encryption_sdk
from aws_encryption_sdk import CommitmentPolicy

Next, we can create a global variable just purely as an example, with a byte string value. Lets also create the SDK+KMS clients and the encryption context:

EXAMPLE_DATA: bytes = b"Hello KMS Learners"


def encrypt_and_decrypt_with_keyring(kms_key_id: str):
    client = aws_encryption_sdk.EncryptionSDKClient(
        commitment_policy=CommitmentPolicy.REQUIRE_ENCRYPT_REQUIRE_DECRYPT
    )

    kms_client = boto3.client('kms', region_name="us-west-2")

    encryption_context: Dict[str, str] = {
        "encryption": "example context",
        "is not": "secret",
        "but adds": "useful metadata",
        "that can help you": "be confident that",
        "the data you are handling": "is what you think it is",
    }

We need to create the KMS keyring next:

material_provider: AwsCryptographicMaterialProviders = AwsCryptographicMaterialProviders(
        config=MaterialProvidersConfig()
    )

    keyring_input: CreateAwsKmsKeyringInput = CreateAwsKmsKeyringInput(
        kms_key_id=kms_key_id,
        kms_client=kms_client
    )

    kms_keyring: IKeyring = material_provider.create_aws_kms_keyring(
        input=keyring_input
    )

Next up we want to encrypt our example data, and do a quick assertion to confirm its now ciphertext (encrypted message):

ciphertext, _ = client.encrypt(
        source=EXAMPLE_DATA,
        keyring=kms_keyring,
        encryption_context=encryption_context
    )

    assert ciphertext != EXAMPLE_DATA, \
        "Ciphertext and plaintext data are the same. Invalid encryption"

And lastly, do a quick decryption test:

plaintext_bytes, _ = client.decrypt(
        source=ciphertext,
        keyring=kms_keyring,
        # Provide the encryption context that was supplied to the encrypt method
        encryption_context=encryption_context,
    )

    assert plaintext_bytes == EXAMPLE_DATA, \
        "Decrypted plaintext should be identical to the original plaintext. Invalid decryption"

This playground example is just demonstrating a single key keyring, have a look at discovery_multi_keyring for multi key keyrings.

Letssss Goooo

Thank you for reading this post and hopefully you found some value in it! Some quick takeaway points:

• We learnt what KMS and the AWS Encryption SDK is, as well as Envelope Encryption.

• We learnt about the KMS key types, and a little bit of information on each type.

• You can have multiple wrapping keys for one data key, which is a really nice fault tolerance feature in the event of a disaster.

• Do you still need to think about things like salt for additional hashing of ciphertext? Yes, I highly recommend adding these additional layers of defence in. Remember, your security is only as good as the layers of controls you put in-place to protect your data.

• I do recommend thinking about using Customer Managed Keys over AWS Managed/Owned keys, and also thinking about multi-region and multiple wrapping keys. Think about how to reduce blast radiuses in the event of a major security incident.

Thanks all, and see you next time!

tech, devops, aws, security, software_engineering

Introduction

You know the score by now and I won’t delve into a full explanation or analogy of observability engineering again (for a refresher, please check out the first post in the series), but let’s quickly recap. Observability engineering involves measuring the internal states of a system or application by examining its outputs. It’s straightforward – no secret sauce, and no hocus pocus! In this series, we’ll cover important topics within observability engineering that should benefit both newcomers and seasoned engineers alike.

I enjoy having my opinions challenged and changed through healthy discussion.

Pillar Three - Tracing

What is tracing?

In modern distributed systems, particularly those built on microservices or serverless architectures, different services often need to interact with each other to fulfil a single user request. This interconnectedness makes it incredibly challenging to identify performance bottlenecks, diagnose issues, and analyse overall system behaviour. The difficulty is amplified when these services span multiple domains and are managed by different teams.

Consider a simple example of an online bookstore. Zod, the senior engineer on the orders team, notices that requests are timing out. His team is responsible for the orders microservice, which interacts with the inventory microservice to check the stock of an item during checkout. After receiving the stock information, the orders service places an order by sending another request to the inventory service. The inventory service then contacts the logistics service to get an estimated delivery time. However, unbeknownst to Zod and his team, the logistics service recently made some changes to their backend queries and inadvertently missed some crucial index updates in their NoSQL database. As a result, queries to the logistics service’s database are taking much longer than usual, ultimately causing the orders service to time out because it can’t provide the user with a delivery ETA.

I know, I know—it’s clear I’ve never worked in the orders or logistics domains before, but you get the point. In distributed systems supported by multiple teams, often across different domains, pinpointing the root cause of an issue like this would be nearly impossible without tracing. In my opinion, of the three main pillars of observability, tracing is the most crucial.

The anatomy of traces and spans

To keep it very simple, a trace consists of a series of interconnected spans. Each span represents an individual operation or activity within a specific service or component, ie. a database query like SELECT product_description, product_stock_count FROM Inventory WHERE product_id="xxxxx". The crucial piece of information about tracing and the passing of requests is that when a request enters a service or component, the trace context is propagated along with the request. This usually involves injecting trace. headers (including the trace_id) into the request, allowing downstream services to participate in the same trace.

Right, I am going to do my obligatory thing in this series and mention OpenTelemetry again. Deal with it! But seriously, the easiest and most standardised way of getting started is to pick an OpenTelemetry APM and off you go. Most modern vendors (ie. Datadog, Sumologic, Honeycomb) will provide exporters and well tested documentation.

Let’s quickly cover the make up of spans, the core component of a trace. A span represents an operation (or a unit of work) in a trace. A span could be a database query, or an in-process function call, or even a remote procedure call (RPC). A span has all of these things:

• A span name (operation name).

• A parent span.

• A span kind.

• A start and end time.

• A status that reports whether operation succeeded or failed.

• A set of key-value attributes describing the operation.

• A timeline of events.

• A list of links to other spans.

• A span context that propagates trace ID and other data between different services.

A trace is a tree of spans that shows the path that a request makes through an app. The root span is the first span in a trace. An example:

Now, time for some sage advice. All OpenTelemetry backends use span names and some attributes to group similar spans together. To group spans properly, I highly recommend giving them short and concise names. You should aim to have less than 1000 unique span names, for performance reasons. Let’s look at some good and bad span names:

Good

Bad

Lastly on spans, every span must have a kind/type, and it must be one of these values:

• server for server operations, for example, HTTP server handler.

• client for client operations, for example, HTTP client requests.

• producer for message producers, for example, a Kafka producer.

• consumer for message consumers and async functions, for example, a Kafka consumer.

• internal for internal operations.

…also, spans must have a status code of one of the following values:

• ok - success.

• error - failure.

• unset - the default value which allows the backends to assign the status.

Some additional features of traces and spans

Attributes

If you wish to record contextual information, you can annotate spans with attributes that carry information specific to the operation. Let’s give a basic example such as a HTTP endpoint, which may have attributes like http.method = GET and http.route = /orders/:id.

You have the freedom to name attributes as you want, but for common operations you should use the OpenTel semantic attributes convention.

Events

You have the option to annotate spans with events that have a start time and an arbitary number of attributes. The main difference between events and spans is that events don’t have an end time (and therefore no duration).

Events can usually represent exceptions, errors, logs, and messages (such as in RPC), but you can also create custom events if you so wish. For example, you may have a telemetry wrapper that your engineeing team uses to annotate spans in a standardised way, including sending your logs via tracing as well.

The observant of you may remember back in my first post in this series, you will note that the first pillar of observability is on Logs. So, if you can send event-logs via spans in a trace, why do you need logs as a seperate pillar? Well, the simple answer is, you probably don’t. However, not all systems architecture is created equally, and through acquisitions or divergent paths in your tech choices, you may still want to collect logs from different components. So, if your tech stack is fairly simple, and you just instrument your code to send traces to your backend/observability platform of choice, then you can likely just stick with this approach.

Context

Context is an important feature of spans. The span context carries information about the span as it propagates through different components and services in the tree.

The trace/span context is a request-scoped data object such as:

• Trace ID. A globally unique identifier that represents the entire trace or query. All spans within a trace have the same trace ID.

• Span ID. A unique identifier for the specific span within a trace. Each span within a trace has a different span ID.

• Trace flags. Flags that indicate various properties of the trace, such as whether it’s sampled or not. Sampling refers to the process of determining which spans should be recorded and reported to the observability backend.

• Trace State. An optional field that contains additional vendor or application-specific data related to the trace.

The span context is incredibly important for maintaining the continuity and correlation of spans within a distributed system. It allows different services and components to associate their spans with the correct trace and provides true end-to-end visibility into the flow of requests or transactions. The span context is typically propagated using headers or metadata of the communication protocols between services, similar to how baggage data is propagated, which we will cover in a minute. This is to make sure that when a service receives a request, it can extract the span context, and associate the ingress span with the correct trace.

You can use data from the context for spans correlation or sampling. For example, you can use the trace_id to know which spans belong to which traces, which is obviously incredibly important during troubleshooting or sampling!

Lastly on context, read up on context propagation from the OpenTelemetry docs here. There is a section on supported serialisation and deserialisation protocols on that page as well, which is helpful!

Baggage

We all come with baggage, and fortunately so does tracing! Baggage allows you to propgate custom key:value pairs (attributes) from one service to another. The example on the OpenTelemetry documentation is fantastic, so I won’t give another - just have a read.

What should we instrument then and how?

You really do not need to instrument every operation in your code to get the most out of tracing - it would be very time consuming and it’s not really necessary, or even valuable for your observability practices. Consider prioritising these operations:

• Network operations, for example, HTTP requests or RPC calls.

• Filesystem operations, for example, reading/writing to files.

• Database queries which combine network and filesystem operations.

• Errors and logs, for example, using structured logging, which from my first post in this series.

Ok, so now a quick example in Golang before the wrap up!

Step 1. Let’s instrument the following example function where we are inserting a new order:

func insertOrder(ctx context.Context, order *Order) error {
  if _, err := db.NewInsert().Model(order).Exec(ctx); err != nil {
    return err
  }
  return nil
}

Step 2. Let’s wrap the operation with a span:

import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("app_or_package_name")

func insertOrder(ctx context.Context, order *Order) error {
  ctx, span := tracer.Start(ctx, "insert-order")
  defer span.End()

  if _, err := db.NewInsert().Model(order).Exec(ctx); err != nil {
    return err
  }
  return nil
}

Step 3. Let’s record errors and set a status code:

import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("app_or_package_name")

func insertOrder(ctx context.Context, order *Order) error {
  ctx, span := tracer.Start(ctx, "insert-order")
  defer span.End()

  if _, err := db.NewInsert().Model(order).Exec(ctx); err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
    return err
  }
  return nil
}

Step 4. We should also record some contextual information with attributes:

import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("app_or_package_name")

func insertOrder(ctx context.Context, order *Order) error {
  ctx, span := tracer.Start(ctx, "insert-order")
  defer span.End()

  if _, err := db.NewInsert().Model(order).Exec(ctx); err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
    return err
  }

  if span.IsRecording() {
        span.SetAttributes(
            attribute.Int64("endorder.id", order.ID),
            attribute.String("endorder.description", order.Description),
        )
    }

  return nil
}

Let’s wrap it up

I hope you have found this blog post helpful, and if anything, you have some takeaway pointers to use in your observability practices. Let’s do a quick a list of the most important takeaways from this post:

• Use OpenTelemetry for your Tracing! You will not regret it one bit and most repetable vendors support OpenTel standards, which is fantastic!

• You probably won’t need logs as well as traces! As mentioned in the events section above, you can send event-logs in your span payloads, which means you won’t need to send log’s seperately. This is why I will always recommend collaborating inside your engineering team to come up with solid observability patterns, and even wrapper libraries to help with consistency across all of your services and systems.

• Context is really important when your request paths traverse many different services and components within your distributed architecture. Always populate the context and remember to check out the supported standards that OpenTel offers!

• Only instrument key components! We covered a database insert in our example, which is considered a key component to the service/system.

• Use Semantic Conventions! When you add instrumentation to your code, it is important to follow semantic conventions. This means using standardised attribute names, span names, and span tags as all defined within the OpenTel specifications. Doing so ensures consistency and interoperability across different instrumentation libraries and backends.

Thanks again for reading and I hope you’re looking forward to the next blog post in this series, which will be about centralised observability platforms (also labelled as backends).

Introduction

I won’t delve into a full explanation or analogy of observability engineering again (for a refresher, please refer to the first post in the series); however, let’s quickly recap. Observability engineering involves measuring the internal states of a system or application by examining its outputs. It’s straightforward – no secret sauce, and no hocus pocus! In this series, we’ll cover essential topics within observability engineering that should benefit both newcomers and seasoned engineers alike.

As always, my strong opinions are based on extensive experience, but I hold them lightly. I take great pride in my ability to change my views when presented with new information or different experiences.

Pillar Two - Metrics

What are metrics?

First, what are metrics? Metrics are simply the numeric representation of data measured over intervals of time. Using my pilot analogy from the first post, a commercial pilot needs real-time metrics to make informed decisions throughout a flight. This data is crucial for the pilot’s situational awareness. If it were in log format, the pilot wouldn’t have time to sift through detailed logs to get the necessary information, which could be disastrous!

For the aeroplane manufacturer, having historical numeric data points is essential for hypothesis-based investigations. For instance, if a new engine version is introduced, manufacturers might hypothesise that these engines run significantly hotter than previous models. By analysing historical data points, they can mark when the new engines were installed and validate or refute their hypothesis. Metrics might not explain why the engines run hotter, but they provide a clear picture of when and what.

The incredible value of metrics

My first piece of advice is, if anyone ever tells you that metrics are unnecessary in modern observability engineering, take a deep breath and firmly disagree. While it might not warrant drastic actions (unless they’ve committed equally heinous acts like stealing your lunch from the office fridge), it’s essential to remind them of the immense value numerical data points provide for single-pane-of-glass troubleshooting. Metrics offer an additional data dimension, giving you immediate, data-based insights.

For instance, imagine a service or application reporting numerous errors early on a Monday morning at 9am. Your service might emit a metric through a framework like python_gc_objects_uncollectable_total, and your cloud provider might collect a memory_utilised metric. These numerical data points are unique to metrics and cannot be easily derived from other observability pillars.

Now, consider Bob, the senior engineer, strolling into the office at 10am with his frappamochachino iced coffee and a croissant from his favourite bakery. Bob recalls a Python version bump on Friday afternoon, “but it had been tested and approved,” he mentions, mid-bite. What if Bob wasn’t there to provide that critical piece of information? No worries – you’re a super observability engineer who thought ahead and emitted a metric indicating the Python version of your service, such as python_version. Brilliant!

With this metric, you can easily correlate the version bump with the rise in uncollectable garbage collection objects due to increased request throughput on Monday morning. The team can then roll back the Python version, averting any major issues. Disaster mitigated.

I acknowledge this is a basic example, and something you might also catch in your logs, like SystemOutOfMemory exceptions. However, not all systems or services operate at the same scale. Engineers shouldn’t have to hunt for a needle in a haystack. So, always strive for clarity and precision in your observability practices.

Get collecting

Great stuff! We have covered how important metrics are, but what is the anatomy of a modern metric?

First of all, my strong opinion here is that you should pick OpenTelemetry to instrument and collect your metrics. Trust me and save a significant amount of time by making that choice early on. The OpenTel standards for metrics collection are supported by most major centralised observability platforms. We will cover centralised observability platforms in a future blog post.

The OpenTel metrics data model structure can be found here. OpenTelemetry has three models:

• The Event model, in which you instrument metrics as the software engineer

• The Stream model, which OpenTelemetry uses for transport

• The Timeseries model, which OpenTelemetry uses for storage

The OpenTelemetry metrics are constructed by using the global MeterProvider to create a Meter, and associating it with one or more instruments. An instrument is a specific type of metric (e.g., a counter, gauge, histogram) that you use to collect data about a particular aspect of your service or application’s behaviour. You capture measurements by creating instruments that are comprised of:

• A unique name, for example, http.proxy.request.duration

• An instrument type, for example, Histogram

• An optional unit of measure, for example, milliseconds or bytes

• An optional description for the instrument

A single instrument can produce multiple timeseries. A timeseries is a metric model with a unique set of attributes. For example, let’s say you have a Kubernetes cluster; each host in the cluster has a separate timeseries for the same metric name.

It’s very important to mention additive instruments at this stage. Additive or summable instruments produce timeseries that, when added up together, produce another meaningful and accurate timeseries. Additive instruments that measure non-decreasing numbers are also called monotonic. For example, http.server.requests is an additive timeseries because it can be summed from multiple hosts to get the actual total number of requests from your service if you load balance requests, which you should be doing, of course! There are also synchronous instruments, which are invoked together with the operations they are measuring. For example, +1 to a counter when a request is fired off for your service or application. Lastly, there are also asynchronous instruments, which periodically invoke a callback function to collect measurements. Asynchronous instruments are also known as observers, and observers can be used to periodically measure things like system memory or CPU usage.

When to use what type and some examples!

Some simple guidance on when to use what:

1. If you need to measure request_latency or maybe request_size, pick a Histogram

2. If you need to measure things like processed_requests, errors, received_bytes, disk_reads, pick a Counter if the value is monotonic. Otherwise, use UpDownCounter as your instrument type

3. If you need to measure things like cpu_time, memory_usage_bytes, memory_utilisation_percentage, if the value is additive/summable and if the value is monotonic, use CounterObserver; otherwise, use UpDownCounterObserver. Lastly, if the value is NOT additive/summable, use the GaugeObserver type.

Lets make this fun by doing a few examples! I’ve picked golang in the following examples, but there are well supported SDK’s/API’s in most popular programming languages. We want to measure the number of requests our startup app sends to social media API’s, so we should pick a Counter, and increment whenever a request is sent:

import "go.opentelemetry.io/otel/metric"

socialMediaAPIRequestCounter, _ := meter.Int64Counter(
	"some.prefix.api.requests",
	metric.WithDescription("Number of sent Social Media API requests"),
)
// Your secret sauce code does things here
socialMediaAPIRequestCounter.Add(ctx, 1)

simple example, but you can see we are incrementing the counter each time our secret sauce code runs. Lets do another very quick example with a Histogram:

import "go.opentelemetry.io/otel/metric"

opHistogram, _ := meter.Int64Histogram(
	"some.prefix.process.image.duration",
	metric.WithDescription("Duration of image enhancement"),
)

t1 := time.Now()
op(ctx)
dur := time.Since(t1)

opHistogram.Record(ctx, dur.Microseconds())

That example is showing the time it takes to do an image enchancement in our CSI: New York image enchancement app.

Lets cover one last example of cache hit rates, because we are using CounterObserver which if we remember correctly is used to instrument monotonic numbers that are additive/summable:

import "go.opentelemetry.io/otel/metric"

counter, _ := meter.Int64ObservableCounter("some.prefix.request.cache")

// Arbitrary key/value labels.
hits := []attribute.KeyValue{attribute.String("type", "hits")}
misses := []attribute.KeyValue{attribute.String("type", "misses")}
errors := []attribute.KeyValue{attribute.String("type", "errors")}

if _, err := meter.RegisterCallback(
	func(ctx context.Context, o metric.Observer) error {
		stats := cache.Stats()

		o.ObserveInt64(counter, stats.Hits, metric.WithAttributes(hits...))
		o.ObserveInt64(counter, stats.Misses, metric.WithAttributes(misses...))
		o.ObserveInt64(counter, stats.Errors, metric.WithAttributes(errors...))

		return nil
	},
	counter,
); err != nil {
	panic(err)
}

We should also note that CounterObserver is an asynchronous instrument, and that means it will periodically invoke a callback function to collect measurements. They were very basic examples that I have provided, but hopefully in the future I can link a repository with some better and more fleshed out real-world examples.

The last step is to choose a backend to send your timeseries metrics to, but as mentioned previously we will cover centralised observability platforms in a later post.

Wrap Up Part Deux

A recap of the advice and guidance I have given in this post:

• Metrics are incredibly useful, and don’t let anyone tell you otherwise. I can share countless stories on the importance of having dashboarded data points during late-night troubleshooting sessions while on-call.

• Choose OpenTelemetry to model, transport, and store your metrics. You can’t go wrong by making this choice early on.

• Learn the basics of the anatomy of an OpenTel metric, and familiarise yourself with the instrument types to help you make informed decisions on which instruments to pick for your requirements.

Thank you so much for reading, and as always, I hope this post was helpful! Keep an eye out for my next post on Tracing and APM!

Introduction

So, what exactly is observability engineering? Simply put, it’s the ability to monitor, measure, and understand the state of a system or application through its external outputs.

Let’s use an analogy to illustrate. Consider a commercial aircraft. An aeronautical engineer relies on data from the aircraft to make informed maintenance decisions, such as monitoring the oil pressure of the jet engines. Pilots, on the other hand, need real-time metrics like altitude and cabin pressure to ensure safe flights. All this data is displayed on a dashboard, providing a comprehensive view of the aircraft’s performance. Manufacturers also analyse this data to plan improvements. To put it in perspective, a single commercial aircraft can generate up to 20 terabytes of data per engine, per hour of flight – a staggering amount of information!

In the same way, observability is crucial in modern software engineering. Mastering it means you can confidently answer how your system or application is performing without constantly checking if it’s still running. Instead, you’ll have a robust alerting and incident response platform, giving you peace of mind. After all, you want to make sure you know about an issue before your customers do!

As for my background, I’ve been in the observability field for years, working with network devices, server hardware, monolithic applications, micro-services, and event-driven architecture. I’ve designed and implemented platforms capable of handling tens of thousands of data points per second. My experience spans creating “observability in a box” solutions, offering platform-as-a-product services that software engineering teams can easily integrate with. I bring a wealth of knowledge and strong opinions on the subject.

The Three Pillars of O11y - Logs

Warning, there be strong opionions here.

What are these three pillars of observability then? Great question, and this isn’t a conclusion and term that I have coined, this was created by super smart engineers in the observability space. I am only going to be covering logs in this post, because during the draft it was getting quite long and I want to make this a fairly digestable series.

Pillar One - Logs/Event-Logs

Wait a minute, why Logs and/or Event-Logs? Whats the difference? Dont panic, we will get into that in a minute or two. Essentially, logs are human-readable flat text files that are used by engineers to capture useful data about their systems/services. Log messages occur when a developer deems it important to tell the system or application owner that something happened that they should probably know about. For example, your service could be dropping requests, and you should probably know why sooner rather than later.

Let’s look at a good example of a log file:

[2021-02-23T13:26:23.505892 #22473] INFO -- : [6459ffe1-ea53-4044-aaa3-bf902868f730] Started GET "/" for ::1 at 2021-02-23 13:26:23 -0800

Source: The Path from Logs to Traces, by Alex Vondrak

The example log line starts with a timestamp and a PID (Process ID) [2021-02-23T13:26:23.505892 #22473], which is incredibly important for time-series investigation. If the logging entity has incorrectly configured time, then this data is effectively useless. Precision timing is crucial in modern software engineering. Please accept that as my first piece of sage advice - make sure you write timezone aware code, and the systems you host on are all connected to accurate centralised time protocol servers.

Next up we have the logging level, INFO in the example log file. Logging level basically means the “importance” of the log message to the system owner or operator. The logging level should ALWAYS be set at the application/service/runtime vars config stage, and the reason for that is that if you have to change it out in the wild for investigative purposes, ie. change to DEBUG level, you don’t want to have to trawl through code in the early hours of the morning and update everywhere where the logging level is set.

Next up we have something called a Universally unique identifier (uuid) v4 6459ffe1-ea53-4044-aaa3-bf902868f730, which is basically a randomly generated id to represent the request ID. This request ID is important, to help chain events/units of work together for a given request.

We have a GET request next, which is one of the HTTP verbs. It is useful to know the HTTP method used when making a request to a service or web app.

We then see a request path /. In this case, this is the root path of the application/service. We then see ::1 which is the host network address, an IPv6 localhost address.

Finally, we have the request start time with the timezone UTC offset 2021-02-23 13:26:23 -0800. This is also important to know when the request hit your service or web app, so that you can accurately piece together the events up until a given point.

This was just purely a random example of how an application or service might log a message. There are many other examples out there where a lot of the formatting choices are relatively sensible out-of-the-box. I give another piece of advice to always find a good logging library for your chosen programming language, unless the standard library logger is really good already. You shouldn’t want to reinvent the wheel for something like logging, just pick the most popular and easy to use open source solution. If you wan’t to make modifications, then fork it, and create a new package, but always inner source it for your engineering team to encourage internal contributions. For example, the software engineers in your org might want to add standardised fields, or even masking of data before it is sent out on the wire.

Now, some of the observant amongst you may have remembered that we were going to discuss the difference between logs and event-logs. Don’t Worry, I haven’t forgotten! I also don’t want to cover this topic in too much detail because otherwise the post will bloat quite significantly, but I will link you to some excellent views from the superstar of Observability Engineering, Charity Majors. Event-logs are just better than logs. Thats right, to the annals of history with you logs! “But before we stand on that hill with you, Mark, can you tell us what the difference is, please?” Alright, I hear you. Event-logs are structured logs, and they follow a standardised format (JSON), which makes it trivial to parse and query for most centralised observability platforms. Let’s take the log example from before and put that into an event-log:

{
  "name": "request",
  "timestamp": "2021-02-23T13:26:23.505892",
  "pid": 22743,
  "level": "info",
  "request_id": "6459ffe1-ea53-4044-aaa3-bf902868f730",
  "request.method": "GET",
  "request.path": "/",
  "request.ip": "::1",
  "request.start_at": "2021-02-23 13:26:23 -0800"
}

Source: The Path from Logs to Traces, by Alex Vondrak

I think you’ll agree it’s even easier to read in it’s raw format. Lastly on event-logs - always emit a single event per request per service that it hits. Use the Open Telemetry conventions and ALWAYS fire off an event before the request errors or exits the service, otherwise you have no breadcrumbs for your investigation! Also, if you have distributed systems, include a trace_id to pass onto other services in the stream.

Wrap Up

Ohh, that isn’t the last piece of advice regarding logs actually. My last piece of advice is to just avoid putting Personally Identifiable Information (PII) into your logs. You don’t need it. If you have an anonymized user id in your data model, you do not need to log PII. Your security/compliance team will keep their hair, and you will also be way happier and content. But lets say you must absolutely include the event payloads from publisher event buses in your event-logs, then, please please think about your event schemas. I will give a very basic example, but hopefully this highlights the point:

{
  "name": "request",
  "timestamp": "2021-02-23T13:26:23.505892",
  "pid": 22743,
  "level": "info",
  "request_id": "6459ffe1-ea53-4044-aaa3-bf902868f730",
  "request.method": "GET",
  "request.path": "/",
  "request.ip": "::1",
  "request.start_at": "2021-02-23 13:26:23 -0800"
  "request.payload": {
    "event": {
      "user.data" : {
        "id": "d6279b68-3460-4799-b26d-ea87a865f7fc"
        "private" {
          "full_name": "Joe Bloggs"
          "email": "joe.bloggs@example.com",
          "address": "1 Mount Olympus"
        }
      }
    }
  }
}

With the field request.payload, and the full field path of request.payload.event.user.data.private I now know everything under that path is private user data, and I can filter/mask that at ingestion time. This is why engineering standardisation is so important in modern engineering orgs. You should agree with your fellow engineering community how to design your schemas to avoid problems later down the line, where mitigating PII data leaking into your observability platforms will be very difficult, and will force you down paths you will not want to go. I am talking reducing data retention periods to 30 days of warm data kinda craziness.

Let me provide a quick summary of all the sage advice and tips we have covered in this post:

• Be smart, be cool, and write timezone aware code that is hosted on infrastructure that have accurate time servers to hand.

• Always make sure you centralise your logging level settings, at the config or runtime variables stage.

• Always pick a good logging package or module for your chosen programming language. Open source solutions are perfect, and if you must customise it, then inner source so that you and your colleagues can modify as desired.

• Event-logs are structured logs and are just way better than bog standard logs. If you had to see some of the regex parsing patterns I have had to create over the years to get logs indexable in observability platforms, you will understand why I stand on that hill.

• Always emit a single event per request to your service before exiting or erroring, and stream over the wire to your observability platform of choice, using Open Telemetry Standards.

• Lastly, do not put PII in your logs. If you absolutely must, work with your fellow engineers to standardise your logging structure, and or your event schemas if you run Event Driven Architecture.

I hope I didn’t miss anything important about logging in the world of observability, but if you feel I did, please tweet me and we can chat (my handle is linked on the left). Thanks for reading!