Operations Engineering Runbooks

These runbooks are designed to provide information and instructions to the Operations Engineering Team when provisioning and supporting tools and services.

Services

What We Don’t Support

Refer to this runbook for a list of services not supported by Operations Engineering, Slack channel contacts, etc to refer users to.

• Not Supported by Operations Engineering

1Password

• Exporting passwords from 1Password
• 1Password User Account Recovery
• Groups and Vaults
• 1Password Chrome extension shortcut conflict
• Users unable to import via the 1Password App

AWS

• Requests for AWS Account Access
• AWS Credentials Remediation Process
• Add OIDC between AWS and GitHub
• YJAF AWS Password Resets
• AWS Root Account Working Group

Auth0

• Creating an Auth0 Tenant

Certificates

• Manual SSL Certificate Processes
• Respond to expired certificates
• Configuring the Certificate Mappings File

CircleCI

• CircleCI
• CircleCI Troubleshooting

DNS

• BT DNS Change Process Pre-requisites
• How to check for domain activity before decommissioning
• Decommissioning Domains
• Supporting migrations/DNS cutovers
• How to manually recover deleted DNS records
• Redirecting Domains
• Delegation of subdomains
• Domain Transfers for Non-gov.uk subdomains
• Changes to Nameserver Records involving GDS Domains Team
• DNS for services using e-mail
• Restoring Route 53 from Backup

Docker

• Add Docker User (Manual Process)

GitHub

• Add GitHub User
• Adding an SSH Key to GitHub
• Handling Third Party GitHub Requests
• GitHub Repository Archiving
• Add GitHub collaborators from a fork PR
• Branch Protection Settings and Issues
• How to respond to a low GitHub seats alert
• How to respond to a low Github Actions minutes alert
• Dormant User Process
• Monthly Upload of Github User Data for JML4 Tool
• Repository Terraform
• Review Organisation PAT Requests

OS Data Hub

• OS Data Hub API Key Management

PagerDuty

• Add a new Slack channel to a PagerDuty service

Renovate

• Renovate

Sentry.io

• Create a Sentry Internal Integration
• Disabling sending errors to a project in Sentry
• Respond to Sentry Usage Alerts

SonarCloud

• Adding a GitHub repository to MoJ SonarCloud

SSO

• Adding SSO to a tool

Internal

How to be Support

• How to be Support

Operations Engineering Communication Plan

• Operations Engineering Communication Plan

Operations Engineering Team

• New Joiners Guide
• Leavers Guide
• Ways of Working
• Ways of Engineering

MoJ Organisation Leavers

• Respond to Leavers

Internal Processes

• Add a Runbook
• Add a Slack Alert to our Alert Channel
• Manage Slack RSS Feeds
• Python Best Practice
• Risk Review
• Post-Incident Review Proceses
• Incident Log
• Responding to Dependency Alerts

Architecture Decision Records

This is a record of architectural decisions made by the Operations Engineering Team

To understand why we are recording decisions and how we are doing it, please see ADR-000

Status	ADR no.	Title
✅	ADR-000	Record Architecture Decisions
✅	ADR-001	Github Failover
⌛️	ADR-002	Sentry Spike Protection
✅	ADR-003	DNS Failover
✅	ADR-004	Docker SSO
✅	ADR-005	Github Standards Branch Protection
✅	ADR-006	Require Authentication for Team Reports
✅	ADR-007	Decomission Developer Portal
✅	ADR-008	Sentry Spike Protection
✅	ADR-009	1Password SSO
✅	ADR-010	1Password Manager Permissions
✅	ADR-011	GitHub Features as Opt In
✅	ADR-012	RSS Feed Aggregation Channel
❌	ADR-013	Archiving the DNS repo
✅	ADR-014	Risk Review
❌	ADR-015	Use of GitHub Actions Runner Controller
✅	ADR-016	Archive DNS-IAC
✅	ADR-017	Revert decision to mandate that repository access must be via a Team
✅	ADR-018	Standardisation of Repository Naming
✅	ADR-019	Management of Github Repositories through Terraform
✅	ADR-020	Bot Account Personal Access Token Standards
✅	ADR-021	Management of DNS Records through OctoDNS

Statuses:

• 🤔 Proposed
• ✅ Accepted
• ❌ Rejected
• ⌛️ Superseded
• ♻️ Amended

This page was last reviewed on 24 June 2024. It needs to be reviewed again on 24 December 2024 by the page owner #operations-engineering-alerts .

This page was set to be reviewed before 24 December 2024 by the page owner #operations-engineering-alerts. This might mean the content is out of date.