A GitHub Actions Pipeline to Generate OpenAPI Documentation

Introduction

This is my first post on GitHub Actions (GHA), the pipeline part of GitHub (GH), and GitHub Pages (GHP), the static hosting service. I will give an example of a pipeline I built for producing documentation for an OpenAPI spec. Seasoned devops/platform engineering people will probably find this very basic, but I hope it can have interest to someone who is starting to play with GHA! I chose the GitHub products because, essentially, they are available for free, and yet are very powerful.

Desired Functionality

I would like to:

  • Produce HTML documentation from an OpenAPI specification located in some URL, or from a local file, and make it available somewhere
  • Store each HTML version of the specification independently, e.g., one file for version 1.0.0, another one for 1.0.1, etc
  • Have a template for the documentation, with the ability to add additional contents - header, footer - optionally, as well as insert custom JavaScript and CSS
  • Have the generation run on demand
  • Be able to change the OpenAPI specification URL easily

Let's get started!

OpenAPI

By now, you probably know what OpenAPI is; previously called Swagger, it is a specification for describing REST web APIs. Majour programming languages and framework allow building web APIs that conform to this specification, like ASP.NET Core does.

GitHub Actions

GitHub Actions is the Continuous Integration/Continuous Deployment (CI/CD) feature of GitHub. It allows building workflows to automate the build, test, and deployment process. Learn about it here.

GitHub Pages

GitHub Pages is a site hosting service that takes static assets such as HTML, CSS, and JavaScript files straight from a repository in GitHub, and optionally runs them through a pipeline defined as a GHA. Read more about GitHub Pages here.

External Libs

To help with this, I will be using some external tools as NPM packages or Linux commands. Here is a list of them.

Curl

Curl should not require introduction: it is a full-featured command line HTTP client. We'll use it to retrieve the OpenAPI specification and store it locally.

Redocly

Redocly is an open-source tool, available as NPM, for generating HTML pages from an OpenAPI specification. It uses an Handlebars file as a template for the page to produce.

markdown-to-html

markdown-to-html is an NPM tool to convert Markdown (MD) files into HTML. We'll use this for the case when we want to add custom MD files to the header and/or footer of the page template.

Redocly Template

We will be using this template, a slightly modified version of the default one (template.hbs):

<!DOCTYPE html>
<html>

<head>
  <meta charset="utf8" />
  <title>{{title}}</title>
  <!-- needed for adaptive design -->
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <style>
    body {
      padding: 0;
      margin: 0;
    }
  </style>
  {{{redocHead}}}
  {{#unless disableGoogleFont}}<link href="https://fonts.googleapis.com/css?family=Montserrat:300,400,700|Roboto:300,400,700" rel="stylesheet">{{/unless}}
</head>

<body>
  <header>
  {header}
  </header>
  {{{redocHTML}}}
  <footer>
  {footer}
  </footer>
</body>

<script>
{script}
</script>

<style>
{style}
</style>

</html>

The syntax comes from Handlebars, it is a mix of pure HTML and some specific tokens. You may have noticed the {header}, {footer}, {script}, and {style} tokens, these are not Handlebars, we defined them, they will be used by us to inject bespoke contents.

Workflow Definition

Now, this is the actual workflow definition:

name: Generate Redoc documentation
on:
  push:
    branches: [ master ]
  workflow_dispatch:
    branches: [ master ]
env:
  EMAIL_DOMAIN: 'users.noreply.github.com'
  REPO: '${{ github.event.repository_owner }}'
  USER: '${{ github.actor }}'
  API_URL: '${{ vars.API_URL }}'
jobs:
  generate-redoc-documentation:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Install Node.js
        uses: actions/setup-node@v4
      - name: Install NPM packages
        run: |
          npm install -g @redocly/cli@latest
          npm install -g markdown-to-html
      - name: Get OpenAPI spec file
        if: ${{ github.event_name == 'workflow_dispatch' }}
        run: curl $API_URL > openapi.json
      - name: Set env variables
        run: |
          echo "NOW=$(date +'%Y%m%dT%H%M%S')" >> $GITHUB_ENV
          echo "VERSION=$(grep version openapi.json | head -1 | awk '{split($0,a,"\""); print a[4]}')" >> $GITHUB_ENV
      - name: Modify template
        run: |
          markdown HEADER.md > HEADER.html 2>/dev/null
          markdown FOOTER.md > FOOTER.html 2>/dev/null
          sh replace.sh
      - name: Run Redocly
        run: |
          [ ! -d "$REPO.github.io" ] && mkdir "$REPO.github.io"
          redocly build-docs openapi.json -t template_complete.hbs -o $REPO.github.io/index.html
          cp $REPO.github.io/index.html $REPO.github.io/index-$VERSION.html
      - name: Commit
        run: |
          git config user.email $USER@$EMAIL_DOMAIN
          git config user.name $USER
          git add openapi.json
          git add $REPO.github.io/index.html
          git add $REPO.github.io/index-$VERSION.html
          git diff-index --quiet HEAD || git commit -m "Generated at $NOW for $USER and version $VERSION"
          git push

I won't go into all of the detail, but I will explain how this works.

GitHub Action Variables

Here we make some use of GitHub Action variables, these can be stored at different levels: 

  • Organisation: for an entire GitHub organisation, all repositories, environments, and workflows, will be able to see them
  • Repository: for a specific repository inside an organisation, inherited by all environments and workflows
  • Environment: for a build environment, such as Dev or Prod, which will be accessible by all workflows running under that environment
  • Workflow: for individual workflow, such as the one we are defining now, and all steps will have access to them
  • Step: defined inside a particular step of a workflow

Organisation, repository, and environment-level variables are configured on the GitHub user interface, which means they can be changed easily. 

The variables we will use are:

  • EMAIL_DOMAIN: a workflow variable that is unchanging, meaning, it contains a constant string; this will be used for the GitHub configuration of the user's email
  • REPO: also a workflow variable that is obtained automatically from the execution context and contains the current repository name (github.event.repository_owner); this is the name of the user or organisation, such as “rjperes”
  • USER: also a workflow variable that contains the username of the person who triggered the workflow (github.actor)
  • NOW: a step variable that will hold the current timestamp from the date command
  • VERSION: another step variable that contains the OpenAPI file version and will be used for the  filename generation, this is obtained from the OpenAPI spec file; for this, we use a number of commands (grep, head, awk)
We also check how the workflow was triggered using a GH context variable (github.event_name), but we're not storing it anywhere.

There are other ways to store configuration, such as secrets, but I won't cover them here, maybe on a future post.

For the purpose of this post, I created a repository variable called API_URL:

A repository variable

Workflow Steps

A quick explanation of the workflow steps is in order:

  1. This workflow will be triggered by any of two events, both on the master branch:
  2. Runs on Ubuntu Linux, latest release available (runs_on)
  3. Some workflow variables are defined from different sources, as explained (env)
  4. The first steps first install some required libs (actions/checkout, actions/setup-node, the @redocly/cli and markdown-to-html NPM packages)
  5. If the workflow was manually triggered (workflow_dispatch), it fetches the OpenAPI specification using curl and stores it locally on a file called openapi.json
  6. It then sets two step environment variables, NOW from the current timestamp, and VERSION, from the version extracted from the OpenAPI specification file
  7. We then proceed to generating HTML files from the MD ones, and replacing the tokens on the template file
  8. Before we run Redocly we create the target folder, if it does not exist, and then output to the GitHub Pages folder (the repo name plus .github.io) twice, one with name index.html and the other index-VERSION.html, where VERSION is the version extracted from the OpenAPI spec
  9. Finally, we set some git configuration properties, add the generated files, and check if there is anything to commit, in which case, we do so and push to the local repository (this is needed)

I think these should be easy to understand, as I said, I won't go into too much detail about building a workflow, there are lots of tutorials out there.

Template Customisation

We want to be able to add external files to the "donut holes"/tokens ({header}, {footer}, etc) that we added to the template file, for that, we will use sed, in a custom shell script (replace.sh):

rm -f template_complete_*.hbs

sed '/{header}/ {
  s/{header}//g
  r HEADER.html
}' template.hbs > template_header.hbs

sed '/{script}/ {
  s/{script}//g
  r SCRIPT.js
}' template_header.hbs > template_script.hbs

sed '/{style}/ {
  s/{style}//g
  r STYLE.css
}' template_script.hbs > template_style.hbs

sed '/{footer}/ {
  s/{footer}//g
  r FOOTER.html
}' template_style.hbs > template_complete.hbs

If the files exist and are not empty, then they are inserted where the tokens are found, otherwise, the tokens are just removed. In the end, this produces a file named template_complete.hbs, which is then passed along to redocly.

Conclusion

I think this should be enough to spike your curiosity and to realise that GH, GHA, and GHP make up a powerful CI/CD environment. Hopefully, for those who are less familiar, this may be useful to get started! As always, looking forward for your comments!

Location: Coimbra, Portugal

Comments

Post a Comment

Popular posts from this blog

ASP.NET Core Middleware

Image
Introduction This post is another of those "back to basics", still, there was one thing or two that I feel were not properly discussed elsewhere, hence this post. The ASP.NET Core Pipeline and Middleware You may know that ASP.NET Core defines a pipeline. This is where the incoming requests are processed and the response submitted. Each component in the pipeline - a middleware - can do things with the message (the request), such as check if it is authenticated and have rights to access the requested resource, turn the request into an MVC-style controller and action invocation, log it, and what not. You can read more about it here . The ASP.NET Core pipeline, from Microsoft The order by which the middleware is added to the pipeline matters. For example, the authorisation middleware must come after the authentication one, static files must come before routing, and exception handling must come before everything. The middleware order, from Microsoft A middleware is just a piece of...
Read more

.NET Cancellation Tokens

Introduction Cancellation tokens in .NET were introduced as part of the  .NET 4 Cancellation Framework . Essentially, it is a standard, cooperative, way to request for the cancellation of the execution of an operation. By cooperative it means that the code that uses it must abide to some rules, as opposed to the cancellation just stopping (killing) the execution. One example of cancellation is, on a web app, when the client closes the connection while the server is performing some long operation. In cases such as this, we may want to abort the operation, as we are not sending the results anywhere (or not, as we shall see). Because of this, ASP.NET Core allows us to add a parameter of type  CancellationToken  to our asynchronous actions, it is automatically associated with the client and is therefore signalled when the client connection is closed. A  CancellationToken  is, essentially, behind the scenes, a ManualResetEvent . Throughout this article I will either ...
Read more

Audit Trails in EF Core

Introduction Audit trails is a common feature in database systems. Essentially, it's about knowing, for every table (or some selected ones), who created/updated each record, and when the creates/updates happened. It can also be useful to store the previous values too. Some regulations, such as SOX , require this for compliance. There are some libraries that can do this automatically in EF Core , and there's also the Change Data Capture in SQL Server (and possibly other databases too), of which I wrote about some time ago, but, hey, I wanted to write my own, so here is EFAuditable , my library for doing audit trails in EF Core. This one is designed with some extensibility in mind, so, please, read on. Scenarios EFAuditable  allows the following scenarios: Entities marked for auditing will have the audit columns added and persisted automatically to the database, on the same table Entities marked for history keeping will have their old values (in the case of updates or deletes) ...
Read more