Using AWS Locally with MiniStack and .NET

Introduction

When using AWS for development, sometimes it is useful to have some sort of emulator, so that we don't incur in costs while doing development and debugging. For that, we have local cloud emulators, and LocalStack used to be the standard one not too long ago; the problem is, its licence changed and it is now generally not free. The good news is that there are alternatives that are free, such as MiniStack (GitHub: MiniStack), a free and open-source AWS cloud emulator that can emulate more than 55 AWS services and that is very similar to LocalStack. On this post I'll be talking about how to set it up so that we can point to it and use it as if it were the real thing with .NET. We will be using Docker Compose to spin up the local environment.

Docker Compose

We want to use the latest MiniStack image, which is made available from https://hub.docker.com/r/ministackorg/ministack. Our docker-compose.yml file will look like this:

services:

  ministack:
    image: ministackorg/ministack:latest
    container_name: ministack
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3,sqs,ssm
      - AWS_DEFAULT_REGION=eu-west-2
      - AWS_ENDPOINT_URL=http:/localhost:4566
    volumes:
      - ./init:/etc/localstack/init/ready.d:ro
      - /var/run/docker.sock:/var/run/docker.sock
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4566/_ministack/health')"]
      interval: 10s
      timeout: 3s
      retries: 3

Noteworthy:

  • We are exposing port 4566 (default port for LocalStack and MiniStack)
  • Services to start are defined using the SERVICES environment variable; in this case, I'm starting S3, SQS, and SSM Parameter Store
  • Setting the AWS_ENDPOINT_URL environment variable to point to the local setup
  • Setting the default region in AWS_DEFAULT_REGION
  • Mounting a local init folder to the container's /etc/localstack/init/ready.d folder as read-only (ro); this is so that we can run an initialisation script (more on this in a minute)
  • For the health check, because the ministackorg/ministack image does not include curl, we use Python with the urlib.request library for making HTTP requests to monitor our container
  • I didn't include any virtual network configuration or anything else, this is just the bare minimum.

For more info on MiniStack configuration, including how to set up durable storage for all of its services, please check out https://ministack.org/docs.

Initialisation

We need to perform some initialisation, like, creating S3 buckets, SQS queues, and reference SSM configuration; by design, MiniStack (also like LocalStack) will run any scripts, ordered by name, that exist on folder /etc/localstack/init/ready.d after it starts, that's why we mapped it to a local folder. Let's create a local folder (on our Windows/Mac/Linux/whatever) called init (the same name as in docker-compose.yml) and inside of it let's create a file called 01-init.sh (the actual name does not matter much, but scripts are sorted by it in alphabetical order) with this content:

#!/bin/sh

# fail in case there are errors (return code not 0)
set -euo pipefail

# log to both stdout and stderr for visibility in Docker logs
echo "Initialising" | tee /dev/stder

# give some time for the services to start
sleep 5

# create a bucket in s3
aws s3 mb s3://bucket-name | tee /dev/stder

# create a queue
aws sqs create-queue --queue-name queue_name | tee /dev/stder

# set some value in parameter store
aws ssm put-parameter --name "parameter_name" --value '{ "foo": "bar" }' --type "String" --overwrite | tee /dev/stder

# all done
echo "Initialisation complete" | tee /dev/stderr

Very important: this file needs to be saved using the UNIX line terminator (LF), not the Windows (CR+LF). Also, it must be saved with the UTF-8 encoding!

What this does is:

  • Sets error handling, so that if any command fails, the script aborts with the command's error code
  • Sleeps for a while, to give services time to start
  • Creates an S3 bucket called bucket-name
  • Creates a SQS queue called queue_name
  • Creates a parameter in SSM Parameter Store called parameter_name with some simple JSON
  • All output is also sent to standard error output using tee

For the full list of parameters to aws, please have a look here: https://docs.aws.amazon.com/cli/latest/reference.

Usage

To start our local service, we just call:

docker-compose up

And to test the services, like listing existing S3 buckets:

aws s3 ls

Note: we can skip the --endpoint-url and --region parameters if we have set the AWS_ENDPOINT_URL and AWS_DEFAULT_REGION environment variables.

If we want to use MiniStack with .NET, we can add the following to our appsettings.json file for the default settings:

{
  "AWS": {
    "Region": "eu-west-2",
    "UseMiniStack": false
  }
}

And, possibly on appsettings.Development.json, we can have an override for the Development environment to use the local emulator:

{
  "AWS": {
    "UseMiniStack": true,
    "ServiceUrl": "http://localhost:4566"
  }
}

This way we know that when UseMiniStack is enabled, the local emulator will be used, together with the ServiceUrl that points to our local emulator.

Now, for actually using this configuration, for example, for accessing S3:

// get config section
var awsSection = builder.Configuration.GetSection("AWS");
var region = awsSection["Region"] ?? "eu-west-2";
var useMiniStack = awsSection.GetValue<bool>("UseMiniStack");
var serviceUrl = awsSection["ServiceUrl"];

// create a config object to be reused by all services
var config = new AmazonS3Config
{
    RegionEndpoint = RegionEndpoint.GetBySystemName(region)
};

if (useMiniStack && !string.IsNullOrWhiteSpace(serviceUrl))
{
    config.ServiceURL = serviceUrl;
    config.ForcePathStyle = true;
    config.UseHttp = true;
}

// register S3 client
builder.Services.AddSingleton<IAmazonS3>(_ => new AmazonS3Client(config));

I think you got the idea, we just inject IAmazonS3/AmazonS3Client service wherever it is needed and off we go! We will need the AWSSDK.S3 NuGet package for S3, AWSSDK.SQS for SQS, and AWSSDK.SimpleSystemsManagement for SSM Parameter Store. Similar registrations for IAmazonSQS/AmazonSQSClient and IAmazonSimpleSystemsManagement/AmazonSimpleSystemsManagementClient should be trivial.

Conclusion

I find using MiniStack very convenient and easy to use, it is free and full open-source, and it has a great user base. Of course, do keep in mind that you will still have to test this with the real AWS, but for daily development, this should be more than enough. I hope you find this useful, let me hear your thoughts!

Location: Coimbra, Portugal

Comments

Post a Comment

Popular posts from this blog

Modern Mapping with EF Core

Image
Introduction In EF Core (as in most O/RMs) we can map and store multiple things: Objects that hold a single value, like int , string , bool , float , Guid , DateTime , TimeSpan , Uri , etc. These are (somewhat wrongly) called primitive types  and are always included inside some container, like the following Objects that hold multiple single values and which have an identity, for example, Customer , Product , Order . These are called entity types  and they must have an identity that makes them unique amongst all the other objects of the same type Objects that are structured to hold multiple values, but have no identity, such as Address , Coordinate . These are called value types  and are merely a collection of (possibly related) properties We can also have collections of the previous things There's more to it, of course, for example, it is even possible to represent  inheritance of entities , of which I talked about before , In this post I am going to cover so...
Read more

C# Magical Syntax

Introduction This post is an update to an old one on my  old blog  and it was inspired by my latest posts on C# versus Java . You may not have realized that some of the patterns that you’ve been using for ages in your C# programs are based on conventions, rather than on a specific API. What I mean by this is, some constructs in C# are based on some magical methods with well-defined names which are not defined in any base class or interface, but yet just work. Let’s see what they are. Enumerating Collections You are probably used to iterating through collections using the foreach statement. If you are, you may know that foreach actually wraps a call to a method called GetEnumerator , like the one that is defined by the IEnumerable and IEnumerable<T> interfaces. Thus, you might think, the magic occurs because the collection implements one or the two, but you would be wrong: it turns out that in order to iterate through a class using foreach all it takes is that the c...
Read more

.NET 10 Validation

Note: this is an update to my original post . Introduction I wrote about  entity validation  in the past, the reason I'm coming back to it is that there are some changes in .NET 10, related to validation. Let's cover the original validation API and then explain what has changed. .NET Validation Validation is implemented in the System.ComponentModel.DataAnnotations namespace. I'll describe it first and then move to the small changes in .NET 10. .NET has featured a way to validate classes and their properties for a long time, the API is called  Data Annotations Validation , and it integrates nicely with ASP.NET Core MVC and other frameworks (not Entity Framework Core, as  I've talked recently , nor ASP.NET Core Minimal APIs ). It is essentially, but not just, based on a set of attributes, one interface, and a class that performs validations on a target class and its properties, from the validation attributes it contain. The base class for attribute validation is ca...
Read more