microservices with .net - ndc sydney, 2016
TRANSCRIPT
OVERVIEW1. Why? (the short version)2. Architecture3. Implementation4. Deployment
PREAMBLEYou probably don’t need themTooling is still improvingMany implementations aren’t ‘pure’… that’s OK
I’m showing ONE way, not THE ONLY way.
THERE’S ALWAYS A BIGGER FISHThere’s plenty of other approaches:• AWS Lambda / Azure Functions• Azure Service Fabric• Akka/Akka.NET
THE MICROSERVICES PITCH
SHINY! SHINY! SHINY! SHINY! SHINY! Shiny! Uber! Shiny! Shiny! Shiny! Shiny! Shiny! Shiny! Netflix!! Shiny!Shiny! Shiny! Amazon!! Shiny! Shiny! Shiny! Shiny! Unicorns!! Shiny! Shiny! Shiny! Shiny!
OK. MORE SERIOUSLY…Greater flexibility & scalabilityMore evolvableIndependently deployable servicesImproved technical agilityIndependent development teams
A FEW MORE REASONS? SURE!Resilience. A failure in one service shouldn’t wipe out the whole system.Tech flexibility. Right tool for the right job.Smaller services are easier to understand and maintain.A potential migration approach for legacy systems
THE REALITY
WTF! OMG! GAH! <UPDATE RESUME />Isn’t this meant to be easy?!I can’t tell how it fits together anymore!It’s more brittle now than it ever was!Performance is terrible!!I need to deploy all my services together and in a specific order!
WHY IS IT SO?Distributed systems are HARD!!Eventual consistency is a paradigm shiftLegacy habits create a distributed “big ball of mud”People and culture problems.
ARCHITECTURE
INDIVIDUALS AND INTERACTIONS...Architecture is never just about the technology.Can your team(s) create a well built monolith?Are you agile, do you “do agile”, or is it neither?Have you got a DevOps culture?Is there an underlying business reason driving the change?
A QUICK REMINDERKeep it simple! Always.Don’t build what you don’t need.Don’t build what you might need.ROI & TCO are still incredibly important!
YOU’RE STILL HERE?If those warnings didn’t scare you off, we’ll continue.YOU HAVE BEEN WARNED :-)
KEY GOALSIndependent, loosely coupled servicesCheap to replace, easy to scaleFault tolerant, version tolerant services
PORTS AND ADAPTERS(VARIATIONS: HEXAGONAL, ONION, AND CLEAN ARCHITECTURE)
http://blog.mattwynne.net/2012/05/31/hexagonal-rails-objects-values-and-hexagons/http://www.slideshare.net/fabricioepa/hexagonal-architecture-for-java-applications/10
AN OVER SIMPLIFIED, CONCEPTUAL VIEW
http://www.kennybastani.com/2015/08/polyglot-persistence-spring-cloud-docker.html
HOW DO SERVICES COMMUNICATE?
http://www.slideshare.net/adriancockcroft/monitorama-please-no-more/31
KEEP COMMUNICATION SIMPLE & GENERICBe language & platform agnosticOne synchronous approach (JSON over HTTP)
One asynchronous approach (AMQP via RabbitMQ)
Why? Consistency reduces complexity.
USE API GATEWAYS/EDGE SERVICESClient applications should not call microservices directly.Have clients call an API/Application Gateway. This then calls your microservices.Why? Encapsulate and isolate change.
SYNCHRONOUS == TEMPORAL COUPLINGIf you use synchronous comms, you need to handle failures and timeouts.Use a circuit breaker pattern & design with failures in mind (and test for it!)Why? Uptime is the product of the individual components ( = 2+ hrs/mth)
http://www.lybecker.com/blog/2013/08/07/automatic-retry-and-circuit-breaker-made-easy/http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
IDENTIFY YOUR BUSINESS TRANSACTIONSOne client request may trigger hundreds of microservice calls. How do we trace a request?Treat each client request as a logical business transaction.Add a Correlation ID to every client request and include it in all internal communications.Why? Traceability aids debugging and performance tuning.
SERVICE DISCOVERYLoose coupling implies no hard coded URLs.Service discovery isn’t new (remember UDDI?)Microservices need a discovery mechanism.
E.g. Consul.io & Microphonehttps://github.com/rogeralsing/Microphone
EMBRACE DATA DUPLICATIONFor services to be independent……they cannot rely on another service being available (temporal coupling), and…they should cache any external data they need.Be prepared for this in your design.
VERSIONED, EVOLVABLE APIS“Services aren't really loosely coupled if all parties to a piece of service functionality must change at the same time.”
Consumer Driven Contracts are concept from the SOA days:WSDLs and XSDs were the SOAP attempt to solve this. With synchronous HTTP calls, have a look at Pact
http://www.infoq.com/articles/consumer-driven-contractshttps://github.com/SEEK-Jobs/pact-nethttps://www.youtube.com/watch?v=SMadH_ALLII
IMPLEMENTATION PATTERNS
DESIGN PATTERNS & COMPONENTSDomain Driven DesignAlign microservices to Domain Contexts, Aggregates & Domain ServicesCQRSCommand Query Responsibility Segregation.Scale reads and writes independently.SQL or NoSQLUse persistent, easily rebuilt caches for query services.VersioningAPIs are your contracts, not versions of binaries.
DESIGN PATTERNS & COMPONENTSMessage BusReliable, async comms. Optimistic ConcurrencyAvoid locking of any kind.Event SourcingPersist events, not state. Avoid 2-PC hassles.API GatewayEncapsulate access to microservices; optimise for client needs.
EVENT SOURCING?When a domain object is updated, we need to communicate the domain event(s) to all the other interested microservices.We could use 2-phase commit for this… and we could also drink battery acid.Why not just persist these events to a database instead of state, and publish those same events on the message bus.
HOW SMALL SHOULD THEY BE?The “100 line” rule is a bit silly.nano-services are effectively a service-per-method.Don’t turn your app into thousands of RPC calls! (unless you want to use AWS Lambda?)It’s about units of functionality, not lines of code.
RULE OF THUMB FOR SIZINGHave a single purpose
E.g. manage state of aggregates/entitiesE.g. send emailsE.g. calculate commissions
Be unaware of other services (in the “core”)Think about your Use Cases/Bounded Contexts
ARCHITECTURE PICTURESIt’s not architecture if there’s no boxes and lines!
Application Services(Gateway/Edge Service)
UI Request (HTTP)
Query MicroService
Data Cache(Redis)
Overall Approach Commands & Queries
Database(EventStore)
Domain MicroService
Message Bus(RabbitMQ)
Commands Queries
Event Sourcing Domain EventsPrecomputedResults
Web API Controller
Request (HTTP)
Aggregate
Event Handler(s)
Event Store
Domain MicroService
Command
Message Bus (publish)
Command Handler Command(s)
Event Store Repository
Save New Events
Event(s) Event(s)
Web API Controller
Query (HTTP)
Query Handler
Event Handler(s)
Message Bus (subscribe)
Query Micro Service Event(s)
Data Cache(Redis)
Consider splitting here when scaling beyond a single instance to avoid competing consumers
Query
Updates
SPECIFIC SOFTWARE & LIBRARIESRabbitMQ + EasyNetQEventStoreRedis + StackExchange.RedisASP.NET Web API
IMPLEMENTATIONSample code is for inspiration, not duplicationhttps://github.com/rbanks54/microcafe
THE MICRO-CAFÉ Inspired by:
Starbucks does not use two phase commithttp://www.enterpriseintegrationpatterns.com/docs/IEEE_Software_Design_2PC.pdf
WHAT ARE THE DOMAIN CONTEXTS? Cashier/Barista/Customer? Coffee Shop/Customer?
What about ‘Master Data’?
Which context owns the “product” entity?
BOUNDARIES?User Story?As the coffee shop ownerI want to define the products that are offered for saleSo I can set my menu
Use Cases?Manage Products (CRUD)
View MenuRun a promotion
CODE WALKTHROUGH
SCENE SETTINGDomain entities form the application core.Commands & Queries are the adapters and ports of our servicesUse CQRS; separate microservices forcommands and queries
ADMIN MICROSERVICE
Products
Admin Domain
CommandHandlersWeb API
Repository
Bus Publisher
EventStoreEvent
Handlers
Bus Subscriber
Admin Microservice
Memory Store
Event Store
RabbitMQ Memory Bus
EVENT SOURCING IMPACTS DESIGNCommands do not update state of any domain objects. They raise domain events.Events are processed by domain objects, who update their own internal state.This pattern makes it very easy to replay events and rebuild state quickly.
public class Product : Aggregate
{
private Product() { }
public Product(Guid id, string name, string description, decimal price)
{
ValidateName(name);
ApplyEvent(new ProductCreated(id, name, description, price));
}
private void Apply(ProductCreated e)
{
Id = e.Id;
Name = e.Name;
Description = e.Description;
Price = e.Price;
}
Methods Create Events
Apply an Event to change state
Apply an Event to change state
AGGREGATE BASE CLASSHolds unsaved events.Helper method to reapply events when rehydrating an object from an event stream.Provides a helper method to apply an event of any type and increment the entity’s version property.
public abstract class Aggregate
{
public void LoadStateFromHistory(IEnumerable<Event> history)
{
foreach (var e in history) ApplyEvent(e, false);
}
protected internal void ApplyEvent(Event @event) { ApplyEvent(@event, true); }
protected virtual void ApplyEvent(Event @event, bool isNew)
{
this.AsDynamic().Apply(@event);
if (isNew)
{
@event.Version = ++Version;
events.Add(@event);
}
else Version = @event.Version;
}
Cast as Dynamic so we don’t need to know all strongly typed Events
beforehand
New Events cause version to increment
Replaying events
public class Product : Aggregate
{
private void Apply(ProductNameChanged e)
{
Name = e.NewName;
}
public void ChangeName(string newName, int originalVersion)
{
ValidateName(newName);
ValidateVersion(originalVersion);
ApplyEvent(new ProductNameChanged(Id, newName));
}
Domain Command
Commands raise Events
COMMANDS (PORTS AND ADAPTERS)We separate the commands from the queries in our design. CQRS approach.
Ports: Command Handlers/ServicesAdapters: HTTP API (ASP.NET Web API)
PORT: COMMAND HANDLERSCommands do not have to map 1:1 to our internal domain methods.Commands Handlers (the ports) act on the inbound contract our adapters (the API) expose.Internal implementation and any created domain events are up to us.Command objects are just POCOs. No behaviour.
public class ProductCommandHandlers
{
private readonly IRepository repository;
public ProductCommandHandlers(IRepository repository)
{
this.repository = repository;
}
public void Handle(CreateProduct message)
{
var product = new Products.Domain.Product(message.Id, message.Name,
message.Description, message.Price);
repository.Save(product);
}
Outgoing “Port”
Incoming “Port”Commands don’t return
values
Act on the domainPersist
ADAPTERS: HTTP APIDoesn’t need to be RESTful.Could be also have a SOAP API.Could also have a Web Sockets API.
Secure your adapters. Flow identity to your microservices
[HttpPost]
public IHttpActionResult Post(CreateProductCommand cmd)
{
if (string.IsNullOrWhiteSpace(cmd.Name))
{
var response = new HttpResponseMessage(HttpStatusCode.Forbidden) { //… }
throw new HttpResponseException(response);
}
try
{
var command = new CreateProduct(Guid.NewGuid(), cmd.Name, cmd.Description, cmd.Price);
handler.Handle(command);
var link = new Uri(string.Format("http://localhost:8181/api/products/{0}", command.Id));
return Created<CreateProduct>(link, command);
}
catch (AggregateNotFoundException) { return NotFound(); }
catch (AggregateDeletedException) { return Conflict(); }
}
Incoming “adapter”
Pass through to the internal
“port”
Commands either succeed or throw an
error
OUTGOING INTERACTIONS(PORTS AND ADAPTERS)Repository Interface for data persistenceMessage Bus interface for publishing events
Ports: Repository / Message BusAdapters: EventStore API / EasyNetQ
ADAPTERS: DATABASE & MESSAGE BUSRepository pattern to encapsulate data accessEvent sourcing; persist events not state.Immediately publish an event on the bus
Note: This approach may fail to publish an eventCan be prevented by using Event Store as the pub/sub mechanismCan be prevented by only publishing to the bus. Use a separate microservice to persist events to the EventStore (extra complexity)Personal choice: RabbitMQ for ease of use & HA/clustering.
public async Task SaveAsync<TAggregate>(TAggregate aggregate) where TAggregate : Aggregate
{
//...
var streamName = AggregateIdToStreamName(aggregate.GetType(), aggregate.Id);
var eventsToPublish = aggregate.GetUncommittedEvents();
//...
if (eventsToSave.Count < WritePageSize)
{
await eventStoreConnection.AppendToStreamAsync(streamName, expectedVersion, eventsToSave);
}
else { //... multiple writes to event store, in a transaction }
if (bus != null)
{
foreach (var e in eventsToPublish) { bus.Publish(e); }
}
aggregate.MarkEventsAsCommitted();
}
Repository method
Persist via the event store “adapter”
Publish events onto the bus
QUERY MICROSERVICE
Product View
Read Model(s)Query
HandlersWeb API
Repository
Persistence
EventHandlers
Bus Subscriber
Admin Read ModelMicroservice
RedisRabbitMQ
QUERY SERVICES (A.K.A. VIEWS)Subscribe to domain events, and Update their read models based on those events (i.e. their cached data)Optimise for querying with minimal I/O
ADAPTERS: MESSAGE BUS SUBSCRIPTIONSSubscribe to messages from the message bus at startupUse Topic Filters to only subscribe to events of interest
var eventMappings = new EventHandlerDiscovery().Scan(productView).Handlers;
var subscriptionName = "admin_readmodel";
var topicFilter1 = "Admin.Common.Events";
var b = RabbitHutch.CreateBus("host=localhost");
b.Subscribe<PublishedMessage>(subscriptionName, m =>
{
Aggregate handler;
var messageType = Type.GetType(m.MessageTypeName);
var handlerFound = eventMappings.TryGetValue(messageType, out handler);
if (handlerFound)
{
var @event = JsonConvert.DeserializeObject(m.SerialisedMessage, messageType);
handler.AsDynamic().ApplyEvent(@event, ((Event)@event).Version);
}
},
q => q.WithTopic(topicFilter1));
Uses reflection and convention over configuration
All events subclass this
Dynamic call to avoid tight coupling with types
Filter to subset of events
PORTS: EVENT HANDLERSQuery microservices determine events they are interested in.Handle events using the same Event Handling pattern as used in the domain objects.Consistency reduces complexity.
public class ProductView : ReadModelAggregate,
IHandle<ProductCreated>, IHandle<ProductDescriptionChanged>,
IHandle<ProductNameChanged>, IHandle<ProductPriceChanged>
{
//...
public void Apply(ProductCreated e)
{
var dto = new ProductDto
{
Id = e.Id,
Name = e.Name,
Description = e.Description,
Price = e.Price,
Version = e.Version,
DisplayName = string.Format(displayFormat, e.Name, e.Description),
};
repository.Insert(dto);
}
Interested in 4 eventsLook familiar?
Queries return DTOs/Result Objects. Not domain objects.
Persist the DTO’s. Denormalised data is OK.
QUERIESQueries are simply WebAPI methodsSimple lookups of precomputed result(s) in the cached data.
REDIS REPOSITORYRedis: A key/value store, with friesCollections stored as ‘sets’Convention approach to ease implementation
Single objects stored using FQ type nameKey = MyApp.TypeName:ID | Value = JSON serialised object
All keys stored in a set, named using FQTNKey = MyApp.TypeNameSet | Values = MyApp.TypeName:ID1, MyApp.TypeName:ID2, etc
Redis can dereference keys in a Set, avoiding N+1 queries.
public IEnumerable<T> GetAll()
{
var get = new RedisValue[] { InstanceName() + "*" };
var result = database.SortAsync(SetName(), sortType: SortType.Alphabetic, by: "nosort", get: get).Result;
var readObjects = result.Select(v => JsonConvert.DeserializeObject<T>(v)).AsEnumerable();
return readObjects;
}
public void Insert(T t)
{
var serialised = JsonConvert.SerializeObject(t);
var key = Key(t.Id);
var transaction = database.CreateTransaction();
transaction.StringSetAsync(key, serialised);
transaction.SetAddAsync(SetName(), t.Id.ToString("N"));
var committed = transaction.ExecuteAsync().Result;
if (!committed)
{
throw new ApplicationException("transaction failed. Now what?");
}
}
Updating the Redis Cache
We cache JSON strings.
Simple Redis query
Return the DTOs we’d previously
persisted
DEPLOYMENT (& DOCKER)
LOCAL DEVELOPMENTBefore we deploy to <environment />, how do we test our microservices in concert?
WHAT IS THE VERSION OF THE APP? Consider having an environment configuration file
List the version of each microservice that has been tested as part of a “known good” configuration
-- OR --Ignore versioning!
Rely on production monitoring to discover problems, and quickly rollback changes
HOW DO YOU UPGRADE A MICROSERVICE?Microservices are small, replaceable units of functionality, right?Stop thinking about upgrading them.You don’t upgrade them; you replace them.Best approach? Isolate the service and it’s execution environment. Replace both at once.
DOCKER: TERMINOLOGYImage: a read only template for a container. Not runnable.Container: a runnable instance of an image.Registry: a collection of Docker images
CONTAINERS AND VERSIONINGContainers are immutable.You don’t upgrade them; you replace them.
No binary promotion to a production container.You promote the container itself to production.Use a repository to store images (e.g. artifactory)
DOCKER: PRODUCTION IN A BOXUse Docker-Compose to automatically build and run a set of containers that matches production.You may be limited by the resources of your dev box (RAM, CPU cores, disk).You could use Azure Container Services to spin up your configuration in the cloud instead.
OR USE A SUBSET OF YOUR CONTAINERSUse test/mock containers or microservices.Only spin up the services you need to test your work, and avoid all the other services that exist.Requires a bit more knowledge around what services to start, what to mock and what to ignore.
Could also use tools like wiremock to intercept and respond to HTTP requests. (more complex)
OR FOCUS ON PROVING YOUR CONTRACTSIf you’ve proven your microservice supports the defined contracts…
- HTTP API (consumer based contracts)- Events on a Message Bus
…then your microservice should work with everything else. Just deploy it!But you MUST have great testing, and strong operational monitoring in place.
DOCKER BASED DEVELOPMENT WORKFLOW 1. Build and test locally in a container2. Push code to source control. Automated build creates new container image.4. Image is pushed to image repository5. Image gets promoted through environments to prod.
HEY, MISTER!I DON’T WANT YOUR DOCKER KOOL-AID!
That’s cool. You don’t need Docker (or containers).
1. Always get the latest code you need.2. Manually build & run all of the services on your dev box
each time you test.3. Use scripting to make it a little less painful.
Side-effect: Encourages a low number of services.
RECAP1. Why?2. Architecture3. Implementation4. Deployment
Q&ARichard Banks - @rbanks54