I recall that in my junior years as a professional developer lot of systems I worked on introduced tons of duplication in regards to cross-cutting concerns. For example, most of the application-layers` methods would follow such a template:

public ResultClass DoSomething(InputClass input) {
    Log.Information("DoSomethin started");
    using(var unitOfWork = unitOfWorkProvider.Get()) {
        // .. actual work 
        unitOfWork.Complete();
    }
    Log.Information("DoSomething completed");       
}

Code regarding the unit of work, logging, error handling, etc. was mostly copied from method to method without much of a thought. Obviously, such an approach regularly resulted in various bugs. Sometimes the Complete method would be accidentally lost during the copying, other times log message was not adjusted so it conveyed incorrect information.

Additionally, adding any other cross-cutting concern was out of the question because it would impact all methods in all services.

At that time, I thought that only compilation-time tools like postsharp could help us, by generating a "boilerplate" code automatically. Luckily, later I realized that standard design patterns were perfectly capable of handling cross cutting concerns without compromising SOLID or DRY principles.

Constructing better alternative

In general, the code I showed above followed the abstract pattern.

Handle concern A
Handle concern B
Handle concern C, ...
Do Actual Work
Complete concern ... , C
Complete concern B
Complete concern A

We could point out that basically we first build the stack of concerns and after the actual work is done we pop each concern and complete it. That could be a major design tip for us. We could devise a solution based on the actual stack of objects each handling some cross-cutting concern.

Obviously, for this approach to be extensible, each "handler" should not know about other handlers, yet, they need to cooperate. Order of the handlers does matter, so some kind of root object must be responsible for arranging them.

That's when our pattern comes into play. It is designed to do exactly what we just described. If you don't know this pattern already take a look at those great explanations on refactoring.guru or sourcemaking.com

Chain of responsibility in the wild

Although almost no one declares their system as implementing a chain of responsibility pattern, most successful, extensible libraries do, in one form or the other. Let's take a look at a few examples

ASP.NET Core middlewares

Maybe one of the most familiar ones is the concept of ASP.NET Core middleware. The documentation defines middleware following:

Middleware is software that's assembled into an app pipeline to handle requests and responses. Each component:

• Chooses whether to pass the request to the next component in the pipeline.
• Can perform work before and after the next component in the pipeline.

As in every application of design pattern, they have their own vocabulary to describe actors in this system. Still, the cooperation pattern is implementing a chain of responsibility.

MediatR behaviors

MediatR is a library that on the top level implements the mediator pattern. It introduces an abstraction layer between the source of the command and its handler. However, each command, before it will be passed to the handler is processed by all registered pipeline behaviors.

Take a look at typical behavior for wrapping action in a transaction scope. Again, we can see the familiar next parameter wrapped by some logic preceding and following the invocation.

public class TransactionalBehavior<TRequest, TResponse> : IPipelineBehavior<TRequest, TResponse> {
    public async Task<TResponse> Handle(TRequest request, CancellationToken cancellationToken, RequestHandlerDelegate<TResponse> next) {
        using (var tx = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
        {
            await SomeMethodInTheCallStackAsync()
                .ConfigureAwait(false);

            tx.Complete();
        }
    }
}

Angular interceptors

@Injectable()
export class AuthorizationInterceptor implements HttpInterceptor {
    intercept(req: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>> {
        token = this.auth.getToken() // auth is provided via constructor.
        if (token) {
          // Logged in. Add Bearer token.
          return next.handle(
            req.clone({
              headers: req.headers.append('Authorization', 'Bearer ' + token)
            })
          );
        }
        // Not logged in. Continue without modification.
        return next.handle(req);
    }
}

Snippet source

Same story. If you know what to look for (request object req and next handler) you immediately recognize the chain of responsibility pattern.

For more examples of angular interceptors take a look at this post from angulardepth.com

Others

• Serilog enrichers
• NServiceBus pipeline behaviors
• ASP.NET MVC filters
• Entity Framework interceptors

Applying the pattern in applications

To use this pattern effectively you need to first identify all composition roots in your application. By composition root, I mean every kind of entry point to your application. Typical ones include:

• Web endpoint
• Message queue listener
• Batch jobs
• Background tasks

It is necessary to devise a solution per composition-root because cross-cutting concerns are going to be specific for each entry-point. As an example, your web endpoints will probably need some kind of "authorization handler" whereas message queue listeners probably not. On the other hand, both endpoints could use the same "transactions handler".

After you identified the composition roots you need to make sure that each composition root's dependency injection container can be configured independently. Obviously we aim to use full-blown DI for our solution and common configuration for all composition roots would render our efforts unnecessarily harder or even impossible for less sophisticated containers.

In the end, set up a "pipeline" in front of each of your composition roots. Each request/action should go through the pipeline and be processed by handlers of cross-cutting concerns. The actual handler executing the core logic should be free of dependencies to cross cutting concerns, focused on domain logic.

From my experience, such setup is the pillar of SOLID architecture and maintainable software.

Among many concerns that could be handled in such a manner you can find:

• Logging
• Transactions
• Authorization
• Profiling
• Exception-handling
• Throttling
• Caching
• Retrying
• Timeouts

Summary

Hopefully, I conveyed the following key points:

• Chain of responsibility it a simple yet powerful solution to handling cross-cutting concerns.
• Many extensible libraries implement the pattern, knowingly or not.
• You should have a "pipeline" in front of each composition root's entry-point.

Below you will find some valuable practices I introduced for my current employer during last year.
If you already use Serilog and want to improve your logging process, be my guest.

1. Aggregate all the logs streams in one system

That may be a no-brainer for experienced developers of distributed systems, but still, it is worth mentioning explicitly. You should have one place where all the logs are stored. I have been using Seq but lots of other products are available.

Having one dedicated system makes it so much easier to extract information from the data. It is also more efficient in terms of computing resources. Last, but not least, it is easier to configure and maintain one system.

2. Apply properties to identify each service

After all log streams are forwarded into one system, you will need a flexible way to filter them. One way to distinguish events from various services is to enrich events with a Service property on the code level.

loggerConfiguration
    .Enrich.FromLogContext()
    .Enrich.WithProperty("Service", "OrderService")

That way, however, we are hardcoding deployment-side configuration. Much more flexible approach is to use server-side property enrichment.

Seq has the ability to apply properties at the time of arrival from the source system. API keys are used to differentiate log events.

After a few attempts to structure those properties, I found out that the two-level hierarchy of properties is the most useful. The first level is for Application (for example "Shop") whereas the second level represents a Service within the application. Of course, general services that live outside any particular application are enriched only with 'Service' property.

3. Correlate log events related to a request end-to-end

When any kind of request arrives at the edges of your system, it should immediately get a globally unique identifier which should be attached to all the log events generated in response. The identifier should be persisted through time and space.

A typical scenario that illustrates the lifetime of the correlation id:

1. User clicks some button on your web/mobile app.
2. The application generates GUID and attaches it as an HTTP header.
3. Backend server recognizes the header and enriches all the subsequent events.

4. Backend server decides that it needs to send some asynchronous message via service bus. Message metadata is filled with correlation id.

5. Another service receives the message, recognizes the correlation id and again uses .PushProperty to enrich its log events.

6. In the end, the service decides to prepare an email to be sent later. Correlation id is persisted along to be used when the time to send the email will come.

This example shows how correlation id should be alive as long as any consequence of the initial request is in progress. Practice shows that company-wide correlation id is a crucial part of microservices architecture.

using (LogContext.PushProperty("CorrelationId", correlationId))
{
    await next(context);
}

Note that every system in the chain should expect correlation id to be passed but generate its own if none is received. That would ensure that area as big as possible is covered.

4. Install Serilog.Exceptions package

Serilog.Exceptions is an innocuously simple but indispensable tool. Without it, your exceptions are going to be logged using .ToString() which almost always skips some important pieces of information. The package will destructure your exceptions (including lists, dictionaries, nested objects and so on) and attach rich representation to your log events.

If you are using EntityFrameworkCode, there is an additional must-have package Serilog.Exceptions.EntityFrameworkCore.

BTW, I am the maintainer of the Serilog.Exceptions package. Visit github page for details.

5. Install SerilogAnalyzer package

Just do it. It uses Roslyn code analyzer infrastructure to pinpoint common pitfalls even before the code is compiled. There is no reason to refuse getting free help :-).

6. Use Fatal and Error only for events that need human attention

That point may be controversial but bear with me for a sec. In Serilog, we have six levels of log events: Fatal, Error, Warning, Information, Debug and Verbose. There is no logging semantics attached to any level. It is your choice as a developer on how to treat them. You could log events with exceptions at Debug level or use Fatal level to notify that user has provided an incorrect password.

On the other hand, levels can serve us best, if there will be some rules governing them, so the person analyzing the logs would be able to make sense of them. I found that if events related to known and anticipated situations like provided incomplete data for HTTP request(for example user did not provide mandatory last name) are logged on level Error, that the level is no longer indicator of a real problem. On the other hand, when only the unexpected exceptions and errors are logged on the Error level, we could take a glance at the logs and find "real problems" with our services.

Some of the expected situations I had moved to Information or Warning levels:

• Transient errors from systems like databases or event buses. They will happen, even more in the cloud environment.
• User requests validation errors. Assuming that your UI guides the user, there is noting erroneous about this situation.
• Rejected requests from the system under maintenance. Again, they are expected so there is nothing to do about that.

7. Configure alerts for errors

The more services your organization has the harder it is to keep up and analyze all the logs. Instead of polling your logging server once in a while I advise you to set up some automated process that would monitor some carefully chosen metrics and alert development team when they are exceeded.

Typical metric to be monitored are "errors/minute", "percentage of failed requests" or
90th percentile of request processing time above threshold".

However, no metric makes sense for everybody, so you will just need to select those that work for you.

8. Correlate on various levels

In one of the previous points, we discussed correlation id related to some external endpoint. This kind of correlation is the most important one, but not the only useful one.

You can correlate log events related to a particular run of the batch process. If the batch process is transforming files, you can then recursively correlate events related to particular file. ASP.NET Core automatically correlates event that happed within the connection by enriching events with ConnectionId property.

You will thank yourself if you will specify lots of those "correlation-scopes". When you have a problem and need to understand what happened based on logs, they will be your best friend.

9. Each state change should emit the event on the Information level

Even the smallest action matter when you are trying to analyze a particular scenario from logs. Adopt the rule of leaving at least one log event for each state change.

10. Don't forget to .CloseAndFlush

By default Serilog sends log events asynchronously. That means that there is a chance that your application could end before all the events are actually sent over the network. It is especially important for batch applications, that starts and stops on a regular basis.

public static async Task Main(string[] args)
{
    try
    {
        var host = hostBuilder.Build();
        await host.RunAsync();
    }
    catch (Exception e)
    {
        Log.Fatal(e, "Unexpected exception caught, shutting down");
        throw;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}

Explicitly flushing the buffer is not necessary for non-static loggers that are disposed of when they are done.

I will describe what I mean by "closing" in software development and why I think it is a very valuable and practical principle to follow.

The opposite of closing

Let's start from the opposite side and take a look at a few examples of behaviour I am sure everybody can relate to.

• Probably you worked with an analyst or product owner who often argues that additional requirements shall be part of next release after the scope has already been fixed. He may extend user stories after they have been developed. He tries to fix UX problems before users even have a chance to touch the product. He may tolerate but does not accept the idea of iterative development.

• You may know or work with managers who almost always say yes to a neat idea, but initial enthusiasm is rarely followed with any plan of action, not even mentioning successful implementation.

• Almost every larger project I saw have a huge backlog with hundreds of old user stories that nobody will ever consider for the upcoming sprints. Same goes for piles of TODOs scattered around the code, doomed to be forgotten, but still adding complexity.

• Developers that have an unbearable number of topics "open". The most pathological example I saw was a guy with tens of open issues in the tracker. There was no way to tell what this person was working on.

I can extend the list even more but I think you get the idea by now. Even though the behaviours I just described are from different branches and one may think they have nothing to do with other, I suggest that there is one underlying principle.

Those kinds of behaviour emerge when people, for some reason, can't mentally close topics they started in the past. They cannot accept that "later means never" and "there is no trying". I think it is caused by that tempting feeling of hope that "in the future" we will have some free time and we will fix the problem at hand. So, just in case, we keep that "TODO: design to be improved" comment that week from now, nobody will remember what was about. Just in case, we keep backlog with years of work, when we are not even sure if the project will last 6 more months.

Unfortunately, reality does not care about our hopes and dreams. I have yet to see a phase in the project when the team would have some "free time" to fix TODOs. Your neat idea from last week, if not aggressively pursued will perish under the road roller of new more important work items.

Better alternative: closing mindset

People are ridiculously bad at remembering stuff, especially in the complex and ever-changing field of software development. Therefore, we need to work in such a way that leaves minimal (even better nothing) amount of information to remember. Every time we have something "open" it takes our brain capacity and prevents us from focusing.

The closing mindset I advocate here means being realistic about ourselves and give up unfounded hope for "better" times. Either I am committed to the issue at hand and develop a plan to implement it immediately or not. In the latter case, we still need to apply closing mindset and either forget about the issue(remove user story, delete the todo etc.) or schedule plan to review the issue with a deadline.

Let's see how "closing" mindset works in examples:

• During sprint planning, product owner sees that one of the user stories is not groomed enough, there are many questions regarding business impact or implementation details. He immediately says "put that out of the sprint", which immediately shuts down unnecessary discussion and sends clear message that only well understood can be committed to. Developers feel respected and can commit to other well-specified user stories for that particular iteration.

• During code review, the developer finds out few TODOs left by a fellow colleague. She refuses to accept those and suggests either to remove them or add appropriate entries to the issue tracker, which can be later prioritized. The fellow developer, fixes one of the TODOs immediately, because it was really important but removes the others, which were not worth the effort. Code stays clean, and trust between developer is maintained.

• During the implementation phase of the iteration, the business analyst discovers that one user story could be improved by using some other user interface approach. He asks the developer directly if he can implement it on time. Developer kindly responds that it was not what the team committed to and asks the product owner to choose one of the alternatives. Either kill the sprint and start a new one or carry on with previous design and prioritize new idea for next iteration. Management does not want to disrupt developers and carries on with the sprint. During the next sprint planning, it turned out that there are many other more important user stories and proposed improvement was never implemented. Development time was saved and the team stayed on track to deliver the final product.

These are just a few examples of situations that happen all the time in real projects. By following the "closing" mindset we are gaining lots of advantages.

How exactly it helps?

• Builds trust. How can I trust that a developer is going to deliver when I can see tons of open issues he/she left in the past? How can you trust a manager that raises your hopes and promises to apply some changes for the better, but then forgets about them? Answer "yes" to few questions, but then commit 100% to them. If for some reason you need to back up from the promise, do that in the spirit of self-responsibility and explicitly close the topic.

• Encourages focus. Having a minimal number of clearly defined goals is a huge performance boost. Some distractions, like TODOs scattered here and there are relatively small, while others like extending the scope of user story during sprints are devastating. All of them, however, degrade performance and morale of the team.

• Maintains alignment between various stakeholders. Just think about the clarity that short backlog brings to the table. Anyone can take a look and instantly know the priorities of the project in a given point of time.

• Keeps morale high. Completing a successful iteration is one of the most gratifying moment for the development team. After a few successful sprints teams seem to get into the habit of "winning" which is huge in terms of team-bonding. However, this can be easily disrupted by constantly changing goals or leaving some work behind, for "later". Managers often are not aware of how much does overall mess demoralize developers.