Awesome

Microsoft.Diagnostics.EventFlow

Introduction

The EventFlow library suite allows applications to define what diagnostics data to collect, and where they should be outputted to. Diagnostics data can be anything from performance counters to application traces. It runs in the same process as the application, so communication overhead is minimized. It also has an extensibility mechanism so additional inputs and outputs can be created and plugged into the framework.

The EventFlow suite supports .NET applications and .NET Core applications. The core of the library, as well as inputs and outputs listed below are available as NuGet packages.

Topics

Getting started

Inputs

• Trace (a.k.a. System.Diagnostics.Trace)
• EventSource
• PerformanceCounter
• Serilog
• Microsoft.Extensions.Logging
• ETW (Event Tracing for Windows)
• Application Insights
• Log4net
• NLog
• DiagnosticSource
• ActivitySource

Outputs

• StdOutput (console output)
• HTTP (json via http)
• Application Insights
• Azure EventHub
• Elasticsearch
• Azure Monitor Logs

There are several EventFlow extensions available from non-Microsoft authors and vendors, including:

• Dynatrace output
• Google Big Query output
• SQL Server output
• ReflectInsight output
• Splunk output

Filters

Standard metatadata types

Health reporter

Pipeline settings

Service Fabric support

Filter expressions

Secret storage

Extensibility

Troubleshooting

Platform support

Contributing to EventFlow

Getting Started

The EventFlow pipeline is built around three core concepts: inputs, outputs, and filters. The number of inputs, outputs, and filters depend on the need of diagnostics. The configuration also has a healthReporter and settings section for configuring settings fundamental to the pipeline operation. Finally, the extensions section allows declaration of custom developed plugins. These extension declarations act like references. On pipeline initialization, EventFlow will search extensions to instantiate custom inputs, outputs, or filters.

1. To quickly get started, you can create a simple console application in VisualStudio and install the following Nuget packages:

   • Microsoft.Diagnostics.EventFlow.Inputs.Trace
   • Microsoft.Diagnostics.EventFlow.Outputs.ApplicationInsights
   • Microsoft.Diagnostics.EventFlow.Outputs.StdOutput

2. Add a JSON file named "eventFlowConfig.json" to your project and set the Build Action property of the file to "Copy if Newer". Set the content of the file to the following:

{
    "inputs": [
    {
      "type": "Trace",
      "traceLevel": "Warning"
    }
  ],
  "outputs": [
    // Please update the instrumentationKey.
    {
      "type": "ApplicationInsights",
      "instrumentationKey": "00000000-0000-0000-0000-000000000000"
    }
  ],
  "schemaVersion": "2016-08-11"
}

Note: if you are using VisualStudio for Mac, you might need to edit the project file directly. Make the following snippet for eventFlowConfig.json file is included in the project definition:

<ItemGroup>
 <None Include="eventFlowConfig.json">
   <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
 </None>
</ItemGroup>

3. If you wish to send diagnostics data to Application Insights, fill in the value for the instrumentationKey. If not, you can send traces to console output instead by replacing the Application Insights output with the standard output. The outputs property in the configuration should then look like this:

    "outputs": [
    {
      "type": "StdOutput"
    }
  ],

4. Create an EventFlow pipeline in your application code using the code below. Make sure there is at least one output defined in the configuration file. Run your application and see your traces in console output or in Application Insights.

    using (var pipeline = DiagnosticPipelineFactory.CreatePipeline("eventFlowConfig.json"))
    {
        System.Diagnostics.Trace.TraceWarning("EventFlow is working!");
        Console.WriteLine("Trace sent to Application Insights. Press any key to exit...");
        Console.ReadKey(intercept: true);
    }

It usually takes a couple of minutes for the traces to show in Application Insights Azure portal.

Back to Topics

Inputs

These define what data will flow into the engine. At least one input is required. Each input type has its own set of parameters.

Trace

Nuget Package: Microsoft.Diagnostics.EventFlow.Inputs.Trace

This input listens to traces written with System.Diagnostics.Trace API. Here is an example showing all possible settings:

{
    "type": "Trace",
    "traceLevel":  "Warning"
}

Back to Topics

EventSource

Nuget Package: Microsoft.Diagnostics.EventFlow.Inputs.EventSource

This input listens to EventSource traces. EventSource classes can be created in the application by deriving from the System.Diagnostics.Tracing.EventSource class. Here is an example showing all possible settings:

{
    "type": "EventSource",
    "sources": [
        {
            "providerName": "MyEventSource",
            "level": "Informational",
            "keywords": "0x7F"
        }
    ]
}

Top object

Source object (element of the sources array)

Remarks

(*) Out of providerName, providerNamePrefix and disabledProviderNamePrefix, only one can be used for a single source. In other words, with a single source one can enable an EventSource by name, or enable a set of EventSources by prefix, or disable a set of EventSources by prefix.

(**) level and keywords can be used for enabling EventSources, but not for disabling them. Disabling events using level and keywords is not supported (but one can use level and/or keywords to selectively enable a subset of events from a given EventSource).

(***) There is an issue with .NET frameworks 4.6 and 4.7, and .NET Core framework 1.1 and 2.0 where dynamically created EventSource events are dispatched to all listeners, regardless whether listeners subscribe to events from these EventSources; for more information see https://github.com/dotnet/coreclr/issues/14434 disabledProviderNamePrefix property can be usesd to suppress these events.<br/> Disabling EventSources is not recommended under normal circumstances, as it introduces a slight performance penalty. Instead, selectively enable necessary events through combination of EventSource names, event levels, and keywords.

Back to Topics

DiagnosticSource

Nuget Package: Microsoft.Diagnostics.EventFlow.Inputs.DiagnosticSource

This input listens to System.Diagnostics.DiagnosticSource sources. See the DiagnosticSource User's Guide for details of usage. Here is an example showing all possible settings:

{
    "type": "DiagnosticSource",
    "sources": [
        { "providerName": "MyDiagnosticSource" }
    ]
}

Top object

Source object (element of the sources array)

Back to Topics

ActivitySource

Nuget Package: Microsoft.Diagnostics.EventFlow.Inputs.ActivitySource

This input listens to System.Diagnostics.ActivitySource sources (introduced in .NET 5.0). ActivitySource is designed to emit telemetry in a way that is compatible with OpenTelemetry specification. Here is an example showing all possible settings:

{
    "type": "ActivitySource",
    "sources": [
        { 
            "ActivitySourceName": "JobSchedulingActivitySource",
            "ActivityName": "JobStatusRequest",
            "CapturedData": "PropagationData"
        },
        { 
            "ActivitySourceName": "JobSchedulingActivitySource",
            "ActivityName": "NewJobRequest",
            "CapturedData": "AllDataAndRecorded",
            "CapturedEvents": "Both" 
        }
    ]
}

Top object

Source object (element of the sources array)

Notes

Be careful about leaving ActivitySourceName and ActivityName blank. It might be tempting to capture all activity data for the process, but it might have a significant, negative impact on performance, and result in a lot of data that will be expensive to process and store.

It is possible to come up with configuration that will "match" a given activity multiple times. Records in the sources array are matched to activities in the order in which they appear in configuration; the first record "wins" and determines what settings will be used. For example, if the configuration is

{
    "Type": "ActivitySource",
    "Sources": [
        { "ActivitySourceName": "S", "CapturedData": "PropagationData" },
        { "ActivitySourceName": "S", "ActivityName": "A", "CapturedData": "AllData" }
    ]
}

and activity A from source S occurs, the first sources record will match the activity, and only propagation data will be captured for activity A. That is why more specific sources should generally precede less specific sources (ones that omit ActivitySourceName or ActivityName). A common scenario when this rule comes handy is when it is necessary to capture all data from specific subset of activities from a given source, and then capture propagation data for all other activities from the same source.

Back to Topics

PerformanceCounter

Nuget Package: Microsoft.Diagnostics.EventFlow.Inputs.PerformanceCounter

This input enables gathering data from Windows performance counters. Only process-specific counters are supported, that is, the counter must have an instance that is associated with the current process. For machine-wide counters use an external agent such as Azure Diagnostics Agent or create a custom input.

Finding the counter instance that corresponds to the current process

In general there is no canonical way to find a performance counter instance that corresponds to current process. Two methods are commonly used in practice:

• A special performance counter that provides instance name to process ID mapping. This solution involves a set of counters that use the same instance name for a given process. Among them there is a special counter with a value that is the process ID of the corresponding process. Searching for the instance of the special counter with a value equal to the current process ID allows to discover what the instance name is used for the current process. Examples of this approach include the Windows Process category (special counter "ID Process") and all .NET counters (special counter "Process ID" in the ".NET CLR Memory" category).

• Process ID can be encoded directly into the instance name. .NET performance counters can use this approach when ProcessNameFormat flag is set in the in the registry.

EventFlow PerformanceCounter input supports the first method of deterimining counter instance name for current process via configuration settings. It also supports the second method, but only for .NET performance counters.

Configuration example

{
    "type": "PerformanceCounter",
    "sampleIntervalMsec": "5000",
    "counters": [
        {
            "counterCategory": "Process",
            "counterName": "Private Bytes"
        }, 
        {
            "counterCategory": ".NET CLR Exceptions",
            "counterName": "# of Exceps Thrown / sec"
        }
    ]
}

Top-level configuration settings

Counter class

Important usage note

Some performance counters require the user to be a member of the Performance Monitor Users system group. This can manifest itself by health reporter reporting "category does not exist" errors from PerformanceCounter output, despite the fact that the category and counter are properly configured and clearly visible in Windows Performance Monitor. If you need to consume such counters, make sure the account your process runs under belongs to Performance Monitor Users group.

Back to Topics

Serilog

Nuget package: Microsoft.Diagnostics.EventFlow.Inputs.Serilog

This input enables capturing diagnostic data created through Serilog library.

Configuration example

{
  "type": "Serilog",
  "useSerilogDepthLevel": true
}

Top-level configuration settings

Example: instantiating a Serilog logger that uses EventFlow Serilog input

using System;
using System.Linq;
using Microsoft.Diagnostics.EventFlow;
using Serilog;

namespace SerilogEventFlow
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var pipeline = DiagnosticPipelineFactory.CreatePipeline(".\\eventFlowConfig.json"))
            {
                Log.Logger = new LoggerConfiguration()
                    .WriteTo.EventFlow(pipeline)
                    .CreateLogger();

                Log.Information("Hello from {friend} for {family}!", "SerilogInput", "EventFlow");
                
                Log.CloseAndFlush();
                Console.ReadKey();
            }
        }
    }
}

Back to Topics

Microsoft.Extensions.Logging

Nuget package: Microsoft.Diagnostics.EventFlow.Inputs.MicrosoftLogging

This input enables capturing diagnostic data created through Microsoft.Extensions.Logging library and ILogger interface.

Configuration example The ILogger input has no configuration, other than the "type" property that specifies the type of the input (must be "Microsoft.Extensions.Logging"):

{
  "type": "Microsoft.Extensions.Logging"
}

Example: instantiating a ILogger that uses EventFlow ILogger input

using Microsoft.Diagnostics.EventFlow;
using Microsoft.Extensions.Logging;

namespace LoggerEventFlow
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var pipeline = DiagnosticPipelineFactory.CreatePipeline(".\\eventFlowConfig.json"))
            {
                var factory = new LoggerFactory()
                    .AddEventFlow(pipeline);

                var logger = new Logger<Program>(factory);
                using (logger.BeginScope(myState))
                {
                    logger.LogInformation("Hello from {friend} for {family}!", "LoggerInput", "EventFlow");
                }
            }
        }
    }
}

Example: using EventFlow ILogger input with ASP.NET Core The following example shows how to enable EventFlow ILogger inside a Service Fabric stateless service that uses ASP.NET Core.

1. Modify the service class so that its constructor takes a DiagnosticPipeline instance as a parameter:

        private static void Main()
        {
            try
            {
                using (ManualResetEvent terminationEvent = new ManualResetEvent(initialState: false))
                using (var pipeline = ServiceFabricDiagnosticPipelineFactory.CreatePipeline("CoreBasedFabricPlusEventFlow-Diagnostics"))
                {
                    Console.CancelKeyPress += (sender, eventArgs) => Shutdown(diagnosticsPipeline, terminationEvent);

                    AppDomain.CurrentDomain.UnhandledException += (sender, unhandledExceptionArgs) =>
                    {
                        ServiceEventSource.Current.UnhandledException(unhandledExceptionArgs.ExceptionObject?.ToString() ?? "(no exception information)");
                        Shutdown(diagnosticsPipeline, terminationEvent);
                    };

                    ServiceRuntime.RegisterServiceAsync("Web1Type",
                        context => new Web1(context, pipeline)).GetAwaiter().GetResult();

                    ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(Web1).Name);

                    terminationEvent.WaitOne();
                }
            }
            catch (Exception e)
            {
                ServiceEventSource.Current.ServiceHostInitializationFailed(e.ToString());
                throw;
            }
        }

        private static void Shutdown(IDisposable disposable, ManualResetEvent terminationEvent)
        {
            try
            {
                disposable.Dispose();
            }
            finally
            {
                terminationEvent.Set();
            }
        }

2. In the CreateServiceInstanceListeners() method add the pipeline as a singleton service to ASP.NET dependency injection container

        protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
        {
            return new ServiceInstanceListener[]
            {
                new ServiceInstanceListener(serviceContext =>
                    new WebListenerCommunicationListener(serviceContext, "ServiceEndpoint", (url, listener) =>
                    {
                        ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting WebListener on {url}");

                        return new WebHostBuilder().UseWebListener()
                                    .ConfigureServices(
                                        services => services
                                            .AddSingleton<StatelessServiceContext>(serviceContext)
                                            .AddSingleton<DiagnosticPipeline>(this.diagnosticPipeline))
                                    .UseContentRoot(Directory.GetCurrentDirectory())
                                    .UseStartup<Startup>()
                                    .UseApplicationInsights()
                                    .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
                                    .UseUrls(url)
                                    .Build();
                    }))
            };
        }

3. In the Startup class configure the loggerFactory by calling AddEventFlow on it:

        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
        {
            loggerFactory.AddConsole(Configuration.GetSection("Logging"));
            loggerFactory.AddDebug();
            var diagnosticPipeline = app.ApplicationServices.GetRequiredService<DiagnosticPipeline>();
            loggerFactory.AddEventFlow(diagnosticPipeline);

            app.UseMvc();
        }

4. Now you can assume the logger factory will be constructor-injected into your controllers:

    [Route("api/[controller]")]
    public class ValuesController : Controller
    {
        private readonly ILogger<ValuesController> logger;

        public ValuesController(ILogger<ValuesController> logger)
        {
            this.logger = logger;
        }

        // GET api/values
        [HttpGet]
        public IEnumerable<string> Get()
        {
            this.logger.LogInformation("Hey, someone just called us!");
            return new string[] { "value1", "value2" };
        }

      // (rest of controller code is irrelevant)

Back to Topics

ETW (Event Tracing for Windows)

Nuget package: Microsoft.Diagnostics.EventFlow.Inputs.Etw

This input captures data from Microsoft Event Tracing for Windows (ETW) providers. Both manifest-based providers as well as providers based on managed EventSource infrastructure are supported. The data is captured machine-wide and requires that the identity the process uses belongs to Performance Log Users built-in administrative group.

Note

To capture data from EventSources running in the same process as EventFlow, the EventSource input is a better choice, with better performance and no additional security requirements.

Configuration example

{
    "type": "ETW",
    "sessionNamePrefix": "MyCompany-MyApplication",
    "cleanupOldSessions": true,
    "reuseExistingSession": true,
    "providers": [
        {
            "providerName": "Microsoft-ServiceFabric",
            "level": "Warning",
            "keywords": "0x7F"
        }
    ]
}

Top Object

Providers object

(*) Either providerName, or providerGuid must be specified. When both are specified, provider GUID takes precedence.

Back to Topics

Application Insights input

Nuget package: Microsoft.Diagnostics.EventFlow.Inputs.ApplicationInsights

Application Insights input is designed for the following scenario:

1. You have an application that uses Application Insights for monitoring and diagnostics.
2. You want to send a portion of your Application Insights telemetry to some destination other than Application Insights (e.g. Azure EventHub or Elasticsearch; the assumption is there is an EventFlow output for where the data needs to go).

For example, you might want to leverage Application Insights sampling capabilities to reduce the amount of data analyzed by Application Insights without losing analysis fidelity, while sending full raw logs to Elasticsearch to do detailed log search during problem troubleshooting.

Application Insights input supports all standard Application Insights telemetry types: trace, request, event, dependency, metric, exception, page view and availability.

Usage–leveraging EventFlow configuration file

To use Application Insights input in an application that creates EventFlow pipeline using a configuration file, do the following:

1. Add the EventFlowTelemetryProcessor to your Application Insights configuration file (it goes into TelemetryProcessors element):

   <ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings" >
     <!-- ... -->
     <TelemetryProcessors>
        <!-- ... -->
        <Add Type="Microsoft.Diagnostics.EventFlow.ApplicationInsights.EventFlowTelemetryProcessor, Microsoft.Diagnostics.EventFlow.Inputs.ApplicationInsights" />
        <!-- ... -->
     </TelemetryProcessors>
     <!-- ... -->
   </ApplicationInsights>
   

   Note that the order of telemetry processors does matter. In particular, if the EventFlowTelemetryProcessor is placed before the Application Insights sampling processor, EventFlow will capture all telemetry, but if the EventFlowTelemetryProcessor is placed after the sampling processor, it will only "see" telemetry that was sampled in. For more information on configuring Application Insights see Application Insights configuration documentation.

2. In the EventFlow configuration make sure to include the Application Insights input. It does not take any parameters:

   { "type": "ApplicationInsights" }
   

3. In your application code, after the EventFlow pipeline is created, find the EventFlowTelemetryProcessor and set its Pipeline property to the instance of the EventFlow pipeline:

   using (var pipeline = DiagnosticPipelineFactory.CreatePipeline("eventFlowConfig.json"))
   {
       // ...
       EventFlowTelemetryProcessor efTelemetryProcessor = TelemetryConfiguration.Active.TelemetryProcessors.OfType<EventFlowTelemetryProcessor>().First();
       efTelemetryProcessor.Pipeline = pipeline;
       // ...
   }
   

This is it–after the EventFlowTelemetryProcessor.Pipeline property is set, the EventFlowTelemetryProcessor will start sending AI telemetry into the EventFlow pipeline.

Usage–ASP.NET Core

ASP.NET Core application tend to use code to create various parts of their request processing pipeline, and both EventFlow and Application Insights follow that model. Application Insights input in EventFlow provides a class called EventFlowTelemetryProcessorFactory that helps connecting Application Insights with EventFlow in ASP.NET Core environment. Here is an example how one could set up Application Insights input to send telemetry to Elasticsearch:

using System;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Diagnostics.EventFlow;
using Microsoft.Diagnostics.EventFlow.ApplicationInsights;
using Microsoft.Diagnostics.EventFlow.HealthReporters;
using Microsoft.Diagnostics.EventFlow.Inputs;
using Microsoft.Diagnostics.EventFlow.Outputs;
using Microsoft.Diagnostics.EventFlow.Configuration;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.ApplicationInsights.AspNetCore;

namespace AspNetCoreEventFlow
{
    public class Program
    {
        public static void Main(string[] args)
        {
            using (var eventFlow = CreateEventFlow(args))
            {
                BuildWebHost(args, eventFlow).Run();
            }
        }

        public static IWebHost BuildWebHost(string[] args, DiagnosticPipeline eventFlow) =>
            WebHost.CreateDefaultBuilder(args)
                .ConfigureServices(services => services.AddSingleton<ITelemetryProcessorFactory>(sp => new EventFlowTelemetryProcessorFactory(eventFlow)))
                .UseStartup<Startup>()
                .UseApplicationInsights()
                .Build();

        private static DiagnosticPipeline CreateEventFlow(string[] args)
        {
            // Create configuration instance to access configuration information for EventFlow pipeline
            // To learn about common configuration sources take a peek at https://github.com/aspnet/MetaPackages/blob/master/src/Microsoft.AspNetCore/WebHost.cs (CreateDefaultBuilder method). 
            // Here we assume all necessary information comes from command-line arguments and environment variables.
            var configBuilder = new ConfigurationBuilder()
                .AddEnvironmentVariables();
            if (args != null)
            {
                configBuilder.AddCommandLine(args);
            }
            var config = configBuilder.Build();

            var healthReporter = new CsvHealthReporter(new CsvHealthReporterConfiguration());
            var aiInput = new ApplicationInsightsInputFactory().CreateItem(null, healthReporter);
            var inputs = new IObservable<EventData>[] { aiInput };
            var sinks = new EventSink[]
            {
                new EventSink(new ElasticSearchOutput(new ElasticSearchOutputConfiguration {
                    ServiceUri = config["ElasticsearchServiceUri"]
                    // Set other configuration settings, as necessary
                }, healthReporter), null)
            };

            return new DiagnosticPipeline(healthReporter, inputs, null, sinks, null, disposeDependencies: true);
        }
    }
}

Back to Topics

Log4net

Nuget package: Microsoft.Diagnostics.EventFlow.Inputs.Log4net

This input enables capturing diagnostic data sent to the Log4net project.

Configuration example The Log4net input has one configuration, the Log4net Level:

{
  "type": "Log4net",
  "logLevel": "Debug"
}

Example: instantiating a Log4net logger that uses EventFlow Log4net input

using System;
using Microsoft.Diagnostics.EventFlow;
using log4net;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var pipeline = DiagnosticPipelineFactory.CreatePipeline(".\\eventFlowConfig.json"))
            {
                var logger = LogManager.GetLogger("EventFlowRepo", "MY_LOGGER_NAME");

                logger.Debug("Hey! Listen!", new Exception("uhoh"));
            }
        }
    }
}

Back to Topics

NLog

Nuget package: Microsoft.Diagnostics.EventFlow.Inputs.NLog

This input enables capturing diagnostic data sent to the NLog library.

Configuration example The NLog input has no configuration, other than the "type" property that specifies the type of the input (must be "NLog"):

{
  "type": "NLog"
}

Example: instantiating a NLog logger that uses EventFlow NLog input

using System;
using Microsoft.Diagnostics.EventFlow;
using NLog;

namespace NLogEventFlow
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var pipeline = DiagnosticPipelineFactory.CreatePipeline(".\\eventFlowConfig.json"))
            {
                var nlogTarget = pipeline.ConfigureNLogInput(pipeline, NLog.LogLevel.Info);
                var logger = NLog.LogManager.GetCurrentClassLogger();
                logger.Info("Hello from {friend} for {family}!", "NLogInput", "EventFlow");
                NLog.LogManager.Shutdown();
                Console.ReadKey();
            }
        }
    }
}

Back to Topics

Outputs

Outputs define where data will be published from the engine. It's an error if there are no outputs defined. Each output type has its own set of parameters.

StdOutput

Nuget Package: Microsoft.Diagnostics.EventFlow.Outputs.StdOutput

This output writes data to the console window. Here is an example showing all possible settings:

{
    "type": "StdOutput"
}

Back to Topics

Http

Nuget Package: Microsoft.Diagnostics.EventFlow.Outputs.HttpOutput

This output writes data to a webserver using diffent encoding methods (Json or JsonLines, eg. for logstash). Here is an example showing all possible settings:

{
    "type": "Http",
    "serviceUri": "https://example.com/",
    "format": "Json",
    "httpContentType": "application/x-custom-type",
    "basicAuthenticationUserName": "httpUser1",
    "basicAuthenticationUserPassword": "<MyPassword>"
}

Back to Topics

Event Hub

Nuget Package: Microsoft.Diagnostics.EventFlow.Outputs.EventHub

This output writes data to the Azure Event Hub.

Here is an example showing configuration using connection string:

{
    "type": "EventHub",
    "eventHubName": "myEventHub",
    "connectionString": "Endpoint=sb://<myEventHubNamespace>.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=<MySharedAccessKey>"
}

Here is an example showing configuration using Azure Identity:

{
    "type": "EventHub",
    "useAzureIdentity": true,
    "fullyQualifiedNamespace": "<myEventHubNamespace>.servicebus.windows.net",
    "eventHubName": "myEventHub"
}

(*) Either connectionString or useAzureIdentity must be specified. useAzureIdentity takes precedence.

(**) When useAzureIdentity is set to true then both eventHubName and fullyQualifiedNamespace must be specified.

Back to Topics

Application Insights

Nuget Package: Microsoft.Diagnostics.EventFlow.Outputs.ApplicationInsights

This output writes data to the Azure Application Insights service. Here is an example showing all possible settings:

{
    "type": "ApplicationInsights",
    "instrumentationKey": "00000000-0000-0000-0000-000000000000",
    "configurationFilePath": "path-to-ApplicationInsights.config"
}

(*) At least one of the parameters: instrumentationKey, connectionString, or configurationFilePath must be provided for the Application Insights output configuration to be valid.

In Service Fabric environment the Application Insights configuration file can should be part of the default service configuration package (the 'Config' package). To resolve the path of the configuration file within the service configuration package set the value of configurationFilePath to servicefabricfile:/ApplicationInsights.config. For more information on this syntax see Service Fabric support

Standard metadata support

Application Insights output supports all standard metadata (request, metric, dependency and exception). Each of these metadata types corresponds to a native Application Insights telemetry type, enabling rich support for visualization and alerting that Application Insights provides. The Application Insights output also supports event metadata (corresponding to AI event telemetry type). This metadata is meant to represent significant application events, like a new user registered with the system or a new version of the code being deployed. Event metadata supports following properties:

Remarks:

1. Either eventNameProperty or eventName must be given.

All other events will be reported as Application Insights traces (telemetry of type Trace).

Back to Topics

Elasticsearch

Nuget Package: Microsoft.Diagnostics.EventFlow.Outputs.ElasticSearch

This output writes data to the Elasticsearch. Here is an example showing all possible settings:

{
    "type": "ElasticSearch",
    "indexNamePrefix": "app1",
    "indexFormat": "yyyy.MM.dd",
    "serviceUri": "https://myElasticSearchCluster:9200;https://myElasticSearchCluster:9201;https://myElasticSearchCluster:9202",
    "connectionPoolType": "Sniffing",
    "basicAuthenticationUserName": "esUser1",
    "basicAuthenticationUserPassword": "<MyPassword>",
    "numberOfShards": 1,
    "numberOfReplicas": 1,
    "refreshInterval": "15s",
    "defaultPipeline": "my-pipeline",
    "mappings": {
        "properties": {
            "timestamp": {
                "type": "date_nanos"
            }
        }
    },
    "proxy": {
        "uri": "https://myESProxy/"
    }
}

Standard metadata support

Elasticsearch output supports all standard metadata types. Events decorated with metadata will get additional properties when sent to Elasticsearch.

Fields injected by metric metadata are:

Fields injected byt the request metadata are:

Elasticsearch version support

Back to Topics

Azure Monitor Logs

Nuget package: Microsoft.Diagnostics.EventFlow.Outputs.AzureMonitorLogs

The Azure Monitor Logs output writes data to Azure Monitor Logs service (also knowns as Log Analytics service) via HTTP Data Collector API. You will need to create a Log Analytics workspace in Azure and know its ID and key before using Azure Monitor Logs output. Here is a sample configuration fragment enabling the output:

{
  "type": "AzureMonitorLogs",
  "workspaceId": "<workspace-GUID>",
  "workspaceKey": "<base-64-encoded workspace key>",
  "serviceDomain" : "<optional domain for Log Analytics workspace>"
}

Supported configuration settings are:

Back to Topics

Filters

As data comes through the EventFlow pipeline, the application can add extra processing or tagging to them. These optional operations are accomplished with filters. Filters can transform, drop, or tag data with extra metadata, with rules based on custom expressions. With metadata tags, filters and outputs operating further down the pipeline can apply different processing for different data. For example, an output component can choose to send only data with a certain tag. Each filter type has its own set of parameters.

Filters can appear in two places in the EventFlow configuration: on the same level as inputs and outputs (global filters) and as part of output declaration (output-specific filters). Global filters are applied to all data coming from the inputs. Output-specific filters are applied to just one output, just before the data reaches the output. Here is an example with two global filters and one output-specific filter:

{
    "inputs": [...],

    "filters": [
        // The following are global filters
        {
            "type": "drop",
            "include": "..."
        },
        {
            "type": "drop",
            "include": "..."
        }
    ],

    "outputs": [
        {
            "type": "ApplicationInsights",
            "instrumentationKey": "00000000-0000-0000-0000-000000000000",
            "filters": [
                {
                    "type": "metadata",
                    "metadata": "metric",
                    // ... 
                }
            ]
        }
    ]
}

EventFlow comes with two standard filter types: drop and metadata.

Drop

Nuget Package: Microsoft.Diagnostics.EventFlow.Core

This filter discards all data that satisfies the include expression. Here is an example showing all possible settings:

{
    "type": "drop",
    "include": "Level == Verbose || Level == Informational"
}

Back to Topics

Metadata

Nuget Package: Microsoft.Diagnostics.EventFlow.Core

This filter adds additional metadata to all event data that satisfies the include expression. The filter recognizes a few standard properties (type, metadata and include); the rest are custom properties, specific for the given metadata type:

{
    "type": "metadata",
    "metadata": "metric",
    "include": "ProviderName == MyEventProvider && EventId == 3",
    "customTag1": "tag1",
    "customTag2": "tag2"
}

Here are a few examples of using the metadata filter:

1. Submit a metric with a value of 1 (a counter) whenever there is a Service Fabric stateful service run failure

   {
   "type": "metadata",
   "metadata": "metric",
   "include": "ProviderName==Microsoft-ServiceFabric-Services 
               && EventName==StatefulRunAsyncFailure",
   "metricName": "StatefulRunAsyncFailure",
   "metricValue": "1.0"
   }
   

2. Turn processor time performance counter into a metric

   {
     "type": "metadata",
     "metadata": "metric",
     "include": "ProviderName==EventFlow-PerformanceCounterInput && CounterCategory==Process 
                 && CounterName==\"% Processor Time\"",
     "metricName": "MyServiceProcessorTimePercent",
     "metricValueProperty": "Value"
   }
   

3. Turn a custom EventSource event into a request. The event has 3 interesting properties: requestTypeName indicates what kind of request it was; durationMsec has the total request processing duration and ‘isSuccess’ indicates whether the processing succeeded or failed

   {
     "type": "metadata",
     "metadata": "request",
     "include": "ProviderName==MyCompany-MyApplication-FrontEndService 
                 && EventName==ServiceRequestStop",
     "requestNameProperty": "requestTypeName",
     "durationProperty": "durationMsec",
     "isSuccessProperty": "isSuccess"
   }
   

Back to Topics

Standard metadata types

EventFlow core library defines several standard metadata types. They have pre-defined set of fields and are recognized by Application Insights and Elasticsearch outputs (see documentation for each output, respectively, to learn how they handle standard metadata).

Metric metadata type

Metrics are named time series of floating-point values. Metric metadata defines how metrics are derived from ordinary events. Following fields are supported:

Remarks:

1. Either metricName or metricNameProperty must be specified.
2. Either metricValue or metricValueProperty must be specified.

Back to Topics

Request metadata type

Requests are special events that represent invocations of a network service by its clients. Request metadata defines how requests are derived from ordinary events. Following fields are supported:

Back to Topics

Dependency metadata type

Dependency event represents the act of calling a service that your service depends on. It has the following properties:

Back to Topics

Exception metadata type

Exception event corresponds to an occurrence of an unexpected exception. Usually a small amount of exceptions is continuously being thrown, caught and handled by a .NET process, this is normal and should not raise a concern. On the other hand, if an exception is unhandled, or unexpected, it needs to be logged and examined. This metadata is meant to cover the second case. It has the following properties:

Also see Application Insight Exception metadata type with EvenSource input issue

Back to Topics

Health Reporter

Every software component can generate errors or warnings the developer should be aware of. The EventFlow library is no exception. An EventFlow health reporter reports errors and warnings generated by any components in the EventFlow pipeline. In what format the report is presented depends on the implementation of the health reporter. The EventFlow library suite includes two health reporters: CsvHealthReporter and ServiceFabricHealthReporter.

The CsvHealthReporter is the default health reporter for EventFlow library and is used if the health reporter configuration section is omitted. Its configuration parameters are described below.

ServiceFabricHealthReporter is described in the Service Fabric support paragraph. It is designed to be used in the context of Service Fabric applications and does not need any configuration.

CsvHealthReporter

Nuget Package: Microsoft.Diagnostics.EventFlow.Core

This health reporter writes all errors, warnings, and informational traces generated from the pipeline into a CSV file. Here is an example showing all possible settings:

"healthReporter": {
    "type": "CsvHealthReporter",
    "logFileFolder": ".",
    "logFilePrefix": "HealthReport",
    "minReportLevel": "Warning",
    "throttlingPeriodMsec": "1000",
    "singleLogFileMaximumSizeInMBytes": "8192",
    "logRetentionInDays": "30",
    "ensureOutputCanBeSaved": "false",
    "assumeSharedLogFolder": "false"
}

CsvHealthReporter will try to open the log file for writing during initialization. If it can't, by default, a debug message will be output to the debugger viewer like Visual Studio Output window, etc. This can happen especially if a value for the log file path is not provided (default is used, which is application executables folder) and the application executables are residing on a read-only file system. Docker tools for Visual Studio use this configuration during debugging, so for containerized services the recommended practice is to specify the log file path explicitly.

Back to Topics

Pipeline Settings

The EventFlow configuration has settings allowing the application to adjust certain behaviors of the pipeline. These range from how many events the pipeline buffer, to the timeout the pipeline should use when waiting for an operation. If this section is omitted, the pipeline will use default settings. Here is an example of all the possible settings:

"settings": {
    "pipelineBufferSize": "1000",
    "maxEventBatchSize": "100",
    "maxBatchDelayMsec": "500",
    "maxConcurrency": "8",
    "pipelineCompletionTimeoutMsec": "30000"
}

Back to Topics

Service Fabric Support

Nuget Package: Microsoft.Diagnostics.EventFlow.ServiceFabric

This package contains two components that make it easier to include EventFlow in Service Fabric applications: the ServiceFabricDiagnosticPipelineFactory and ServiceFabricHealthReporter. ServiceFabricHealthReporter is used automatically by ServiceFabricDiagnosticPipelineFactory. It does not require any configuration and does not need to be listed in the pipeline configuration file.

The ServiceFabricDiagnosticPipelineFactory is a replacement for the standard DiagnosticPipelineFactory, one that uses Service Fabric configuration support to load pipeline configuration. The resulting pipeline reports any execution problems through the Service Fabric health subsystem. The factory exposes a static Create() method that takes the following parameters:

The recommended place to create the diagnostic pipeline is in the service Main() method. The following code will work all types of Service Fabric services (statefull, stateless and actor):

public static void Main(string[] args)
{
    try
    {
        using (ManualResetEvent terminationEvent = new ManualResetEvent(initialState: false))
        using (var diagnosticsPipeline = ServiceFabricDiagnosticPipelineFactory.CreatePipeline("MyApplication-MyService-DiagnosticsPipeline"))
        {
            Console.CancelKeyPress += (sender, eventArgs) => Shutdown(diagnosticsPipeline, terminationEvent);

            AppDomain.CurrentDomain.UnhandledException += (sender, unhandledExceptionArgs) =>
            {
                ServiceEventSource.Current.UnhandledException(unhandledExceptionArgs.ExceptionObject?.ToString() ?? "(no exception information)");
                Shutdown(diagnosticsPipeline, terminationEvent);
            };

            ServiceRuntime.RegisterServiceAsync("MyServiceType", ctx => new MyService(ctx)).Wait();

            ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(MyService).Name);

            terminationEvent.WaitOne();
        }
    }
    catch (Exception e)
    {
        ServiceEventSource.Current.ServiceHostInitializationFailed(e.ToString());
        throw;
    }    
}

private static void Shutdown(IDisposable disposable, ManualResetEvent terminationEvent)
{
    try
    {
        disposable.Dispose();
    }
    finally
    {
        terminationEvent.Set();
    }
}

The purpose of handling CancelKeyPress and UnhandledException events (the latter for full .NET Framework only) is to ensure that the EventFlow pipeline is cleanly disposed. This is important for pipeline elements that rely on system-level resources. For example, Event Tracing for Windows (ETW) input creates a system-wide ETW listening session, which must be disposed when the EventFlow pipeline is shut down. Ctrl-C signal is the standard way the Service Fabric runtime uses to notify service processes that they need to perform cleanup and exit. By default the process has 30 seconds to react.

The UnhandledException event method is a very simple addition to the standard ServiceEventSource:

    [Event(UnhandledExceptionEventId, Level = EventLevel.Error, Message = "An unhandled exception has occurred")]
    public void UnhandledException(string exception)
    {
       WriteEvent(UnhandledExceptionEventId, exception);
    }

Depending on the type of inputs and outputs used, additional startup code may be necessary. For example Microsoft.Extensions.Logging input requires a call to LoggerFactory.AddEventFlow() method to register EventFlow logger provider.

Back to Topics

Support for Service Fabric settings and application parameters

Version 1.0.1 of the EventFlow Service Fabric NuGet package introduced the ability to refer to Service Fabric settings from EventFlow configuration using special syntax for values:

servicefabric:/<section-name>/<setting-name>

where <section-name> is the name of Service Fabric configuration section and <setting-name> is the name of the Service Fabric configuration setting that is providing the value for some EventFlow setting. For example:

"basicAuthenticationUserPassword": "servicefabric:/DiagnosticPipelineParameters/ElasticSearchUserPassword"

The EventFlow configuration entry above means "take the ElasticSearchUserPassword setting from DiagnosticPipelineParameters section of the Service Fabric service configuration and use its value as the value for the EventFlow basicAuthenticationUserPassword setting". As with other Service Fabric settings, the values can also be overriden by Service Fabric application parameters. For more information on this see Manage application parameters for multiple environments topic in Service Fabric documentation.

Version 1.1.2 added support for resolving paths to other configuration files that are part of the default service configuration package. The syntax is:

servicefabricfile:/<configuration-file-name>

At run time this value will be substituted with a full path to the configuration file with the given name. This is especially useful if an EventFlow pipeline element wraps an existing library that has its own configuration file format (as is the case with Application Insights, for example).

Back to Topics

Using Application Insights ServerTelemetryChannel

For Service Fabric applications running in production, we recommend to use Application Insights ServerTelemetryChannel. This channel has the capability to store data on disk during periods of intermittent connectivity and is a better choice for long-running server processes than the default in-memory channel.

To use the ServerMemoryChannel you need to create an EventFlow configuration file and an Application Insights configuration file in the same Service Fabric configuration package:

eventFlowConfig.json

{
  "inputs": [
    {
      "type": "EventSource",
      "sources": [
        { "providerName": "Microsoft-ServiceFabric-Services" },
        // Other event sources, as necessary...
      ]
    }
  ],
  "filters": [
    // Filters, as necessary
  ],
  "outputs": [
    {
      "type": "ApplicationInsights",      
      "configurationFilePath":  "servicefabricfile:/ApplicationInsights.config"
    }
  ]
}

ApplicationInsights.config

<?xml version="1.0" encoding="utf-8" ?>
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">

  <TelemetryModules>    
  <!-- Telemetry module configuration, as necessary -->
  </TelemetryModules>

  <TelemetryProcessors>
  <!-- Telemetry processor configuration, as necessary -->    
  </TelemetryProcessors> 

  <TelemetryChannel Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel, Microsoft.AI.ServerTelemetryChannel"/>

</ApplicationInsights>

Ensure that your service consumes Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel NuGet package.

Back to Topics

Filter Expressions

Filter expressions allows you to filter events based on the event properties. For example, you can have an expression like ProviderName == MyEventProvider && EventId == 3, where you specify the event property name on the left side and the value to compare on the right side. If the value on the right side contains special characters, you can enclose it in double quotes.

Standard and custom (payload) properties are treated equally; there is no need to prefix custom properties in any way. For example, expression TenantId != Unknown && ProviderName == MyEventProvider will evaluate to true for events that

• were created by MyEventProvider provider (ProviderName is a standard property available for all events), and
• have a TenantId custom property, with value that is different from "Unknown" string.

The following table lists operators supported by logical expressions

Back to Topics

Store Secrets Securely

If you don't want to put sensitive information in the EventFlow configuration file, you can store the information at a secured place and pass it to the configuration at run time. Here is the sample code:

string configFilePath = @".\eventFlowConfig.json";
IConfiguration config = new ConfigurationBuilder().AddJsonFile(configFilePath).Build();
IConfiguration eventHubOutput = config.GetSection("outputs").GetChildren().FirstOrDefault(c => c["type"] == "EventHub");

if (eventHubOutput != null)
{
    string eventHubConnectionString = GetEventHubConnectionStringFromSecuredPlace();
    eventHubOutput["connectionString"] = eventHubConnectionString;
}

using (DiagnosticPipeline pipeline = DiagnosticPipelineFactory.CreatePipeline(config))
{
    // ...
}

Back to Topics

Extensibility

Every pipeline element type (input, filter, output and health reporter) is a point of extensibility, that is, custom elements of these types can be used inside the pipeline. Contracts for all EventFlow element types are provided by EventFlow.Core assembly (except from the input type, see below).

EventData class

EventFlow pipelines operate on EventData objects. Each EventData object represents a single telemetry record (event) and has the following public properties and methods:

Note that EventData type is not thread-safe. Don't try to use it concurrently from multiple threads.

Back to Topics

EventFlow pipeline element types

Inputs

Inputs are producing new events (EventData instances). Anything that implements IObservable<EventData> can be used as an input for EventFlow. IObservable<T> is a standard .NET interface in the System namespace.

Filters

Filters have a dual role: they modify events (e.g. by decorating them with metadata) and instruct EventFlow to keep or discard the event. They are expected to implement the IFilter interface:

public enum FilterResult
{
    KeepEvent = 0,
    DiscardEvent = 1
}
public interface IFilter
{
    FilterResult Evaluate(EventData eventData);
}

Outputs

Output's purpose is to send data to its final destination. It is expected to implement IOutput interface:

public interface IOutput
{
    Task SendEventsAsync(IReadOnlyCollection<EventData> events, long transmissionSequenceNumber, CancellationToken cancellationToken);
}

The output receives a batch of events, along with transmission sequence number and a cancellation token. The transmission sequence number can be treated as an identifier for the SendEventsAsync() method invocation; it is guaranteed to be unique for each invocation, but there is no guarantee that there will be no "gaps", nor that it will be strictly increasing. The cancellation token should be used to cancel long-running operations if necessary; typically it is passed as a parameter to asynchronous i/o operations.

Back to Topics

Pipeline structure and threading considerations

Every EventFlow pipeline can be created imperatively, i.e. in the program code. The structure of every pipeline is reflected in the constructor of the DiagnosticPipeline class and is as follows:

1. One or more inputs produce events.
2. A set of common filters (also known as 'global' filters) does initial filtering.
3. Then the events are sent to one or more outputs (by cloning them as necessary). Each output can have its own set of filters. A combination of an output and a set of filters associated with that output is called a 'sink'.

EventFlow employs the following policies with regards to concurrency:

1. Inputs are free to call OnNext() on associated observers using any thread.
2. EventFlow will ensure that only one filter at a time is evaluating any given EventData object. That said, the same filter can be invoked concurrently for different events.
3. Outputs will invoked concurrently for different batches of data.

Back to Topics

Using custom pipeline items imperatively

The simplest way to use custom EventFlow elements is to create a pipeline imperatively, in code. You just need to create a read-only collections of the inputs, global filters and sinks and pass them to DiagnosticPipeline constructor. Custom and standard elements can be combined freely; each of the standard pipeline elements has a public constructor and associated public configuration class and can be created imperatively.

Creating a pipeline with custom elements using configuration

To create an EventFlow pipeline with custom elements from configuration each custom element needs a factory. The factory is an object implementing IPipelineItemFactory<TPipelineItem> and is expected to have a parameter-less constructor.

The factory's CreateItem(IConfiguration configuration, IHealthReporter healthReporter) method will receive a configuration fragment that represents the pipeline item being created. The health reporter is available to report issues in case configuration is corrupt or some other problem occurrs during item creation. The health reporter can also be passed to and used by the created pipeline item.

For EventFlow to know about the item factory it must appear in the 'extensions' section of the configuration file. Each extension record has 3 properties:

1. "category" identifies extension type. Currently types recognized by DiagnosticPipelineFactory are inputFactory, filterFactory, outputFactory or healthReporter.
2. "type" is the tag that identifies the extension in other parts of the configuration document(s). It is totally up to the user of an extension what she uses here.
3. "qualifiedTypeName" is the string that allows DiagnosticPipelineFactory to instantiate the extension factory

Here is a very simple example that illustrates how to create a custom output and instantiate it from a configuration file.

namespace EventFlowCustomOutput
{
    class Program
    {
        static void Main(string[] args)
        {
            using (DiagnosticPipelineFactory.CreatePipeline("eventFlowConfig.json"))
            {
                Trace.TraceError("Hello, world!");
                Thread.Sleep(1000);
            }
        }
    }

    class CustomOutput : IOutput
    {
        public Task SendEventsAsync(IReadOnlyCollection<EventData> events, long transmissionSequenceNumber, CancellationToken cancellationToken)
        {
            foreach(var e in events)
            {
                Console.WriteLine($"CustomOutput: event says '{e.Payload["Message"]?.ToString() ?? "nothing"}'");
            }
            return Task.CompletedTask;
        }
    }

    class CustomOutputFactory : IPipelineItemFactory<CustomOutput>
    {
        public CustomOutput CreateItem(IConfiguration configuration, IHealthReporter healthReporter)
        {
            return new CustomOutput();
        }
    }
}

(content of eventFlowConfig.json)

{
  "inputs": [
    {
      "type": "Trace"
    }
  ],
  "filters": [
  ],
  "outputs": [
    {
      "type":  "CustomOutput"
    }
  ],
  "schemaVersion": "2016-08-11",

  "extensions": [
    {
      "category": "outputFactory",
      "type": "CustomOutput",
      "qualifiedTypeName": "EventFlowCustomOutput.CustomOutputFactory, EventFlowCustomOutput"
    }
  ]
}

Back to Topics

Troubleshooting

Events are getting dropped

Under high load you might emit a health warning that says

An event was dropped from the diagnostic pipeline because there was not enough capacity

The “capacity” referred to in the error message is the size of the event buffer that EventFlow has. By default the buffer can hold 1000 events, although that can be changed by configuration settings. The “not enough capacity” error indicates EventFlow output(s) cannot consume incoming events fast enough. The buffer is exhausted and events are starting to get dropped.

If the event inflow is quite bursty, increasing the buffer size might help.

For ApplicationInsightsOutput make you are using the ServerTelemetryChannel for sending data to Application Insights. This channel has the ability to buffer data on disk if the connectivity to Application Insights is intermittent. It can be enabled via output configuration, using a separate ApplicationInsights configuration file.

Ultimately it might be necessary just to reduce the number of events produced by the service

Back to Topics

Platform Support

EventFlow supports full .NET Framework (.NET 4.5 series and 4.6 series) and .NET Core, but not all inputs and outputs are supported on all platforms. The following table lists platform support for standard inputs and outputs.

Back to Topics

Contributions

Refer to contribution guide.

Code of Conduct

Refer to Code of Conduct guide.

Back to Topics