Request filtering in Application Insights with Telemetry Processor

October 19, 2015 jason Uncategorized 0 Comments

After great feedback from the community, we have invested in a new feature: Telemetry Processor. You can now inspect, filter and modify telemetry generated by the Application Insights SDK before it is sent to the backend service. Telemetry Processor is included in the new version of the SDK.


Now that you can initialize, filter, and sample your data before sending it to the backend you can, for example, filter out telemetry about requests from robots, or successful dependency calls. This means you can reduce the volume of data collected which allows you to focus more on the telemetry of interest to you.


To create a filter, implement ITelemetryProcessor. This is another extensibility point like telemetry module, telemetry initializer and telemetry channel


Create a Telemetry Processor


  1. Update your SDK to the latest version. Note: This version is in prerelease, actual API may change in the future.
  2. Create a new file for your telemetry processor. The class you create should implement ITelemetryProcessor interface. Telemetry Processors construct a chain of processing. This is why the next telemetry processor in the chain will be passed into the constructor. Interface has the method Process which applies logic that may modify ITelemetry item and decide whether to pass it further to the chain of telemetry processors. 
    
    namespace FilteringTelemetryProcessor
    {
        using Microsoft.ApplicationInsights.Channel;
        using Microsoft.ApplicationInsights.DataContracts;
    
        class UnauthorizedRequestFilteringProcessor : ITelemetryProcessor
        {
            public UnauthorizedRequestFilteringProcessor(ITelemetryProcessor next)
    		//Initialization will fail without this constructor. Link processors to each other
            {
                this.Next = next;
            }
            public void Process(ITelemetry item)
            {
                // To filter out an item, just return 
                if (!OKtoSend(item)) { return; }
                // Modify the item if required 
                ModifyItem(item);
    
                this.Next.Process(item);
            }      private ITelemetryProcessor Next { get; set; }
        }
    }
    

  3. Create a new TelemetryConfiguration in your main source file, typically Global.asax.cs.

    Here is an example of creating a new TelemetryConfiguration with one processor: UnauthorizedRequestFilteringProcessor


    
    TelemetryConfiguration.Active.TelemetryChannel = new TelemetryChannelBuilder().Use((next) => new UnauthorizedRequestFilteringProcessor(next)).Build(); 
    
    TelemetryClient client = new TelemetryClient();
    

When creating your new TelemetryConfirguration you may have more than one processor. You can use multiple processors, but it is important to note the order in which the processors occur is the same as the order in which you call .Use for that processor.


Here is an example of creating a new TelemetryConfiguration with two processors: UnauthorizedRequestFilteringProcessor and MySecondFilteringProcessor. These processors will create a chain. Any item filtered out in UnauthorizedRequestFilteringProcessor will not reach MySecondFilteringProcessor.



Var builder = new TelemetryChannelBuilder(); 
Builder = builder.Use((next) => new UnauthorizedRequestFilteringProcessor(next)); 
Builder = builder.Use((next) => new MySecondFilteringProcessor(next)); 
TelemetryConfiguration.Active.TelemetryChannel = builder.Build(); 
TelemetryClient client = new TelemetryClient();


Short hand for the two processor TelemetryConfiguration above.



TelemetryConfiguration.Active.TelemetryChannel = new TelemetryChannelBuilder().Use((next) => new UnauthorizedRequestFilteringProcessor(next)).Use((next) => new MySEcondFilteringProcessor(next)).Build(); 
TelemetryClient client = new TelemetryClient();


Filtering


Many customers have asked for a filtering feature, for example, filtering of sensitive information. Application Insights is proud to now offer this functionality. Here are some examples of useful filters.


Filter out synthetic requests


Metrics explorer gives you the option to filter out requests from search engines and web tests. You can reduce traffic and improve your quota of data points by not sending them from the SDK.



public void Process(ITelemetry item)
{
    if (!string.IsNullOrEmpty(item.Context.Operation.SyntheticSource))
    { return; }

    this.Next.Process(item);
}


Note:  SyntheticSource is set by TelemetryInitialzier.


Filter out requests where authentication failed


This filters out requests with a “401” response. 401 responses: authentication failed – bad name in many cases means “negotiation” response and you don’t want to track it as separate request item. You may have more complicated logic in place that distinguishes between negotiation response and actual failure. 



public void Process(ITelemetry item)
{
    var request = item as RequestTelemetry;

    if (request != null &&
    request.ResponseCode.Equals("401", StringComparison.OrdinalIgnoreCase))
    {
        // To filter out an item, just terminate the chain: 
        return;
    }
    // Send everything else: 
    this.Next.Process(item);
}


Filter out “fast” remote dependency calls


This filters out a dependency call that is under 100 milliseconds. In many cases, fast queries are not interesting to diagnose.



public void Process(ITelemetry item)
{
    var request = item as DependencyTelemetry;
            
    if (request != null && request.Duration.Milliseconds < 100)
    {
        return;
    }
    this.Next.Process(item);
}


Now you can filter out whatever telemetry you desire. With Telemetry Processor you have more control regarding what data is collected.


Future plans


Currently our SDK does not support setting up Telemetry Processor from ApplicationInsights.config, but we plan to support this in the future. The Application Insights team is committed to providing quality tools for developers. Our APIs will likely change as we develop new features and as such, we would greatly appreciate any feedback or recommendation for new features.


Source: Microsoft Azure News

Could not resolve host: urls.api.twitter.com