- AVC: Application Visibility and Control. Cisco Deep Packet Inspection feature embedded in routers. it enables application recognition based on signature and fields extraction.
- ART: Application Response Time. Cisco passive probe embedded in routers. It enables TCP application performance metrics such as latency per network part.
- BR: Border Router. Cisco PfR component
- CSR: Cloud Services Router
- FNF: Flexible NetFlow. Latest Netflow version.
- IP: Internet Protocol – Layer 3 Datagram Protocol.
- IPFIX: Internet Protocol Flow Information Export. Industry standard for Netflow
- LAN: Local Area Network
- MACE: Measurement, Aggregation, and Correlation Engine
- MC: Master Controller Cisco PfR component
- MMA: Cisco Metric Mediation Agent. Cisco passive probe embedded in routers. It enables TCP & UDP application performance metrics.
- NAT: Network Address Translation
- DPI: Deep Packet Inspection. Generic name for application recognition engine over telecommunication network.
- NBAR2: Network Based Application Recognition. Cisco Deep Packet Inspection Engine
- PfR: Performance Routing
- SNMP: Simple Network Management Protocol
- TCP: Transmission Control Protocol—L4 Reliable Transport Mechanism.
- UDP: User Datagram Protocol—L4 Transport Mechanism. Connectionless transport layer protocol
- VRF: Virtual Routing and Forwarding
- WAAS: Wide Area Application Services
- WAN: Wide Area Network
Cisco is a feature bundle embedded in routers, targeted at improving end-user experience when they use applications over Wide Area Network (WAN). Cisco IWAN provides the ability to report your application performance metrics, enables per-application policy for granular control of application bandwidth use (Application Visibilty and Control, AVC), monitors network performance and selects the best path for each Class of Service (Performance Routing, PfR), and optimize application traffic for faster response time and less bandwidth (Wide Area Application Services, WaaS).
Netflow version 9 and IPFIX are the protocols of choices for Cisco IWAN to export information from the routers.
eye.lo is a probe-less application-aware platform. Network information are exported by IWAN features towards eye.lo through an open export format Netflow Version 9 and IP Flow Information Export (IPFIX).
Eye.lo leverages Cisco AVC application recognition and performance metrics to:
- Provide visibility into different types of traffic,
- Quickly isolate and troubleshoot application performance issues,
- Check CoS policy consistency across the Network
eye.lo leverages Cisco PfR data to:
- Display synthetic view of hybrid infrastructure based on path change & out-of-policy event.
- Provide full visibility on traffic per path when PfR intelligently load balances traffic over all WAN paths to protect critical applications from fluctuating WAN performance.
- Understand performance issue based metrics (Jitter, latency, loss)
eye.lo is a multi-tenant software platform dedicated to Service Provider. Its distributed architecture, based on redundant nodes, insures smooth scalability, optimal level of use of resources, fault tolerance and low latency response for the concurrent requests and big data.
LivingObjects recommends a centralized collection to ease implementation of new customers. However, eye.lo can also be deployed in isolated environment.
eye.lo installation and operation don’t need any additional 3rd party license or tool to deliver full visibility on critical application. eye.lo is a probe-less solution which leverages advanced metrics provided by Cisco Intelligent WAN features embedded in Customers Premises Equipment (CPE). No additional box is needed to provide full visibility on critical application over the network.
Cisco IWAN + eye.lo, without any additional box, enables:
- Full visibility on application traffic and optimization
- Multi-tenant massive deployment
- Sober storage strategy & best responsiveness
Highly customizable environment
eye.lo is deployed in Service Providers’ DCs on a single server or a distributed infrastructure. Above 3000 routers, LivingObjects recommends a distributed architecture which enables smooth scalability across time. I/O are optimized for random data access. Data storage needs to be implemented on physical machine with SSD. The other components can be virtualized, but need dedicated VM infrastructure not to compete with other software for I/O.
LivingObjects recommends Linux Debian distribution with Kernel version greater than 3.16. For other distribution contact LivingObjects at firstname.lastname@example.org. To access eye.lo, end-customers can use any recent web browsers.
eye.lo is a multi-tenant platform that delivers Communication Service Providers (CSP), Managed Service Providers (MSP) and Network Integrators with a powerful application-aware management tool, helping them to assure the delivery of WAN connectivity services to their business customers.
For all the traffic and applications going through the WAN links and accesses, for thousands of enterprise customers, eye.lo™ is divided into several modules that allow to :
- Display executive view of network and critical application KPIs.
- Drill down near real time dashboards and assess end-user experience with fine-grain end-to-end application performance metrics across LAN, WAN and Application servers.
- Troubleshoot hybrid network path (PfR) issues
- Alert when critical application performance is out-of-policy
- Report daily, executive view of your Network SLA
- Closely customize customers’ network (Dashboard, alerting, site clustering, custom applications,…)
- Easily admin multi-tenant profiles and rights
Monitor flow and polling collection
Launching IWAN services is:
- Deciding service scope available for enterprise customers.
- Adapting user rights and eye.lo features accordingly,
- Reviewing the network customer topologies to define routers templates and check IWAN impact on existing CPE portfolio
- Sizing the collection link and eye.lo hosting hardware
- Adapting the provisioning system to automatically deliver the seed files to eye.lo platform every night
- Training the sales and manage services team
Based on experience, LivingObjects can deliver a full set of professional services to help deploy IWAN services in less than three months.
In the past, typical network traffic could easily be identified using well known port number. HTTP, HTTPS, POP3, or IMAP were among common traffic seen in enterprise. Today, there is increasing number of applications which is delivered over HTTP – both business and recreational applications. And many applications use dynamic ports such as Exchange, and voice and video which are delivered over RTP. This makes them impossible to be identified by looking at port number.
NBAR2 is Cisco’s Deep Packet Inspection (DPI), based on application signature, and Field Extraction (FE) technologies, to retrieve fields such as HTTP URL, SIP domain, mail server, and so on. Application information such as Sharepoint, Netflix, or Google Docs is provided by Nbar2 signature dictionary, called protocol pack. The protocol pack is updated several times a year to include new applications. Version 16.0 includes more than 1500 signatures.
PfR is part of Cisco IWAN. PfR monitors network performance and routes applications based on application performance policies and load balances traffic based upon link utilization levels to efficiently utilize all available WAN bandwidth. PfR is comprised of two major Cisco IOS components, a Master Controller (MC) and a Border Router (BR).
The Master Controller is a policy decision point at which policies are applied to various traffic classes that traverse the Border Router systems.
- The hub master controller is the master controller at the hub-site, which is either a data center or a head quarter. This is the device where all policies are configured. It also acts as master controller for that site and makes optimization decision.
- Branch Master Controller: The Branch Master Controller is the master controller at the branch-site. There is no policy configuration on this device. It receives policy from the Hub MC. This device acts as master controller for that site for making optimization decision.
Border Routers (BRs) are in the data forwarding path. Border Routers collect data from their Performance Monitor cache and smart probe results, provide a degree of aggregation of this information, and influence the packet forwarding path as directed by the Master Controller to manage user traffic.
NetFlow provides the ability to collect IP network information as it enters or exits an interface. A Flow Record consists of keyed fields and non-keyed fields. Keyed fields are all field(s) which need to be unique in order for a new Flow Record cache entry to be created in the CPE memory. Non-keyed fields provide information such as metrics (byte count, packet count, latency or jitter). For every record, a cache table is created to track and store flow entry. A new cache entry is created when the keyed field(s) of the packet does not match existing cache entry. Otherwise, only the non-keyed fields are updated, e.g. byte count is incremented.
The key difference resides in the information access: SNMP requires collectors to request the information. Netflow collectors passively receive and process flows from all devices. For first case (polling), devices need to store the data available on request. With Netflow, devices send data once processed. Thus, if devices embed the right processing engines (Deep Packet Inspection, passive probe, …), one could have much more detail on traffic and performance using Netflow.
1. Netflow 5: (IPV4-specific,) NFV5 is the most commonly deployed version. The flows exported by the equipment provides 5-tuple keyed fields, Source IP/port, Destination IP/port and protocol, to describe the identities of the systems involved in the conversation and the amount of data transferred.
2. Flexible NetFlow FNF V9: (IPv4&6 compatible) Version v9 has brought FNF capability, which makes Netflow a highly versatile protocol. Its flexibility makes it particularly more relevant for complex reporting and heterogeneous data.
- Flexible key field aggregation
- variable number of data fields.
- unidirectional or bidirectional
- sampled or not
- multi-vendor (430 standardized fields, thousands vendor-specific fields)
- aggregated, synchronized or not for exports
3. IPFIX: (“IP Flow Information eXport”) also referred to as NFv10, IPFIX is the industry standardized version of Netflow. It builds on NFv9 for most of the features, and brings additional flexibility (variable-length fields, sub -application extracted fileds, options-data…)
Note: Netflow version 9 and IPFIX are the export protocols of choices for AVC, because they can accommodate flexible record format and multiple records required by Flexible Netflow infrastructure. IPFix is recommended.
Dashboards are made of customizable dashlets. A dashlet may use various visualization format and can be configured as easily as you build an excell timeline graph.
- Pie, to display Top nodes on one specific metrics (Ex: Top loaded interfaces).
- Bar graph, to display aggregated view for several metrics and nodes (Ex: Traffic repartition per DSCP for citrix, gmail and ftp).
- Timeline, to visualize trend of KPIs (Ex: Traffic per primary/secondary Network for hybrid network)
- Gauge, to highlight performance status of a KPI (Ex: Salesforce response time over the end-customer network).
- Table, to detail performance metrics for Top nodes (Ex: Top path changes with jitter/drop/latency out-of-policy event count).
- Pie-line, to mix gauge and timeline visualization (Ex: Business/leisure traffic repartition).
Reports can include image, text, date and shape, to have a powerful communication tool.
eye.lo is designed to give both non-specialist and specialist the right insight of Network SLAs and end-user experience.
- The Landing page module provides a workspace to build an overview of network usage, performance and status.
- The dashboard module provides a fully customizable tab-centric workspace. Admins browse the KPI library and mix in a same tab metrics coming from snmp polling and IWAN features. End-users drill down near real time data from one dashboard to another for more details or to spot the root cause of an issue.
The report module helps Service Providers structure their communication with their multiple enterprise customers. It turns application or network information from the eye.lo™ platform into synthetic, decision-driven and good looking PDF report . Read a report End-customer chose a template in the report library. They customize the timeframe and schedule a pdf e-mailing for daily/weekly/monthly reporting. Each widget is a dynamic object that is automatically updated when the report is sent.
eye.lo includes several resources out of the box: Default dashboards, reports, alerting, landing page, KPI, poller, … These resources are based on years of network monitoring expertise and make eye.lo available day one for many use cases.
Yes, The “drill down” feature is available across the platform. It will help IT managers to deep-dive, step-by-step by displaying the useful dashboards/visualization of their network.
When you select an element (site, application, DSCP, ..), using the magnifying glass, from any module (landing page, flow map, ..), eye.lo will automatically display the available dashboards. Pick the view you need and eye.lo will switche on the new view, with the right scope on right time frame.
For example, you detect unusual spikes of traffic for ms-update. You need more details to understand who is updating its PC during working hours (thus generating a spike of WAN load). Click on the magnifying glass beside the ms-update legend. eye.lo will automatically propose the compliant dashboards.
5 minutes is the default recommended granularity for all counters.
When received, data is aggregated to minimize storage without deleting history. Available granularities are 5 minutes, hour and day. By default, data retention is 3 week for 5 minutes granularity, 3 months for 1 hour granularity, and day. Data are never purged. Specific KPI such as daily 5 minutes max period can however be stored for longer period for capacity planning needs.
CPEs send flows on a scheduled period. Let’s take 5 minutes as the export period: For five minutes, the CPEs aggregate traffic. Then, they export the aggregated flows, the next five minutes (smooth export to prevent spike of traffic in the collection link). eye.lo collects the data on the fly and starts data processing at the end of the five minutes export period. So when an event occurs, it will be displayed between 5:30 minutes (best case) to 14 minutes after the event (Worst case with heavy load on the eye.lo servers).
End-customers (or account manager) can define a dedicated environment including dashboard, reports, alerting, custom application, site clustering.
They can group sites in cluster to focus on specific area of their network.
They can build their custom application on top on NBAR2 dictionary.
Service providers administrators, operations team access eye.lo using login and password or via Single Sign On (SSO) from their internal portal. Customers access eye.lo preferably using SSO, via existing external portal. It prevents customer Login/Password administration in eye.lo platform.
eye.lo is compliant with any SSO configuration. As SSO implementations are often different, LivingObjects professional services team provides specific plugin compliant with your SSO environment (preferably RESTful or SOAP Web Services).
eye.lo allows admin to configure alerts based on network or application metrics in order to increase visibility of end-user experience or network events. When the chosen KPI is above a threshold, an alert is raised.
eye.lo as a dedicated solution for Service Provider enables customer profiling in a unique instance. Depending on the offer a customer has subscribed to, end-user access specific features, KPI and resources (Dashboard, report, ..). Service Provider can also decide to keep specific features for internal use in order to improve their operating efficiency and proactivity (scheduled reports, alerting).
When a VPN architecture is used and application traffic is transiting between sites, obtaining visibility on the flows is an important facet of the QoE management. With dynamic WAN paths, Cisco PfR, in-depth knowledge of the application flows is becoming even more critical.
With its WAN path module, eye.lo offers a simple and innovative visualization of the end-to-end flows. Designed for VPN set across a few or thousands of sites, it adapts to each enterprise context and builds maps showing the flow going through their network.
Combining traffic metrics and distribution with performance metrics, the flow map is an extremely effective tool to troubleshoot or optimize the network and application delivery infrastructure.
eye.lo needs to be provisioned with the same information a multi-tenant SNMP collection tool needs and some specific fields dedicated to IWAN: IP address, Client, WAN interfaces and additional data for smoother UI browsing, such as site name, customer contract, address or coordinate for mapping, …. Additional information is auto-discovered by eye.lo. Seed file is provisioned manually using a .csv file or scheduled for automatic import.
See Installation Guide for required fields format.
You need to configure data export on CPEs. Data will be exported using Netflow/IPFIX towards eye.lo.
- The Exporter defines the source interface (LAN/WAN) and the destination IP address of the remote collector
- The Monitor attaches records and exporters to the devices interfaces to activate the monitoring. It also configures the cache maximum size (32k recommended for small branch routers to 200k for DCs) and strategy (Synchronized 5 minutes recommended).
You may configure one of the following options (see LLD for details): static LO Performance Monitors, custom LO EZPM, EZPM ‘application-performance’ profile, FNF, TNF.
Compared to the relative simplicity of SNMP monitoring metrics, IWAN features comes at an expense on the Network device in terms of memory (need for anticipation in configuring the cache size) and CPU (for advanced processing). Cisco has introduced EZPM in latest version of IWAN to decrease CPU load. Service providers have to check if the needed additional resources on routers is compliant with their existing portfolio (ie: Which router for which contracted bandwidth).
To access eye.lo, end-customers may use these supported web browsers:
- Mozilla Firefox: 40 and upper
- Apple Safari: 9 and upper
- Google Chrome: 45 and upper
- Microsoft Internet Explorer: 11
- Microsoft Edge
On Top of Cisco IWAN dictionary, eye.lo can recognize traffic based on IP, port, HTTP hostname, SSL (HTTPS) server name. The workflow is the following:
- Does the flow match an embedded eye.lo customer specific application ?
- If no match, does it belong to the Nbar2 dictionary?
- If no match, does it match with a IANA service based on well-known ports?
If no match then the flow is classified as unknown
With up to 50 CPEs, the service runs on a VM with 4CPUs/8GB RAM/100GB disk. A Debian Linux distribution is recommended + installation of Docker 1.9 and Ansible.
To poll data from devices, eye.lo embeds SNMP V2/V3, Telnet, TL1 and SSH
For flow collection, eye.lo is compliant with Traditional Netflow and IPFIX.
eye.lo embeds traditional ways to collect information among IP Network devices: Polling (snmp V2/V3, SSH, Telnet) and Flow collection (Traditional Netflow). Using these communication protocols, eye.lo can collect information on a multi-vendor Network.
As a multi-tenant platform, eye.lo does not support network discovery. eye.lo needs information that can not be discovered such as the client name for a specific IP address, and particular IWAN fields to process IWAN information. A seed file is required to enable eye.lo.
Standard SNMP polling mode help admin to quickly add metrics based on OID (For example: CPU, traffic, drop, ….).
Expert mode leverages a scripting interface to connect the monitored device and its counterparts (Provider Edge, IPSLA probe) and collect advanced metrics (CoS, IPSLA, Metrics, ..). Admins can mix CLI, attributes coming from the topology and routers template to build advanced metrics such as traffic per CoS, IPSLA jitter, …
eye.lo architecture divides in layers: Proxy, collection node, storage cluster and services. Each component embeds his scalability strategy for a light-touch implementation. The non-blocking technology and distributed architecture used to collect data let eye.lo to achieve a high response rate for hundreds of thousands of multi-vendor devices.
The extreme flexibility and scalability yields a very short lead-time from the order to the delivery, resulting in value from day one.
See Low Level design Guide for more detail.
The provisioning process is highly parallel: Using snmp protocol, all tasks requests are asynchronously sent on the network and smoothed over time. For large seed files, processed equipment responses are batched onto groups of 1000, and returned to the administration UI for the user to track the provisioning progress.
If service providers choose a centralized collection, they have to size the collection link properly. Link sizing recommendation depends on :
- IWAN features enabled: More features = more data to export.
- Bandwidth per site repartition: headquarter with 500 employees will have more variety (=more export) than a small office with 20 employees)
- The time traffic distribution: the CPEs don’t have their max traffic at the same time.
A typical 10 000 CPE enterprise SP IWAN Network requires an 200 Mbps bandwidth
Collection link Max speed = Average flow size x Predicted Max aggregated traffic x Flow count at this max traffic
Please contact LivingObjects for customized sizing.
eye.lo processes and aggregates raw flows coming from the routers to store what is needed for a customer/service profile. Storage size varies depending on:
- Customer profiles: For example, an internet connectivity customer will access only application traffic metrics and a VPN connectivity customer will access traffic and performance metrics.
- The data retention: For example, an internet connectivity customer will access 5 minutes metrics for 1 week, and a VPN connectivity customer will access traffic and performance metrics for 3 weeks.
Storage sizing = Client profile X Data retention rule X Predicted Average aggregated traffic
A typical 10000 CPE, scalable infrastructure is
- Proxy: 1 x 4 CPU, 4 GO RAM, 1 SAS 20 GB (scalable)
- Collect: 2 x 8 CPU, 16 GO RAM, SAS 1 TB (scalable)
- Services : 4 x 8 CPU, 8 GO RAM, 1 SAS 200 GB
Physical environment for DB:
- 2 x (20 CPU/64GB RAM/ 2*1TB SSD disk (RAID1)) (Scalable)
Please contact LivingObjects for customized sizing.
An expert mode is available on eye.lo. It records raw flows (data + template) coming from the specified CPE and make the the data available for analysis. It helps admin troubleshoot issues on router template configuration.
The Flow Record represents the atomic building block exported by the device. NFv9 and IPFIX define 4 types of flow records: template, data, options template, options data. While template and data records describe the actual live traffic, the two latter stand for static mapping information, such as devices, interfaces, applications…
By using NBAR2 in the class-map, routers can identify traffic by NBAR2 application signature. This allows per-application policy control such as QoS, e.g. limit traffic rate for Netflix, Pandora, and iTunes applications, or guarantee bandwidth for business applications such as WebEx, Office 365, or Sharepoint.
Eye.lo stores and reports metrics down to the DSCP level per application and interface. This helps know whether a given application may have been misclassified when there is a DSCP-based QoS.
HTTPS is the default transport protocol for service requests. User passwords are encrypted. Network counters over time are stored in a proprietary, compact, binary format and split by customer (to protect data confidentiality per tenant).
Yes. High availability for eye.lo requires the solution to be installed on two data centers, and declaring one instance as a backup to the other.
Note: See Low Level Design guide for more details.
IWAN monitoring level may be configured through eye.lo provisioning for each CPE. If the CPE exports PFR data flows but provisioned only for AVC, the PFR flows will be ignored by eye.lo.
Is it possible to define his own Key Performance Indicators based on raw counters provided by the collection ?
Beside default KPI included in the KPI library, admins can define custom KPIs they need to assess the Network and end-user experience. They use a graphical interface to build formulas mixing raw counters and operand. New KPIs are immediately available for building new graph or tables.
eye.lo architecture relies on proprietary TSDB, optimized for time series. Time Series data are stored every 5 minutes in read-only binary files. Map Reduce algorithms fetch and calculate values displayed on end-user graphs: this requires random, non linear access in those files, which explains why SSD drives is mandatory when eye.lo platform embeds more than a few hundreds of CPEs.
This optimized end-to-end architecture insures high responsiveness to end-users requests even for tens of thousands od CPEs.
eye.lo is based on a micro-service architecture. New modules can easily be added and existing modules upgraded as per the end-customers or service providers requests.
LivingObjects default recommendation is to define custom applications directly in eye.lo, rather than touching the router configurations: traffic will get classified in reports for all the exporting devices at once.
However, there may be cases justifying the creation of NBAR custom applications on routers (for example QoS, or PfR application-based policies): NBAR on each device would then generate a new entry in its application dictionary, with an identifier code that eye.lo would not understand in the flows. For eye.lo to support customapps on routers, please follow these guidelines:
- Specify on router CLI for each application the same id across all routers of the client
(For example ‘ip nbar custom cisco_site composite server-name *cisco.com id 50’)
- Manually create this definition in eye.lo, on the application criterion
(‘application=6:50’, 6 being the ‘User-Defined’ NBAR classification engine)
Eye.lo is compatible with NBAR custom apps defined on routers.
NBAR on each device generates an entry in its application dictionary, with an identifier code that eye.lo needs to understand in the flows. NBAR allows to specify the identifier in the custom application definitions: we only ask that you assign one common identifier per custom application on the routers.
Most architecture are compatible with Cisco IWAN, as it runs on top of an overlay transport protocol (DMVPN for IWAN 2). However, the concrete deployment of IWAN requires a Cisco Validated Design: Cisco provides online in-depth PDF guides for deployment and configuration.
Note: although eye.lo counts among the top 3 leading reporting tools for IWAN, it may provide application visibility and control on CPEs for networks where IWAN is not yet implemented.
We’re using bcrypt as our password encryption mechanism: it is an OpenBSD Blowfish password hashing algorithm.
Here are the supported Linux distributions in order of preference and support level:
- Debian 8 Jessie 64 bits: direct installation with no other prerequisites
- Ubuntu Server 16.04 LTS: direct installation with no other prerequisites
- Redhat Enterprise Linux (RHEL) 7: re quires Ansible and Docker
- Oracle Linux (OEL) 6.7: requires Ansible and Docker
(All other distributions in a recent version may work but are not supported today)
From version 3.5/3.6 administrators will be able to upgrade the Cisco application baseline for all clients, by importing into eye.lo the ‘XML taxonomy file’. This taxonomy file is retrievable from any router with the command ‘sh ip nbar protocol-pack active taxonomy’.
A single Proxy can deal with hundreds of thousands of CPEs (the memory and CPU footprint is very low for this function)
Any UDP port can be used for Netflow collection – the default is 2055 for Netflow (just a matter of configuration at startup time)
There is a small (mostly negligible) footprint on CPU, but it obviously depends on the CPE model and family (ISRG2, ISR4K, ASR1K). Please contact Cisco for precise numbers.