context
Center, network slowdowns may appear out of nowhere. A sudden burst of traffic from a distributed system, microservices, or AI training job can overwhelm the buffer in seconds. The problem isn't knowing when something isn't going well. You can see it coming before it happens.
Telemetry systems are widely used to monitor network health, but most operate in reactive mode. They flag crowds only after performance has slowed down. Once the link is saturated or the queue is full, it is already past the point of early diagnosis and it becomes significantly more difficult to track the original cause.
In-band network telemetry, or INT, attempts to resolve that gap by tagging live packets with metadata as they pass through the network. It provides a real-time view of how traffic flows, where queues are piling up, where delays are creeping up, and how each switch is being forwarded. It is a powerful tool when used carefully. But that costs money. Enabling INT on all packets introduces serious overhead and allows flooding of telemetry data to be pushed into the control plane.
What if there is a possibility of being more selective? Instead of tracking everything, predict where troubles are likely to form and enable INTs just for those areas and in a short time. This gives you detailed visibility, where most importantly, without paying the full cost of monitoring at all times.
Always On Telemetry Issues
Int provides a powerful, detailed perspective on what's going on within your network. You can track queue length, hop-by-hop latency, and timestamp directly from the packet path. However, there is a cost. This telemetry data can consume significant bandwidth and processing power when added weight to all packets and applied to all traffic.
To avoid that, many systems take shortcuts:
sampling: Tag only the fractions (for example – 1%) of packets using telemetry data.
Event Trigger Telemetry: Just turn on INT if something bad is already happening, like a queue passing a threshold.
These techniques help to control overhead, but miss out on the important early moments of traffic surges. This is the part you want to understand most if you are trying to prevent slowing down.
Introduction of a predictive approach
Instead of responding to symptoms, we designed a system that could predict congestion before it occurred, and actively activated detailed telemetry. The idea is simple. If you can predict when and where traffic will spike, then you can only have that hotspot and the right time window to select your INT.
This will lower your head, but gives you a deeper view when it is actually important.
System Design
I came up with an easy approach that makes network monitoring more intelligent. It can predict when and where monitoring is actually needed. The idea is not to sample every packet, but to wait for congestion to occur. Instead, you need a system that can catch signs of trouble early and allow you to choose high fidelity monitoring only when necessary.
So how did you get this done? We have created four important components: Each of them achieved a clear task.
Data Collector
Starting by collecting network data, you monitor the amount of data moving across different network ports at a particular moment. SFLOW uses SFLOW as it helps to collect important metrics without affecting network performance. These metrics are captured periodically to get a real-time view of the network at any time.
Prediction Engine
A prediction engine is the most important component of a system. It is built using a long-term long-term memory (LSTM) model. Because we learn how patterns evolve over time using LSTM, and are suitable for network traffic. I'm not looking for perfection here. The important thing is to find the unusual traffic spikes that are normally displayed before congestion begins.
Telemetry Controller
The controller listens to these predictions and makes decisions. The system responds when the predicted spike exceeds the alert threshold. Send a command to the switch to enter detailed monitoring mode, but only for critical flows or ports. Also, I know when I'll back off when the conditions return to normal.
Programmable Data Plane
The last piece is the switch itself. The setup allows you to adjust the behavior of the packets on the fly using a P4 programmable BMV2 switch. In most cases, the switch simply forwards traffic without making any changes. However, when the controller turns on INT, the switch starts embedding telemetry metadata in packets that match a particular rule. These rules are pushed by the controller and only target traffic that concerns you.
This avoids the trade-off between constant monitoring and blind sampling. Instead, you get detailed visibility exactly when you need it, without flooding your system with unnecessary data for the rest of your time.
Experiment setup
I created a complete simulation of this system using the following:
- Mini Net To emulate the leaf spinal network
- BMV2 (P4 software switch) For programmable data plane operation
- SFLOW-RT For real-time traffic statistics
- Tensorflow + Keras For LSTM prediction models
- python + grpc + p4runtime For controller logic
LSTM was trained with synthetic traffic traces generated by Mininet using IPERF. Once trained, the model runs in a loop, making predictions every 30 seconds and storing predictions that the controller works.
This is a simplified version of the prediction loop:
For every 30 seconds:
latest_sample = data_collector.current_traffic()
slinding_window += latest_sample
if sliding_window size >= window size:
forecast = forecast_engine.predict_upcoming_traffic()
if forecast > alert_threshold:
telem_controller.trigger_INT()The switch responds immediately by switching the telemetry mode for a particular flow.
Why LSTM?
We used the LSTM model because network traffic tends to have a structure. It's not completely random. There are patterns tied to time, background load, or batch processing jobs, and LSTM is particularly good at picking up these temporal relationships. Unlike simpler models that handle each data point individually, LSTM can remember what came before and use its memory to improve short-term predictions. In our use case, just looking at how the last few minutes behaves means finding early signs of a surge in the future. We didn't need it to predict the exact number just to flag something unusual when something unusual might come. LSTM provided enough accuracy to trigger proactive telemetry without overfitting noise.
evaluation
Although we did not perform any large performance benchmarks, the prototype and system behavior in test conditions allow us to outline the actual benefits of this design approach.
Benefits of lead time
One of the main advantages of such prediction systems is its ability to catch trouble early. Reactive telemetry solutions typically wait until queue thresholds cross or performance degrade. In other words, it's already behind the curve. In contrast, our design predicts congestion based on traffic trends, activates detailed monitoring in advance, and gives operators a clear insight into what has not only emerged symptoms but also what has led to problems.
Monitoring efficiency
The key goal of this project was to keep it low overhead without compromising your vision. Instead of applying full INTs to all traffic or relying on coarse grain sampling, the system selectively allows for high fidelity telemetry for short bursts, and only if the prediction indicates a potential problem. Although it doesn't quantify accurate cost savings, the design naturally limits overhead by focusing on Int and keeping it short-lived.
A comparison of telemetry strategies concepts
Although we did not record overhead metrics, the design intent was to find a midpoint that offered deeper visibility than a sampling or reactive system, but was only a small fraction of the cost of telemetry that was always on. Here's how the approach is compared at a high level:
Conclusion
I wanted to find a better way to monitor network traffic. By combining machine learning and programmable switches, we built a system that predicts congestion before congestion occurred, and activated detailed telemetry at the right place and time.
Predicting instead of a response seems like a small change, but it opens up a new level of observability. As telemetry becomes more and more important with AI-scale data centers and low-latency services, this kind of intelligent monitoring becomes a baseline expectation.
reference
- https://www.researchgate.net/publication/340034106_adaptive_telemetry_for_software-defined_mobile_networks
- https://liyuliang001.github.io/publications/hpcc.pdf
