Log anomaly detector for a web service

The brief

An operations team gets millions of log lines a day and only looks at them after something breaks. They want a tool that learns what normal logs look like and flags the unusual stretches early, with the lines that explain why. Good looks like: anomalies caught on the labelled logs with few false alarms, and an alert an engineer can act on.

What you will build

1. Parse raw logs into templates and events
2. Turn events into time windows and counts
3. Train an unsupervised anomaly detector (Isolation Forest or an autoencoder)
4. Measure precision and recall on the labelled anomalies
5. Show alerts with the log lines behind them in a small dashboard

Free, open data to build it with

• Loghub: system log datasets (HDFS, BGL, Apache…) ↗ LogPAI (GitHub)

Check each dataset's licence before you publish your work. Your mentor gives your team its own version of the brief (data, scope or region), so every team's project is its own.