Last Updated: 13th August, 2022

Real-Time Rich Metadata (First Step): The concept that a security incident is not really an “if,” but instead a “when” has been embraced by several other enterprises. Some of us are still trying to turn this into a more proactive security architecture and approach. But the reality and fact are that we need more than prevention when it comes to cybersecurity and real-time rich metadata are the first key to unlock the hindrance and as well as our mindsets.

Teams need the right tools to detect and investigate critical security threats, tools for hunting, and performing diagnostics. Some organizations try to “capture everything” at enormous infrastructure and workforce cost, only to find they can’t analyze or operationalize it effectively come crunch time due to the same reasons.

Hunting for signs of an attack on your network is like searching for needles in a haystack. Rich metadata is a part of the solution that allows you to decode the haystack and drive insights to find the attacker, halt their intrusion or exfiltration of data and stop the next attack.

What Is Real-Time Rich Metadata And Why We Need It?

Real-time rich metadata accumulated out of your network could access more nearly 90% of the valuable information that would have been apprehended by a full packet capture framework. More importantly, you can actually store and analyze it in real-time so you can actually impact attacks, insights to find the attacker and halt their intrusion or exfiltration that you might never have been able to discover otherwise.

Incorporating real-time rich metadata in this way can lower the overall cost of storing the actual packet captures (PCAP) while providing nearly the same level of visibility into the communication. Metadata can be stored as flat text which has the benefit of optimal compression rates for long-term storage. Real-time rich metadata has the ability to be stored in many common formats like JSON or XML rendering it searchable and reference-able by standard libraries.

Think of a conversation over the phone. If you had a recording of a conversation, you could listen to every word that was said. It does, however, take time with patience. If you had an easily searchable description of everything that was said, you could get almost the same value in a format much easier to consume. The richer the metadata you have, the richer the set of questions you can ask and answer quickly and without specific expertise.

For instance, you may start answering the foregoing dimensions: web application logging information, protocol layer and database real-time rich metadata to IP address as well as the location layer, email interaction rich metadata, internal file sharing, document author records accessed/transferred, filename/file hash, header-footer data, and creation date.

• Do we see the document before ever being transmitted?
• Who authored the document and when?
• Does the documentation include tags representing sensitive data?
• Who else in the enterprise has a copy of the document?
• Was there any personally identifiable information (PII) or protected health information (PHI) in the document?
• Who has been logged into the document’s computer system?

Extracting data like this from real-time rich metadata in as close to near real-time as possible should be the new standard for cybersecurity within your enterprise team.

What Secrets Your Real-Time Rich Metadata Can Show You?

Every incident responder or security analyst can show you what happens when you get a “serious” alert. You swing into investigation mode – pulling logs, triaging endpoints, and piecing together disparate data. The data you need is simply not feasible in several instances, or that can sometimes take weeks to procure and process it, or perhaps even need competence to interpret it.

By providing content-enriched metadata in near-real time, security teams can investigate suspected incidents in seconds and get answers to questions that were previously impossible to know.

For example, organizations can leverage real-time rich metadata to routinely detect multi-vector attacks such as the Angler Exploit Kit by correlating related activity across multiple sessions. Rather than just simply warning on such an access to the landing page or transmitting the exploit, metadata could link every phase of the exploitation chain next to each other, enabling teams to identify how they have been actually breached, which malware has been downloaded, insights to find the attacker, halt their intrusion or exfiltration, allowing the teams to remediate instantly and extensively.

The rich metadata capture every session that the network sensor can see in the network so that teams can investigate immediately. Placement of these sensors enables both server operations teams to hunt, and incident response (IR) teams gather needed information for all packets that move across such a sensor.

Enriching these real-time rich metadata is the next important step in gaining context on both tactics, techniques and procedures (TTPs) that adversary’s use along with their intent. Adversary intent is critical to adversary intelligence plans of battle (IPB) which constitute a critical part in cyber battle planning by adversaries.

You can apply new threat intelligence and indicators of compromise (IOC) to all real-time rich metadata from the network sensor that gathered the traffic. Storing this data over time facilitates retrospective analyses through data mining tools that can be tuned and instrumented around criteria and associated probability of ‘bad.’ This simply means that you can look back in time and determine if you were affected by the threat.

Retrospective analyses also provide a deeper understanding around adversary intent producing deeper details to support a comprehensive analysis of the adversary IPB. In this respect, leveraging well-understood military doctrine principles along with enriching captured metadata with new threat intelligence to historical data is an incredibly unique capability.

Not only will it enable you to confidently answer the question: “are we safe?” but it will equip you to detect attacks, intuit adversary intent and piece together their intent through their approach toward IPB.

Enterprises that store their historical rich metadata for several months can use quantitative techniques along with customized security analytics to support forensic analysis, including retrospective analysis of host activity to attempt to understand the initial event ‘blast radius.’

Without this ability, security teams often spend days or weeks piecing together information to determine exactly how their cybersecurity defense was penetrated, what the threat did and what needs to be done to prevent future breaches.

Real-time rich metadata is crucial to find a needle in the haystack and drive insights to find the attacker, halt their intrusion or exfiltration of data and stop the next attack.