SecurityTechnology
February 17, 2016

Logging and Security Information and Event Management

Application, operating system, and device logs contain essential security information, but many organizations struggle to collect and analyze them. Smaller businesses are particularly vulnerable because they often do not have dedicated security resources and centralized log collection is perceived as unnecessarily expensive.

From a security perspective, log data can be used retrospectively for investigations and proactively analyzed to detect and respond to security incidents. Both use cases benefit significantly from centralized log collection.

Cybersecurity incidents are often detected after significant damage has been done. For retailers, the first sign of a payment card breach may be notification by law enforcement or their acquiring bank. Victims may learn of incidents when they receive larger than expected bills for cloud computing or VoIP services, when confidential information is published to the Internet, or on receipt of a blackmail attempt. During compromises, intruders often delete system logs. While the potential for forensic recovery exists, real-time centralized log collection is often key to understanding what happened.

Decades ago, security literature advised administrators to review logs daily, but the volume of logs created by even a single web server makes this infeasible. Logs contain valuable information that can be used to detect and respond to threats, but this requires the use of analytic techniques. In some cases, simple analytics will reveal unusual behaviour that warrants further investigation. In other cases, more sophisticated correlation is required.

Many products exist to address log collection and analysis. They can be broadly divided into two categories: Log Management (LM) and Security Information and Event Management (SIEM). LM products such as Splunk and Graylog focus more broadly on log aggregation and search capabilities. SIEM products focus more tightly on event correlation and security analytics.

Splunk is the Cadillac of LM. The product will ingest logs in virtually any format and allow free-form searching. Agents to collect logs from Windows, Linux, and other operating systems are included. This provides broad value to IT. Splunk also offers SIEM-like capabilities. These are generally realized using scheduled or real-time searches. For example, a Splunk query might be written to detect failed login attempts and notify security operations when a threshold is exceeded. The downside of this approach is that each real-time search runs in parallel, consuming significant CPU resources. Spunk is licenced by ingested data volume, making it an expensive product.

Graylog, the leading open-source LM solution, takes a different approach. The product does not provide collection agents, but logs can be ingested in several formats, including syslog and Graylog Extended Log Format (GELF). Received data is written to the database to facilitate future searches, and is matched against various criteria for entry into streams. This approach is particularly powerful because criteria can be applied to each log entry as it arrives instead of requiring repeated searches. Streams can be used to tag relevant log entries and optionally forward them to another system for further analysis. Alerts can be configured on stream data as well, but are somewhat limited. For example, an alert can be raised when the count of stream events exceeds a threshold, but not, for example, limited to when a single IP address exceeds a threshold count. Graylog does not natively provide SIEM capabilities. However, Graylog’s extensible, open-source framework enables the creation of SIEM functionality as well as pulling in additional data to augment logs as they are received.

HP ArcSight is the best known product in the SIEM space. Unlike LM products, ArcSight requires much more structure when ingesting logs. This is accomplished with the Smart Connector approach — a lightweight module must be able to parse and understand logs as received. If a connector does not exist, it must be created so that logs can be ingested and normalized.  ArcSight is designed specifically to correlate log data and provide security analysis. Using the product involves a significant learning curve, but it remains popular in security operation centers.

SIEM products can be very useful if organizations can afford the licensing, training, and employee time required to operate the system. Outsourcing to a managed security provider is another option. The debate for many companies is whether to collect data directly into a SIEM, or collect into an LM and then forward data to a SIEM. From a security operations perspective, direct collection into a SIEM eliminates the cost of the LM solution while still providing the data SOCs desire. However, this approach does not provide the greater value to IT that can be realized with an LM solution.

Small organizations without a dedicated security team may find the cost and complexity of SIEM solutions may make deploying an organization-wide LM solution a better choice. SIEM capabilities can subsequently be outsourced, purchased, or built using API capabilities of the LM product.

About this author

Avatar

Eric Jacksch

Eric Jacksch is a leading cybersecurity analyst with over 20 years of practical security experience. He has consulted to some of the world's largest banks, governments, automakers, insurance companies and postal organizations. Eric is a regular columnist for IT in Canada and was a regular columnist for Monitor Magazine and has contributed to several other publications.

0 comments

There are no comments for this post yet.

Be the first to comment. Click here.

Security
 
Governments around the world are increasingly relying on cloud-based IT services...
 
For a few years now, there’s been a throwaway metaphor bounced...
 
According to a 2018 study led by Dr. Michael McGuire, Senior...
 
Cloud technology is a game changer! Successful implementation in both the...
 
For over two days at the end of January this year,...
 
Earlier this month I had the privilege of testifying as an...
 
Canadian Government Executive Media, (CGE) publisher of Canadian Government Executive magazine...
 
In the last few years, we’ve seen various federal governments warning...
 
Canadian Government Executive is excited to announce the agenda for TechGov...
 
In the wake of the WannaCry outbreak, corporate executives, IT professionals,...
 
Facebook Pages can be an essential tool for businesses and charities,...
 
Cybersecurity professionals have sounded the alarm for years, and they are...
 
CBC deserves full credit for exposing the presence of IMSI catchers...
 
Security professionals have an obligation to communicate risks and recommendations to...
 
Over the decades, technology has been grafted into governments around the...
 
In this episode, J. Richard Jones talks about being candid about...
 
Criminals have reportedly threatened to take over 250 million Apple accounts...
 
In this episode, hear more about how Canada is a prime...
 
While the incoming administration of President-elect Donald Trump is being buffeted...
 
In the world that we are living in today, free and...
 
The RCMP adopted a new media strategy earlier this month by...
 
What would tomorrow’s cybersecurity look like? That’s an intriguing question to...
 
Terrorism operates with deadly regularity. In June 2016, a gunman who...
 
Just as the federal government has begun consultations on cyber security,...
 
Efforts by the government to counter the radicalization of young Canadians...
 
Canadian healthcare organizations and businesses in the financial industry are the...
 
Global market trends are accelerating to increase the pressure on commercial...
 
A recent report suggests several strategies how governments and the private...
 
The latest information from IBM Cloud covers: Consolidating Complex Environments Consolidating...
 
IBM Cloud is the first cloud provider to use Intel TXT...
 
Signaling a realignment of Canada’s involvement with NATO, Prime Minister Justin...
 
United States President Barack Obama, speaking before Parliament last night, urged...
 
Yes, according to the former head of the Canadian Security Intelligence...
 
Early this morning, Philippine police confirmed that the severed head found...
 
The challenge is clear: a fast-paced industry pressures organizations to simultaneously...
 
As populations grow and age, the demand for services increases. As...
 
The agency responsible for safeguarding the Pentagon and several other buildings...
 
By Michael Murphy Not all assets can and should be equally...
 
Government agencies, international businesses, as well as, European organizations that comply...
 
The Royal Canadian Mounted Police (RCMP) is poised to launch an...
 
One of Canada’s largest integrated oil companies said it is not...
 
Associates of Russian President Vladimir Putin, the king of Saudi Arabia,...
 
Now more than ever, organizations in both the public and private...
 
The Federal Bureau of Investigation announced that it has managed to...
 
IT organizations, especially those in healthcare facilities and government institutions that...
 
Last year, the Canada Revenue Agency rolled out a pilot program...
 
Strong cryptography is clearly required to protect sensitive government, business, and...
 
As the battle between the FBI and Apple continues to escalate,...
 
“I don’t think that backdoors into encryption is going to increase...
 
Hackers are zeroing in on users of SSL/TLS encryption and no...
 
Meet Bob Heart.  He is an outstanding employee who works hard...
 
The CEO of Google Sundar Pichai has come out in support...
 
A new study released yesterday, Securing the C-Suite, Cybersecurity Perspectives from...
 
Application, operating system, and device logs contain essential security information, but...
 
Yesterday, Ontario Supreme Court Justice John Sproat ruled that the Peel...
 
I wrote about accountability more than a year ago. Recently, a...
 
Intelligence agencies have had widespread and long-running programs to gather, analyze...
 
What concerns me is whether or not we’ve got the balance...
 
One of the consequences of the Information Age in which we...
 
In March of 2011, the east coast of Japan was rocked...
 
BYOD is hot! But is it for you? If yes, which...
 
Protecting critical infrastructure from cyber threats is the shared responsibility of...
 
In numerous interviews with senior military commanders over the past several...
 
In early February, James R. Clapper, the U.S. director of national...
 
The widespread adoption of mobile devices as enterprise-level tools is occurring...
 
CGE Vol.13 No.2 February 2007 Public security, once a task relegated...
 
CGE Vol. 14 No.4 April 2008 In recent years, policy makers...
 
The changing face of public and personal privacy in the face...
 
The announcement regarding the establishment of Shared Services Canada (SSC) was...
 
What role should governments and public servants play in safeguarding personal...
 
L’univers de la sécurité des TI évolue rapidement. À mesure que...
 
The world of IT security is rapidly evolving. As quickly as...
 
There was probably a day in spring of AD 72 that...
 
Cyber attacks don’t have to look highly sophisticated. Hackers are purposely...
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
 
Some title Some author
Some excerpt
Governments around the world are increasingly relying on cloud-based IT services...