Vic Hargrave coder on the loose Enjoying life one cup of coffee at a time.

Packet Capture with Pyshark and Elasticsearch

Network packet capture and analysis are commonly done with tools like tcpdump, snort, and Wireshark. These tools provide the capability to capture packets live from networks and store the captures in PCAP files for later analysis. A much better way to store packets is to index them in Elasticsearch where you can easily search for packets based on any combination of packet fields.

Pyshark is a module that provides a wrapper API to tshark – the command line version of Wireshark – with which you can build packet capture applications that take advantage of all the Wireshark protocol dissectors. With Pyshark and the Elasticsearch Python client library you can easily create a simple packet capture application in Python that can index packets in Elasticsearch.

Packet Capture with Pyshark and Elasticsearch feature image

Create an OSSEC Log Management Console with Kibana 4

It’s been awhile since Kibana 4 was released, so I figured it was about time I updated my OSSEC Log Management Console to use the latest and greatest Kibana. The look and feel of Kibana has changed quite a bit, with a new data discovery mode that let’s you browse your data quickly before you create any visualizations. The visualization panels are fluidly moveable to any position, query results are displayed very rapidly and you can even embed your dashboards into static web pages with the dashboard export feature.

In this article I’ll go over how to create a security event dashboard with KIbana 4. I’ll forgo discussing the details on setting up Elasticsearch and Logstash since they have been covered in my previous OSSEC log management and logstash blogs. Read those first to get an idea of how the system described here parses OSSEC alert logs and indexes them with Elasticsearch.

Create an OSSEC Log Management Console with Kibana 4 feature image

RESTful Web Services with Jersey

Although Eclipse is commonly used for Java web service programming, it can be challenging to configure and use for testing. Luckily Eclipse is not the only IDE game in town.

NetBeans provides a rich environment for developing Java web services with Jersey – the reference implementation of JAX-RS RESTful web services. This article will explain how to create a simple RESTful service and test it with the Apache Tomcat Server from the NetBeans IDE.

RESTful Web Services with Jersey feature image

Creating Elasticsearch Snapshots

Benjamin Franklin once wrote “…in this world nothing can be said to be certain, except death and taxes”. In this computerized world of ours, I would add having to backup your data to free up disk space to that list of eventualities.

For Elasticsearch users, backups are done using the Elasticsearch snapshot facility. In this article I’ll go through the design of an Elasticsearch backup system that you can use to create snapshots of your cluster’s indices and documents.

Creating Elasticsearch Snapshots feature image

Improved OSSEC Log Parsing with Logstash

The ELK stack (Elasticsearch-Logstash-Kibana) provides a cost effective alternative to commercial SIEMs for ingesting and managing OSSEC alert logs. Previously I wrote a blog – OSSEC Log Management with Elasticsearch – that discusses the design of an ELK based log system.

Since then some readers have asked for and suggested ways to parse additional fields from the OSSEC alert log stream. For example, the IP addresses of systems that cause certain security events is buried down in the Details field. So I have created a Logstash configuration file that does just that.

Improved OSSEC Log Parsing with Logstash feature image

Elasticsearch Client Programming - Python

The first article in this two part series focused on developing Elasticsearch clients with Perl. Elasticsearch also has an excellent Python library which lets you search for and analyze your data with one of the many mathematics and machine learning libraries available for Python.

In this article I’ll cover how to create an Elasticsearch client using Python that has the same capabilities as the Perl client from the part 1 article.

Elasticsearch Client Programming - Python feature image

Elasticsearch Client Programming - Perl

Since creating a log management system for the OSSEC HIDS with Elasticsearch, I have been busy applying this useful search technology in other projects. Elasticsearch is a marvelous system for ingesting streaming data that gets indexed on the fly and quickly searching your data.

The Elasticsearch community provides client libraries that expose their search API in several popular languages, including Perl and Python. This article is the first of a two part series where I show you how to write an Elasticsearch search client application in both of these languages, starting with Perl.

Elasticsearch Client Programming - Perl feature image

TCP/IP Sockets with Time Out Capabilities

Recently I had a question from one of my readers about how to close connections on a server when there are no requests received after a certain period of time. The question was asked with regard to the tcpsockets classes I covered in my blog TCP Network Programming Design Patterns in C++, none of which support time out capabilities.

Timing out on both receive and connect operations are common use cases. So in this article I’ll update my tcpsockets classes to provide these capabilities.

TCP/IP Sockets with Time Out Capabilities feature image

OSSEC Log Management with Elasticsearch

Among the many useful features of OSSEC is its capability to send alerts to any system that can consume syslog data. This makes it easy to combine OSSEC with a number of 3rd party SIEMs to store, search and visualize security events. Splunk for OSSEC is one such system that works on top of the Splunk platform.

Splunk can be expensive though, particularly if you collect a lot of log data. So I’ve been working on a solution for collecting OSSEC security alerts based on Elasticsearch that provides a cost effective alternative to Splunk.

OSSEC Log Management with Elasticsearch feature image

XML Creation with DOM in Java

Awhile back I wrote a blog about parsing XML documents with the DOM facilities provided by Java. As it turns out you can use the same API to programmatically create an XML document.

So I’ve dusted off the old XmlDomDocument class and added some methods that make it easy to build an XML document from scratch.

XML Creation with DOM in Java feature image

Securing Hadoop with OSSEC

Hadoop has built-in security facilities such as kerberos user authentication, encrypted RPC between system nodes and file encryption. While these are important features, Hadoop clusters can benefit from host intrusion detection to round out the security picture.

That’s where a system like OSSEC comes in. Find out how I’ve applied OSSEC to monitor and detect security events in Hadoop and HBase clusters.

Securing Hadoop with OSSEC feature image

Condition Variable Class in C++

Condition variables are used in conjunction with mutexes by one thread to signal other threads that it has changed the state of a given variable. Synchronizing threads with the standard Pthreads functions is straightforward, but wrapping these calls in C++ classes makes them all the easier to use.

In my last article I showed you how to build a Mutex class in C++. This time around I’ll use that class to develop a C++ wrapper for condition variables.

Condition Variable Class in C++ feature image

Mutex Class in C++

When multiple threads want to synchronize access to a shared resource they use mutexes and condition variables. A mutex enables a thread to block other threads from accessing a shared resource, while a condition variable is used by a thread to make itself wait for the state of a resource to be changed by other threads.

In my next two blogs I’ll illustrate how to simplify Pthread mutex and condition variable functions by wrapping them in C++ classes, starting with mutexes.

Mutex Class in C++ feature image

Multithreaded Work Queue Based Server in C++

Creating a multithreaded TCP/IP protocol based server requires the capabilities to handle network communication, multithreading and transferring data between threads.

I have described how to build C++ components to handle this functionality in previous blogs. This time I’ll show you how to combine these components to create a simple multithreaded server.

Multithreaded Work Queue Based Server in C++ feature image

XML Parsing with DOM in Java

In my blog XML Parsing with DOM in C++, I used the Xerces-C++ XML Parser as the foundation for the XML parsing API. The classes from that article are also useful for and can be implemented in Java. The difference is Java includes support for XML parsing with both the SAX and DOM models.

You can read up on the specifics of the DOM model in my previous article, so let’s dive right into the API code.

XML Parsing with DOM in Java feature image

TCP/IP Network Programming Design Pattens in C++

Network programming with the BSD Sockets API involves making a series of boilerplate calls to several operating system level functions every time you want to create connections and transfer data over TCP/IP networks. This process can be both cumbersome and error prone.

Fortunately there is an easier way to develop network applications. By thinking in terms of design patterns, we can devise abstractions for creating connections and transferring data between network peers that encapsulate socket calls in easy to use C++ classes.

TCP/IP Network Programming Design Pattens in C++ feature image

XML Parsing with DOM in C++

Having the ability to parse XML files is a requirement for a lot of applications these days. XML is a standard format for exchanging data between programs and storing configuration data.

If you want to parse XML documents in C++ you can benefit from using an external library like the Xerces-C++ XML Parser. Xerces provides an elaborate, but somewhat complex API for navigating XML files. To simplify matters, I’ll describe a C++ class that encapsulate the Xerces calls to index and retrieve XML element values and attributes.

XML Parsing with DOM in C++ feature image

Reversing a Linked List

In all my years as a software developer, reversing a singly linked list is not something I’ve had to do. But it is a commonly asked question in interviews for programming positions. Of course the easy solution to the problem is to place items in a Standard C++ collection, like a vector, then apply the reverse() function to the collection.

For those of you who want or have to roll your own solution, here is an example of a simple list structure and a reverse list function.

Reversing a Linked List feature image

Multithreaded Work Queue in C++

In a previous blog Java Style Thread Class in C++ I discussed how to develop a C++ class that enables you to create Pthread based objects that resemble Java threads. The next step to building a multithreaded application is to devise a means to distribute tasks between threads so they can be processed concurrently.

Queues are good devices for transferring work items from one thread to another. In this article I’ll discuss the design of a work queue class implemented in C++ that can be used with Thread class objects to easily build a multithreaded application.

Multithreaded Work Queue in C++ feature image

Java Style Thread Class in C++

Java provides the Thread class that enables developers to easily create multiple threads in Java applications. Pthreads is commonly used for multithreaded programming in C and C++, but this API is procedural and somewhat less convenient to use than Java’s object oriented thread interface.

However, it is possible to mimic Java threads behaviors by wrapping Pthreads calls in a C++ class that will let you create threads Java style.

Java Style Thread Class in C++ feature image

Develop a Packet Sniffer with Libpcap

Libpcap is an open source C library that provides an API for capturing packets directly from the datalink layer of Unix derived operating systems. It is used by popular packet capture applications such as tcpdump and snort that enables them to run on just about any flavor of Unix.

Here’s an example of a simple packet sniffer application based on libpcap that displays packet information in a snort-like format.

Develop a Packet Sniffer with Libpcap feature image

Java CSV Parser Using Regular Expressions

Parsing files is a real pain. You can find programming tools to help, regular expressions for example, and even languages that are built for that sort of thing, like Perl. But sooner or later you forget how to use them, unless you have a very good long term memory, in your head that is.

Java CSV Parser Using Regular Expressions feature image