Problem to Solve – My company is starting a Big Data / “Internet of Things” initiative as a Business Strategy. How can I use my APM/NPM tools in this strategic effort?
Yes, yes, we all remember the classic movie, Animal House from the early 80’s. To quickly catch everyone up, a student named “Flounder”, had just been given a grade point average for the semester of “0.0”. The Dean of Faber College, Vernon Wormer then provided this classic piece of advice, “Fat, Drunk and Stupid is no way to go thru life son”. (YouTube link to refresh your memory. https://www.youtube.com/watch?v=bK-Dqj4fHmM ) I know I was able to get through college with that mantra, and it helped push me on to better and brighter things. This is an ode to you Dean Wormer. 🙂 🙂 🙂
And this is related to Big Data How ????
Well all of this reminiscing is a lot of fun, but exactly what does this have to do with Big Data? Well, if I told you nothing, it would really make for a short blog wouldn’t it. I have had many discussions with my customers in the recent months about them considering a Big Data project. Big Data is new and sexy and will very likely be a tremendously useful business initiative for many of you that could really “slingshot” traditional IT departments into key business strategy roles. I have seen lots of companies developing the Chief Digital Officer role and really lining up IT resources, skills, and information to support the effort. The role of the Big Data initiative and the Internet of Things (IoT) go hand in hand, as it is really all about leveraging structure and unstructured data sets to provide automated information and analytics across industries. That means “all of your data” in most cases. All of the industries that I work with are all getting on board quickly … Health Care, Manufacturing, Retail, Banking, etc. all have different but important use cases for pushing forward. Information is king !!
So again, what does Fat, Drunk and Stupid have to do with Big Data? Well, you are going to hear buzz words around Big Data called the three V’s.. Volume – Velocity – Variety. While these are the components related to your data, we are going to look at this strictly from an APM/NPM perspective. We will break this down into the basic (3) components as I see it shaping up. This is the first article in a series of (4) on this topic, so let’s get started.
Fat? No, Aggregate the Traffic
While “fat” is probably a stretch here, a better word might be the word “many”. The simple reality is that Big Data is going to take data from about any data source, and there will be lots of them. You have probably already guessed such things as mainframes, ERP systems, CRM systems, Cloud environments, CAD/CAM, engineering systems, patient health records, as all potential data sets. They will be, but let’s not forget about sources of data that are not sitting in a traditional structure data set. Things such as social media like Facebook or Twitter can be very rich data sets. This information usually comes in over your Internet Links. So what about other data sets that might fit into your strategy that you didn’t consider. Think about the rich call center, application, server, and network data that is available in your company from a Co-location, hosted Data Center, or Cloud. These could include any or all from provisioning databases, CMDB, ITIL service deployments, flow based data, packets, log files, network devices, servers, etc.
Why is this Valuable? Again, strictly from an APM/NPM perspective, big data sets that are potentially being pulled in over the network can get “fat” (as in a lot of them) if your goal is to connect them all up individually to your network via a TAP or Mirror (SPAN) port. There is usually a hard limit on the number of available mirror (SPAN) ports available and other solutions for cybersecurity, compliance and now potentially for Big Data. An efficient strategy to address this type of solution by deploying an aggregation switch strategy where you can pull in the traffic from various points of your network. And then SHARE them with various tools sets, including Big Data nodes. I will continue this topic in a separate article later.
Drunk? No, Monitor the Big Data Environment
Again, “drunk” might not be the best adjective, but let’s consider this again from an APM/NPM perspective. If your company executives have begun to consider Big Data as a business strategy / initiative, then it seems like a good idea to actually monitor the Big Data traffic as a service. (And you thought I couldn’t pull in “drunk”) The complexities of Big Data will include such things at IPV6, actual communications of Big Data nodes and clusters (i.e. Hadoop), security of data, potential cyber attacks and denial of service attacks against Big Data sources, availability, etc. Shall I really go on?
Why is this Valuable? From an APM/NPM solution viewpoint, the solution should have the ability to address monitoring BigData as a service. Again, if Big Data is important to your executives and/or company, then it is your best interest career wise and company wise to monitor the traffic / service / availability / security / etc. of the environment as one of your key applications. This takes on even more importance when your company will be gathering data from your devices that your company may produce and sell. Think of “Internet of Things” and the “why” behind your company creating a strategy around it. The data that is collected from your own products is used for better engineering, better service record, better product development, better customer service, and competitive differentiator. Again, the need to monitor this information will likely become high priority. I will continue this topic in a separate article later.
Stupid? No, Use Intelligent Data Sources
The final frontier for Big Data here is not to be “stupid”, but actually to be the polar opposite — intelligent. Think about your current APM/NPM solution, but not from a traditional monitoring perspective. Let’s take a fresh look at your solution, and this time from an INPUT. Whatever you are using as your APM/NPM product set for other use cases like application, performance, VoIP/Video, compliance, cybersecurity, ITIL, etc. could also potentially be used as an INPUT to your Big Data strategy.
Why is this Valuable? Think about it, many times these types of solutions have actual packet level data that could be collected, extracted and pulled into your Hadoop Cluster for processing as an input along with your CRM data records and a piece of twitter information. A hypothetical example yes, but perhaps there is a key piece of data or field from an HTTP based application, or even from your own custom application that your Chief Data Scientist would find valuable for some Big Data analysis or project. The idea that I am trying to convey here is to open your mind to “what is possible” for your Big Data or Internet of Things strategy. Those of us in IT usually think about IT services, performance, infrastructure, virtualization, etc. and there is nothing wrong with that notion. But in a Big Data conceptualization, anything can really be a data set, so why not include the data from your APM/NPM solution as a data set? Quite simply, it increases the value of these types of solutions and broadens the use case that your company can take reap and harvest information. I will continue this topic in a separate article later.
