Skip to Content
+ Menu

Wiki Toolbox

Search

Collectors

From Developer Community

Developing Sentinel Collector Plug-ins

Overview

Sentinel Collector plug-ins are used to process incoming data from event and data sources, and to transform that data into data objects usable by Sentinel. Most Collectors process real-time events generated by enterprise systems, but this SDK also supports development of Collectors to gather and store other types of information (Identities, Assets, Vulnerabilities).

Collector plug-ins are managed within the Event Source Management (ESM) framework, which is a GUI tool that allows you to deploy and configure Collectors and attach them to Connectors and Event Sources. Collector typically use Connectors to gather the data they will process; the Connector performs the protocol-level connection and handles errors and retries, ensuring that the Collector's task is simpler.

Collectors take input records from the event sources and process them into Sentinel data objects. The primary such object is the Event, which represents the actual parsed event that is sent to the Sentinel backend. The Collector must produce valid, normalized Events which follow the Sentinel Event schema and Sentinel Taxonomy.

Collectors may also produce Identity and Account objects, which are stored in the Sentinel DB, exported into a map applied to incoming events, and available via the Identity Browser.

Coming soon we will add JS versions of Asset and Vulnerability objects; you can use legacy code in the meantime to store those types of objects.

Custom Execution Mode

This site is primarily about developing new Collectors based on the SDK template. In some circumstances, however, customers wish to simply extend the operation of an existing Collector, in which case the Collector will operate in custom mode. A separate section will describe how to create and install parsing logic for Custom Execution Mode.

Getting Started

This section assumes that you have already completed the setup steps detailed in SDK Pre-Requisites and SDK Installation and Configuration.

To begin creating a Sentinel Collector plug-in:

Expand the "Collectors" entry under the Ant tab (Window > Show View > Ant if not available) (you should hide internal targets by selecting the target/slash icon in the toolbar)
Double-click on the "new" entry under "Collectors", and follow the prompts
1. When prompted, enter the vendor that sells the product whose data you will be processing
2. When prompted, enter the name of the product whose data you will be processing
3. Select the language you will use (Novell recommends using the newer JavaScript language)
4. Select the category of the event source, e.g. what type of product you will be collecting data from.
5. Select the license type of the event source and continue
In the Package Explorer tab, select the "collectors" project and hit F5 to refresh the listing
- You should see your new Collector development directory appear in the list under "collectors" under the vendor and product names that you entered

This procedure will clone the Collector template to create a new skeleton Collector in a development directory. You will edit the files in this new Collector plug-in to add code, meta-information, documentation, and other key features.

Building the Plug-in

In order to build your plug-in from the development directory into a functional plug-in (the default template will work as a fully-functional plug-in), perform the following steps:

Expand the "collectors" project in the Package Explorer tab
Expand the directory that contains your Collector development directory
Expand the "6.1" directory to reveal the development files
Drag the "build.xml" file in that directory over to the Ant tab
- You should see a new Ant project with the name of your Collector
Expand your Collector under the Ant tab
Double-click on the "build" target and follow the prompts
1. Select "n" when asked to increment the release number

The resulting built plug-in and associated files will be created under the directory /proj/content/build/<plug-in name>.
The plug-in will be named <plug-in name>_<template version>r<release version>.clz
This initial build is just to test that things are working; as you develop your plug-in you can rebuild as many times as you like to debug and test your work
When development is complete, you can remove the project from the Ant tab to reduce clutter; if you go back to working on the same plug-in, you can re-drop the "build.xml" to the Ant tab

It is expected that you will build your plug-in many times during development; the build process is fully automated. In addition to the ant build command, which does a complete build, you can also use the ant build-zip which only builds the portion of the plug-in based on code (e.g. no documentation etc), for more rapid development cycles.

After building the Collector, import it into ESM using the Scripts palette. If the Collector already exists in ESM, it will be replaced. If the Collector is deployed and running or being debugged, you will have to restart the Collector to load any changes.

Development Process

The development process consists of:

Clone the template to make a new skeleton plug-in (see above)
Modify the development code (see below) and meta-information files
Build for test and debug (see above)
- Repeat previous two steps
Perform a final build and release your plug-in

Exploring the Template

The newly created skeleton Collector development directory has the structure displayed at right.

The top level directory is given the vendor name of the Collector, as you specified during the ant new process.
The next level lists the product name you entered for this Collector
- If you change these names, the plugin will take on the new name
  - But note that the name in the Ant tab will retain the old name unless you edit build.xml
The archive directory is a place to contain old copies of the plug-in, for reference
The research directory is a place to contain research, design, and integration documentation
The sample directory is a place to put sample input or output files
The 6.1 directory is the actual development directory where the plug-in resides
- The "6.1" refers to the version of the template in use

Under 6.1
- The docs directory contains the embedded plug-in documentation template (see below)
- The pack directory contains the template Collector Pack which applies to this Collector (a set of controls to assist in management of this Collector) (see below)
  - The package.xml file contains the empty release Collector Pack
- The parameters directory contains the plug-in parameters (see below)
- The build.xml file contains the build instructions for this specific plug-in
  - Drag the build.xml file over to the Ant tab to run the build
  - If you change the name of the plug-in, also edit this file and change the project name on the first line
- The connectionMethods.xml file contains the list of supported Connectors that work with this Collector (see below)
- The controls.exclude file lists the template controls that should be excluded from the final Collector Pack
- The files.include file lists files that should be added to the plug-in during the build; these are typically common code, common maps, or similar
- The deviceSupport.xml contains the list of event sources that this Collector can process data from (see below)
- The plugin.properties file contains meta-information about the plug-in, such as a description (see below)
- The protoEvt.map file contains the static Event fields that are preset for data from this Collector (see below)
- The Rec2Evt.map contains the Record to Event conversion DataMap (see below)
- The release.js file contains the actual code that is run when the plug-in is executed.
  - The overall control structure of the plug-in is defined elsewhere in the template; you just need to build the methods defined in this file
  - You can create additional *.js files as necessary to create new classes or methods
- The release.properties file contains the current release number of the plug-in
  - This is incremented if you select "y" to the corresponding question during the build
  - In general we recommend you increment this for any publicly-released plug-ins.
- The sqlquery.base file contains the template SQL query used by this Collector (if it gets data from a DB) (see below)
- The taxonomy.map file contains a KeyMap used to generate taxonomy for each event (see below)

JavaScript Files

The Collector execution structure is not actually defined in the *.js file in the development directory. Instead it is defined in a template which calls out to the methods that you will define in that file. The two are combined during the build process that creates the actual plug-in.

The overall flow of Collector execution looks like this:

The steps in light green are the methods that you will create; the rest of the template will handle the control flow necessary to make the Collector work correctly.

Detailed walkthrough: Implementing the Collector methods defined in the template

JavaScript Notes

When building your code, there are several things to remember:

The JavaScript interpreter is based on Rhino
You can use all of the standard JS objects including E4X
Many Sentinel-specific objects are defined, like Identities and Accounts, that you can access and use in your code
- See the Sentinel JavaScript API for details
Some sample code is provided to, for example, get data from a Connector
[Advanced] You can call Java methods by importing the appropriate JDK classes
[Advanced] You can also define your own Java objects and methods in your own JAR files, and include them in the Action plug-in

Meta-information Files

The template also includes several files that contain meta-information used to help define properties of the plug-in.

pack

This directory contains the empty release Collector Pack. When it is first copied into the dev directory, it's basically an empty Collector Pack; you add additional categories and controls as necessary to support your Collector. At build time, the materials you build to support the Collector release will be merged with a "Base Collector Pack" which includes commonly-used controls to create the complete release Collector Pack.

Use the 'pack-edit' Ant target to open and edit the Collector pack

Review the Novell documentation on Solution Packs (Collector Packs are just special-purpose Solution Packs) regarding how to build your own controls. You can include reports, correlation rules, Actions, iTRAC workflows, and other content to enforce whatever policies or provide whatever visibility you need.
Review the Base Collector Pack to see what controls are provided by default. The easiest way to do that is to:
1. Create a new empty Collector plug-in (call it anything)
2. Delete the contents of the controls.exclude file (see below) in that Collector
3. Build the Collector
4. Open the resulting Collector Pack in Solution Designer to explore the categories and controls
When you build controls for your Collector, make sure you filter to only use data from your Collector. To do so, filter for your Collector name in the "CollectorScript" field (internal tag "agent").

controls.exclude

This file lists the template controls that should be excluded from the final Collector Pack. By default the file lists all controls that depend on specific data from the Collector; delete entries from this file if your Collector will generate data that feeds a specific control. In general this is dependent on the taxonomy of the events produced.

For example, if your event source includes the concept of user management and generates events relating to creation of users, remove the "User Account Management" exclusion from this file.

files.include

This file lists the common library code and mapping files that should be included in the output Collector. Enter each file name (from the //content/dev/collectors/AA-common/src directory) that you wish to be included in the build, separated by commas.

connectionMethods.xml

This file contains the list of supported Connectors that work with this Collector. In general most event sources will support collecting event information using a specific protocol like SNMP, Syslog, WMI, etc. Some sources will support multiple methods, for example Snort allows you to send event data to both Syslog and a database. The job of this file is to list which connection types are supported, along with associated settings.

The template file copied in contains entries for all common Connectors; eliminate those which are not used for your event source.
Note that the last method listed is used for debugging by replaying data captured to file using the "Save raw data to file" option available on all Connectors. Do not delete this connection method.
Each Connector supports certain options that are passed to the Connector at deployment time and which control how the Connector works, including what the Connector output looks like. The Connector documentation for each Connector has details; make sure you specify the correct options for the Connectors your event source will use. See the template file for examples.

In general, the format if this file is:

ConnectionMethods element - top level
- ConnectionMethod element - Include one of these for each supported connection method (in some cases, you may have multiple methods for a single type of Connector)
  - ConnectorName element - The name of the Connector - get this from the Connector documentation (or use the samples)
  - IsDefault element - Set to "1" for only one ConnectionMethod in the file to specify the default method
  - ConnectionModes element - specifies different modes in which this Connector can operate. These modes are actually applied per Event Source
    - ConnectionMode element - specifies a specific connection mode
      - InternalName element - an internal name for this mode
      - DisplayName element - the name of this mode which will be displayed in the "Connection Mode (Advanced)" tab on each Event Source's properties.
      - Description element - A description of this Connection Mode (not sure this is displayed anywhere, so don't spend too much time worrying about it)
      - IsDefault element - Set to "1" for only one ConnectionMode under this ConnectionMethod to specify the default mode
      - Properties element - The properties that will be applied to the Connector/Event Source when this mode is selected (you will see the name/value pairs for each property appear in the "Customize Settings" table in the "Connection Mode (Advanced)" tab on each Event Source's properties.
        Properties are made up of names and values that specify parameters for the Connector/Event Source. See the samples and the Connector documentation.

deviceSupport.xml

This file contains the list of event sources that this Collector can process data from. In essence this is used to indicate which versions of event sources a particular Collector supports. During Collector deployment, the customer actually starts by selecting the name of the event source application/device; this then leads to the Collector(s) that support that device.

SupportedDevices element - containing element that holds the list of supported devices
- CollectorSupportedDevice element - contains the specification for a single supported device/event source - repeat this element for each version
  - Vendor element - The vendor that sells/distributes this specific event source
  - Name element - The product name of the event source
  - Version element - The supported version of the event source
  - Description element - A description of this event source (not displayed anywhere as far as I know)

plugin.properties

The plugin.properties file contains some meta-information about the Collector. One of the first and last things you should do is review this file and make any necessary adjustments:

uuid: the unique ID for this plug-in (autogenerated, DO NOT EDIT)
entitlement: Not used
plugindesc: description of the plug-in; edit this to describe what the plug-in does
pcat: Product category - see Sentinel_Taxonomy for details
includejars: if you create custom JARs for your Collector, list them here so they will be in the classpath

There may be other elements in this file - do not edit them.

protoEvt.map

This file contains the static Event fields that are preset for data from this Collector. Data that is configured in this file will be hard-coded to appear in every event that passes through this Collector. If you would like to set one of these fields, cut the relevant line from the Rec2Evt.map file (see below) and paste it here with the hardcoded data. You can use the same replacement filters available for the SDK Documentation.

Do not edit the last few lines in the file as those are universal defaults

Rec2Evt.map

This file contains the Record to Event conversion DataMap. This determines which fields in the input record get placed into which fields in the output Event.

This file is extremely useful for event sources that have regular output formats like databases, name-value pair formats, or delimited text formats where the meanings of event fields are relatively fixed. You can do a minimal amount of parsing of data to extract each data element, and then simply specify where to put that data in the output event.

For event sources with variable output formats or where the meaning of a given field changes, you might need to do some additional manipulation to the input data before you can apply this map. This will be a balancing act for each Collector you develop.

As you implement your preParse(), parse(), and normalize() methods, for the most part you will be chopping up the input Record object and storing parts of the input data back into attributes of that same object. You may need to apply conditional logic in some cases to determine which Record attribute to store a particular piece of data in. Then you simply list the source fields in the Record object on the corresponding lines of this map file. The alternative is to pull data out of the Record object and store it directly into the output Event object (which is passed in as variable 'e' to each method). This is not the preferred method, however, as later modifications to the schema will then force more difficult code changes.

The format of the file is:

<field name of event object>,<attribute in Record object>

' The field names of the output Event are pre-assigned and should not be edited. When specifying the source fields, do not include the object name itself, i.e. if you want to copy rec.object[0].username into the Event field InitUserName, then the file will look like:

InitUserName,object[0].username

After you are done specifying all the fields you want to capture, you can delete any lines that have nothing in the second column.

release.properties

The release.properties file contains the current release version number for the plug-in. It is automatically maintained by the build process; you are prompted to increment the release number if you wish:

Novell recommends that you increment the release number when you begin work on a new version that will be release to production or the public. In other words, if the current production version is "5", select "y" to the "increment release number?" question when you begin working on the next release and build it for the first time.
The only circumstance in which you should manually edit this file is if you accidentally increment the release number.
The release number appears in the filename of the built plug-in: "Novell_Identity-Manager_6.1r1.clz"

sqlquery.base

This file contains the template SQL query used by this Collector (if it gets data from a DB). Basically you want to put the SQL query that must be issued against the event source's DB exactly as it should be issued, except with two replacement parameters:

%d : this parameter will be replaced with the maximum number of rows to return on a single query against the database. You must add the max_rows.xml parameter specification to your template.pml file so that the correct parameter is passed in to the Collector.
%s : this parameter is replaced with a string used to define the starting offset for the database query. For SQL Server you might use a number that is compared to the AutoID column; for other databases you may use a date/time field or some other incrementing field. Your query should request rows that are greater than this offset, and you must pass a starting offset in the correct format as part of your connectionMethods.xml file (see above, and see the template file).

If you are not using the DB Connector, you may delete this file from your dev directory.
The sample template version includes comments to help guide the structure; you may include your own comments with '~' as the first character on a line which will be ignored

taxonomy.map

This file contains a KeyMap used to generate taxonomy for each event.

You must define a taxonomy key for each event received from the event source and set that key for the current event using the Event.setTaxKey() method. This key will be used to select the corresponding row in this map file and return the appropriate taxonomy. Use the taxonomy specified on the Sentinel Taxonomy page.

For example, if your input contains a field called "eventname" that is set to a unique identifier for each event, you might use:

this.s_RXBufferString.search(/eventname: (.*)/);
this.evt = RegExp.$1;
e.setTaxKey(this.evt);

Suppose that the event name in one case is "User Create"; your taxonomy file might then say:

User Create,SYSTEM,USER,CREATE,,XDAS_AE_CREATE_ACCOUNT,XDAS_OUT_SUCCESS

Note that you might need to make compound keys to handle different outcomes, suppose that this.evt contains the event name and this.result contains the outcome of the attempt:

e.setTaxKey(this.evt + "_" + this.result);

And in the taxonomy file:

User Create_OK,SYSTEM,USER,CREATE,,XDAS_AE_CREATE_ACCOUNT,XDAS_OUT_SUCCESS
User Create_SYSFAIL,SYSTEM,USER,CREATE FAILED,,XDAS_AE_CREATE_ACCOUNT,XDAS_OUT_FAILED
User Create_DENIED,SYSTEM,USER,CREATE FAILED,,XDAS_AE_CREATE_ACCOUNT,XDAS_OUT_DENIED

Plug-in Documentation Template

(follow heading link)

Parameters

(follow heading link)

Back up to Develop to Sentinel

Retrieved from "http://developer.novell.com/wiki/index.php/Collectors"

Novell® Making IT Work As One™

© 2009 Novell, Inc. All Rights Reserved.