Genfiledriver

From Developer Community

1 Generic file driver for IDM
2 Downloads
3 Browse the CVS Tree
4 CVS Access
5 Bug Tracker

[edit]

Generic file driver for IDM

[edit]

News

[edit]

11 April 2007

First release of GenFileDriver as uploaded to cooltools.

[edit]

Description

The Genric File Driver is similar to the Text Driver shipped with IDM, but has more options, and has the capability to read virtually any file type. Out of the box, the following file types are supported: XML, CSV and XLS (the latter using POI 2.5.1 from apache).

[edit]

Main Features

Pluggable setup: the driver uses pluggable strategies to perform tasks such as what files should be read, how should a file be read/written, when a new file should be created, etc. The driver is shipped with strategies for each of the options.
Currently support for reading and writing XML, CSV and XLS files. Possible support (via custom plugins) for virtually every file type.
When reading XML or CSV files, maximum 20 records are in memory at any given time. This allow for reading of large files.
Input and output format are independent. You can eg read CSV files and write XML files.
The records in the input file can be enriched with meta data (eg: record number, filename, last record indicator, etc).
Restarts reading of input files at the point it was interrupted when it was interrupted (eg: shutdown during reading of a file).

[edit]

Progress

First build has been released.
Still to do:

create a preconfigured driver
update documentation

[edit]

Downloads

Nothing released yet via forge. See cooltools for downloading and installation instructions

driver config does not work as posted in coolsolutions, here's a properly escaped one:

<?xml version="1.0" encoding="UTF-8"?><driver-configuration dn="GenFileDriver.Driver Set.admin" driver-set-dn="Driver Set.admin" name="Generic File Driver">
	<attributes>
		<configuration-manifest>
			<manifest/>
		</configuration-manifest>
		<driver-filter-xml>
			<filter/>
		</driver-filter-xml>
		<driver-image/>
		<log-events inherit="true"/>
		<trace-file inherit="true"/>
		<trace-size-limit inherit="true"/>
		<trace-level inherit="true"/>
		<driver-trace-level inherit="true"/>
		<log-limit inherit="true"/>
		<java-module value="info.vancauwenberge.filedriver.shim.driver.GenericFileDriverShim"/>
		<policy-linkage/>
		<driver-cache-limit value="0"/>
		<shim-auth-password-query/>
		<driver-password-query/>
		<driver-start-option no-resync="false" value="0"/>
		<shim-config-info-xml>
			<driver-config name="GenFileDriver">
				<driver-options>
					<schema display-name="Field Names (Field1,Field2,Field3)" id="100">LastName,FirstName,Title,Email,WorkPhone,Fax,WirelessPhone,Description</schema>
					<objectClass display-name="Object Class Name" id="101">User</objectClass>
				</driver-options>
				<publisher-options>
					<pub_fileLocator display-name="Publisher: File Locator Strategy (Class):" id="124">info.vancauwenberge.filedriver.filelocator.RegExpFileLocator</pub_fileLocator>
					<pub_FileSorter display-name="Publisher: File Sorter Strategy (Class):" id="125">info.vancauwenberge.filedriver.filesorter.FilePropertySorter</pub_FileSorter>
					<pub_fileReader display-name="Publisher: File Reader Strategy (Class):" id="126">info.vancauwenberge.filedriver.filereader.csv.CSVFileReader</pub_fileReader>
					<pub_pollingInterval display-name="Publisher: Polling interval (seconds):" id="127">5</pub_pollingInterval>
					<pub_metaData display-name="Publisher: List of meta data elements that should be added(recordNumber,isLastRecord,filePath,fileName,fileSize):" id="128">recordNumber,isLastRecord</pub_metaData>
					<pub_heartbeatInterval display-name="Publisher: Heartbeat interval (minutes):" id="129">1</pub_heartbeatInterval>
					<pub_workDir display-name="Publisher: Temporary work folder:" id="130">/home/nlv10194/filedriver/temp</pub_workDir>
					<regExp-sourceFolder display-name="RegExpFileLocator: Source folder for files:" id="131">/home/nlv10194/filedriver</regExp-sourceFolder>
					<regExp-regExp display-name="RegExpFileLocator: Regular expression for finding files.Use (?i) at start to make case ignore regexp:" id="132">(?i).*\.csv</regExp-regExp>
					<fileSort_SortMethod display-name="FileSorter: File sort methode:" id="133">getName</fileSort_SortMethod>
					<fileSort_SortOrderAsc display-name="FileSorter: Sort ascending (true/false):" id="134">true</fileSort_SortOrderAsc>
					<xmlReader_useTagNames display-name="XMLFileReader: Use the tag names from the XML document (true) or use the driver schema given (false):" id="135">true</xmlReader_useTagNames>
					<xmlReader_preXslt display-name="XMLFileReader: xsl to apply prior to processing the XML file (leave empty for no pre-processing):" id="136"><?xml version="1.0" encoding="UTF-8"?><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:fo="http://www.w3.org/1999/XSL/Format"><xsl:output method="xml" indent="yes"/><xsl:template match="root"><xsl:copy><xsl:applytemplates/&gt;</xsl:copy></xsl:template><xsl:template match="record"><xsl:copy><xsl:applytemplates/&gt;</xsl:copy></xsl:template><xsl:template match="LastName"><newLastName><xsl:value-of select="."/></newLastName></xsl:template><xsl:template match="*"><xsl:copy><xsl:applytemplates/&gt;</xsl:copy></xsl:template></xsl:stylesheet></xmlWriter_PostXSL></xmlReader_preXslt>
					<xmlReader_forcedEncoding display-name="XMLFileReader: forced enoding of the xml file. Leave blank to use the encoding attribute inside the xml file (or system default when attribute not given)." id="137"/>
					<csvReader_skipEmptyLines display-name="CSVReader: skip empty lines (true/false)" id="138">true</csvReader_skipEmptyLines>
					<csvReader_UseHeaderNames display-name="CSVReader: use the header names (true) or the driver schema given (false) " id="139">true</csvReader_UseHeaderNames>
					<csvReader_hasHeader display-name="CSVReader: has the CSV file a header (true/false)" id="140">true</csvReader_hasHeader>
					<csvReader_forcedEncoding display-name="CSVReader: forced enoding of the xml file. Lea ve blank to use system default encoding." id="141">UTF-8</csvReader_forcedEncoding>
					<csvReader_seperator display-name="CSVReader: field seperator" id="142">,</csvReader_seperator>
					<xlsReader_SheetName display-name="XLSReader: Sheet name:" id="143">Sheet1</xlsReader_SheetName>
					<xlsReader_HasHeader display-name="XLSReader: has the XLS sheet a header (true/false):" id="144">true</xlsReader_HasHeader>
					<xlsReader_UseHeaderNames display-name="XLSReader: use the header names -if any- (true) or the driver schema given (false):" id="145">true</xlsReader_UseHeaderNames>
				</publisher-options>
				<subscriber-options>
					<sub_FileStartStrategy display-name="Subscriber: File Start Strategy (Class):" id="102">info.vancauwenberge.filedriver.filestart.BasicNewFileDecider</sub_FileStartStrategy>
					<sub_FileNameStrategy display-name="Subscriber: File Name Strategy (Class):" id="103">info.vancauwenberge.filedriver.filename.SimpleDateFormatFileNameStrategy</sub_FileNameStrategy>
					<sub_FileWriteStrategy display-name="Subscriber: File Write Strategy (Class):" id="104">info.vancauwenberge.filedriver.filewriter.XMLFileWriter</sub_FileWriteStrategy>
					<sub_WorkDir display-name="Subscriber: Work directory:" id="105">C:\workspace\DirXMLFileDriver\temp</sub_WorkDir>
					<sub_DestFolder display-name="Subscriber: Destination directory:" id="106">C:\workspace\DirXMLFileDriver</sub_DestFolder>
					<newFile_MaxRecords display-name="NewFile: Maximum number of data records in a file:" id="107">100</newFile_MaxRecords>
					<newFile_MaxFileAge display-name="NewFile: Maximum 'age' of a file (in seconds):" id="108">60</newFile_MaxFileAge>
					<newFile_InactiveSaveInterval display-name="NewFile: Save file after nnn seconds of inactivity:" id="109">0</newFile_InactiveSaveInterval>
					<newFile_FieldName display-name="NewFile: Fieldname used to manully control the creation of new files:" id="110"/>
					<csvWriter_WriteHeader display-name="CSVWriter: Write a header record(true/false):" id="111">true</csvWriter_WriteHeader>
					<csvWriter_Seperator display-name="CSVWriter: Seperator character:" id="112">,</csvWriter_Seperator>
					<csvWriter_ForcedEncoding display-name="CSVWriter: Forced file encoding. Leave blank to use system default encoding.:" id="113">UTF-8</csvWriter_ForcedEncoding>
					<xmlWriter_RootName display-name="XMLWriter: Root element name.:" id="114">root</xmlWriter_RootName>
					<xmlWriter_RecordName display-name="XMLWriter: Record element name:" id="115">record</xmlWriter_RecordName>
					<xmlWriter_ForcedEncoding display-name="XMLWriter: Forced file encoding. Leave blank to use system default encoding.:" id="116">UTF-8</xmlWriter_ForcedEncoding>
					<xmlWriter_PostXSL display-name="XMLWriter: Xsl to apply after saving the XML file (leave empty for no postprocessing):" id="117"><?xml version="1.0" encoding="UTF-8"?><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:fo="http://www.w3.org/1999/XSL/Format"><xsl:output method="xml" indent="yes"/><xsl:template match="root"><xsl:copy><xsl:applytemplates/&gt;</xsl:copy></xsl:template><xsl:template match="aRecord"><xsl:copy><xsl:applytemplates/&gt;</xsl:copy></xsl:template><xsl:template match="schema"><xsl:copy><xsl:applytemplates/&gt;</xsl:copy></xsl:template><xsl:template match="objectClass"><newObjectClass><xsl:value-of select="."/></newObjectClass></xsl:template></xsl:stylesheet></xmlReader_preXslt></xmlWriter_PostXSL>
					<xlsWriter_SheetName display-name="XLSWriter: Sheet name:" id="118">Sheet1</xlsWriter_SheetName>
					<xlsWriter_AddHeader display-name="XLSWriter: Add a header row (true/false):" id="119">true</xlsWriter_AddHeader>
					<GUIDNamer_FilePostFix display-name="GUID FileNamer: Optional postfix for the filename (eg: file extention:" id="120">.csv.out</GUIDNamer_FilePostFix>
					<GUIDNamer_FilePreFix display-name="GUID FileNamer: Optional prefix for the filename:" id="121">NEW_FILE_</GUIDNamer_FilePreFix>
					<simpleDateNamer_FormatString display-name="Date FileNamer: Date format string(according to jave.util.SimpleDateFormat):" id="122">'NEW'_'FILE'_yyyyMMdd-HHmmssSSS.'out'</simpleDateNamer_FormatString>
					<simpleDateNamer_TimeZone display-name="GUID FileNamer: Timezone for the dataformatter. Leave blank to use system default timezone.:" id="123"/>
				</subscriber-options>
			</driver-config>
		</shim-config-info-xml>
		<global-engine-values/>
		<global-config-values/>
		<application-schema/>
	</attributes>
	<children>
		<publisher name="Publisher">
			<attributes/>
			<children/>
		</publisher>
		<subscriber name="Subscriber">
			<attributes/>
			<children/>
		</subscriber>
	</children>
</driver-configuration>

[edit]

Browse the CVS Tree

Browsing the CVS tree gives you a great view into the current status of this project's code. You may also view the complete histories of any file in the repository.