Novell is closing it's Forge project site and so the project has to move. The SVN server at Novell doesn't support SVN Replay so the source repository not moving yet.
The new home will be courtesy of Google Code:
update: Novell will be providing a SVN dump so I can use that to populate the Google repository. After that is done I can start migrating the documentation from the wiki.
update 2: Novell have dumped and locked the SVN repository and mailing list. A new mailing list has been setup on Google Groups. Awaiting access to said dump in order to upload to Google.
update 3: Invitations have been sent to all current list members to join the new Google Group hosted email list.
update 4: Source code has been migrated to Google SVN.
The 2.0 branch is starting to stabilize, passing unit and network tests. Many bug fixes and performance enhancements with bus based transports, i.e. both send and receive enabled transports.
Download the tarball via the Novell Wiki or FTP site:
The new 1.2 branch changes the default functionality of UDP encapsulated sockets to have SO_REUSEADDR set. This means multiple applications can listen on the same port with UDP sockets exactly the same as CAP_RAW privileged applications can with IP/PGM sockets.
Labeled versions have now been updated to match the internal API numbers, meaning 1-0-0 release is actually 1-0-12, with the latest stable update at 1-0-13. The new 2.0 beta is currently at 2-0-6.
Details of how to migrate existing version 1.0 to version 2.0 code is now available on the wiki:
OpenPGM 2.0 is now functional and supported on OpenSolaris 2008.11 and 2009.06. Support can be found in the latest beta tarball:
Or with the 2.0 beta 2 tag in the SVN repository. To access this version of the 2.0 release use the following commands:
svn checkout https://forgesvn1.novell.com/…/release-2-0-0beta2
Build instructions continue as per installation platform detailed on the Build Library instructions page.
New documentation covering the changes from version 1.0 is now becoming available on the wiki, find listed as Beta under the documentation section.
The beta build is now functional on AMD64 and i386 releases of Ubuntu, Debian, and RHEL/CentOS. A Win32 cross-compile build is available using the MinGW tool chain on Linux. OpenSolaris support is currently incomplete.
The 1.1 development branch is now bumped to 2.0 beta as pretty much every API has been updated. The latest changes include unit testing coverage of every source module and inline timer processing: no need for an extra timer thread. As a beta, functionality is complete but full platform support is not currently available, only SCons 0.97 building is functional on Ubuntu AMD64.
Other News
ØMQ version 1.0.0 has been released bundling a fully integrated OpenPGM subsystem.
The 1.x development tree is progressing with the following changes:
As a stable release updates to 1.0 are being maintained in the 1.0 branch in the SVN repository. To access the latest stable version of the 1.0 release use the following commands:
svn checkout https://forgesvn1.novell.com/…/release-1-0-0
Build instructions continue as per installation platform detailed on the Build Library instructions page.
OpenPGM 1.0.0 Press Release from Miru Limited.
The following changes apply to the last release candidate.
Known Issues
Future Plans
For 1.1 releases the follow features are being implemented.
openSUSE from Novell is now a supported platform with rc8. One small minor difference is the lack of a ncurses panel library and so the ncurses example application is not supported.
This release candidate adds OpenSolaris support for IA32/AMD64 architectures, SPARC is untested, along with validation for Ubuntu 9.04 and RHEL/CentOS 5.3. Build scripts have been updated for Scons version 1.0 and newer including libsoup 2.4 support as 2.2 is not readily available on OpenSolaris. Building with Scons is now modular enabling the core library to be built without libsoup or net-snmp dependencies.
For routed environments OpenPGM now tags IP ToS fields as per DSCP definitions (RFC 3246) as the previous interpretation is obsolete (RFC 1349).
There are no API changes to the previous release candidate 1.0.0rc7. This is the last expected release candidate pending major bug reports or new architecture support.
(Update) Link to the rc8 download package, the Novell Wiki appears out of synchronisation with the FTP site contents:
This release candidate makes available the senders TSI for application layer session tracking, along with improved rate handling at gigabit data rates, and bug fixes.
The ABI is broken for the pgm_msgv_t structure adding a pgm_tsi_t identifier, and adding a new function pgm_transport_recvfrom() to capture the TSI with basic single APDU buffer usage.
API documentation has been updated to reflect the changes of the rc7 releases.
This release candidate introduces compatibility with Microsoft's PGM implementation in Windows 2008, Source-Specific Multicast (SSM), notification of affected source when unrecoverable data loss is detected, support for Net-SNMP 5.4.1, and general conformance, performance, and reliability improvements.
The ABI is broken again, to implement both Any-Source Multicast (ASM) and Source-Specific Multicast (SSM) on pgm_transport_create(). The gsr_source members actually mean the address of the multicast source for SSM or should be the same as gsr_group for ASM. pgm_create_ipv4_gsi() now assumes the node address for easier usage.
API documentation has been updated to reflect the changes of the rc6 releases.
This release candidate introduces further test suites to ensure stable and reliable code. The ABI is broken again, to reduce confusion by matching parameter order on pgm_transport_create() and pgm_if_parse_transport(), and adding a boolean parameter to pgm_transport_set_close_on_failure() and pgm_transport_set_send_only() for better clarity.
API documentation has been updated to reflect the changes of the rc5 releases.
PGM experimental support is now available as part of ØMQ version 0.5. ØMQ or ZeroMQ is a messaging middleware that adds high level functionality like routing and queueing which is typically required for advanced application communication.
Development on OpenPGM 1.0 rc5 is continuing with improvements to the network and unit test suite to ensure correct operation of all modules.
Other News:
Red Hat releases Red Hat MRG 1.1 with support for InfiniBand, .NET, and additional queueing functions.
Demo Gstreamer plugins acting as a Gstreamer sink/PGM source (pgmsink) and Gstreamer source/PGM source (pgmsrc) have been updated to meet the 1.0.0rc4 ABI. These can be used with the gst-launch command line or any Gstreamer compatible application. Example pipelines are provided for h263, x264, theora, and JPEG multi-part streams. Multicast loop is enabled by default for single host testing.
Example x264 source:
GST_PLUGIN_PATH=. gst-launch-0.10 \
v4lsrc ! \
videorate ! \
video/x-raw-yuv,width=320,height=240,framerate=15/1 ! \
x264enc byte-stream=true bitrate=128 bframes=4 b_pyramid=true \
me=dia trellis=false key_int_max=4000 threads=1 ! \
rtph264pay ! \
pgmsink
Example x264 receiver:
GST_PLUGIN_PATH=. gst-launch-0.10 \ pgmsrc caps="application/x-rtp" ! \ rtph264depay ! \ ffdec_h264 ! \ videoscale ! \ video/x-raw-yuv,width=640,height=480 ! \ timeoverlay ! \ autovideosink
Download and build with SCons. Requires GStreamer 0.10 and dependencies.
http://developer.novell.com/wiki/index.php/Special:Downloads/openpgm/gstpgm3.tar.bz2
This release candidate transitions over to the CMake build system together with improved 32-bit architecture support. The ABI is broken again to add support for setting the PGM source port.
API documentation has been updated to reflect the changes of the rc3 and rc4 releases.
Changes in this revision compared with rc2, minor changes to application code are required to handle IPv6 multicast group specifications.
OpenPGM has been engineered for dual stack support, both IPv4 and IPv6, but to-date only IPv4 has been functional. As of revision 271 in the development trunk initial functioning IPv6 support now exists with the help of transition to full RFC 3678 multicast source filter socket interface objects.
This is a relatively minor API change which should only consist of renaming pgm_sock_mreq structures to RFC 3678 compliant group_source_req, and if using UDP encapsulation updating smr_multiaddr and smr_interface member variable names to gsr_group and gsr_source. See the pgmsend diffs as an example of required code changes.
A method to build on RHEL/CentOS 4 is now available, however it does not work with the default GLib tree. RHEL bundles an updated GLib tree for the Evolution mail client and this has been used to enable OpenPGM construction. Support from Red Hat is likely to be limited for this library set.
Support for the latest RHEL/CentOS supported platform is now bumped to 5.2. Upgrading from 5.1 is a simple yum update. Further details available in the CentOS 5.2 release announcement.
Other News:
Red Hat release version 1.0 of Red Hat Enterprise MRG.
The Build Library instructions have been updated for the recent Ubuntu 8.04 release. The latest RHEL/CentOS supported platform remains version 5.1.
RedHat have announced the release of RedHat Enterprise Linux 5.2, whilst the CentOS team guestimate a mid-June CentOS 5.2 release.
Other News:
Cisco announces an open source messaging protocol called Etch, no official details on cisco.com, but CIO.com details Etch beta release expected mid-July.
update: Etch project page now available in the Cisco Developer Community as part of the Cisco Unified Application Environment (CUAE), with plans to join the Apache Incubator.
Forward error correction (FEC) provides two advantages, the ability to recover lost packets from extra redundant information sent with original data, and the ability to recover any lost packet with each extra parity packet. The cost is extra processing time calculating the redundant parity packets and regenerating lost packets.
The PGM protocol transport can elect to offer either or both forms of error correction together with regulation selective re-transmission requests.
Other News:
The MIT DARPA Urban Challenge Team published their Lightweight Communications and Marshalling (LCM) library. Similar to Rendezvous 8 it provides an in-process daemon but with basic subject naming, XDR message format, and optional reliability. APIs for C, Java, and Python.
The OpenCV library from Intel aimed at real time computer vision. It includes a high level API for capturing from any local video source and displaying in a window with only a few lines. PGM enables the distribution of video sources across a network.
The example application cvdd sources from a local camera, compresses each frame with JPEG 2000 and multicasts over the network with the OpenPGM stack, with a different command line parameter the same application can subscribe to PGM sources and display on the local monitor. Native IPL images are used and so the code can be extended to implement network computer vision systems.
The sending side also receives a re-factoring and now features non-blocking support with respect to the rate control parameters and the ability to send a vector of APDUs.
The OpenPGM C Reference is now up to date for both new API updates.
The receive side of the OpenPGM stack is undergoing a huge overhaul, it will now allow direct synchronous non-blocking receiving of message vectors of TPDU sized scatter/gather IO vectors. Unfortunately there is no matching Linux API for zero-copy UDP or message vectors (vectors of recvmsg()), but this reduces the overhead the stack imposes for significantly high data rates.
Updated documentation covering the new APIs with examples for epoll(), poll(), select() and their asynchronous counterparts will follow shortly.
New documentation has been started to help newcomers understand and use OpenPGM, a Concepts guide covering the fundamentals and programming guidelines, and C Reference for the developers API.
Other News:
Red Hat announces Red Hat MRG (/mer.ʤe/ or merge), a new platform label for the combination of AMQP compliant messaging software, Apache Qpid, a new RHEL 5.1 real time kernel, and grid scheduler (RHG) for High Throughput Computing (HTC) and High Performance Computing (HPC).
Current research and development is on the AMD64 (also known as x86-64 as a generic term including Intel's EMT64) platform with only 64 bit code on Linux, progress can be tracked on the miru blog.
Other News:
RMM is a Reliable Multicast Messaging kit from IBM Haifa Labs with extensions available for large object transfer. IBM previously detailed RMTP, Reliable Multicast Transport Protocol in 1998 in partnership with NTT.
There are many published papers on UDP bulk transfer protocols and algorithms, these stem from Internet2 research between universities transferring gigantic datasets. The goal is high bandwidth and reliability, TCP over long distances has problems with maintaining high-speed data transfers and is only ever reliable: when transfer gigabyte to terabyte datasets a guarantee of transfer is preferred as re-downloading is not practical. Some of these concepts can be used with PGM however the other goal of PGM is for latency critical applications, over low latency links, that would like high bandwidth: the best of all worlds. The side effects are that NAK's have to be sent expediently rather than delayed (to cope with high latency links), and that many small packets need to be transmitted rather than large jumbo frames, or even jumbograms.
The TCP/IP stack has seen many developments to help speed up the internet with Scatter/Gather I/O, and new functions like sendfile() implementing zero copy semantics to reduce moving around blocks of memory which slows things down. However there is not currently support for sending many small packets, each currently require a context switch between user space and the kernel slowing handling down. One option is implementing the PGM protocol inside the kernel, however there are many problems with how the sending window is kept and managed. Microsoft Windows XP has an implementation of PGM that uses Microsoft Message Queuing (MSMQ) and file stores for reliability.
Other News:
Emcaster is a high-speed PGM interface in .NET (C#) using MSMQ for its sending window and claims performance on modest hardware to 0.5 Gb per second.
© 2009 Novell, Inc. All Rights Reserved.
