SAS Event Stream Processing Features List
In-memory, distributed & optimized processing that scales
- SAS Event Stream Processing server for processing millions of events per second and low-latency response times (millisecond, submillisecond).
- Built-in metering server for monitoring and recording event consumption for each SAS Event Stream Processing project, input window and production SAS Event Stream Processing server – speeding the collection of event consumption data.
- Retained and aggregated data kept in memory for maximum performance.
- Ability to take advantage of distributed grid architectures, SAS Cloud Analytic Services (CAS), or on public and private cloud providers (Azure and AWS support).
- Processing speeds can be customized with flexible thread-pool sizing, caching stores and more.
- Includes patented, instantaneous 1+N way failover, native failover, guaranteed delivery without the use of persistence and other fault-tolerance functions to ensure successful event stream processing activity.
- Full and open access to all event metadata.
Cloud native
- New SAS Event Stream Processing lightweight deployment scripts allow you to deploy containerized SAS Event Stream Processing for the cloud faster, easier and with fewer resources, but all the power of SAS Event Stream Processing.
- Scale resources dynamically on cloud with SAS Event Stream Processing’s new, built-in Kubernetes Operator framework for simplified deployment, upgrades and scalability in the cloud, public or private.
- Enables Microsoft Azure Event Hub integration for ingesting events from Azure IoT Event Hub using Kafka or native SDK support. Azure IoT Edge integration supported with SAS Event Stream Processing's Azure IoT Edge Hub integration.
- SAS ESP Kubernetes Load Balancer supports automated distribution of events using multicast, hash and round-robin techniques with simple deployment-time configuration for each project.
- Provides Amazon Web Services Kinesis and Redshift connectors for streaming data source connections.
- Enables multitenancy-ready deployments that can be integrated with your multitenant and multiuser environments.
In-stream learning model windows
- Allows you to combine different window types to specify data stream input sources, patterns of interest and derived output actions. Streaming model windows include:
- Train – Develop an advanced analytical model in stream and pass the resulting model updates to a score window.
- Score – Apply the trained SAS and open source (Python and ONNX format) model to current events in stream to produce score output, as well as support for learning models that use both training and scoring together.
- Calculate – Use with offline ASTORE models, Python code, data normalization and transformation methods, as well as learning models that bundle training and scoring together.
- Model supervisor – Control what model to deploy, and when and where to deploy it (for example, to the score window).
- Model reader – Integrate SAS and open source (Python and ONNX format) models and publish a model to another streaming analytics window, such as the score window.
Ability to consume & connect streaming data
- Extensive suite of data connectors for publishing and subscribing to live data streams of both structured and unstructured data, including videos, audio and images.
- Prebuilt, out-of-the-box connectors include read and write (i.e., publish and subscribe):
- Adapter connector makes it easier to manage adapters from within a SAS Event Stream Processing project, simplifying adapter orchestration (similar to connector orchestration).
- Aerospike (using plug-in technology).
- Amazon Web Services: Kinesis and Redshift.
- Apache Camel.
- Audio data.
- Axeda.
- BACNET.
- Cassandra.
- Common Event Format (CEF).
- Database ODBC: supports a variety of databases such as IBM DB2, Oracle, IBM Netezza, Sybase ASE and others. See Data Driver support for a complete list.
- Event Stream Processor.
- File/socket.
- HDAT Reader.
- HDFS.
- IBM WebSphere MQ.
- JMS.
- Kafka.
- MapR.
- Modbus.
- MQTT.
- Nurego.
- OPC-UA.
- OPC-DA.
- OSIsoft PI Asset Framework.
- Project Publish.
- QuasarDB.
- RabbitMQ.
- RADAR(Asterix CAT240 format support).
- REST.
- SAS Cloud Analytic Services.
- SAS data sets.
- Customized publish/subscribe APIs can also be written in C or Java.
- SAS LASR Analytic Server.
- Solace.
- Teradata.
- Tervela Data Fabric.
- TIBCO Rendezvous.
- Timer.
- URL.
- WebSocket.
- XML/JSON file socket adapter.
- Publish only to SAS Event Stream Processing from the following:
- BoardReader.
- HTTP RESTful interfaces.
- Log sniffers (Oracle, Greenplum).
- Network sniffer.
- SYSLOG.
- Subscribe only from SAS Event Stream Processing to:
- SOAP.
- SMTP.
- Connectors built for IoT event sources:
- Modbus.
- MQTT.
- OPC-UA.
- OPC-DA.
- UVC Camera.
- PylonCamera.
- Modbus.
- OSI PI Historian.
- BACnet gateway devices.
- Kafka.
- Cassandra (adapter only).
- Boar.
- dRea.
- der.
- Data stream support:
- Azure IoT Event Hub.
- Azure IoT Edge Connector.
- Hortonworks DataFlow (HDF) NiFi integration – SAS and HDF can provide immediate, streaming and deep intelligence.
- Static data joins – Integrate static data from databases or files to enrich streaming data using ODBC database adaptor and connector support in conjunction with database drivers.
Adaptable, in-stream analytics & data manipulation
- Machine learning streaming-algorithm support lets you create scoring and learning windows for various continuous learning algorithms. A combination of train and score windows are used to periodically update the model. These include:
- Streaming linear regression.
- Streaming logistic regression.
- Support vector machine.
- K-means clustering.
- Recommender.
- t-Distributed Stochastic Neighbor Embedding (t-SNE).
- In-stream analytics packaged with SAS Event Stream Processing includes:
- Built-in Image processing(crop, resize, rotate, flip)
- Video encoding.
- Butterworth Filter.
- Cepstrum Transformation.
- Change detection.
- Chebyshev Type I or Type II Filter.
- Density-based clustering (DBSCAN).
- Dirichlet Gaussian Mixture Model.
- Distribution Fitting.
- Fit statistics.
- Histogram.
- Kalman filter.
- Kernel Principal Components Analysis.
- Lag monitoring.
- Moving relative range (MRR).
- Parametric Power Spectral Density.
- Peak Finder.
- Segmented Correlation.
- Streaming Pearson’s correlation.
- Receiver operating characteristic (ROC).
- Robust Principal Components Analysis.
- Segmented correlation.
- Shapley Values.
- Short-time Fourier transform.
- Slice operations.
- Slice Peak Finder.
- Smoothing.
- Stability Monitoring Scoring.
- Streaming audio feature computation (speech to text).
- Streaming summary (univariate / multivariate statistics).
- Subspace tracking.
- T Test.
- Text tokenization.
- Text vectorization.
- Text Transcription (speech to text).
- Weibull distribution fitting.
- Convolution Algorithm.
- High-Dimensional Digital Signal Processing.
- Hampel Filter.
- Online Recursive ICA.
- Streaming KT Chart Monitoring.
- Algorithms for offline training packaged with SAS Event Stream Processing include:
- Robust principal components analysis.
- Bayesian network.
- Deep neural networks:
- Convolutional neural networks.
- Recurrent Neural Networks
- Factorization Machine.
- General Linear Model.
- Generalized Additive Models.
- Generalized Linear Multi-task Learning.
- Generalized Linear Regression Model.
- Gradient Boosting Tree.
- Random Forest.
- Recommender (implicit and explicit feedback).
- Regularized Matric Factorization (RMF).
- Support Vector Data Description.
- Support Vector Machine.
- Term Frequency-Inverse Document Frequency (TFIDF).
- Flexible, modular, window-driven architecture to define complex continuous queries:
- Based on an extensive suite of interchangeable window types and operators to detect an unlimited number of patterns, correlations, computations and aggregations.
- Prebuilt, common data quality routines are used to cleanse, standardize and filter livestream data before it’s stored, reducing downstream processing.
- Patterns of interest can include nearly unlimited advanced analytics calculations with in-stream, machine learning k-means clusters and livestream analytical scoring.
- Event stream windows to transform event state and data, and manipulate inbound streaming images:
- Combine image resizing, rotating, cropping, and flipping operations with powerful computer vision algorithms for object detection and classification use cases.
- The geofence window type allows you to track the location of objects relative to borders of a geofence. Alert when an object approaches, enters or leaves the defined geofence boundaries – and track entities within the geofence boundaries.
- Transpose: Enables you to interchange an event’s rows as columns, or columns as rows.
- Remove state: Facilitates the transition of a stateful part of a model to a stateless part of a model.
- Multiple object tracker (MOT): Enables you to perform multiobject tracking (MOT) in real time.
- Train: Model training on historical data (for accurate model development) complements high-performance analytics for at-rest data.
- Define and customize notifications by SMS, email and other alerts as part of event stream model workflow.
- Lua Window: Allows for the execution of Lua functions to enable high-speed mathematical operations, data transformation and XML/JSON string management.
- StateDB Windows: The StateDB windows enable you to use an externally deployed, high-performance database to store and maintain the state that is required for joins and aggregations.
- Analytical model support includes:
- SAS ASTORE.
- DATAStep2.
- DATAStep.
- ONNX format from SAS or third-party frameworks (e.g., TensorFlow, PyTorch).
- Python code.
- C code.
- Embeddable on gateways, edge devices, compute sticks and any existing C++ application (with dedicated thread-pool processing).
Low-code design time environment
- Low-code graphical development environment or the data scientist-friendly Jupyter Lab Python development environment are available, providing open and intuitive options for designing, testing, versioning and publishing ESP streaming projects.
- Analytical models are easily integrated using SAS Model Manager. Model management provides faster, automated integration and monitoring of analytical models.
- Browse the Model Manager model repository to easily locate and integrate advanced analytics to embed in SAS Event Stream Processing projects.
- Filter and Search large sets of analytical models quickly for rapid access.
- Authentication with SAS Logon for simplified and unified user access.
- Uses native Viya Postgres as managed, persistent data store for ESP project storage.
- Multitenancy-enabled using new Kubernetes operator framework and Docker containers delivered for ESP server in the cloud.
- Enhanced test mode, allowing the user to focus on the data that matters the most.
- Automated ESP Server pod creation and tear down for optimal resource utilization.
- Updated UI for faster response and custom output window selection.
- Access to ESP server log for simplified debugging.
- Run projects outside test mode to conduct long-running tests.
- Usability and user experience improvements:
- View and update Kubernetes ESP Server properties from ESP Studio UI.
- Application layout updated to simplify testing and design.
- Integrated, inline, type-ahead expression validation within Compute, Join, Filter and Aggregation window for immediate expression validation.
- Configurable memory, instance and CPU settings for ESP cloud server test mode.
- Diagram and layout improvements for more intuitive views.
- New test server management page for simplified server definition.
- Improved output schema panel with window selection for customized test result layout.
- Enhanced validation of ESP project syntax and properties.
- Easy-to-Use Expression Editor
- User-friendly editor for writing and validating expressions.
- Type-ahead autocompletion for both functions and schema information.
- All ESP-supported functions and operators available.
- Integration with SAS Event Stream Manager.
- Supports project version control including major and minor version selection.
- Minor version updates in SAS Event Stream Manager pushed back to SAS Event Stream Processing Studio.
- Published project surfaced in SAS Event Stream Manager automatically.
ESP operations & monitoring
- Construct and manage repeatable deployment plans with an easy-to-use interface, for projects executing on SAS Event Stream Processing servers at the edge or in the cloud.
- Quickly create deployments to monitor collections of servers and to simplify management.
- Identify deployment errors and retry operations only on servers that need attention.
- Create filtered lists of SAS Event Stream Processing servers to apply deployment operations.
- Create repeatable deployment scripts for rapid automation and user prompts, simplifying SAS Event Stream Processing project operationalization.
- Monitor events consumed with metering server displays to identify event usage patterns per licenses and host.
- Easily add new SAS Event Stream Processing servers for improved monitoring on edge deployments.
- Monitoring of ESP cluster servers supports clustered, multitenant environments in the cloud.
- ESP server monitoring.
- Dynamic ESP cloud server management and configuration to support clustered Kubernetes pods and containers.
- Automatically manages creation and tear down of ESP servers through the ESP Kubernetes operator framework.
- Monitoring includes ESP heartbeat monitoring, ESP server status reporting and ESP server performance stats.
- Prometheus integration for SAS ESP Server monitoring.
- View historical resource usage within a cluster over time.
- Identify resource problems early by exploring pod statistics, all provided in a clear, easy-to-follow UI.
- View log information for both actively running and previously running ESP pods.
- Usability and user experience improvements.
- Search and filter capabilities for quickly deploying SAS Event Stream Processing projects to the cloud.
- More visual indicators of issues when executing projects in production.
- More responsive UI for faster updates.
- Improved deployment management with alerts from SAS Model Manager for new champion models to enhance collaboration with data scientist teams.
- Enhanced server config reporting to highlight cluster memory and CPU utilization.
- Deployment controls for streamlined operations and rapid deployment.
- Load and start projects without templates with a single click from UI.
- Stop and Unload projects without templates with a single click from UI.
- Integration with SAS Event Stream Processing Studio.
- Published project surfaced in SAS Event Stream Manager automatically with shared file locations.
Expanded deployments & open source support
- Deploy SAS Event Stream Processing at the edge for IoT applications:
- SAS Event Stream Processing for Edge Computing provides a smaller, configurable disk footprint for simplified deployment to smaller edge devices.
- Support for Docker container deployment for SAS Event Stream Processing for Edge Computing for standardized deployment.
- SAS Event Stream Processing Python development interface:
- Speed development time with a familiar, open and flexible Python interface for developing, publishing, testing and streaming events through SAS Event Stream Processing projects.
- Python publish/subscribe API – Publish events and subscribe to SAS Event Stream Processing using Python.
Use SAS & open source languages
- SAS Event Stream Processing for SAS Viya® and CAS – Deploy SAS Data Mining and Machine Learning models to SAS Event Stream Processing for streaming analytics.
- Support for Python execution in stream within SAS Event Stream Processing projects. Native in-memory execution of Python using patented microanalytic service technology.
- Support for native ONNX model inferencing on GPU platforms for edge (on-site) and for cloud (Docker).
- SAS programming languages supported include SAS DATAStep2 and SAS DATAStep.
- MapR streams support – The Kafka adapter is certified to work with the MapR converged data platform for publishing and subscribing.
- Support for Python publish/subscribe API to publish and subscribe to SAS Event Stream Processing server to inject events to SAS Event Stream Processing and listen to insights from SAS Event Stream Processing.
Visual monitoring of event streams
- Support for visualizing streaming data and SAS Event Stream Processing project insights includes:
- Secure access with log-on support for authorized access to SAS Event Stream Processing Streamviewer application.
- Streamviewer support for SAS graphs in real-time streaming dashboards.
- Streamviewer components support embedding within applications to deliver Streamviewer real-time insights.
- WebSocket support for reliable and fast SAS Event Stream Processing server connectivity.
- User configurable dashboard for customized testing of streaming activity.
- Interactively filter and query livestream activity to examine specific behavior of elements.
- Faster response with new web socket support to monitor events from SAS Event Stream Processing server.
- Compare historical activity with current processing using graphical representations.
- Monitor stream processing detail by subscribing to events of interest.