Over the past months we have been working hard on WDP (Wizzie Data Platform) to get it to the next level. These are the most important highlights you can find in this release:

Organization based authentication & authorization system

Now, Wizz-In and Wizz-Vis work together to create the user authentication and data authorization system based on what we call “organizations”. This new feature allows a user to only view and work with dashboards and data sources that belong to a specific organization.

Smart microservices distribution

WDP “Dance” has become smarter than its predecessor, WDP “Country”. Now, when the WDP is distributed across a Kubernetes cluster it uses its pod priority, affinity, and anti-affinity logic. Using this new logic, WDP can optimize the resource usage and improve the deployment stability.

Visualization interface

We have improved our visualization interface adding some analytics features. With “Dance”, Wizz-Vis receives two important analytics features: Global Time Interval & Dimension Filters.

This allows applying global filters to a dashboard and provides the capability to move the time window to a specific moment, or to use filters on all the widgets at the same time. The dimension filter also provides an automatic discovery for dimensions and values. You can find more info regarding the time selector here.

For this release, we have also developed two Druid aggregators that we have contributed to the community, called stringFirst and stringLast. They are available on the Druid 0.13.0 release. Additionally, now some widgets supporting these new aggregators, will improve their functionality, performance and accuracy.

Wizzie Workflow

This release adds the Workflow Package Management feature. This feature allows to export, import and remove all the workflow configuration for a specific organization. You can compress and save the full use-case inside a sigle file, and load complete use-cases for different organization using a simple request.

The Normalizer engine has also received new functions to operate on the streams and the Enricher now supports global tables joiners. You can build global table states from the streams and enrich the data streams without having to repartition the full stream. Want to know more about the Global Tables? Look here. And, for the Normalizer Mapping Functions, have a look here.

Static Datasets Support

The Batch Data Loader has arrived! Now, Batuta (which is our Configuration Control Manager) has a new endpoint that allows loading a batch static dataset from external sources like Hadoop HDFS, Amazon S3, Google Storage, Azure Storage and others. This new feature provides the capability to load and work with raw historical data using the power of WDP.

CLI editor mode

“Dance” also helps WDP administrators to speed up their work. Now the WDP-CLI provides a new command, called edit, that allows configuring the platform and user variables with your favorite editor.

Aerohive RTLS support

Dance loves moving things and now comes with the new Aerohive RTLS connector. WDP can collect, transform, enrich and index the Aerohive stream data. This new connector, which comes with capabilities for our own Mobility Engine allows making awesome location intelligence use cases.

https://wizzie-io.github.io/prozzie/protocols/aerohive

Changelog

Features

• Support GlobalKTable at enricher engine
• [Wizz-Vis] User invitation process for a organization in wizz-in
• Global interval per dashboard.
• WDP smart microservices distribution
• Caching system on GUI
• Implement druid mobility support
• Authentication system on Wizz-Vis
• Text widget
• Threshold lines for graphs
• Aerohive support
• Enricher capability to repartition output streams before send to Kafka
• Normalizer map flatten function
• WDP integration in AMI
• Config vars management on Prelude
• WDP code protection in AMI
• Heatmap legend
• Dashboard search bar
• [Wizz-In] Organizations and API Tokens
• [Normalizer] Support UpperCase and LowerCase String function
• [Terra-box] Update Terra-box to support terraform-provider version 1.2.2
• [Wizz-Vis] Default granularity
• [Wizz-Vis] Dashboard visualization and management based on organizations
• [Wizz-Vis] Global Dimension Filter
• [Terrabox] Add support for enricher global topics
• Terraform plan support on WDP-CLI
• [Batuta] Allow to load batch data indo Druid
• [Enricher] Create BaseEnrich to enrich with static data
• [Batuta] Allow to control druid supervisor management properties
• [Wizz-Vis] Back Link on WidgetImage
• [Normalizer] DecimalPrecision Mapper Function
• WidgetPlane customization

Improvements

• Control Kong memory management
• [Terra-box] Automate wizzin and wizzvis URL configuration in terra-box
• [WDP-CLI] Return error codes when execution fails.
• [WDP-CLI] Automatize Init command execution when performing changes on deploy configuration
• [Prelude] Avoid user-component plugin deletion when deleting plugin from global. All user-component plugins must be deleted first.
• [Prelude] Interruption management to ensure terraform state is saved
• [Wizz-Vis] Possibility of hiding the legend at WidgetHeatmap and WidgetPlane
• [Wizz-Vis] WidgetValue resize text value inside widget size
• [Enricher] Review docs
• [OTX-Reputation] Migrate to Circle-CI 2.0
• Support __MATCH filter on FormatterFlatMapper function
• [Normalizer] Allow to support insert multiple input topics inside the same stream
• [Batuta] Check that kafka topics exist when send new stream-plan
• [Enricher] Update MaxMind GeoIP Service to GeoIP2
• [Batuta] Control overlapping dimensions and metrics
• [WDP-CLI] Improve wdp-cli lib to avoid duplicate code
• [Wizz-Vis] Improve the text on the navbar to make it more clear
• [Wizz-Vis] Update the dashboard associated to a widget through the API
• [Wizz-Vis][Documentation] Improve wizz-vis documentation
• [Wizz-Vis] Homogenize the attribute metrics inside options.
• [Wizz-In] Limit the access to /oauth/applications only to administrator users
• [Wizz-Vis Enterprise] Limit the access to sidekiq only to administrator users
• [Mobility Engine] Add consolidated dimension to global stream
• [Normalizer] Add filter sink validation to check if the filter is valid before try to build
• [Normalizer] Review Try-Loop exception cases
• [Prelude] Improve terra-box flavours configuration for WDP deploys via prelude
• [WDP-CLI] Allow branches and commits id as versions when using wdp deploy set-version command and plugins
• [AMI] SSH Keys management in AMI wdp command, to ensure that prelude can be used in all aws cluster nodes
• [Wizz-Vis] Avoid docker failure if database can’t be created but already exists
• Improve wizz-vis login interface design
• Automatic EKS authorization
• Modify module source parameter in terrabox to use local paths instead of git paths
• Include PropTypes and defaultProps properties at React components.
• Use redux on Wizz-Vis for managing states.
• [Wizz-Vis] Control datasource’s authorization at Wizz-Vis API
• [Wizz-Vis] Dynamic time ranges
• [Normalizer] Improve geofencing function
• [Prozzie] Configure kafka logs rotation
• [Wizz-Vis] Extend the color palette used on graphs.
• [Wizz-Vis] Prevent WidgetImage and WidgetText to execute a Druid query.
• [Wizz-Vis] round on hyperunique metrics.
• [Wizz-Vis] Reduce webpack bundle size for production

Bugs

• Prozzie monitor environment variables do not match with container ones
• [WDP-CLI] Unmanaged exception when executing wdp config deploy current with an existing but empty cli.yml
• [Wizz-Vis] Global Time selector doesn’t allow to write date with Safari
• [Normalizer] JoinMapper use delimitier property instead of delimiter
• [Wizz-Vis] Prevent all widgets crash when one widget doesn’t work on the same dashboard
• [Wizz-In] Error creating an organization without selecting an owner.
• [Wizz-Vis] Change the PRIMARY_COLOR not change the global time selector calendar
• [Wizz-Vis] WidgetTable doesn’t resize the text inside its column
• [Wizz-Vis] Error on WidgetMultiserie when there are only points in a single timestamp moment.
• Fix data repo dependency vulnerabilities
• [Mobility Engine] Mobility send messages with timestamp older than start execution time
• [Mobility Engine] Mobility crash when report.time.seconds property is not defined
• [Prelude] Prelude doesn’t create some component folders
• [Prelude] Plugin removing feature isn’t working correctly
• [Prelude] Prelude plugin config vars doesn’t work
• [WDP-CLI] wdp wrapper script can’t recognize commit message as multiword message, interpreting each word as a different option
• [Prelude] wdp init command removes static and flavour vars
• Batuta remove consumer.id from kafka topics (topic name inclusive)
• MQTT kafka connector stop when receives connection timeout
• Batuta doesn’t control druid redirects when druid (overlord and coordiantor) works on HA
• [Normalizer] maven artifacts have wrong names
• Wizz-In doesn’t work with kong 0.14
• Error displaying charts on Internet Explorer 11
• [Normalizer] ClassificationMapper throws null pointer when dimension value is null
• WidgetValue with compare enable, crash when receive -Infinity druid value
• Mobility wrong output messages on some corner cases.
• [Normalizer] The replaceMapper function doesn’t work with null values
• ZZ-CEP repo doesn’t work with mvn release:perform -P docker
• Heatmap color aggregation doesn’t work properly
• [Wizz-In] When making a Oauth login, redirect correctly to the client after a failed login.
• Bug on compare feature at WidgetTable
• Security issue on rubyzip gem
• Error assigning color to markers at WidgetPlaneLocation
• Error on Wizz-Vis login with OAuth process.
• [Wizz-Vis] Bug building docker container with plugins
• [Wizz-In] Error creating a new organization when there are one or more already created.
• Heatmap legend doesn’t adjust under widget resize (small size)
• Heatmaps plane represents heatmap layer out of widget
• [Wizz-Vis] Security vulnerability on nokogiri gem.
• [Wizz-In] Sendmail not installed on the docker image
• [Wizz-In] Error sending emails using sendmail
• [Wizz-Vis-Enterprise] Error running migrations
• [Wizz-Vis] Error at WidgetValue with compare activated
• Normalizer should close when crashes
• [Wizz-Vis] Widget’s endpoints disappear and are not shown in the API
• [Wizz-Vis & Wizz-In] Buttons are no clickable in the whole area. Only are clickable in the text.
• k2http does not start at prozzie linux_setup.sh end
• [Wizz-Vis] Gem vulnerabilities
• [ZZ-CEP] The zz-cep instance always use a 100% of CPU without stream-plan
• Change mobility general behaviour to avoid problematic corner cases
• [Wizz-Vis] Validate the value of col, row, size_x and size_y of a widget.
• [Wizz-Vis] DrillDowns are not clickable
• [Wizz-Vis] Javascript error on WidgetLocation
• [Wizz-Vis] Fix error on main link at WidgetImage and render of markers at WidgetPlaneLocation
• [Wizz-Vis] loofah gem vulnerability