The Project
ActioNet designed a Big Data Platform (BDP) solution that combines the features and capabilities of several big data applications and utilities within a single solution. This enables organization in developing, deploying, operating and managing a big data environment. The platform also supports continuous monitoring, incident response and enterprise risk management.
The Solution
We met this integration challenge by finding the fine line between emerging and stable technologies, and integrating 40+ analytics tools, dashboards, and virtualizations. For example, the Cyber Awareness Dashboard shows a snapshot of top threats and provides vulnerability scores and actions that will alter risk scores. The environment is secured through encrypted data transport capabilities using Direct Connect and AWS Snowball for mass data ingest.
Our solution had to overcome the cybersecurity challenges facing the agency:
- Volume: The size of data generated by sensors that need to be ingested and the ability for the platform to support the ingest volume (i.e. TB, PB).
- Velocity: The ability to ingest data at the speed that data is flowing from the networks of interest (events per second).
- Variety: The ability to support a wide variety of data types and sensors and the ability to normalize those to a single notional taxonomy (structured and unstructured).
- Veracity: The ability to rapidly find the validity or truth of the data using analytics developed on the platform (what is true vs. what is believed).
The Result
We transformed data fusion workflows to dramatically enhance analytics and position DOE to collect a wide array of cyber metadata enabling Continuous Diagnostics and Mitigation.
- Scalable: Operationally deployed on a 4PB infrastructure, the BDP scales to meet customer demands and has sustained ingestion rates of 4.5Gbps.
- Rapid: BDP provides over 1,200 direct participants from Government centers, labs, FFRDCs, and UARCS, including US Cyber Command (USCYBERCOM), DISA, NSA and Joint Forces, providing automated deployment in the AWS GovCloud in hours, cutting previous times by months.
- Granular: Control through Attribute Based Access Controls (ABAC) for cell-level data security.
- End to End: Data lifecycle support from data ingestion with Storm, Storage and common schema through Hadoop, Analysis and Queries through R, and Dashboards and visualizations through Shiny and Kibana to support Cyber Awareness Dashboards. BDP effectively leverages the right tool for the right job.
