Integrate Twin Talk with Snowflake: Unlocking Real-Time Data Insights
Introduction
In today’s data-driven world, the ability to access and analyze real-time data is crucial for organizations seeking to gain a competitive edge. Twin Talk, developed by the Embassy of Things (EOT), is an event-driven, real-time data sharing engine that empowers businesses to harness the full potential of their industrial data. By seamlessly connecting operational sources to cloud-based analytics systems, Twin Talk facilitates digital transformation, enabling real-time actionable insights, and improving operational efficiency. In this article, we will explore how Twin Talk can be integrated with Snowflake, a cloud-based data warehousing platform, to unlock the power of real-time data analytics.
Understanding Twin Talk
Twin Talk serves as a Cloud Access Security Broker (CASB) for industrial IoT, bridging the gap between operational systems (OT) and cloud solutions (IT). It removes the barriers associated with establishing complex, firewalled networks, making it easier and more cost-effective to transmit sensor data from operational assets to the cloud. Twin Talk accelerates digital transformation by seamlessly integrating AI, ML, analytics applications, event hubs, and data lakes with OPC/SCADA servers using EOT’s real-time data streaming platform, TwinTalk.
The Challenge of Industrial Data
Many industrial companies are eager to leverage artificial intelligence (AI), machine learning (ML), and data science to reduce costs and increase productivity. However, a significant challenge they face is accessing and utilizing the vast amount of sensor data generated by their operational assets. This data is often locked within closed industrial networks, making it difficult to deliver predictive analytics, intelligent solutions, and real-time actionable insights.
How Twin Talk Works
Twin Talk offers two strategies to address this challenge:
Strategy 1: Real-Time Data Integration
For organizations coming from a “batch” world where data is moved to a data lake in the cloud, the process of moving data and running batch queries can take hours or even days. Twin Talk simplifies this process by connecting directly to operational data systems in real-time, allowing AI, ML, and analytics applications to access data instantly.
Strategy 2: Real-Time Streaming Architecture
Twin Talk’s Real-Time Data Platform is designed for securely connecting to OT environments. It supports real-time streaming, enabling the continuous flow of data to AI, ML, and analytics applications while keeping batch environments up to date simultaneously. This approach offers advantages such as simplified integration, real-time speed, high throughput, and low latency.
Under the Hood of EOT’s Twin Talk
EOT’s Twin Talk Real-Time Data Streaming Gateway architecture is a scalable solution for connecting to PI systems, legacy historians, and other data sources. It ensures the efficient and real-time exchange of data with parallelism, making it easy to integrate AI, ML, analytics applications, event hubs, and platforms, whether on-premise or in the cloud.
Twin Talk Modes:
1) Publisher-Subscriber for real-time monitoring and updates
2) Request-Response for historical backfill requests
Twin Talk’s architecture allows external applications to send requests for data, which are processed in parallel, achieving maximum throughput and low latency. This approach ensures real-time access to data from various operational systems.
Realizing the Potential with Snowflake
Snowflake is a cloud-based data warehousing platform known for its scalability, efficient storage, and analytics capabilities. Its unique architecture separates compute and storage components, allowing users to scale these resources independently. This flexibility ensures exceptional performance and concurrency, making it an ideal choice for data-intensive operations.
Twin Talk RT Snowflake Direct SQL Insert
Twin Talk facilitates the integration with Snowflake by connecting to OSIsoft AF Data Archive Server, transforming the data, and storing it in Snowflake tables using Snowflake’s Direct API inserts. This process enables real-time analytics with Snowflake’s SQL-based interface.
Requirements for Direct Insert into Snowflake Tables:
- Snowflake Landing Table or Tables
- Snowflake Twin Talk Service Account
- Snowflake Twin Talk Role with SELECT and INSERT privileges for tables
To enable direct insertion of data into Snowflake tables, organizations must ensure that these requirements are met, including having the necessary tables and configuring service accounts with the right permissions.
Integrating Twin Talk with Snowflake
The integration of Twin Talk with Snowflake involves the transfer of data from industrial sources to Twin Talk, which then transforms and loads the data directly into Snowflake. This seamless connection allows businesses to harness real-time data analytics and derive actionable insights from their operational data.
Twin Talk’s real-time data sharing capabilities combined with Snowflake’s robust data warehousing platform offer organizations a powerful solution to unlock the potential of their operational data. By seamlessly integrating Twin Talk with Snowflake, businesses can gain real-time actionable insights, reduce costs, and improve operational efficiencies, ultimately driving digital transformation and increasing their competitive advantage in the data-driven landscape.
Twin Sight – visualize and navigate through operational, asset model and analytics insights
Twin Sight is an intuitive, simple visualization tool and dashboard builder that provides fast, easy, and secure access to operational, engineering, transactional real-time data and semantic data. You can visualize operational data and semantic metadata in conjunction with analytics, ML insights for anomaly detection, production optimization, and operation monitoring and optimization across all production sites.
Twin Central – intuitive industrial digital twin builder that provides semantic asset modeling, mapping and management from operational, engineering and transactional metadata sources.
Twin Central is a intuitive approach to create and manage an asset-centric, single source of truth and semantic data model across the enterprise. Twin Central can create digital twin data models that map, connect, link, store, and synchronize relationships between assets and their operational, engineering and financial metadata using a unified digital twin relationship graph.
Twin Talk – industrial data ingestion, curation, and contextualization platform
Twin Talk is a secure and scalable way to deliver operational data from SCADA systems and historians to the cloud. It enables cloud-based, event-driven, real-time architecture such as data lakes, mobile monitoring and surveillance apps and unlocks valuable insights from analytics, AI and machine learning. Twin Talk can be integrated with existing or new IT environments within weeks with minimal integration work and costs. It transforms a company into a proactive, event-driven enterprise.