Understand why Spark has experienced such wide adoption and learn about some Spark use cases today. Take a technical, deep dive into the architecture, and the vision for the Hadoop ecosystem and why Spark is the successor to MapReduce for Hadoop data processing.
Just last week Cloudera released some impressive performance numbers showing how the Impala SQL-on-Hadoop engine scales to support concurrent query workloads. The Cloudera blog post confirms what we at AtScale have experienced with real-world customer installations – that Impala plus AtScale is a scalable solution for running concurrent, interactive business intelligence (BI) queries on Hadoop.
Our team is headed to the Javits Convention Center next week for the Strata + Hadoop World Conference. We are excited to meet all you to discuss the future of BI, particularly BI on Hadoop.
This past week was pretty packed in news and surveys highlighting the developments occurring in our industry.
October is an incredibly busy month for Data professionals. First, the Strata O'Reilly Conference takes us East to mingle with thousands of data nerds and debate on the future of Hadoop. Then, the Tableau Conference takes us back West to challenge Directors of Analytics and Business Intelligence to connect their tools to the "Data Lake".
In just a few days, Las Vegas will be invaded by thousands of "BI heroes". If you've been following the Data news, you know that the upcoming Tableau Conference is about to take over the MGM hotel next week.
This morning, Gartner released their famous "Magic Quadrant for Operational Database Management Systems". You can see more about it directly on the Gartner site here
If your team has been trying to connect your Business Intelligence (BI) tools to your Hadoop environment, you are familiar with the typical issues: performance, security and the inability to model the data in a way that business users like to consume it.
Years ago, when our CEO's team processed over 30 billion events per day, managed a 200 Hadoop node cluster and tried to build a system that connected Hadoop to 8,000 Business Intelligence users, very few people knew what Hadoop was.
Nowadays, it's virtually impossible to be in a meeting without Hadoop being mentioned. Luckily, our CMO was recently interviewed about Hadoop and provided a concise answer to this question. Watch the below video to find out. It lasts less than 2 mins.
Every once in a while, our team gets questions about the validity of Hadoop. Why it exists, why people should consider using it...etc. In the below video, I provide a few examples of use-cases across common industries like Financial Services or Retail that provide a clear answer to this question.
As usual, the video lasts less than 2 minutes so don't expect to have sit through long, hesoteric explanations. We get to the point and very quickly at AtScale!
One of the "buzziest" subjects of conversation in the Big Data scene this past year has been Spark. The powerful open source processing engine developed in 2009 has gotten some great traction and many have covered the stories of community adoption and growth.
As one would expect though, some of the buzz spun out of control, so much, some went on to write that "Hadoop was dead" or that "Spark would eventually replace Hadoop".
Such reports are misinformed, overly sensational and inaccurate. If you have a few minutes, I propose you watch the below 2 minute-video on the key role that Spark plays and its relationship with Hadoop. In this interview, we talk about security, speed and other key items that make Hadoop relevant to business users.
For a deeper understanding on the topic, check out the great piece that our VP of Product Management, Josh Klahr, authored for ReadWrite last week. My favorite passage is below (the full piece is here):
"We need to stop playing Spark and Hadoop off each other and understand how they will coexist. Hadoop will continue to be used as a platform for scale-out data storage, parallel processing, and clustered workload management. Spark will continue to be used for both batch-oriented and interactive scale-out data-processing needs."
One of my favorite business quotes is the one featured in this blog's photo: "The future is here. It's just not evenly distributed yet" (William Gibson).
When I look back at 2015 and the amazing accomplishments of our team and our customers, I can't help to think that, we are building the future here. And while many got access to it in 2015 already, many more will be able to benefit from it very soon.
We've had a great time serving you and the community in 2015:
We can't wait to meet you all at next year's Big Data events such as the Strata Conferences, Gartner BI as well as many of our partners' events and roadshows! If you ever have a question about our direction, please email me directly at bruno@atscale.com.
As you consider AtScale, understand what we ARE, and what we are NOT. We are NOT another "Tableau 2.0" company. We are NOT a replacement for your entire BI stack: we are the "Business Interface for Hadoop".
Happy Holidays to all and...Analytically Yours!
2016 is bound to bring a wake-up call to tech industry observers. Many will find out that their favorite “unicorns” were actually “donkeys”, and that forgetting to focus on revenue is a capital sin in the enterprise software world.
We're opening 2016 with great news for anyone who's trying to make BI work on Hadoop! We’ve partnered with Tableau to bring to market an offer that, we believe, will eradicate the issues customers run into when connecting Tableau to Hadoop directly.
This week, Forrester released great insights about the state of the Big Data and Hadoop market. The report points to the maturity of the vendors involved in the space. The research study also acts as yet another data point for what we have been calling the "no-so-silent data revolution".... You can see more about it directly on the Forrester site here
Spark Summit 2016 kicks off next week in NYC and thousands are expected to attend the event, whose theme this year is "Data Science and Engineering At Scale". Great companies will be presenting - from Comcast to Thales and Viacom...
Hadoop Summit 2016 may not be until be June 28 – 30 in San Jose, but already the community is abuzz. What’s in it for you? With Hortonworks, Yahoo!, and MAPR sponsoring and leading companies like Trulia, Macy’s and PayPal speaking ; there's a ton of new insights just waiting for you. If you're the type who needs something more concrete; here are The Big 3 of Hadoop Summit 2016 to get you going!
Trystan here, Software Engineer and doer of all things technical at AtScale. Which SQL-on-Hadoop engine performs best? We get this question all the time!
We looked around and found that no one had done a complete and impartial benchmark test of real-life workloads across multiple SQL-on-Hadoop engines (Impala, Spark, Hive...etc).
So, we decided to put our enterprise experience to work and deliver the world's first BI-on-Hadoop performance benchmark.
What did we find out? Well, turns out that the right question to ask is: "Which engine performs best for Which query type?". We looked across three of the most common types of BI queries and found that each engine had a particular niche. Bottom line: One Engine does NOT fit all.
Read on to find out the details of our environment and configuration, the types of queries we tested... (or download the full whitepaper here)
