Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdf

© 2018 Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building Your Data Lake
on AWS
Luke Anderson
Business Development, AWS

What to expect from the session
1. Defining the Data Lake
2. Reducing Costs
3. Increasing Performance
4. Planning for the Future

Rethink how to become a data-driven business
• Business outcomes
• Experimentation
• Agile and timely

Traditionally, Analytics looked like this
(Duplication & Sprawl)
Hadoop
Spark
NoSQL
Storage
Arrays
Databases
Data
Warehouse
Structured Data
SQL
Raw Data
ETL
Advanced Analytics
ETL

Defining the AWS data lake
Data lake is an architecture with a virtually
limitless centralized storage platform capable
of categorization, processing, analysis, and
consumption of heterogeneous data sets
Key data lake attributes
• Decoupled storage and compute
• Rapid ingest and transformation
• Secure multi-tenancy
• Query in place
• Schema on read

AWS Data Lake Components
Any analytic workload, any scale, at the lowest possible cost
Insights
Analytics
Data Lake
Data Movement
QuickSight SageMaker
Glue
(ETL & Data Catalog)
S3/Glacier
(Storage)
Redshift
+Spectrum
EMR Athena
Elasticsearch service
Kinesis Data Analytics
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams
Real-time
Comprehend
DW Big data processing Interactive

Unmatched durability,
availability, and scalability
Best security, compliance, and audit
capability
Object-level control
at any scale
Business insight into
your data
Twice as many partner
integrations
Most ways to bring
data in
Reasons to choose Amazon S3 for data lake

Reducing Data Lake Costs

Optimize costs with data tiering
Hot
Cold
Amazon
S3 standard
Amazon S3—
infrequent access
Amazon
Glacier
HDFS  Use EMR/Hadoop with local
HDFS for hottest data sets
 Store cooler data in S3 and
cold in Glacier to reduce costs
 Use S3 Analytics to optimize
tiering strategy
S3 Analytics

Process data in place…
Amazon Athena Amazon Redshift
Spectrum
Amazon EMR
AWS Glue
Amazon S3

Amazon EMR: Decouple compute & storage
Highly distributed
processing frameworks such
as Hadoop/Spark
Compress datasets
Columnar file formats
Aggregate small files
S3distcp “group-by” clause

Amazon Redshift Spectrum: Exabyte Scale
query-in-place
Structured data w/ joins
Multiple on-demand
clusters-scale concurrency
Data partitioning
Better query performance
with predicate pushdown

Amazon Athena: Query without ETL
Serverless service
Schema on read
Compress datasets
Optimize file sizes
Optimize querying (Presto
backend)
Query Data in Glacier
(Coming)

Today: All of these tools…
retrieve a lot of data they don’t need and
do the heavy lifting

Today: You need to….
entire object from Amazon Glacier to Amazon S3
and then use it.
Amazon
S3
Amazon
Glacier

Select
Amazon S3 Select and Amazon Glacier Select
Select subset of data from an object based on a SQL expression

Motivation Behind S3 Select
GET all the data from S3 objects, and my application will filter the data that I need
Redshift Spectrum Example:
• Beta customer: Run 50,000 queries
• Amount of data fetched from S3: 6 PBs
• Amount of data used in Redshift: 650 TB
Data needed from S3: 10%

Amazon S3 Select
SELECT a filtered set of data from within an object using standard SQL Statements
• First content aware API within Amazon S3
• Unlike Amazon Athena and Spectrum, operates within the Amazon S3 system
• SQL Statement operates on a per-object basis—not across a group of objects
• Works and scales like GET requests
• Accessible via SDK (Java, Python), AWS CLI and Presto Connector—others to follow
• Who will use it?
• Amazon Redshift Spectrum, Amazon Athena, Presto and other custom Query engines
• Everyone doing log mining

Amazon S3 Select
Output
Format: delimited text (CSV,
TSV), JSON …
Clauses Data types Operators Functions
Select String Conditional String
From Integer, Float, Decimal Math Cast
Where Timestamp Logical Math
Boolean String (Like, ||) Aggregate
Input
Format: delimited text (CSV,
TSV), JSON …
Compression: GZIP …

Amazon S3 Select: Simple pattern matches
…get-object …object… | awk -F ’{ if($4=="x") print $1}’
...select-object …object… ‘SELECT o._1 WHERE o._4 == “x”…’

Amazon S3 Select: Serverless applications
Amazon
S3
AWS
Lambda
Amazon
SNS
S3
Select
Lambda
Trigger

Before
200 seconds and 11.2 cents
# Download and process all keys
for key in src_keys:
response = s3_client.get_object(Bucket=src_bucket, Key=key)
contents = response['Body'].read()
for line in contents.split('n')[:-1]:
line_count +=1
try:
data = line.split(',')
srcIp = data[0][:8]
….
Amazon S3 Select: Serverless MapReduce
After
95 seconds and costs 2.8 cents
# Select IP Address and Keys
for key in src_keys:
response = s3_client.select_object_content
(Bucket=src_bucket, Key=key, expression =
SELECT SUBSTR(obj._1, 1, 8), obj._2 FROM s3object as obj)
contents = response['Body'].read()
for line in contents:
line_count +=1
try:
….
2X Faster at 1/5 of the cost

Demo – S3 Select Timing

Amazon S3 Select with Presto
Works with your existing Hive Metastore
Automatically converts predicates into S3 Select requests
Amazon S3
S3 Select

Before
Amazon S3 Select: Accelerating big data
After
After
5X Faster with 1/40 of the CPU

Using Amazon Glacier Select

How Amazon Glacier Select Works

Delivering Results Faster

Optimizing data lake performance
Aggregate small files
EMR: S3distcp
Amazon Kinesis Firehose
S3 Select
Big data cheaper, faster
Up to 400% faster
Data Formats
Columnar formats
EMRFS consistent view
Amazon
S3
Amazon
DynamoDB

Amazon Kinesis—Real Time
Easily collect, process, and analyze video and data streams in real time
Capture, process,
and store video
streams for analytics
Load data streams
into AWS data stores
Analyze data streams
with SQL
Build custom
applications that
analyze data streams
Kinesis Video Streams Kinesis Data Streams Kinesis Data Firehose Kinesis Data Analytics
SQL
New

Data preparation accounts for ~80% of the
work
Building training sets
Cleaning and organizing data
Collecting data sets
Mining data for patterns
Refining algorithms
Other

AWS Glue—Serverless Data catalog & ETL
service
Data Catalog
ETL Job
authoring
Discover data and
extract schema
Auto-generates
customizable ETL code
in Python and Spark
Automatically discovers data and stores schema
Data searchable, and available for ETL
Generates customizable code
Schedules and runs your ETL jobs
Serverless

Amazon SageMaker (GA)
The quickest and easiest way to get ML models from idea to production
End-to-End
Machine Learning
Platform
Zero setup Flexible Model
Training
Pay by the second
$

Planning for the Future

Transactional Data
Stream Data
Collect Store Analyze Visualize
A
iOS Android
Web Apps
Logstash
Amazon
RDS
Amazon
DynamoDB
Amazon
ES
Amazon
S3
Apache
Kafka
Amazon
Glacier
Amazon
Kinesis
Amazon
DynamoDB
Amazon
Redshift
Impala
Pig
Amazon ML
Streaming
Amazon
Kinesis
AWS
Lambda
Amazon
Elastic
MapReduce
Amazon
ElastiCache
Search
SQL
NoSQL
Cache
Stream
Processing
Batch
Interactive
Logging
Stream
Storage
IoT
Applications
File
Storage
Analysis
&
Visualization
Hot
Cold
Warm
Hot
Slow
Hot
ML
Fast
Fast
Amazon
QuickSight
File Data
Notebooks
Predictions
Apps & APIs
Mobile
Apps
IDE
Search Data
ETL
Evolve As Needed!

AWS Training Offer
Make your data driven decisions count, and make a career in Big
Data on AWS. Follow the Big Data Specialty learning path and
become a specialist in Big Data:
• Implement core AWS Big Data services according to best
practices
• Design and maintain Big Data
• Leverage tools to automate data analysis
Certified Cloud
Practitioner
Associate-level Certification
AWS Certified Big Data - Specialty
• Enterprise solutions
architects
• Data scientists
• Big Data solutions
architects
• Data analysts
Who should attend
Free AWS digital training: Foundational
knowledge
Big Data on AWS – 3-day Classroom Training
Free AWS digital training:
Big Data Technology Fundamentals
Visit www.aws.training to find out more.

We hope you found it interesting! A kind reminder to complete the survey.
Let us know what you thought of today’s event and how we can improve
the event experience for you in the future.
Thank You For Attending
AWS Data Driven Decisions Webinar Series.
aws-apac-marketing@amazon.com
twitter.com/AWSCloud
facebook.com/AmazonWebServices
youtube.com/user/AmazonWebServices
slideshare.net/AmazonWebServices
twitch.tv/aws

Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdf

Recommended

Recommended

More Related Content

Similar to Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdf

Similar to Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdf (20)

Recently uploaded

Recently uploaded (20)

Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdf