Scribd Logo
Încărcați

Bine ați venit la Scribd!

  • Waas

    Încărcat de

    ramanavg
    29 vizualizări9 pagini
    The document discusses testing challenges for Greenplum Database, an open-source massively parallel processing (MPP) database management system. It notes specific challenges including distributed transactions, parallel query processing, and testing fault tolerance through large test matrices and diverse fault scenarios. The document argues that as distributed programming becomes more common, the testing challenges faced by systems like Greenplum will become ubiquitous, requiring new distributed testing methodologies integrated within systems.

    Descriere originală:

    Greenplum

    Drepturi de autor

    © Attribution Non-Commercial (BY-NC)

    Formate disponibile

    PDF, TXT sau citiți online pe Scribd

    Vi se pare util acest document?

    Este necorespunzător acest conținut?

    Raportați acest document
    The document discusses testing challenges for Greenplum Database, an open-source massively parallel processing (MPP) database management system. It notes specific challenges including distributed transactions, parallel query processing, and testing fault tolerance through large test matrices and diverse fault scenarios. The document argues that as distributed programming becomes more common, the testing challenges faced by systems like Greenplum will become ubiquitous, requiring new distributed testing methodologies integrated within systems.

    Drepturi de autor:

    Attribution Non-Commercial (BY-NC)
    Descărcați ca PDF, TXT sau citiți online pe Scribd
    Indicator pentru conținut neadecvat
    29 vizualizări9 pagini

    Waas

    Încărcat de

    ramanavg
    The document discusses testing challenges for Greenplum Database, an open-source massively parallel processing (MPP) database management system. It notes specific challenges including distributed transactions, parallel query processing, and testing fault tolerance through large test matrices and diverse fault scenarios. The document argues that as distributed programming becomes more common, the testing challenges faced by systems like Greenplum will become ubiquitous, requiring new distributed testing methodologies integrated within systems.

    Drepturi de autor:

    Attribution Non-Commercial (BY-NC)
    Descărcați ca PDF, TXT sau citiți online pe Scribd
    Indicator pentru conținut neadecvat
    din 9

    Florian Waas June 7, 2010

    06/23/10 Confidential

    Greenplum Database

    Full-featured database system


    Emphasis on support for OLAP

    Software-only solution
    Not an appliance No proprietary hardware

    Shared-nothing MPP architecture


    High-end data analytics market
    06/23/10 Confidential 2

    Greenplum Database Architecture


    MPP (Massively Parallel Processing) Shared-Nothing Architecture
    Master Severs
    Query planning & dispatch

    SQL MapReduce

    ...

    ...

    Network Interconnect

    Segment Severs
    Query processing & data storage

    ...

    ...

    External Sources
    Loading, streaming, etc.

    06/23/10 Confidential

    Customer Profiles 100+ global enterprise customers 10s of TB to multi-PB 10s of tables to 15,000+ 10s of concurrent users to 150+ Integration w/ existing data analysis ecosystem, e.g., BI tools, ETL/ELT infrastructure Compute intensive, storage intensive Mixed workloads Urban myth: pure DW workloads
    06/23/10 Confidential 4

    Specific Challenges

    Parallel processing
    Distributed transactions Distributed DDL Parallel query processing Multi-core aware query optimization

    Data management
    In-cluster replication DAS = unmanaged storage TBs per box
    ~10PB today, ~100PB tomorrow

    These are general trends for all DBMSs in the future


    06/23/10 Confidential 5

    Test Challenges: General


    Functional tests, e.g., correctness of queries
    Same/similar to conventional DBMS

    Stress tests
    Similar; more components

    Performance
    Same basic principles; much more data to capture

    In-cluster replication
    Special hooks into product

    Scale testing
    True at-scale testing prohibitively expensive

    Fault-tolerance
    Elaborate external harnesses
    06/23/10 Confidential 6

    Test challenges: Fault-tolerance Significantly larger test matrix Need to capture detailed system state
    Observing distributed systems cannot rely on time stamps

    Fault scenarios
    Network bisection Node failures Gradually degrading hardware

    Test strategies
    Random fault injection Network/drive failure simulations Explicit K-safety scenarios
    7

    06/23/10 Confidential

    Up-ing the challenge


    Dont other people have the same problem? Database Systems are true enterprise software
    Not single instances like Google, Yahoo!, Amazon.com, Facebook, etc. No test by controlled-flooding No developers to maintain it

    Higher standards of quality Transactional data management Test methods highly specialized System specific Programming language specific
    06/23/10 Confidential 8

    The Road Ahead Distributed programming will become pervasive Above test challenges will be ubiquitous Cannot afford to implement these as black-box tests outside of system
    Re-implements highly complex logic Too costly to maintain

    Need textbook of distributed test methodologies


    Checklists for fault-scenarios Integration with existing methods Both theory and tools

    Lots of exciting work ahead!


    06/23/10 Confidential 9

    S-ar putea să vă placă și