Requirements for VLDB Support in Objectivity/DB

Requirements for VLDB Support in Objectivity/DB
[ draft ]

Jamie Shiers

CERN/IT/ASD

http://wwwinfo.cern.ch/asd/cernlib/rd45/index.html
VLDB Requirements

Basic Assumptions:

Require an ODBMS architecture that permits federations of at least 100PB without any arbitrary limitations

Changes that invalidate existing federations are not acceptable

this is true in any case, and not just applicable to the VLDB discussion

Changes that are required by HEP today will be needed by others in a few years

Scalability: LHC Data Volumes

Objectivity/DB Scalability

Theoretical limitations of Objy

On 64-bit systems:

Total bytes = 2¹⁶ db/fd x 2¹⁵ c/db x 2¹⁶ pg/c x 2¹⁶ bytes/pg

or 10¹⁹ bytes (10,000 PB!!!)

fd = Federation

db = Database

c = Container

pg = Page

Practical Limitations of Objy

In current version, each DB maps to a file

Although 64-bit filesystems permit files > or >> 2GB, impractical to go to very large files

e.g. requirement that file can be restored from MSS in 10² - 10³ secs

10MB/s transfer rate to/from tape gives 10³ seconds/10GB file

Effective maximum DB size of a 10GB today, perhaps as much as 100GB in 2005…

permits 6.5PB federations - 1-2 orders of magnitude too low!

Options:

permit more DBs / federation

use multiple federations

change logical - physical mapping

[other]

What Would We Like?

Maximum Size of a Federation

preferably no practical limit

A limit of 1EB would probably be flexible enough

(Assume 25-100PB per LHC experiment)

Maintaining current DB - file mapping would require up to 10⁸ DBs/FD (10⁸ x 10GB = 1EB)

16⁷ = 2²⁸ = 268, 435, 456

i.e. DBID would have to be 26-28 bits

storage/performance overhead?

Mapping containers - files & using 4GB containers would permit (2¹⁶ - 1) . (2¹⁵ - 1) . 2³² byte federations

would permit EB federations

catalog overhead?

# Slots / page

Given the maximum size of a DB page and the minimum size of an object, # slots / page clearly too high

Can perhaps save 3 bits here

[ page size implicitly connected to OID; increasing maximum page size will eventually require more SlotIDs to address all possible locations ]

DB Page Size

Current limitation is 2¹⁶ bytes

OK for current networks; perhaps abit low for Ultranet/HiPPi

Page size directly linked to storage overhead/cache efficiency - cannot use maximum page size now!

The possibility to have a variable page size (fixed per container/DB?) would be useful

storage efficiency

The possibility to perform a group I/O would also be useful

performance, e.g. group write for sequential write case

Cannot "recuperate" any bits by reducing maximum page size, but increasing would [eventually] affect SlotID

Container Size

Containers currently limited to 2¹⁵ - 1 (logical) DB pages

Gives a maximum container size of 0.5 - 4GB

except in the case of many large objects

This is too small for some purposes, but increasing the maximum container size would (presumably) also affect locking problems

we have implemented a logical container on top of the physical container to work around this problem

native support in Objy would be just great

No possibility to "recuperate" bits here...

# Containers / DB

Currently limited to 2¹⁵ - 1

BaBar plans to use ~ 10 containers / DB ! (?)

A limit of 2¹⁰ containers / DB would probably be ok (?)

However, this may be too low for some other (non-HEP) applications

OID Changes

Could perhaps save 3 bits in slotID, 5 in containerID

However, this would only give us a maximum of 2²⁴ DBs / federation

10GB databases x 2²⁴= 160PB

10GB = 2¹⁰ containers of 10MB each

Container - File Mapping

Permitting containers, rather than databases (or as well as?) to be mapped to files dramatically increases the maximum size of a federation

Using a container size of ~ 2GB, can theoretically create federations in the EB range

even larger in the case of many large objects/container

Allows more flexibility, particularly when coupled with MSS

container level clustering can map to a "storage class" in the MSS

individual containers can migrate independently

opens the way to container-level replication etc.

Can later be combined with e.g. larger DB page size, extended OID, OID bit-shuffling to permit Zetta-byte (10²¹ B) databases(!)

Conclusions

Can meet our requirements by:

by changing OID mapping but maintaining a 64bit OID

invalidates existing federations

likely to adversely affect performance

by mapping containers, rather than databases, to files

[ increasing the OID ]

Strongly preferred solution would be container - file mapping

consistent lock / migration / clustering strategy at container level

advantages in terms of Objy-HPSS integration

much more "future proof"

Need such changes implemented during 1998

requirement for BaBar/Compass production