OLAP and/or data mining?

Hello,

If I wrote the next ebay (yes I know, yawn-snore) and I had a database
with 5 million auction items in it, what would be a really good
strategy to get a search done very quickly? Would it involve
something called OLAP and/or "data mining"? The only technology I am
familiar with is simply SQL Server databases with stored procedures.
I think I'd be guessing correctly and say that this technology simply
wouldn't be fast enough *on it's own* to do super fast queries against
massive amounts of data.

Any insights would be of great interest. Thanks.

-Frameworker."Framework fan" <tempframeworkfan@.hotmail.com> wrote in message
news:f109ac80.0312221359.2269c2f0@.posting.google.c om...
> Hello,
> If I wrote the next ebay (yes I know, yawn-snore) and I had a database
> with 5 million auction items in it, what would be a really good
> strategy to get a search done very quickly? Would it involve
> something called OLAP and/or "data mining"? The only technology I am
> familiar with is simply SQL Server databases with stored procedures.
> I think I'd be guessing correctly and say that this technology simply
> wouldn't be fast enough *on it's own* to do super fast queries against
> massive amounts of data.

OLAP is exactly what you'd want.

Data mining may be useful post-auction to see who bought what, what things
need more promotion etc.

I don't know why you don't think SQL Server wouldn't be fast enough for a
small setup like 5 million auctions.

I have a database with 14 million rows that's probably as complex as an
auction database and most queries are a couple of seconds or less. (some
are longer because of some fairly complex floating point math that has to be
done along the way.)

I have another database that handles at least that many inserts per day w/o
a major problem. And this is all on 3+ y.o. equipment.

Give me a quad Xeon MP box and the right disk subsystem and this thing could
scream.

> Any insights would be of great interest. Thanks.
> -Frameworker.|||"Greg D. Moore (Strider)" <mooregr@.greenms.com> wrote in message
news:fCNFb.56190$Ug6.2898@.twister.nyroc.rr.com...
> "Framework fan" <tempframeworkfan@.hotmail.com> wrote in message
> news:f109ac80.0312221359.2269c2f0@.posting.google.c om...
> > Hello,
> > If I wrote the next ebay (yes I know, yawn-snore) and I had a database
> > with 5 million auction items in it, what would be a really good
> > strategy to get a search done very quickly? Would it involve
> > something called OLAP and/or "data mining"? The only technology I am
> > familiar with is simply SQL Server databases with stored procedures.

Using stored procedures as component building blocks
you can create all forms of data mining and analytical
processing applications.

> > I think I'd be guessing correctly and say that this technology simply
> > wouldn't be fast enough *on it's own* to do super fast queries against
> > massive amounts of data.
> OLAP is exactly what you'd want.
> Data mining may be useful post-auction to see who bought what, what things
> need more promotion etc.
> I don't know why you don't think SQL Server wouldn't be fast enough for a
> small setup like 5 million auctions.

Seconded.

If you write your procs in the right manner, and maintain
the data integrity in your database then you should have
null performance problems.

> I have a database with 14 million rows that's probably as complex as an
> auction database and most queries are a couple of seconds or less. (some
> are longer because of some fairly complex floating point math that has to
be
> done along the way.)
> I have another database that handles at least that many inserts per day
w/o
> a major problem. And this is all on 3+ y.o. equipment.
> Give me a quad Xeon MP box and the right disk subsystem and this thing
could
> scream.
>
> > Any insights would be of great interest. Thanks.
> > -Frameworker.|||Hi Greg,

I have one follow up question please:

I'd like to incorporate OLAP in to my design. Do I go ahead and
create a sensibly normalised relational database design, with many
dozens of stored procedures to query the tables in "the usual manner",
*then* add on OLAP as a kind of "bolt on", or does OLAP technology
require a totally different database/stored procedures design strategy
right from the word go?

Thanks!

- Frameworker.

"Greg D. Moore \(Strider\)" <mooregr@.greenms.com> wrote in message news:<fCNFb.56190$Ug6.2898@.twister.nyroc.rr.com>...
> "Framework fan" <tempframeworkfan@.hotmail.com> wrote in message
> news:f109ac80.0312221359.2269c2f0@.posting.google.c om...
> > Hello,
> > If I wrote the next ebay (yes I know, yawn-snore) and I had a database
> > with 5 million auction items in it, what would be a really good
> > strategy to get a search done very quickly? Would it involve
> > something called OLAP and/or "data mining"? The only technology I am
> > familiar with is simply SQL Server databases with stored procedures.
> > I think I'd be guessing correctly and say that this technology simply
> > wouldn't be fast enough *on it's own* to do super fast queries against
> > massive amounts of data.
> OLAP is exactly what you'd want.
> Data mining may be useful post-auction to see who bought what, what things
> need more promotion etc.
> I don't know why you don't think SQL Server wouldn't be fast enough for a
> small setup like 5 million auctions.
> I have a database with 14 million rows that's probably as complex as an
> auction database and most queries are a couple of seconds or less. (some
> are longer because of some fairly complex floating point math that has to be
> done along the way.)
> I have another database that handles at least that many inserts per day w/o
> a major problem. And this is all on 3+ y.o. equipment.
> Give me a quad Xeon MP box and the right disk subsystem and this thing could
> scream.
>
> > Any insights would be of great interest. Thanks.
> > -Frameworker.|||I'm a speed freak! If I can get a query to run .25 seconds faster
using OLAP technology, I'm all for it. :o)

"mountain man" <hobbit@.southern_seaweed.com.op> wrote in message news:<f3WFb.62838$aT.6089@.news-server.bigpond.net.au>...
> "Greg D. Moore (Strider)" <mooregr@.greenms.com> wrote in message
> news:fCNFb.56190$Ug6.2898@.twister.nyroc.rr.com...
> > "Framework fan" <tempframeworkfan@.hotmail.com> wrote in message
> > news:f109ac80.0312221359.2269c2f0@.posting.google.c om...
> > > Hello,
> > > > If I wrote the next ebay (yes I know, yawn-snore) and I had a database
> > > with 5 million auction items in it, what would be a really good
> > > strategy to get a search done very quickly? Would it involve
> > > something called OLAP and/or "data mining"? The only technology I am
> > > familiar with is simply SQL Server databases with stored procedures.
>
> Using stored procedures as component building blocks
> you can create all forms of data mining and analytical
> processing applications.
>
> > > I think I'd be guessing correctly and say that this technology simply
> > > wouldn't be fast enough *on it's own* to do super fast queries against
> > > massive amounts of data.
> > OLAP is exactly what you'd want.
> > Data mining may be useful post-auction to see who bought what, what things
> > need more promotion etc.
> > I don't know why you don't think SQL Server wouldn't be fast enough for a
> > small setup like 5 million auctions.
>
> Seconded.
> If you write your procs in the right manner, and maintain
> the data integrity in your database then you should have
> null performance problems.
>
> > I have a database with 14 million rows that's probably as complex as an
> > auction database and most queries are a couple of seconds or less. (some
> > are longer because of some fairly complex floating point math that has to
> be
> > done along the way.)
> > I have another database that handles at least that many inserts per day
> w/o
> > a major problem. And this is all on 3+ y.o. equipment.
> > Give me a quad Xeon MP box and the right disk subsystem and this thing
> could
> > scream.
> > > > Any insights would be of great interest. Thanks.
> > > > -Frameworker.|||"Framework fan" <tempframeworkfan@.hotmail.com> wrote in message
news:f109ac80.0312240543.63e644ac@.posting.google.c om...
> I'm a speed freak! If I can get a query to run .25 seconds faster
> using OLAP technology, I'm all for it. :o)

I think you completely misunderstand what OLAP is or does. (And I mispoke
below (forgive me, I was tired). You want an OLTP system, not an OLAP.

OLAP and getting better performance are orthogonal to each other. i.e.
using OLAP is not a way to increase performance.

>
> "mountain man" <hobbit@.southern_seaweed.com.op> wrote in message
news:<f3WFb.62838$aT.6089@.news-server.bigpond.net.au>...
> > "Greg D. Moore (Strider)" <mooregr@.greenms.com> wrote in message
> > news:fCNFb.56190$Ug6.2898@.twister.nyroc.rr.com...
> > > > "Framework fan" <tempframeworkfan@.hotmail.com> wrote in message
> > > news:f109ac80.0312221359.2269c2f0@.posting.google.c om...
> > > > Hello,
> > > > > > If I wrote the next ebay (yes I know, yawn-snore) and I had a
database
> > > > with 5 million auction items in it, what would be a really good
> > > > strategy to get a search done very quickly? Would it involve
> > > > something called OLAP and/or "data mining"? The only technology I
am
> > > > familiar with is simply SQL Server databases with stored procedures.
> > Using stored procedures as component building blocks
> > you can create all forms of data mining and analytical
> > processing applications.
> > > > I think I'd be guessing correctly and say that this technology
simply
> > > > wouldn't be fast enough *on it's own* to do super fast queries
against
> > > > massive amounts of data.
> > > > OLAP is exactly what you'd want.
> > > > Data mining may be useful post-auction to see who bought what, what
things
> > > need more promotion etc.
> > > > I don't know why you don't think SQL Server wouldn't be fast enough
for a
> > > small setup like 5 million auctions.
> > Seconded.
> > If you write your procs in the right manner, and maintain
> > the data integrity in your database then you should have
> > null performance problems.
> > > I have a database with 14 million rows that's probably as complex as
an
> > > auction database and most queries are a couple of seconds or less.
(some
> > > are longer because of some fairly complex floating point math that has
to
> > be
> > > done along the way.)
> > > > I have another database that handles at least that many inserts per
day
> > w/o
> > > a major problem. And this is all on 3+ y.o. equipment.
> > > > Give me a quad Xeon MP box and the right disk subsystem and this thing
> > could
> > > scream.
> > > > > > > > Any insights would be of great interest. Thanks.
> > > > > > -Frameworker.