RSS

Monthly Archives: August 2012

HBase Regions and Region Servers

Common Question: How to determine the right number of regions in the Region Server ?

1 What is Region? How is region assigned to Region Server http://hbase.apache.org/book/regions.arch.html

2 Bigger or Smaller Region? http://hbase.apache.org/book/important_configurations.html#bigger.regions

The data organization can be affected with the regions distribution?

Monitor the regions.

 
Leave a comment

Posted by on August 19, 2012 in HBase

 

HBase Monitoring

1 HBase Monitoring in the book http://hbase.apache.org/book/ops.monitoring.html

2 HBase provides JMX interface to monitor it and there is a tool can be used http://blog.monitis.com/index.php/2012/03/28/monitoring-hbase/

http://my.safaribooksonline.com/book/databases/database-design/9781449314682/10dot-cluster-monitoring/id3289013

There is a tool kit for accessing MBeans: https://github.com/larsgeorge/jmxtoolkit

The remote service format in jconsole or JMXTookit: service:jmx:rmi:///jndi/rmi://127.0.0.1:10101/jmxrmi

jmx command line client : http://crawler.archive.org/cmdline-jmxclient/downloads.html

3 Visualize the log of HBase can also be a helper to diagnose the problem and root cause. (This is what an intern in our lab did in the summer 2012 using Timeline. Since it has some performance issue, I will continue to make it workable.)

4 Visualize the Hadoop file system might also be a plus to do performance tuning.

http://engineering.twitter.com/2012/08/visualizing-hadoop-with-hdfs-du.html

5 how to monitor hadoop with Catci

http://www.jointhegrid.com/development/

 
Leave a comment

Posted by on August 19, 2012 in HBase

 

Tags:

How to be mentored effectively

1 How to choose a mentor

could be from a variety of environments.

2 What is mentoring?

it is a dynamic, reciprocal relationship of a trusted and experienced individual who freely acts as a friend, advisor, coach, model..

3 Characteristics of mentor

what is your success definition?

Competence, Confidence, and Commitment

4 Mentor vs Expert: Mentor has “own” & “Share”; Expert only has “own”

5 Mentor can be Role model, Advisor, Teacher, Guide, Coach, Friend, Advocate

6 It is important to set up Informational Interview to learn the career you want to do

7 Ask someone to be a mentor, please ask yourself two questions

1) why do you want them to be your mentor?

2) The responsibility and expectations of being a mentor

 

 
Leave a comment

Posted by on August 14, 2012 in Life

 

Tags: ,

HBase Schema Highlights

0 When to use HBase

  • Storing large amounts of data (100s of TBs)
  • need high write throughput
  • need efficient random access (key lookups)
  • need to scale gracefully with data
  • for structured and semi-structured data
  • don’t need full RDMS capabilities (cross row/cross table transactions, joins, etc.)

1 Every region is served by one and only one region

2 An ideal cell size would probably be the size of a block, so 64KB including the keys.

NOTE: It gives a hint about how to organize your data via (row, column, version) design

3 Context on HBase http://ctolist.com/2012/05/context-on-hbase/

Lots of companies are mentioned there which are using HBase.

http://java.dzone.com/videos/hbase-schema-design-things-you

4 HBase schema in sematext

http://blog.sematext.com/2012/04/09/hbasewd-avoid-regionserver-hotspotting-despite-writing-records-with-sequential-keys/

4 HBase ecosystem http://nosql.mypopescu.com/post/1541593207/quick-reference-hadoop-tools-ecosystem

5 An reasonable explanation about column family in HBase

George, here’s a presentation I gave about understanding HBase schemas from HBaseCon 2012:

http://www.hbasecon.com/sessions/hbase-schema-design-2/

In short, each row in HBase is actually a key/value map, where you can have any number of columns (keys), each of which has a value. (And, technically, each of which can have multiple values with different timestamps).

Additionally, “column families” allow you to host multiple key/value maps in the same row, in different physical (on disk) files. This helps optimize in situations where you have sets of values that are usually accessed disjointly from other sets (so you have less stuff to read off disk). The trade off is that, of course, it’s more work to read all the values in a row if you separate columns into two column families, because there are 2x the number of disk accesses needed.

Unlike more standard “column oriented” databases, I’ve never heard of anyone creating an HBase table that had a column family for every logical column. There’s overhead associated with column families, and the general advice is usually to have no more than 3 or 4 of them. Column families are “design time” information, meaning you must specify them at the time you create (or alter) the table.

Generally, I find column families to be an advanced design option that you’d only use once you have a deep understanding of HBase’s architecture and can show that it would be a net benefit.

So overall, while it’s true that HBase can act in a “column oriented” way, it’s not the default nor the most common design pattern in HBase. It’s better to think of it as a row store with key/value maps.

 
Leave a comment

Posted by on August 13, 2012 in HBase

 

Tags:

Questions on HBase in Public

1 HBase uptime issues

Region sever fails due to overload, then GC pauses, then Zookeeper declared the region servers dead. Sometimes, the region server is out of memory

2 How many memory is needed?

  data node + task tracker running on the same node, with 4G memory, is that enough?

3 How much data can be handled by the certain cluster? What is the ideal configuration and load for a region server

 3TB data will be demonstrated in the way: 231 regions, 2055 stores, 1180 storeFiles adn 15MB storeFileIndex

4 ….

 
Leave a comment

Posted by on August 13, 2012 in Uncategorized

 

Tags:

The Rose

Some say love, it is a river,that drowns the tender reed.
有人说,爱,是河流,容易淹沒柔嫩的芦苇

Some say love, it is a razor,that leaves your soul to bleed.
有人说,爱,是利刃,會任由你的灵魂滴血

Some say love, it is a hunger,an endless aching need.
有人说,爱,是飢渴,一種永無止境的渴求

I say love, it is a flower,and you it’s only seed.
我却说,爱,是花朵,而你是那唯一的种子

It’s the heart afraid of breaking,that never learns to dance.
一個害怕受傷的心,永遠不可能學會跳舞

It’s the dream afraid of waking,that never takes the chance.
一個害怕醒來的夢,永遠不可能把握機會

It’s the one who won’t be taken,who can not seem to give,
一個不願吃虧的人,永遠不可能會去付出

And the soul afraid of dying that never learns to live.
而那個害怕死亡的靈魂 ,永遠不可能會懂得生命的真諦

When the night has been too lonely,And the road has been to long
當夜晚一如往常的顯得孤單無比,去路也依舊漫長

And you think that love is only for the lucky and the strong
而你覺得只有幸運者和強者才有資格得到愛

Just remember in the winter,Far beneath the bitter snows lies the seed
請記住,在那冬天的冰雪下靜躺著一粒種子

That with the sun’s love,In the spring becomes the rose.
一旦春天的陽光降臨,就會幻化為一朵玫瑰

=================Guita ===================================

A                     E
Some say love it is a river
      D          E       A
That drowns the tender reed
                         E
Some say love it is a razor
         D         E      A
That leaves your soul to bleed

   A                     D
Some say love it is a hunger
    D        D       E
An endless aching need
       A             E
I say love it is a flower
    D        E        A
And you it’s only seed

A                         E
It’s the heart afraid of breaking
D              E         A
That never learns to dance
 A                        E           D         A    
  A
It’s the dream afraid of waking that never takes the chance
  A                              D
It’s the one who won’t be taken
  D                  E
Who cannot seem to give
  A                     E            d      E     A
And the soul afraid of dying that never learns to live

         A                      E
When the night has been too lonely
         D         E        A
And the road has been too long
           A                E
And you think that love is only
         D     E          A
For the lucky and the strong

 A                      D                           E
Just remember in the winter far beneath the bitter snows
          A
Lies the seed
               E
That with the sun’s love
         D
In the spring
   E            A
Becomes the rose
 
Leave a comment

Posted by on August 11, 2012 in Life

 

Tags:

For JQuery Beginner

Layout with JQuery http://layout.jquery-dev.net/downloads.cfm

 
Leave a comment

Posted by on August 7, 2012 in Web Application

 

Tags: