• “It is not a matter of wanting, it is a matter of doing!”

Self-talk is a wonderful thing. We all do it throughout our waking moments. There is a little voice in the back of your head telling you things. Sometimes I say to my kids “You need to listen to that little voice.” Well I need to clarify something about this piece of advice – you need to listen, but you don’t always need to go along with the input that your little voice is giving.

Here’s an example – the alarm goes off and you are lying there in bed.  The little voice says “I don’t want to get up.” This is an example of one of those places where you need to apply the Rule of Done – “It is not a matter of wanting, it is a matter of doing.” You may not want to, but you get up anyway.

Here’s another example – you know you have an assignment due in a few days. You realize that you need to start working on it so that you can deliver a high-quality result. That little voice might say “But I don’t want to work on it now, I will put it off until tomorrow.” – There is that tip-off phrase “I don’t want…” again. So apply the Rule of Done. It is not a matter of wanting, it is a matter of doing. You overrule that little voice in your head, and you “get-r-done.” You stop arguing with yourself and “just do it.”

Try this out in your every day life and I think you’ll like the results. Tomorrow when that little voice in your head says “I don’t want to work out” say right back to it “It is not a matter of wanting, it is a matter of doing.” Or when that voice says “I don’t want to cold-call that prospect” or “I don’t want to have that conversation” or “I don’t want to….” Apply the Rule of Done. You will find that you can save yourself a lot of time and a lot of angst by following this rule.

Dale and I had a great time rooming together because we and our roommates shared a respect for the Rule of Done.

Databases are a great place to show this because the underpin most business-critical systems out there. I showed some specific examples with SQL Server, but just to prove that this is not just a Microsoft problem, let’s take a look at Oracle.

The Backup Tax – Oracle style:

Oracle RMAN is the default utility for performing database backup, restore and recovery of Oracle databases. Here are some results quantifying the impact of RMAN backups in an Oracle database environment, this is for a 2-Node Oracle 11g R2 RAC cluster running a TPC-C style transactional workload.

There are a couple of key points to note.

• Even though this was a lightly loaded system, CPU Utilization increased 30-40% for the entire duration of the backup.
• Response time and Transactional performance see a negative spike for the first couple of minutes of the backup window.

Conclusion: You have to pay the “tax man” for Oracle when you use RMAN on your Oracle host.

For more info on this testing see: http://dell.to/xAsNCa

Now it is time to hear from you. What is your experience with SLA taxes?

One of the hallmarks of a good pilot is how they can smooth out a ride, and precisely control their aircraft in maneuvers. I was flying with Harry one day and clearly was working hard to keep the plane on track while following his commands during the lesson. Harry didn’t talk much, his philosophy was to let the plane do the teaching. On occasion he would impart a key piece of advice so I always listened carefully to what he would say. On this particular day, after watching me work too hard at the controls he said: ”Hey Bob, try this: A little less aileron…Sooner!”

The ailerons on a plane are control surfaces on the wing. Here is a picture:

They are moved by the pilot, along with the rudder and the elevator to control the flight of a plane.

Essentially his input to me was this: I was using large, abrupt movements of the control surfaces. He wanted me to start more quickly but use subtler and more gradual control movements. It took me a while to apply what he said, but after hours of practice, I learned to slightly anticipate the need for control inputs and make the adjustments more progressive – quickly adding a little and then a little more or taking away a little at a time.

How does this apply to other areas of life? I’ll try a couple, and I bet you can come up with a lot more.

• I coach ski racing on the weekends. As skiers work to control their turn shape, one of the variables is edge angle – the amount the ski is tipped relative to the snow surface. This concept works great here – “a little less edge – sooner!” Perfect for smoothing out jerky turns or improving hold on ice.
• As a Dad, I’m trying to help my children develop good study habits. They are realizing that if they study a little each night instead of cramming right before an exam, they do better in school. Imagine telling your kids ”a little less studying – sooner!”

 So now I’m curious to hear from you, what could you “do less of, sooner” to positive effect?

Those tests showed two examples of an approximate 40% tax rate for data protection, meaning that 40% of the server processing headroom was sitting idle so that performance SLAs could be met during data protection operations.

How to reduce your taxes – So how can you reduce those taxes? This is where storage comes in to the picture. The reason that meeting service levels requires over-provisioning of your database servers in these scenarios is that server resources are being used for tasks that are best managed by a SAN. SAN-based snapshots have several advantages:

• They are quick to create – taking under 10 seconds instead of the 10 minutes it takes to create native SQL server snapshots for the same workload use-case.
• They have a very minor performance impact that is of no consequence to server provisioning.
• Snapshots can be used as the source for an off-host backup, eliminating backup tax.

Here is an example of actual testing of the performance impact of SAN-based snapshots for the same SQL Server transactional database load tested in parts 2 and 3 of this series.

As you can see, CPU utilization and transactional performance are minimally impacted by the SAN-based snapshot operations. The “Smart Copy” terminology refers to the fact that these are application-consistent snapshots which have been created by momentarily quiesceing the database and flushing buffers so that a completely valid recovery point for the database is created. This is done with Microsoft VSS, and took between 4 and 8 seconds for the configuration tested.

Conclusion: By using the data protection capabilities of your SAN, you can ensure more efficient use of your IT resources and reduce your taxes to achieve IT Efficiency.

For more information on the testing that was done please refer to:  http://dell.to/w35fsv

Now it is time to hear from you, what is your preffered “Tax” reduction strategy? Are you using SAN data protection?

Challenge: The Snapshot Tax – The following graph shows the results of testing snapshot processing on a transactional (TPC-C style) workload running in SQL server. It illustrates a 40% tax associated with establishing on-disk recovery points with the native SQL Server snapshot capability.

There are several observations from this data that should cause concern to those of you who would like to reduce your IT taxes:

• Creating native recovery points impacts performance – average response times and transactions per second see a significant negative impact.
• Creating a recovery point takes a long time – in this scenario about 10 minutes. So if you want to keep one recovery point per hour (RPO), around 1/6^th of that hour will be spent with degraded performance.
• The overall impact becomes cumulatively worse as more recovery points are kept.

Conclusion: Attempting to meet service level agreements for performance and data protection by establishing and keeping 5 hourly recovery points with native SQL server snapshots requires at least 40% over-provisioning of database server resources. This is an exorbitant 40% tax rate that you shouldn’t have to pay.

For more information on the testing that was done please refer to:  http://dell.to/w35fsv

Let’s hear from you. What is your RTO? Is one hour reasonable for recovery points for SQL Server transactional databases? What rate do you pay for snapshot taxes?

Challenge: The Backup Tax – The following graph shows the results of testing of backup processing on a transactional (TPC-C style) workload running in SQL Server. It illustrates one example of a tax rate of 40%.

The graph of the results shows that approx. 40% of the CPU cycles provisioned for SQL server must be reserved in order to make sure that service levels for performance are met during backup with the SQL Server native backup utility.

Conclusion: Simultaneously meeting service level agreements for performance and data protection while using the native SQL server backup requires at least 40% over-provisioning of your database server resources. And this assumes that you’re willing to run your database server at 100% utilization during the backup window. This is an exorbitant 40% tax rate that you shouldn’t have to pay.

For more information on the testing that was done please refer to:  http://dell.to/w35fsv

Let’s hear from you, what do you pay for backup taxes? What is your tax reduction strategy?

It comes down to managing SLAs. As you design your IT environment you need to be cognizant of Service Level Agreements. What are the expectations of your users for performance? What is the agreement you have with them about response time? Also what is the agreement you have with management about data protection, what are your Recovery Point Objectives and Recovery Time Objectives?

Databases are a great place to explore this balancing act.  Databases underlie most of our business systems. As you size your database servers, you must ensure that SLAs for performance are met while at the same time making sure those SLAs for data protection are met. It turns out that this usually results in database servers being over-sized to deliver acceptable performance AND accommodate for the processing load of establishing recovery points. This is a “tax” that almost every IT organization is paying, and with the right storage strategy you can reduce, and even eliminate most of this tax.

In this series we’ll look at some lab testing of transactional database systems (TPC-C style workload) under different scenarios to determine what that tax rate might be.  I think you will be surprised.  The data shows on the order of a 40% tax rate being paid by IT departments. I mean by that on average servers must be sized so that 40% of their headroom lies dormant in order to make sure that performance and data protection SLAs can be met. And this assumes that IT departments are willing to use all available headroom and run their servers at 100% utilization during data protection operations, which is rarely the case in the real world.

It is time to hear from you. What is your experience with this issue? Are you paying these taxes? Would you like to reduce your tax rate? Can you share any specific examples?

What was really striking to me though was the fact that people seemed to be most upset about losing connectivity to the phone and internet networks! I live in a town that reportedly has the fastest residential internet speeds in the nation. (see the NY Times article “For Idaho and the Internet, Life in the slow lane.”)  We are used to instant gratification when it comes to all-things-digital. But with the power out, and telephone and internet service disrupted, we were stripped of our information connection. Smart phones either ran out completely or had to be powered off to save what little battery was left. Laptops searched constantly for a not-to-be-found wireless network signal. Even with UPS battery-based power supplies, PCs couldn’t get a signal from the wire.

It seemed to be the younger generations who were most at-a-loss. We had to resort to things like reading the paper to get news, playing scrabble for entertainment, and walking to a friend’s house to check in, connecting by offering to share food and warmth with a neighbor.

I sat with a friend last night and recalled a childhood less connected. If we got to watch an hour of television a day – eg a news program followed by a sit-com on a 13” fuzzy black-and-white screen, we were lucky. We couldn’t use the phone because it cost money. We ran out the back yard to join the neighborhood street hockey game. We sat in the tree fort to have a private chat. We hung out at the store to get the gossip. Mom called us in for dinner by ringing the bell hanging on the back porch. We warmed up by the wood-stove in the kitchen.

If we gained something from our experiences in the Halloween storm, it was a reminder of how dependent we now are on our digital communications.

 One of the benefits of this approach is the ability to achieve more “elastic” application infrastructure, deal with seasonal spikes, and relieve the pressure to reduce over-provisioning of compute cycles. With the growing power of commodity X86-based servers and the Linux operating system, this scale-out has also come to be seen as a cost-effective approach to database scalability and reliability.

 A similar trend is now being played out in the storage components of the database infrastructure. Scale-out storage is built on modular storage building blocks or “nodes”, each of which contains storage media, memory, networking and controller resources. The capacity, performance and throughput of a scale-out storage system grow with the number of storage nodes in the system. The more traditional approach of scale-up storage systems features a single enclosure or “frame” with front-end connectivity, internal bandwidth and storage media. The capacity, throughput and performance are limited by the constraints of the frame.

 Scale-out storage is a great choice to complement an Oracle RAC implementation. The benefits include lower CAPEX because storage can start small and grow over time as needed. Unlike scale-up storage, there is no need to purchase up-front a large frame which can accommodate possible storage needs for the life of the project.  In terms of OPEX, in addition to reducing hardware to manage, many newer scale-out offerings provide very strong benefits in terms of management simplification. These systems virtualize storage to abstract storage objects from the underlying storage hardware allowing the simple and quick provisioning of new storage resources without specialized knowledge of storage hardware configuration.  

 Many of the pure scale-up architectures have been around for a long time and have some mainframe-like characteristics from the perspective of tunability. These systems can be, and must be highly customized. Though this tuning can lead to achieving extreme performance on the high-end, it is a cause for longer lead times when adding new storage and higher cost-of-ownership.  Time consuming and disruptive data migrations are a fact of life when moving to a new storage frame.  A migration is required when moving to a new capacity or in the case of a periodic storage architecture refresh.

 So take a look at your storage options when planning an Oracle RAC deployment. The economic and scalability advantages of scale-out storage are a good match for Linux-x86 grid-based database deployments. In a future post I’ll discuss some performance benchmarking that has been done that demonstrates the simple scalability of these types of systems.

• Most organizations are going to evolve their infrastructure toward cloud computing, and virtualization is the on-ramp to the private cloud.
• Shared storage is a requirement for enabling the mobilized-workload benefits of virtualization and therefore private cloud. 

This where the flight attendant says “If your destination isn’t New York’s Laguardia airport, you’d better de-plane right away!”

Now that you’re on the right plane, let’s get on our way. You’ve begun to virtualize and are wondering at what point you are going to be able to cross that line to private cloud computing. The answer is that you’ll reach private cloud when you have:

1. adopted standardized building blocks for your infrastructure,
2. adopted integrated infrastructure management, and
3. automated service delivery.

Integrating storage management with virtual server management is critical. However, the dirty little secret of the storage industry is that not all shared storage is actually appropriate for private cloud, though some vendors would have you think otherwise. Here’s why: traditional shared storage still binds storage objects to physical resources! If you want to move toward private cloud, you need to adopt Virtualized Storage. With Virtualized Storage there is an abstraction layer between storage hardware and storage objects, this is directly analagous to server virtualization.

But what does this mean – “an abstraction layer between storage hardware and storage objects”? This isn’t just thin provisioning folks, though certainly thin provisioning is one capability of virtualized storage. This means that administrators no longer have to know what volumes are pinned to what disks, or for that matter what disks are tied to what RAID policy, what trays are connected to what loops etc. etc. etc.. Virtualized storage leaves behind the complexity associated with traditional shared storage.

Extensive automation is a hallmark of virtualized storage. Capacity is used automatically based on policies. When new storage hardware is added to the SAN, the SAN recognizes this and non-disruptively starts to incorporate that storage into existing workloads. Performance is optimized automatically by spreading workloads across as many spindles as possible, AND by moving hot data blocks to higher-performing storage tiers without administrative intervention.

Another aspect of abstracting storage objects from the underlying hardware is the scaling methodology. Traditional SANs with frame-based architectures add performance and capacity within the limitations of a “frame” (or enclosure) that forces the IT department to pony up for a pre-determined maximum. On the other hand, modular scale-out SANs eliminate the need to pre-determine a performance or capacity ceiling, and allow your storage to start small and grow as needed.

So as you’re charting your course toward private cloud, keep the storage piece in mind. Match the capabilities of your storage with that of your other virtualized resources. In future posts I’ll explore other aspects of storage for private cloud, including management integration.