Somebody has proposed a SAN. The quote is four times what a NAS costs, the case for it is performance, and nobody in the room can say which workload actually needs that performance. This is the most common shared storage conversation in a company of thirty to eighty people, and it usually has the same answer.
An organization this size that is buying shared storage for file shares, backups and a handful of virtual machines wants a NAS with redundant controllers. A SAN earns its place when a specific application has a latency requirement somebody can name, and almost nobody can.
Step 1Start from the workload, not the product
The question that ends most of these evaluations is simple and nobody asks it: what is going on this, and what does each of those things need?
Write down the real list. Departmental file shares. User home folders. A backup target. Some number of virtual machines. Possibly a line of business database. The list is usually shorter than expected, and once it is written the answer is often visible without any further analysis.
File shares and backups want capacity and simplicity, which is a NAS. Virtual machines want low latency block storage, which is a SAN, or a NAS over NFS, which every hypervisor supports well. A database wants predictable latency, which is the one entry on the list that genuinely argues for block storage.
Step 2Sizing it honestly
Two numbers matter and only one of them is usually collected.
Capacity is the easy one. Current usage, plus the growth rate over the last two years, projected across the life of the box. Then add the overhead of whatever redundancy scheme you choose, which is more than people expect.
Throughput and latency are the ones nobody measures. They are also measurable in an afternoon on the existing servers, before anybody quotes anything. A workload that peaks at 200 operations per second does not need a flash array, and knowing that is worth more than any vendor conversation.
| What to measure | Where | What it decides |
|---|---|---|
| Used capacity and growth | Current file servers | How much to buy |
| Peak read and write operations | Performance counters | Disks against flash |
| Average latency under load | Same counters | Whether a SAN is needed |
| Sequential throughput | A real file copy | Network speed required |
| Backup window | Existing job logs | Whether the target keeps up |
MethodThe counters to read, and what the numbers mean
Measuring this is genuinely an afternoon, and it is worth being specific about what to collect, because the argument is settled by four numbers rather than by an opinion about workloads.
On Windows, log the physical disk counters on each server that matters for a full working day, including whatever happens overnight. The four that decide it are reads per second, writes per second, average seconds per read and average seconds per transfer.
The last two are recorded in seconds, so a value of 0.008 is eight milliseconds, which is the number people misread most often.
logman create counter disk -c "\PhysicalDisk(*)\Disk Reads/sec" "\PhysicalDisk(*)\Disk Writes/sec" "\PhysicalDisk(*)\Avg. Disk sec/Read" "\PhysicalDisk(*)\Avg. Disk sec/Write" -si 10 -f csv -o C:\disk.csv
logman start disk
# On Linux, the same question, live
iostat -x 5
Read the result at the peak rather than the average, because the average is always reassuring and the complaint always comes from the peak. Then apply three thresholds. Under one millisecond is flash behaving normally.
Under ten milliseconds is spinning disk behaving normally and is fine for file shares, backups and most virtual machines. Sustained above twenty milliseconds during the working day is a workload that is already waiting, and it is the only measurement in this article that argues for block storage on its own.
Add the operations per second across every server that will move onto the box. That total, not the largest single number, is what the array has to serve, and at this size it is usually smaller than the quote assumes.
ArithmeticRaw capacity is not usable capacity
The quote is written in raw terabytes and the share is written in usable ones, and the gap between them is bigger than most people expect.
| Redundancy | Usable from 12 disks | Survives | Where it belongs |
|---|---|---|---|
| RAID 5 | 11 of 12 | One disk | Small, fast disks only |
| RAID 6 | 10 of 12 | Two disks | The default for large drives |
| RAID 10 | 6 of 12 | One per mirror pair | Latency sensitive workloads |
| Erasure coding | Varies by scheme | As configured | Scale out systems |
RAID 5 on large capacity drives is the one to argue about. The exposure is not the failure, it is the rebuild: a large array reading every remaining disk end to end for many hours while running with no redundancy left, at exactly the moment the surviving disks are the same age and have done the same work. RAID 6 costs one more disk and removes that window.
Then take more off the top. Formatting and metadata claim a percentage. Snapshots claim whatever you let them, and the reserve is easier to set at the start than to reclaim later.
And an array that is close to full stops performing well before it stops accepting writes, which is why the working rule is to size for eighty percent and treat the last fifth as headroom rather than capacity.
Run those three deductions against the quote before signing it. A twelve disk array sold as a capacity figure regularly delivers little more than half of it once redundancy, formatting, snapshot reserve and headroom are taken out, and finding that out in month two is how a three year box becomes an eighteen month one.
Step 3Budget the network, not just the box
The mistake at this size is buying good storage and connecting it badly. Shared storage moves the bottleneck onto the network, and a single gigabit link that was fine when every server had local disks is not fine afterwards.
The practical minimum is 10 gigabit to the storage, on switches that are not also carrying every desktop. If block storage over iSCSI is in the design, that traffic belongs on its own switches or at minimum its own VLAN, because a saturated link produces storage timeouts rather than slow file copies.
Budget two of everything in that path. The whole point of consolidating onto one array is undone by a single switch that can take it away from every server at once.
Step 4The three things that get forgotten
Backup. Shared storage with redundancy is not a backup. It survives a failed disk and does not survive deletion, ransomware or a controller writing corruption to both copies. The backup target has to be somewhere the array cannot reach.
The single point of failure you just created. Twelve servers that each had local disks now depend on one box. That is usually a net gain, and only if the box has redundant controllers, redundant paths and redundant power, and only if somebody has actually tested pulling a cable.
Who runs it. A NAS is an appliance a generalist administers. A SAN needs somebody who understands zoning, masking and multipathing. In a company of this size that person is often one person, and the risk is what happens when they are on leave.
The answerWhat most organizations this size should buy
A rack mounted NAS with two controllers, enough capacity for three years of measured growth, a flash tier or an all flash configuration if the measurements justify it, connected over 10 gigabit, serving file shares over SMB and hypervisor datastores over NFS.
That configuration covers everything on the list above, is administered by whoever administers the servers, and costs a fraction of the alternative. If the database measurements come back demanding sub millisecond latency, buy block storage for that database specifically rather than rebuilding the whole design around it.
The version to avoid is buying a SAN because it sounds more serious, then serving file shares from a Windows server sitting on top of it. That is a SAN and a file server doing a NAS's job, at three times the price, with two things to keep running.
- Write down what is actually going on it
File shares, home folders, a backup target, some virtual machines, maybe a database. The list is shorter than expected and usually answers the question by itself.
- Measure peak operations and latency on the current servers
An afternoon with performance counters, before any vendor is called. A workload peaking at 200 operations per second does not need a flash array.
- Budget the network as part of the storage
10 gigabit minimum, on switches that are not carrying every desktop, with block storage traffic separated. Two of everything in that path.
- Put the backup somewhere the array cannot reach
Redundancy survives a failed disk. It does not survive deletion, ransomware, or corruption written to both copies at once.
What the measurements decide
Three numbers, collected before anybody quotes anything, replace most of the argument.
FAQQuestions from the comments
Is a NAS really enough for virtual machines?
Over NFS, yes, and every major hypervisor supports it well. What you should not do is put datastores on an SMB share.
When does a company this size genuinely need a SAN?
When a specific application has a latency requirement somebody can state in numbers. A database committing transactions is the usual one. Everything else on the list is a NAS workload.
How much capacity should we buy?
Current usage plus the last two years of growth projected over the life of the box, then add the redundancy overhead. Buying for ten years costs money now for capacity that will be cheap later.
Do we need all flash?
Only if the measurements say so. A hybrid array with a flash tier covers most workloads at this size for considerably less.
What about cloud storage instead?
Reasonable for backup and archive, awkward for anything latency sensitive over an office internet connection. It is a real option for part of the list rather than all of it.
Which counters actually settle the SAN argument?
Reads per second, writes per second, average seconds per read and average seconds per write, logged for a full working day on every server that will move onto the box. Read the peak rather than the average, and remember the latency counters are in seconds, so 0.008 is eight milliseconds.
What latency is too slow?
Under one millisecond is flash working normally. Under ten is spinning disk working normally, and fine for file shares, backups and most virtual machines. Sustained above twenty during the working day means the workload is already waiting, and that is the one measurement that argues for block storage on its own.
How much of the quoted capacity do we actually get?
Often little more than half. Redundancy takes the first slice, formatting and metadata take a percentage, snapshots take whatever reserve you set, and the last fifth is headroom rather than capacity because an array close to full stops performing long before it stops accepting writes.
Is RAID 5 still acceptable?
On small fast disks, yes. On large capacity drives the exposure is the rebuild rather than the failure: many hours reading every remaining disk end to end with no redundancy left, on disks the same age that have done the same work. RAID 6 costs one more disk and removes that window.
Should snapshots count as backup?
No, for the same reason redundancy does not. A snapshot lives on the array, so it survives a deleted file and does not survive the array, ransomware with access to it, or a controller writing corruption. It is a fast restore for the common case and it sits inside the thing you are protecting against.
How full is too full?
Eighty percent is the working rule and it is about performance rather than space. Beyond it, allocation gets harder, thin provisioned volumes and snapshot reserves start competing for what is left, and the symptom is a general slowness that nobody connects to a capacity number nobody is watching.